Posting a BOM file for Dependency-Track with PowerShell

Today I was asked a “HEEEELLLLPPP” question at the end of the day. I typically like those types of questions so I was fully engaged! As a result, I was quite activated for a couple of hours, so I decided to make a blog out of it directly!

Let me provide a little context. The help question was related to Dependency-Track. I won’t go into details on the tool itself but if you are interested I would start at the dependencytrack.org website. Back to the question, there was a working PostMan call but getting a PowerShell script to posting a BOM (Bill Of Material) through posting Form Data is not as simple as you would expect, so this is where we were stuck…

First, the easy part, getting a version of Dependency-Track running locally. This gist shows you that a few lines can get you started with a container to do just that. It is easy to configure and allows you to be up-and-running in no time.


docker pull owasp/dependency-track
docker volume create –name dependency-track
docker run -d -m 8192m -p 8080:8080 –name dependency-track -v dependency-track:/data owasp/dependency-track

While we had a working PostMan Call, we could make use of the PostMan code examples, combined with lot’s of extra search result tabs in my browser, as well as good discussion and screen sharing session we came to the following working script.


try {
Set-Location $PSScriptRoot
$ProjectGuid = "d78bc750-d6db-4805-9836-5d77075ec37a"
$ApiKey = "6Ue2f8uVfRiVGpdowjWfF3yW02ryA7Uc"
$Uri = "http://localhost:8080/api/v1/bom"
$FileName = "bom.xml"
$ContentType = "multipart/form-data"
$xml = Get-Content (Join-Path $PSScriptRoot $FileName) -Raw
$httpClientHandler = [System.Net.Http.HttpClientHandler]::new()
$httpClient = [System.Net.Http.Httpclient]::new($httpClientHandler)
$multipartContent = [System.Net.Http.MultipartFormDataContent]::new()
$projectHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
$projectHeader.Name = "project"
$projectContent = [System.Net.Http.StringContent]::new($ProjectGuid)
$projectContent.Headers.ContentDisposition = $projectHeader
$multipartContent.Add($projectContent)
$bomHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
$bomHeader.Name = "bom"
$bomContent = [System.Net.Http.StringContent]::new($xml)
$bomContent.Headers.ContentDisposition = $bomHeader
$multipartContent.Add($bomContent)
$httpClient.DefaultRequestHeaders.Add("X-API-Key", $ApiKey);
$response = $httpClient.PostAsync($Uri, $multipartContent).Result
$response.Content.ReadAsStringAsync().Result
}
catch {
Write-Host $_
}
finally {
if ($null -ne $httpClient) {
$httpClient.Dispose()
}
if ($null -ne $response) {
$response.Dispose()
}
}

There is no magic but creating the objects to be able to properly post MultipartFormDataContent is not something I do on a daily basis. While we were happy with the working solution, I was not completely satisfied with this result. There must be other ways of doing this. One another way I found is using another API call that also allows larger content. That will come in handy when parsing a combination of BOM files!


try {
$xml = Get-Content (Join-Path $PSScriptRoot ".\bom.xml") -Raw
$ProjectGuid = "d78bc750-d6db-4805-9836-5d77075ec37a"
$ApiKey = "6Ue2f8uVfRiVGpdowjWfF3yW02ryA7Uc"
$Uri = "http://localhost:8080"
$Body = ([PSCustomObject] @{
project = $ProjectGuid
bom = ([Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($xml)))
} | ConvertTo-Json)
$Header = @{ 'X-API-Key' = $ApiKey }
Invoke-RestMethod -Method Put -Uri "$Uri/api/v1/bom" -Headers $Header -ContentType "application/json" -Body $Body
}
catch {
Write-Host $_
}

This file is something like you would expect in the first place. Much more condensed and lot less typing and types flying around. Also working with the Invoke-RestMethod is a lot more common. Great improvement found during the troubleshooting! Convenient is that this also works in PowerShell Core!

I hope this post helps you when you are in search of posting some BOM files to Dependency-Track with PowerShell!

Microsoft DevOps FastTrack & Azure DevOps Migrations

ethan-weil-262745-unsplash

While writing this post Microsoft has re-branded VSTS (Visual Studio Team Services) towards Azure DevOps. I have reflected the new name in this post so that it is up-to-date with latest naming and documentation references.

Introduction

Recently I have completed a very nice project and it finished with a migration weekend bringing several TFS collections towards Azure DevOps. This writeup helps me share my experiences with running the DevOps FastTrack Program as well as my approach on migrating from TFS to Azure DevOps.

Microsoft FastTrack Program

The Microsoft FastTrack (DevOps Accelerator) program is a program for customers that qualify. The FastTrack program consists of a two-week engagement, run by Microsoft selected and Microsoft trained consultants. Together with Rene van Osnabrugge and myself Xpirit has two!

Continue reading “Microsoft DevOps FastTrack & Azure DevOps Migrations”

Provision a VSTS Agent using an ARM Linux Custom Script Extension

There are many ways to get VSTS Agents deployed to a machine. You can find more on that here: https://docs.microsoft.com/en-us/vsts/pipelines/agents/agents?view=vsts. In this post you will find a way to deploy a VSTS Agent on a Linux Azure VM through an ARM template. For this we use a Custom Script Extension.

I intentionally left out the creation of the Linux VM in this post. I used a Packer script for this while my collegue Manuel Riezebosch created a very convenient VSTS Task for that! See this here: https://marketplace.visualstudio.com/items?itemName=riezebosch.Packer

To deploy the agent a couple of steps are involved;

  1. Get the download URL for the agent; blogged here: https://wp.me/p34BgL-81
  2. Encode a Linux script, to install the agent
  3. Use a Linux ARM Custom Script Extension in your ARM template

To create the encoded script I used another Inline PowerShell Task in VSTS. The full script can be found here: https://github.com/JasperGilhuis/VSTS-RestAPI/blob/master/Get-EncodedAgentDeployScript-Linux.ps1

To clarify the details I expanded the script a bit:

1. curl -s $(AgentDownloadUrl) > /tmp/agent.tar.gz;
2. for i in `seq 1 $(AgentsPerVM)`;
3. do mkdir /agent$i &&
4. cd /agent$i &&
5. tar zxf /tmp/agent.tar.gz -C . &&
6. chmod -R 777 . &&
7. sudo -u $(AdminUserName) ./config.sh --unattended --url $(VSTSAccount) --auth pat --token $(PersonalAccessToken) --pool $(AgentPool) --agent $(AgentName)$i --work ./_work --runAsService &&
8. ./svc.sh install &&
9. ./svc.sh start;
10. done;

The following lines comment the script above on line by line basis;
1. Download the VSTS Agent, save in the tmp folder
2. Loop for the desired number of agents
3. Create the agent directory
4. Go to agent specific directory
5. Unpack the agent in folder
6. Set permissions for the directory so that users can access it
7. For the user, configure the agent for to the specified VSTS account, using a PAT and named Pool and provided Agent name.
8. During the configuration a svc.sh file is generated. This needs to be run to install the service.
9. After installation the service can be started using the start method.
10. Done one loop

This script needs to be passed to the ARM template. The Custom Script Extension allows us to send a base64 encoded script. So we encrypt the script first and then encode it.

$Bytes = [System.Text.Encoding]::UTF8.GetBytes($script)
$EncodedText =[Convert]::ToBase64String($Bytes)

The encoded script is stored in a VSTS Variable

Write-Host "##vso[task.setvariable variable=EncodedScript;issecret=true]$EncodedText"

This script can be passed to a section in an ARM template through a parameter for the ARM template. The template can be deployed using the Azure Resource Group Deployment task.

In the ARM template you can add a section that executes the provided script. The section can be found here: https://github.com/JasperGilhuis/VSTS-RestAPI/blob/master/ARM-Linux-Custom-Script-Extension-Snippet.json

More details on the current Custom Script Extensions can be found here: https://github.com/Azure/custom-script-extension-linux/blob/master/README.md and here: https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux

Getting the latest VSTS Agent Download URL for your account

This week I have been playing to automatically provision a VSTS Agent on a Linux Machine. One thing i noticed is that in separate VSTS accounts the latest agent is not always the agent your account supports.

There may be little risk but this PowerShell script, that I use in an Inline PowerShell script in a Task during my provisioning release, helps to get the URL for the account your are targeting. Convenient and checked.

The script requires a few parameters;

  • PersonalAccesToken – A PAT for the VSTS account you are targeting
  • VSTSAccount – The https://account.visualstudio.com url
  • AgentType – The REST API calls for the Agent Type requested, this could be one of three values; “linux-x64”, “windows-x64” or “osx-x64”

The script updates a variable, AgentDownloadUrl, that can be used in the pipeline.

View/Download the script here: https://github.com/JasperGilhuis/VSTS-RestAPI/blob/master/Get-LatestAgentDownload.ps1

 

 

Automatically retain a VSTS release

In some environments it can be convenient to retain production releases automatically. Richard Zaat and I worked on this together. Our objective was to retain a release automatically after it has been succesfully enrolled to an environment. To achieve this we wanted to utilize the PowerShell task to minimize the effort.

First we created a demo release pipeline containing one environment. The Release does not do anything and does not have any artifacts. To the environment we only added the PowerShell task.

We have configured to have it use the Preview 2.* version but this works for version 1.* too. The script we have is the following;

$baseurl = $env:SYSTEM_TEAMFOUNDATIONSERVERURI
$baseurl += $env:SYSTEM_TEAMPROJECT + "/_apis"
$uri = "$baseurl/release/releases/$($env:Release_ReleaseID)?api-version=3.0-preview.2"

$accesstoken = "Bearer $env:System_AccessToken"

Invoke-RestMethod -Uri $uri -Method Patch -ContentType "application/json" -Headers @{Authorization = $accesstoken} -Body "{keepforever:true}"

In the script we construct the URL for VSTS Release Management, together with a ‘template’ to call the Release REST API service, passing the current Release ID. It also constructs the Bearer token to be able to call the REST API authenticated. The last line invokes the contructed REST API call. The call sets the ‘KeepForever‘ attribute of the release. This will exempt it from the release retention policies.

In the release definition the “Agent Phase” needs to be configured to “Allow scripts to access OAuth token”. Listed under ‘Additional Options’ section. This will allow the script to use the $env:System_AccessToken.

The last thing to do is to make sure that the agent account has the “Manage Releases” permissions. This can be done very specifically or for all release definitions.

A few links to usefull resources

VSTS Release API overview
https://www.visualstudio.com/en-us/docs/integrate/api/rm/releases

VSTS Release Retention polocies
https://www.visualstudio.com/en-us/docs/build/concepts/policies/retention#release

Interacting with VSTS and the Rest API’s
https://roadtoalm.com/2017/05/01/only-trigger-a-release-when-the-build-changed/

Enjoy!

Adding a DMZ server to the TrustedHosts list

Today I was working with Release Management in an On-Premise TFS 2015 situation where I had to release into server located in the DMZ.

After getting all kinds of things in place, like installing an agent, having shadow accounts setup and having validated i could reach and use the agent to install the required software I came across another issue.

The issue issue was that to be able to run a PowerShell script on the machine, WinRM is used. When running that PowerShell script from the release pipeline it blew up the pipeline with the following error:

“The WinRM client cannot process the request. If the authentication scheme is different from Kerberos, or if the client computer is not joined to a domain, then HTTPS transport must be used or the destination machine must be added to the TrustedHosts configuration setting. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be authenticated. You can get more information about that by running the following command: winrm help config.”

As the error suggests you need to add the server to the local TrustedHosts list. I first checked the current list with the following command:

get-item wsman:\localhost\Client\TrustedHosts

That returned an empty list. And thus I decided to add the current server to the list, which can be done with the following command:

set-item wsman:\localhost\Client\TrustedHosts -value 192.168.XX.XX

The following screen shows the commands in action, actual IP’s are blurred.

dmz_powershell_trustedhosts

When re-running the deployment all was good in the “safe zone”

Just published: VSTS Extension Token Comparer

Today I published a new Visual Studio Marketplace extension named “Token Comparer”. In this post I will quickly highlight its features and its usage. In a future post will do and end-to-end scenario so in which you will learn about the creation process as well as the delivery process. But first let’s see the extension.

What does the Token Comparer do?

The Token Comparer can parse specified source files for usage of Tokens and it can compare these against available variables defined in your Release Definition. It will detect and compare the results. Based on the settings you can choose to fail, warn or continue your release.

The tasks will provide you with a summary that will show the findings. The list states the findings.

VSTS Token Comparer Summary

Configuring the Token Comparer?

In this version I choose to let you define a generic service endpoint to allow safely storing your credentials. Now VSTS has the ability to access an oAuth token this will be changed in a future version.

How to find the Token Comparer Extension

Navigate to your VSTS Team Project. Click the Marketplace icon. Search for “Token Comparer”. Choose to install it to your VSTS account.

Token Comparer

Happy Releasing!

Post a TFS Team Room Message from Visual Studio Release Management

This is Part 3 in the series on Release Management + PowerShell + TFS TeamRoom API. In this post I am going to utilize the PowerShell Enabled TFS Team Room API logic from a Release Template in Visual Studio Release Management.

I will not cover a ‘usual’ flow that will release an application based on a Team Foundation Server build, if you like more information on this please check out my post covering that here: Release Management with InRelease

Or you can check out my Curation on Release Management here: Getting Started with Release Mangement

In Part 2 of this series I figured out all the PowerShell commands needed to post a message to the Team Room. Now let’s advance.

Visual Studio Release Management allows for creating custom tools. To create a Tool a Custom PowerShell Script can be used, so let’s make a script that can post the message. You can use the PowerShell ISE Editor to create this script.

A little clarification on the script. It has two parameters, TeamRoomMessage and TeamRoomName. I choose not to pass all the TFS configuration to the script, but if you want you can 🙂

After declaring the parameters, the module (Wrapper to the Team Room API) is imported. Then variables are set and the call to the Team Room is made passing the received parameters.

P3_PowerShellSE_script

Creating the tool in Visual Studio Release Management is fairly easy. Most important part is adding the PowerShell script as a resource. On the Execution section we specify that the tool is a PowerShell command.

Our script needs two parameters and those need to be passed as arguments to the script. Mind that the parameters are qouted otherwise spaces in the message might break the script.

P3_ReleaseManagement_AddTool

Having created the tool we need to create an Action to be able to add this to the Release Template. The creation of the Action is quite similar to the creation of the tool.

P3_ReleaseManagement_AddAction

Having the Tool and the Action ready we can continue to create a very simple Release Path to post a message. The Release Path is using an existing stage (Development) and an existing environment (Development Environment). Acceptance and validation are both set to be automatic. This way no manual actions are required.

P3_ReleaseManagement_ReleasePath

Now that we have a Release Path available, we need to create a Release Template. On the Deployment Sequence we drag our target server. Next in to Toolbox we browse to the Custom Section. In the section we can find our Custom Action. Drag this action to the Server surface. It lists the two parameters that our action needs, which now can be provided.

P3_ReleaseManagement_ReleaseTemplate

After saving, on the top menu there is the possibility to create a “New Release”, provide it a unique name and our Release Template is pre-selected.

P3_ReleaseManagement_ReleaseProperties

Let go and start the release! After starting the release you will see the Release Progress. As acceptance was set to be automatic, that step is automatically passed. Next the Deploy step is executed, status is Pending.

P3_ReleaseManagement_ReleaseProgress

After a little waiting (posting the message doesn’t take that long) this will transfer to Done.

P3_ReleaseManagement_ReleaseProgressDone

For the Deploy step there are details available. You can review these by clicking the ‘…’ button. The Deployment Log screen will be shown.

P3_ReleaseManagement_DeploymentLog

Through the View Log button we can see the log that is created during our ‘Add TeamRoomMessage’ action. The log message contains the output generated in our PowerShell Script. As we can see below it says our message is successfully posted!

P3_ReleaseManagement_ActionLog

The only place to find out if that’s true is to head over to our Team Room and see for our self! As expected the message is there!

P3_ReleaseManagement_TeamRoomMessage

As mentioned in the previous posts I promised to all the source code required for this complete series. You can download the source for this here: Release Management – Distribution.zip

Please note that this code was written for demo purposes only and is not intended to run on your production environment.

If you have any questions or remarks please feel free to leave a comment or contact me on twitter @jaspergilhuis

Blog Post Series announcement on Release Management + PowerShell + TFS TeamRoom API

Blog Post Series announcement on Release Management + PowerShell + TFS TeamRoom API

In a new series of posts I will be getting more out of the combination of Visual Studio Release Management + PowerShell + TFS TeamRoom API. The series will consist out of 3 posts that wrap it all together.

  1. Utilize the TFS Team Room REST API.
  2. Encapsulate Team Room API calls with a PowerShell Commandlet
  3. Post a TFS Team Room Message from Visual Studio Release Management

First post will come shortly! Stay tuned!

Update 1: Posted part one on 18 februari 2014
Update 2: Posted part two on 23 februari 2014
Update 3: Posted part three on 28 februari 2014

Release Management and PowerShell

A few weeks ago I wrote a (long) post on InRelease, since it has been released on November 13th, it is now officially called Release Management.

Release Management comes packed with useful Tools, Actions and the ability to create Components. With these items you can really shape your ReleasePath to your needs. But it could happen that there is something you want to achieve that is unavailable.

While there are several other options available in preparing your Web solution for deployment with Release Management, I recommend reading Colin Dembosky’s post on Web Deploy and Release Mangement – the proper way, I want to explore PowerShell options. Another option would be using the Tokonization functionality (not checked this against the Release Management release).

The following steps will guide you through the process on calling a custom PowerShell action from your Release Template.

Disclaimer: I am a PowerShell newbie, so there probably are more powerful or optimized scripts possible…

1. Lets create a PowerShell Script for your desired actions. Using the PowerShell ISE Editor will help you in getting the syntax right.

2. Lets create a very basis script first. Containing all hard-coded raw materials. Test the script to see if you got all the syntax and desired outcome. In this case, ‘BlogEngineData’ is replaced by ‘BlogEngineDEV’.

PowerShellScript-HardCoded

 

 

 

3. Now we have a functional script, lets add some parameters, so we enable reuse of the script. Validate the script by executing it.

PowerShellScript-Variables

 

 

4. Now open [Release Mangement], navigate to [Inventory] and then [Tools]. Then create a new [Tool]. The tool is capable of executing the powershell script, that will be added as a resource.

ReleaseManagement-NewTool

 

 

 

 

 

 

Notice that in the [Arguments] options we use the Release Management parameter syntax ‘__Parameter__’ (yes that is two! underscores to start and end with). All recognized parameters are listed in the Parameters section. For our script that is four.

5. Make sure you upload the PowerShell script as a Resource. Find my sample PowerShell Script here

6. To be able to use a [Tool] you need to create an [Action], so navigate to the [Actions] section within the [Inventory] and create a new [Action]. Follow the options below and you will be OK.

Tip: Create a Category for your ‘Custom’ actions. This will help you in adding them to the Release Template.

ReleaseManagement-NewAction

 

 

 

 

 

 

7. Now we have a new Tool and Action we are ready to use it on our Release Template. Navigate to [Configure Apps], [Release Templates] and open a template. Find your ‘Custom’ Action, drag it to the canvas, and fill all the properties.

ReleaseManagement-AddToolToTemplate

 

 

 

 

 

 

 

8. Repeat this for all your Environments / Templates.

The Release Template is now ready to be used! With this example I hope to show the potential of using a custom PowerShell script to extend the Release Management Toolbox.

Happy Releasing!