Starting From Scratch–Building Your Project Right Part 1

Prologue

Let’s say for the sake of argument that you just uploaded your codebase into TFS2017/VSTS. What do you do? XAML builds are deprecated and the new Build system seems daunting. Again what do you do?  You can watch videos and read tons of different stackoverflow articles and blog posts on how to… yet there are still lingering questions on how to “just” start from scratch.

In my experience with teams from around the world before they used a version control system, would happily code on their local machine, perform a local build (where is just worked), and then using the power of Visual Studio would “Publish” their fixes directly to their remote environments. For one that is poor a ALM practice and two there is just no way to track any of the changes that were either breaking or fixing described typically with screenshots in an email thread. Overall it was the Dark Ages, a chaotic time where teams that were trying to be Agily/Scrummy/etc, yet really having no anchor or starting point to leap off from for how do perform a proper “single source of truth” build; let alone a deployment.

With “DevOps” and “Shift Left” being the buzz words of the day, it can be hard to get your team in the correct cultural mindset of ownership and control. In this article we will dig into the new MS Build system as if you were a newly minted Developer Lead with the appropriate Administrative Rights in your TFS/VSTS project.

Here is a basic scenario and then we will work through how you can build your project right with the new Build system in VSTS/TFS2017.

Assumptions

  • You are just now using source control
  • Your builds consisted of developers performing builds on their local machines
  • You may had a build server in the past, but you have either upgraded or the build templates that you previously used are incompatible
  • You have Post-Build scripting moves files around to make your codebase viable for a manual or even scripted deployment

Scenario

This is barebones. Your particular scenario may not apply and I will discuss in the future how to do advanced builds and deployments.

Your codebase has been freshly checked into TFS/VSTS source control. If you chose Git or TFVC, it doesn’t matter the below techniques will apply for both version control types.  You need to perform a build that is simple and the output needs to packaged and ready to be consumed by either Release Management or Octopus Deploy. So where do we start?

How it’s done

Now for the pretty pictures.

First log into your TFS or VSTS site or account and click on the Build & Release tab. It should be blank if you are first starting out. The screenshot I am providing is from previous posts that build more and more advanced concepts.

image

You will see that there four items in the image called Mine, All Definitions, Queued, and XAML. You will not need the XAML tab; it has been deprecated and you can edit your old XAML builds nor are they compatible with the new Build system. So because of those points we will not be discussing that tab.

  • Mine – represents those build definitions that you the logged in user have created

image

  • All Definitions – represents all the build definitions that have been created for your solutions, branches, etc

image

  • Queued – represents all those builds that have been currently queued or running or completed

image

Now lets create a new build definition for our solution. In our case we are going to do something basic and build up from there.

image

then we get a popup that gives a number of options of generic templates to choose from. For now we will just choose the Visual Studio build. The reason behind this is that most developers are accustomed and acquainted with and it is in keeping with my idea of starting with the basics and building from there.

image

Click Next and you will see a another page where you choose your repository and other settings for your build.

image

The great thing about this page in the Build Definition Wizard is that you can make preliminary adjustments to your build before it is created. For instance you can choose the type (remote or local) of repository that you wish to point at, select the branch you wish to build from, and determine whether or not you wish to have Continuious Integration (build after every check in).  Subsequently you can choose the default agent for your build to be something different that has the capability you require to build your solution.

image

Make your selections and then click Create. Now you will have a “unsaved” generic build definition that you will need to continue editing. But first it will be wise to save you creation so that if you have to leave the page you don’t have to start all over again.

image

image

Now that we have saved the build definition, we can go into each of the build steps and one make adjustments and two add more build steps as necessary to perform our build.

On the left part of the page you should have noticed that there are six steps and descriptions for each. Lets talk about them a little in detail.

First you have the NuGet restore.

image What this little step does is go into your solution and check for the packages.config file and restore or install all of the packages each time you build. In this instance you can ensure that you have the everything restored from NuGet.org or other repo of your choice without having to check in your packages folder like you would have to do for your old XAML builds. With this step you can control a lot of what can happen with the installation or restoration of your packages.

Next is the Build Solution. This is a powerful step very similar to the older XAML Process. One item of note is that there are some overlapping of functionality that I have found for build steps, while it may not hinder a build, it just gives you further options for streamlining your process.

image

This will look a little familiar to those with a XAML build background, but this is a lot cleaner and can be adjusted to suit your needs. MSBuild arguments still work, but in some cases you don’t have add switches like “/m:1” when you can check the Advanced->Build in Parallel check box.

The Text Assemblies Step is standard with the Visual Studio Build Template. It will use Visual Studio Test to perform that function to ensure that you have testings completed and code coverage for the widgets you have on your Dashboard.

imageFrom a testing perspective this a powerful step that allows you to go into and perform advanced testing from the build without a lot of tools that you would assume that you need to perform the reporting, etc.

Also note that you see a image or this imagein each step it will show either a hover tip or open a new tab to more details about the step for your leverage or understand.

Next is the Publish Symbols path. It is a way for you to use your pdb and obj files to help debug your application on a different machine other than where your sources were built.

image

Next is the Copy Files to: step. This step is takes the output from your sources bin directory and copies those files or folders to your artifacts directory.  The artifacts directory is a cleaned directory that ensures that you have just the right objects in your artifacts folder that need to be either packaged or deployed at a later time. Again clicking on the image icon will provide options and parameters for you to use to make your build more coherent and robust.

So after the previous 5 steps, now what happens? Well the build agent then publish your outputs to a Drop folder. Typically this drop folder is within the $(Build.ArtifactStagingFolder). Where might that be? It is located under your build definition directory. Similar to this file path: “E:\TFSBuildAgent\vsts-agent-onprem\_work\5”. Inside of this folder are 4 folders that represent “a” artifacts, “b” build (output), “s” sources, TestResults (obvious).  Look at Resources->Variables for more information.

image

Once we have made our initial edits to your build definition, we can now Click Save and then queue a new Build.

image

and our build succeeded.

image

From our log output we can know that the artifacts are “published” and a little magic later we can compare and see that the artifacts are in the location where they are intended to be.

image

and finally…

Don’t be afraid of the build

As the heading suggests, when you are first starting out with using TFS/VSTS Build, don’t be afraid if your build fails or doesn’t perform the functions you are expecting immediately. I cannot stress this enough. I know that when I am testing builds that I spend a lot of time troubleshooting failed builds before I finally have a successful (green) build.

By working through your issues and using a methodical approach you can be successful. You shouldn’t feel pressured to get it right the first time. You don’t have to feel like you are going to get fired for having a broken build. Work through it, understand the build process, then communicate the understanding to your leadership.

If you are developing a Web Application or web api you may have to consult this how to. But if you are doing a single applicaton (exe, service, etc), then this approach can get you started in the right direction.

Resources

Here is another good resource about build settings and build tokens https://www.visualstudio.com/en-us/docs/build/define/general 

https://www.visualstudio.com/en-us/docs/build/steps/build/visual-studio-build further details about the Visual Studio Build. Word of caution here is that if you are using TFSBuild.proj type of file, you will not be able to use Build because it contains tasks and targets that are supported only for XAML builds.

https://www.visualstudio.com/en-us/docs/build/define/variables this is for getting the different built-in and custom variables to work in your favor.

Advertisements

CI/CD with TFS/VSTS and Octopus Deploy

Building off my previous posts here and here about building multiple projects within a solution and troubleshooting packaging of your projects. We will now delve deeper into the Continuous Integration and Continuous Deployment pipeline that can ease the tensions of Deployment Aversion. Using two very powerful tools in your arsenal, you can streamline your deployment process and ensure that you are getting the biggest bang for your buck. Up until now we have identified our pipeline components and we have successfully, albeit manually, deployed our sample codebase to our destination servers. The question now is: can I streamline my pipeline using good ALM practices?

Assumptions

  1. You have your source code checked in
  2. Your build definition is in place
  3. You either have a partial Octopus Deploy project or similar

Setting up for success

Lets double check our Octopus Deploy Project. We will need to ensure that we have certain steps, configuration values, and settings.

Lets start out with cloning our already working deployment project.

image

Save our newly cloned project. We will get back this Octopus Project in a moment, but first we will need to do somethings with our TFS/VSTS build definition. In this example, we will expand upon our previous build definition and clone it. Why? Well first we know that it works and second it is easier to take away unneeded steps instead of trying to add and configure steps from memory. Cloning a Build Definition is very simple and straight forward. The below gif shows you how its done.

CloneBuildDefinition

Since our build definition is only queued manually we will need to go and make some changes to the Triggers for the build to allow it to perform the CI for the beginning of our CI/CD pipeline. We will edit our newly cloned build definition and choose the Triggers tab and then select.  For now we will just check the Continuous Integration and leave the defaults.

image

Now each time there is a change to the codebase and change is checked in, this build definition will start and finish with our packages being pushed to Octopus Deploy. One of the many benefits here with the use of Triggers is that you can have multiple build definitions that can focus on specific branches and with CI/CD working in your favor your can get your code to testers or others for faster consumption and approval.

 

CI/CD Option 1

You are probably asking yourself why there is an Option 1 and 2. Well the fact of the matter is that with Octopus Deploy Extension you basically  perform can perform all of the steps of packaging, creating a new release, and finally queuing up a deploy.  Essentially, TFS/VSTS is now a one stop shop for building and orchestrating  your build and deployment pipeline.

How to perform Option 1

Going back to our CI/CD build definition we need to add a couple of more steps. Click add build step and select and add Create Octopus Release and Deploy Octopus Release.

image

Now we see that for our entire build definition we can build, test, package, publish, create and finally deploy a release. If you noticed that there was Promote Octopus Release, that task can be used in other build/deployment scenarios. For now, though, we are just going to use the create and deploy release.

image

Editing our create Octopus release step, we see a lot of options that we will need to fill out. We fill out the Octopus Deploy Server connection endpoint, we select the project name. For now we are going to use the Octopus defined version number, but later we can change this to be what ever SemVer compatible version number we want to display. Finally in the Deployment section we select our initial environment we would like to deploy to. After that because I like verbose logging for troubleshooting, select the Show Deployment Progress. This will ensure that we have a complete set of logs for our End-to-End test.

image

One thing to note here is that you add a lot of data for your release notes. This great for troubleshooting or even attempting to rollback to an earlier series of changesets.

We are going to kick off a manual build/release creation first, but to make sure that we are putting everything in the correct order.

imageimageimage

and the end result without doing anything special with Octopus Deploy.

image

and now our Web application is still functioning.

image

Let’s go and make a change now in our source code, check it in and then see what happens.

Made some changes to our index page.

image

checked it in

image 

Synced with the master (because this is a Git repo)

image

Build kicked off

image

Build finished and deployed

SNAGHTML2c00ca50

Now to validate against the working web application.

It worked. I made the change and the change was “automatically” reflected on the site. Now keep in mind this is simple way to perform this function. If you needed to perform a rolling deployment, or some other type, it would make sense t to adjust for Octopus Deploy process to accommodate those requirements.

image

CI/CD Option 2

Option 2 involves some manipulation of Octopus Deploy and not so much TFS/VSTS. If we leave what we were currently performing in TFS/VSTS from my previous post we will need to work and manipulate Octopus Deploy a little more.  So following the same example as above we will clone both the TFS/VSTS build and the Octopus Deploy project/build definitions, to ensure that we have a clean break from what worked  to what could work.

How to perform Option 2

There are a couple of items that we need to accomplish here. Because we have already cloned both the Basic Build definition and the Octopus Project, we will work with those cloned items in this mini exercise. For now we are not going to touch the clone of the build definition because it is generic and just performs the build and then packages and publishes the package( s ) to our Octopus Deploy internal NuGet repository.

First we will need to change how our Releases get created. After selecting your Process, find the Automatic Release Creation in the left hand side of the page. Select Project Triggers, this has moved from previous versions of Octopus Deploy and then select the create a release when a package is pushed.

image

image

 

Then we need to change the behavior of our Lifecycle (more information is here). The important fact here is that if we wish to stay away from modifying our build definition to automatically deploy we can do it within our Lifecycle. One key note here is that the default lifecycle allows for deployments to created environments. While this is not ideal, there is a way for us to create and manipulate a new Lifecycle to benefit our needs. For our cloned project we will need to “Choose a different lifecycle”.

image

Special note here is that the lifecycle once applied, will only be used by your deployment project going forward. Any previous releases that were created will use the previously assigned lifecycle. This can be confusing, but it makes sense because Octopus Deploy takes a snapshot of all of the variables and state of your project for each deployment.

Let’s go ahead and create a new lifecycle:

Click Library->Lifecycles-Add lifecycle

image

Then we need to give the new lifecycle a Name and add a Phase.  This is important because the phase helps us describe what we wish to do in each step of our deployment.

image

Adding a Phase means that you are going to be describing what happens to your deployment. After you click the Add Phase you will get this:

image

Give it a name and then click Add environment

image

Initially you will see and empty drop down box and a couple of radio buttons. Select your dev or other environment that you wish to deploy to first and then select Deploy automatically to this environment as soon as the release enters this phase. Then click Add.

image

Now it should look like this:

image

Click Save and we are almost there.

You can can multiple phases and multiple environments to each phase depending on your need. For this exercise, we are putting out the CI/CD basics that you can build upon for your future development and deployment efforts.

Now that I have created that lifecycle, I need to perform a couple more things to get everything wired up and in sync.

First, I need to go back to my project and change the lifecycle from Default to CI-CD Option2.

image

The next thing we have to do is change the Automatic Release Creation under the Project Triggers.  Warning here is that if you have an external NuGet or other repo that you are dealing with this option will most likely not work for you.  There may be an option 3 workaround, but I haven’t gotten to that state yet.

Click on Project Triggers and then

image

image check the “Create a release when a package is pushed to the built-in…” make sure you select the deployment step that contains your package, then click Save

image

Let’s test this out.

image

Our build was successful.

image

And it just works now.

image

Outcomes

From Option 1 it is clear that you can just use the Create Release step and not be dependent on both the Octopus Create Release and Octopus Create Deployment. Either option is viable and can be used interchangeably depending on your needs. You can have build definitions that work on specific branches of code for CI/CD and then use the promote package to your higher environments. The above technique while focused on a simple web application can be used for more complex deployment scenarios and applications.

Troubleshooting _PublishedWebsites Errors

Leading up to my next post about performing a full CI/CD pipeline with TFS/VSTS and Octopus Deploy, I came across an interesting “glitch” with the prescribed use of the _PublishedWebsites folder from my earlier post. Something was missing or misplaced during either the build or the packaging of my example Visual Studio solution. Using my basic troubleshooting skills, I found that not only were things missing from my packages, but there were items that were definitely misplaced.

(This post is looking at TFS2017/VSTS/Visual Studio 2017 and an OOB MVC Web Application built from a .NET Framework template of 4.5.2)

image

After performing a test deployment of my created packages namely the ones that are related to WebApplication and WebApi’s (in extension to those that use the Web.Application.targets for MSBuild), I did a simple test of trying to view the web site.  Keep in mind that viewing the website via F5 within Visual Studio will be perfect, but when you are attempting to utilize the “single source” of truth (TFS/VSTS) and building and moving your code through a CI/CD pipeline with MSBuild challenges can arise.

If you followed my previous example and created a release/deployment in an Octopus Deploy project, you probably would have found this:

SNAGHTML205cc07c

If you are scratching your head, so was I! _PublishedWebsites should have had everything I needed to run a web application, correct?  What is missing that I need to include possibly?

Digging back into my TFS instance, I slowly came to the realization that the _PublishedWebsites didn’t really contain all of the items required to run a WebApplication or other related project. Case in point is that in my previous post about performing builds and packaging on multiple projects, I found that the “roslyn” folder  was nowhere near _PublishedWebsites. So how could roslyn folder be in an incorrect.  If I removed _PublishedWebsites, the pages, css and other items would be gone and I would still have an issue.

Web Application or Web Application-like build output

image

vs Windows Service build output

image

But then the problem starts to compound itself with:

  • How can I keep my package small and minimalize duplication of files in unwanted directories? –According to Parker’s Packaging Paradigm
  • Why does this seem to just work when I publish a WebApplication directly from Visual Studio?

The first question is more about what I want to discuss and the second question is a philosophical one that expounds upon the larger differences between small and larger teams that approach development/deployment from different angles. When you are a large organization it makes sense from an auditability standpoint to have a central way for you check-in, build, and deploy your codebase.

Approach

Scratching my head some more I went to Bing/Google to see if someone else had:

  1. Experienced what I was experiencing.
  2. Provided an easy repeatable fix.

Apparently not, I did this search and I found a lot of different ways to remove the issue. “Hacking” the config files in the project, “Hacking” the project file, uninstalling packages, etc.

https://blog.gldraphael.com/removing-roslyn-from-asp-net-4-5-2-project-template/

https://blogs.msdn.microsoft.com/webdev/2014/05/12/enabling-the-net-compiler-platform-roslyn-in-asp-net-applications/

All of these were seemingly kludgy. Couldn’t there be another approach to this? I had to try.  The KISS method was beating down my door.

When I tried to perform the “fixes” described in my searching and then performed a deployment and subsequent refresh of the web application I would still get an error. Missing .NET assemblies, complier errors. Something wasn’t right WebApp-ville.

clip_image001

So what could I do? I had to prove that I could simply keep the VS project clean and still be able to push my prescribed changes through to my Octopus Deploy instance and finally view my results.

Resolution (?)

Warning, this may work for some, but not all projects.

Going back to my original build definition, I had to come up with a strategy that could work for the issues that I was facing. How do I keep the project clean? Meaning that I do not add or take away from any of the .csproj files or .sln file. Finally how can I minimize my project package footprint without losing integrity? Meaning what would I need to do keep a small enough package foot print with all of the correct dlls and other appropriate files.

The partial answers to the questions were: I would have to move the roslyn folder and then I would have to clean up the extraneous files that were copied into the bin directory by the MSBuild

 Creating directory “E:\TFSBuildAgent\vsts-agent-onprem\_work\3\a\DemoLabs.WebApi\roslyn”.
Copying file from “E:\TFSBuildAgent\vsts-agent-onprem\_work\3\s\packages\Microsoft.Net.Compilers.1.3.2\tools\csc.exe” to “E:\TFSBuildAgent\vsts-agent-onprem\_work\3\a\DemoLabs.WebApi\roslyn\csc.exe”.

How it’s done

Opening up my build definition, I added two new PowerShell steps and placed them after my Publish Artifacts task and before my Octopus Package tasks. The first step is to move the roslyn folder to its intended location and the second task was for cleaning up the csc/roslyn related files that were copied into the bin directory during the build process.

SNAGHTML25237a8d

The first step of moving the roslyn folder. Here we edit the PowerShell step to be an Inline Script. This could be done another way with a PowerShell script referenced within the codebase, but it was simpler to leverage the power of the build/orchestration steps.image

Here is the code:

 Write-Host "Moving the roslyn directory"  
 $folders=gci $env:BUILD_ARTIFACTSTAGINGDIRECTORY -Recurse -Include "roslyn","_PublishedWebsites"  
 for($d=0;$d -lt $folders.Length;$d+=2)  
 {  
   if($folders[$d].Parent.Name -eq $folders[$d].Parent.Name)   
   {  
     if($folders[$d].Name -eq "roslyn" -and $folders[$d+1].Name -eq "_PublishedWebsites")  
     {  
       move-item -Path $folders[$d].FullName -Destination "$($folders[$d+1].FullName)\$($folders[$d+1].Parent.Name)\bin\"  
     }  
   }  
 }  

Now to get rid of the duplicate files. Because the build actions copy roslyn folder contents directly over to the bin directory the web application complains that it cannot find roslyn folder so if we move the roslyn folder then we will end up with duplicate files one set in the roslyn folder and the other one level up in the bin folder.

image

 Write-Host 'Removing duplicates from bin directory'  
 $folders=gci -Path $env:BUILD_ARTIFACTSTAGINGDIRECTORY -include "bin" -recurse  
 foreach($folder in $folders)
{  
 if(Test-Path "$($folder.fullname)\roslyn")  
 {  
   $Ref = Get-ChildItem -Path $folder.FullName  
   $Diff = Get-ChildItem -Path "$($folder.FullName)\roslyn"  
   $dupes = Compare-Object -ReferenceObject $Ref -DifferenceObject $Diff -Property Name,Length -PassThru -ExcludeDifferent -IncludeEqual   
   foreach($dupe in $dupes)  
   {  
     del $dupe.fullname  
   }  
 }  
}  

Next save the steps in your build definition and queue up a new build.

SNAGHTML25661218

Now that we are good to go, we can check our build artifact location so that we can view the contents of our NuGet package. Once that is done we can see that the library for our NuGet packages have been updated too.

Downloading the package from Octopus we can just change the file extension to .zip and open up the file to see the contents as shown here.

image

Now to confirm that everything is working the way it should I created a new release in Octopus Deploy and Deployed it.

image

image

image

Now that we have a successful deployment, lets look if all that work was worth it.

SNAGHTML25742e95

At the end of the day, without editing the Visual Studio project and leveraging the various utility steps within TFS2017/VSTS, you can “fix”/ “clean” your project for a better, smaller, cleaner pipeline solution.

Next time I’ll start the process of getting your web application and other projects to the next step towards Continuous Integration and Continuous Deployment.