Managing DevOps as a Service (Part 1)

DevOps Challenges

One of the bigger challenges I see in the DevOps space is when you attempt initiate DevOps as a Service (DaaS).

My Definition: DevOps as a Service (DaaS)

  1. Performing actions to allow for teams 1+ to deploy codebase(s) to multiple environments (1+ servers) and maintain those servers within specifications.
  2. Maintaining multiple environments across globally distributed teams with a follow the sun approach
  3. Allowing for just pass through of code and deployment while maintaining infrastructure for enabling teams.

Typically what you see with small and medium sized teams is that one or two of the members of the development team are involved in the operations space as well. This works from the standpoint of smaller teams who have intimate knowledge of their environments, code base, and configurations. Yet, what if you had to control hundreds of teams? Different code bases? Different time zones? Different environments? Hundreds, if not, thousands of servers spanning on-premise, AWS, Azure? What you do and what can you do?

A friend of mine Damian Brady wrote about DevOps as a culture. Well truly it is a culture, you have to have ownership of the work that your are doing.  All too often, I see where teams develop and test locally and assume that their codebase is going to work in the various environment’s that they deploy to. A “throw it over the fence” style of development and deployment. This can be problematic when the assumption from the developers is that the operations team knows and understand the nuances of their codebase.

Categories of Development and Operations

Image result for devops images

Let’s define in simplistic terms both what I mean by categories of Development and Operations. Development is really the design and coding of an application or API to be consumed by internal or external parties. Operations is about the continuous maintenance of the application or API once the Development portion of the job is completed and the team that originally developed the code has rolled off on to other projects or clients. In nutshell, this “old school” approach has left many a maintainer performing Development work that really isn’t their forte. Developers are so focused on their timeline, their code that sometimes don’t understand or don’t want to understand the underlying infrastructure that serves up their application. This leads to the dreaded WOMM effect and its consequences.

My definition of WOMM is “Works on My Machine”. This the most basic build on the local machine that “just works”. F5 and it just works. It is also, from my point of view as a bacteria or viral disease that at some point most if not all development teams contract in their effort to quickly get code developed for consumption.

Many consulting teams in my experience fall into the Development Only category because of a few reasons:

  • Contract is Fixed Price or has limited scope
  • Operations, like Documentation, is the first to be cut from a contract to make the cost of work attractive
  • Business Developers sometimes fail to grasp long term affects of a short term project and too narrowly scope the deal in the hopes that contract once signed can lead to further work. Sometimes this bet pays off; but most others it does not.

Operations teams on the other hand sometimes have unfounded aversions towards developers because of a few reasons of their own:

  • Developers inherently create bad code that breaks functioning applications and infrastructure – emphasis on the infrastructure. 
  • Developers don’t really understand infrastructure or how their codebase can work on one set of servers and yet not work at all in Production

Merging of Development and Operations

Now that we have the basics, we build up and look at the merging of developers and operations. In my experience I have seen a few larger organizations where there is a such a large divide between developers and operations that members of the operations team end up becoming the defacto hidden developer, bug fixer, tester for developer teams. I have in the past fallen into that category when I started branching out from development and into the operations space.

In my opinion it is very good for developers and operations teams to have a rudimentary understanding of each others space. Yes there are purists out there that would contend otherwise, but in order to be an effective team and have a strong application for consumption this is a critical piece within DevOps. Not only should the developer understand the infrastructure that they are going to host their application on, but they should be developing on it as well.  There are edge cases about this argument, but in general this is a good practice. Likewise it is good for the operations team to understand the developer space and be given a crash course when the code becomes un-configurable and broken.

DevOps is about continuous ownership from planning the application, to IDE development, to source control, to build, and finally to deployment. It ensures that the developers understand that to put their code into production is not the responsibility of a select few Wizards of Oz, but developer and the operations team go along the journey together to ensure a smooth and proper deployment.

Advertisements

How to Update Octopus Deploy Tentacle to Restart Automatically

Seen this too many times before, where the users (Developers were unaware of a server reboot that caused their Tentacle to stop. Here is a defensive way to make sure your Tentacles start consistently.

Here is how to update the Octopus Tentacle in the case that the server is scheduled for downtime or just in general to avoid Tentacles being or considered offline within the Octopus Deploy server UI. Typically during installation of a tentacle on a destination server you don’t have the ability to recover gracefully if something goes wrong. This can be troublesome during patching or long server reboot times. Hmmm. For some automation here.

Open PowerShell as Administrator:

clip_image002

Open services.msc

clip_image004

Scroll to OctopusDeploy Tentacle service:

Right click and select properties:

clip_image006

Change Startup type to: Automatic (Delayed Start)

Switch to the Recovery tab:

clip_image008

Change the 1st, 2nd, Subsequent failures to Restart the service

clip_image010

Click apply or OK and you will now have a tentacle service that will be more stable after your machine boots from either a scheduled or maintenance reboot.

Starting From Scratch–Building Your Project Right Part 1

Prologue

Let’s say for the sake of argument that you just uploaded your codebase into TFS2017/VSTS. What do you do? XAML builds are deprecated and the new Build system seems daunting. Again what do you do?  You can watch videos and read tons of different stackoverflow articles and blog posts on how to… yet there are still lingering questions on how to “just” start from scratch.

In my experience with teams from around the world before they used a version control system, would happily code on their local machine, perform a local build (where is just worked), and then using the power of Visual Studio would “Publish” their fixes directly to their remote environments. For one that is poor a ALM practice and two there is just no way to track any of the changes that were either breaking or fixing described typically with screenshots in an email thread. Overall it was the Dark Ages, a chaotic time where teams that were trying to be Agily/Scrummy/etc, yet really having no anchor or starting point to leap off from for how do perform a proper “single source of truth” build; let alone a deployment.

With “DevOps” and “Shift Left” being the buzz words of the day, it can be hard to get your team in the correct cultural mindset of ownership and control. In this article we will dig into the new MS Build system as if you were a newly minted Developer Lead with the appropriate Administrative Rights in your TFS/VSTS project.

Here is a basic scenario and then we will work through how you can build your project right with the new Build system in VSTS/TFS2017.

Assumptions

  • You are just now using source control
  • Your builds consisted of developers performing builds on their local machines
  • You may had a build server in the past, but you have either upgraded or the build templates that you previously used are incompatible
  • You have Post-Build scripting moves files around to make your codebase viable for a manual or even scripted deployment

Scenario

This is barebones. Your particular scenario may not apply and I will discuss in the future how to do advanced builds and deployments.

Your codebase has been freshly checked into TFS/VSTS source control. If you chose Git or TFVC, it doesn’t matter the below techniques will apply for both version control types.  You need to perform a build that is simple and the output needs to packaged and ready to be consumed by either Release Management or Octopus Deploy. So where do we start?

How it’s done

Now for the pretty pictures.

First log into your TFS or VSTS site or account and click on the Build & Release tab. It should be blank if you are first starting out. The screenshot I am providing is from previous posts that build more and more advanced concepts.

image

You will see that there four items in the image called Mine, All Definitions, Queued, and XAML. You will not need the XAML tab; it has been deprecated and you can edit your old XAML builds nor are they compatible with the new Build system. So because of those points we will not be discussing that tab.

  • Mine – represents those build definitions that you the logged in user have created

image

  • All Definitions – represents all the build definitions that have been created for your solutions, branches, etc

image

  • Queued – represents all those builds that have been currently queued or running or completed

image

Now lets create a new build definition for our solution. In our case we are going to do something basic and build up from there.

image

then we get a popup that gives a number of options of generic templates to choose from. For now we will just choose the Visual Studio build. The reason behind this is that most developers are accustomed and acquainted with and it is in keeping with my idea of starting with the basics and building from there.

image

Click Next and you will see a another page where you choose your repository and other settings for your build.

image

The great thing about this page in the Build Definition Wizard is that you can make preliminary adjustments to your build before it is created. For instance you can choose the type (remote or local) of repository that you wish to point at, select the branch you wish to build from, and determine whether or not you wish to have Continuious Integration (build after every check in).  Subsequently you can choose the default agent for your build to be something different that has the capability you require to build your solution.

image

Make your selections and then click Create. Now you will have a “unsaved” generic build definition that you will need to continue editing. But first it will be wise to save you creation so that if you have to leave the page you don’t have to start all over again.

image

image

Now that we have saved the build definition, we can go into each of the build steps and one make adjustments and two add more build steps as necessary to perform our build.

On the left part of the page you should have noticed that there are six steps and descriptions for each. Lets talk about them a little in detail.

First you have the NuGet restore.

image What this little step does is go into your solution and check for the packages.config file and restore or install all of the packages each time you build. In this instance you can ensure that you have the everything restored from NuGet.org or other repo of your choice without having to check in your packages folder like you would have to do for your old XAML builds. With this step you can control a lot of what can happen with the installation or restoration of your packages.

Next is the Build Solution. This is a powerful step very similar to the older XAML Process. One item of note is that there are some overlapping of functionality that I have found for build steps, while it may not hinder a build, it just gives you further options for streamlining your process.

image

This will look a little familiar to those with a XAML build background, but this is a lot cleaner and can be adjusted to suit your needs. MSBuild arguments still work, but in some cases you don’t have add switches like “/m:1” when you can check the Advanced->Build in Parallel check box.

The Text Assemblies Step is standard with the Visual Studio Build Template. It will use Visual Studio Test to perform that function to ensure that you have testings completed and code coverage for the widgets you have on your Dashboard.

imageFrom a testing perspective this a powerful step that allows you to go into and perform advanced testing from the build without a lot of tools that you would assume that you need to perform the reporting, etc.

Also note that you see a image or this imagein each step it will show either a hover tip or open a new tab to more details about the step for your leverage or understand.

Next is the Publish Symbols path. It is a way for you to use your pdb and obj files to help debug your application on a different machine other than where your sources were built.

image

Next is the Copy Files to: step. This step is takes the output from your sources bin directory and copies those files or folders to your artifacts directory.  The artifacts directory is a cleaned directory that ensures that you have just the right objects in your artifacts folder that need to be either packaged or deployed at a later time. Again clicking on the image icon will provide options and parameters for you to use to make your build more coherent and robust.

So after the previous 5 steps, now what happens? Well the build agent then publish your outputs to a Drop folder. Typically this drop folder is within the $(Build.ArtifactStagingFolder). Where might that be? It is located under your build definition directory. Similar to this file path: “E:\TFSBuildAgent\vsts-agent-onprem\_work\5”. Inside of this folder are 4 folders that represent “a” artifacts, “b” build (output), “s” sources, TestResults (obvious).  Look at Resources->Variables for more information.

image

Once we have made our initial edits to your build definition, we can now Click Save and then queue a new Build.

image

and our build succeeded.

image

From our log output we can know that the artifacts are “published” and a little magic later we can compare and see that the artifacts are in the location where they are intended to be.

image

and finally…

Don’t be afraid of the build

As the heading suggests, when you are first starting out with using TFS/VSTS Build, don’t be afraid if your build fails or doesn’t perform the functions you are expecting immediately. I cannot stress this enough. I know that when I am testing builds that I spend a lot of time troubleshooting failed builds before I finally have a successful (green) build.

By working through your issues and using a methodical approach you can be successful. You shouldn’t feel pressured to get it right the first time. You don’t have to feel like you are going to get fired for having a broken build. Work through it, understand the build process, then communicate the understanding to your leadership.

If you are developing a Web Application or web api you may have to consult this how to. But if you are doing a single applicaton (exe, service, etc), then this approach can get you started in the right direction.

Resources

Here is another good resource about build settings and build tokens https://www.visualstudio.com/en-us/docs/build/define/general 

https://www.visualstudio.com/en-us/docs/build/steps/build/visual-studio-build further details about the Visual Studio Build. Word of caution here is that if you are using TFSBuild.proj type of file, you will not be able to use Build because it contains tasks and targets that are supported only for XAML builds.

https://www.visualstudio.com/en-us/docs/build/define/variables this is for getting the different built-in and custom variables to work in your favor.

Building Multiple Projects and Pushing to Octopus Deploy with Build vNext

In some cases there is need to have multiple projects within a solution that each require being packaged and then published or pushed to your repository of choice for deployment.  Typically from a Web Application perspective you could have Web Services, Web Api, and finally the Web Application itself.

What I am about to detail for you is performing these actions with success using TFS 2017 and Octopus Deploy.  These same actions can be applied to Visual Studio Team Services (VSTS). Be warned that this technique will not apply to older XAML build definitions.

Assumptions

We are going to assume that you have basic solution with .Net MVC Web Application, MVC WebApi, and Windows Services projects. You can more than this, but for the purposes of this demonstration this is what we have. Something like this:

image

And we have the solution checked into Source Control like this:image

You will also want to make sure that you have Octopus Deploy extension installed or applied to your account or TFS instance.

Where to Start

Well the most common thing here is that you will need to create is a basic Visual Studio vNext Build Definition. Log into your Project and Navigate to Build & Release and then click New in the right hand corner.

imageimage

Now we are going to Scroll down and choose a Visual Studio build template. I won’t go into the details of all the different templates are, but for this exercise we will choose Visual Studio because it is a very quick way for you to get started.

image image

Now that we have our basic build definition in place it should look something like this.

image 

Now save what we have so that we can start with the meat of this exercise. The next little preparation that you have to do make sure that you create a Service Endpoint for your Octopus Deploy input. I will be using Built-in Nuget Repository, but can also use and external NuGet repository of your choice. The fields that you will need to fill out are listed at points 3, 4, 5. You need to give the connection a meaningful name that you can remember. You must provide a url to the octopus instance that you intend to publish to and finally you will need to have an API key with the minimum permissions to publish to the Octopus internal NuGet Repository. Rule of thumb here is to create or have a Octopus Service Account that has the appropriate permissions.

image

How its done

We now have the basics set up. So lets get into the requirements for getting multiple projects to be compiled, packaged, and pushed to Octopus Deploy. Previously you would have to have a lot of pre-work done to your actual project/solution, e.g., adding OctoPack, creating a nuspec file, etc. With the new build system and its extensibility, it makes your projects/solutions much cleaner and much simpler to use especially if you add more projects to your solution that require packaging and publishing later on.

Edit your saved build definition and select Build Solution step. We need to make an adjustment here to ensure that

  1. We get a separate folder for each project
  2. We ensure our Output Directory for the solution is pointing to our ArtifactsStagingDirectory

Like so.

 image

What this will do will ensure that the build output from the step will go to the build artifacts directory on your build agent, similar to this:image

The “a” in this case means Artifact.

We will not worry about the Test Assemblies step right now and we can also leave the Publish Symbols and Copy Files Step and the Publish Artifact alone for now. The point of this post is to get your projects built, packaged, and published for deployment. Next what we need to do is add another few build steps to our Build definition.

Click on Add Build Step

image

And then we are going to Add the Octopus Package Application step, one for each of our projects that we wish to publish. Then we are going to add the Octopus Push Package(s) to Octopus step to finish up our Build.

image

Now for each of the projects you will need to edit the Package Id, Version, but more importantly you need to know what and where your source and destination path is going to be. If you don’t these two items correct, your build will fail.

image

We do know that our output is going to the artifacts directory and we know that each project is going to have its directory. But a critical component here is what the Octopus Step is going to do. It is important to note that we want, must, need to do is

  • Keep your package small
  • Take only the most relevant codebase

I have seen so many teams in the past and currently take everything from their source tree and package it into a 100MB+ file and attempt to push it to Octopus Deploy then deploy all of source tree to production. It is really hard to contemplate the security risks of putting your entire codebase for someone to steal and compromise.

For web applications and web api projects there is typically a _PublishedWebsites folder, this folder contains a reduced build output with only the most relevant dlls, config files, and other content for your web application or web api to run. Everything else is irrelevant and if you choose the folder at the higher level you will essentially get two copies of the same code base in your package. Citing the two points above, keep the package small and only package the most relevant code! This doesn’t mean only the files that have changed! This means your most minimal amount required run your application.

Now that we have edited the Package steps, we can move on the other important final element to this post, which is the Push Packages to Octopus.

Select the Push Packages step and you will notice that you need to edit the Octopus Deploy Server (this is the Service Endpoint that created earlier) and the most important item is the the location of the packages.  Earlier in this post I told you that you didn’t really need to worry about the Publish Artifact step. The drop directory is where we are going to put our package after the build and packaging steps and that same directory is going be where we find those packages and publish them to our Octopus Deploy NuGet feed.

image

Now we are done! Save all the work that we have done and then queue a build. Don’t get frustrated if you your build doesn’t register success. Take a look at the logs for each build you attempt and step through to ensure that your folder paths are correct. Ensure that the API Key has correct permissions to publish to the Octopus NuGet Repository.

If you do have a successful build it tell you that it performed the compilation, packaging and publishing of the packages. Similar to this.

image

SNAGHTML10847104

image

Checking each step to ensure that the logs are telling me what is happening.

image

image

SNAGHTML108a0b87

and finally verified it has been pushed to our Octopus Deploy instance.

SNAGHTML108ba689

Next Steps

Now that I showed you how get your build to package and publish to Octopus. You may wonder what about CI/CD? or what can I do now?

To answer the second question, you can actually go ahead and make this a task group and save it for other teams to utilize in the future. You can also use this technique for other project types as well. Once you understand how you can create a package with the Octopus Deploy Extension, you can change the build definition to a CI/Timed Build and get those packages published for the next phase.

To answer the first question: That is for another post.