Poor man's CI for BizTalk Server

Many have tried, but very few are telling success stories when it comes to continuous integration with BizTalk Server. In this post I will show one way to tackle some of the management issues related to deploying integration solutions built on top of BizTalk.

Event though I have been working with lots of different integration environments that run BizTalk Server, I haven’t seen one with a real CI setup. Normally nothing gets build from the version control data, instead it is always the codebase from the developer’s machine that gets built. And one day you realize that the developer has left the company and has never checked in that one cruzial map to version control. Most of the time artifacts end up being stored somewhere in the server’s disk or even worse in someone’s email. After a while you don’t know which assembly contained which changes and you end up comparing timestamps and assemby sizes. Deploying new versions involves manual steps and getting new versions through deployment pipeline can take months in the worst cases. Automated unit testing seems like a distant dream.

A nightmare from a CI perspective.

For years I have been thinking that there must be a better way to manage deployments of the integration solutions. Even my Masters thesis was related to the deployment process, but didn’t take any stand on the technical aspects of it. Lately I have started to think whether there would be at least some part of the process that could be somewhat automated and taken to more controlled level. I have no high hopes about ever seeing an BizTalk environment with dedicated build servers and logic that can intelligently, and automatically, deploy the most complex integration solutions without any interventions. But maybe there are some low hanging fruits that could be picked without reching to the moon? Let me introduce to you: Poor man’s CI for Biztalk Server.

What do I mean by Poor man’s CI? For me it means that getting new integration solutions and especially changes to the existing ones to production is as controlled as possible - without spending weeks of manhours or thousands of dollars to setting up the process. For now it means that the process has some similarities to a real CI process when it comes to what is deployed and how, but still involves rather lot manual steps. Someone has said that if you don’t do multiple deployments to production per day, then you are not doing proper CI. In that sense, this system certainly is not real CI.

Building from the version control

This is a part of the process where I really don’t have a good automated workflow. It would require a real build server. One way to secure that everything actually is checked in to version control and the codebase builds is to make sure that building the actual deployment packages is done as part of the code review process by someone else than the developer who has implemented the code changes. This way the code has to be pulled from the version control and only that version is taken forward. Not 100% bulletproof, but poor man’s tactics.

Versioning

Versioning BizTalk solutions, ouch. Even though being technically possible, how many of us really want to take that path? In my experience, using assembly versions to differentiate versions of a certain integration solution will only cause headache when it comes to BizTalk. And that is sad.

I really like the way you can have automated builds and deployments directly from version control for example with a normal ASP.NET project running in Azure. And you can go back to any previous version just with the click of the button. To have similar ideology with integration solutions, I have come up with a manual way of achieving some of the same aspects. Instead of versioning the assemblies (or other components related to the solution), manual versioning is used: all the solution changes that are being deployed in the same time, are put to same “version”. Couple of benefits that comes with this approach:

• You can have a clear change log for the solution
• You can easily have a managed way to store snapshots of the complete solution
• It is easier to communicate the changes to the stake holders as they are in most cases used to software versioning
  

The downside is that this involves manual steps and versions will be somewhat artificial. You also have to have the guts to say no to the stake holders when they want that one more change to the package event though it has already been shipped forward. No updating one tiny assembly on the fly, its always a totally new version that goes to the next deployment round.

What this means in practice:

• Deployment packages always contain the whole solutions, not just the changed bits. Including scripts to create/update database components. Also configuration information for all environments are included (bindings, sso-settings, etc.)
• Deployment packages are stored in a formal folder structure in a central location accessible from all needed environments. The folder structure that I have found to work quite well is following:

As you probably realize this approach is trusting that developers and other people involved pay extra attention that the folder structure is not messed up and old versions are not touched. On the other hand you can take any version of the solution and deploy it to any of your environments from one central location. No more moving deployment packages and figuring out what to update. It’s all there in one location.

Tooling

There are some tools that can ease the pain related to managing all the bits and pieces involved in the deployment process like Deployment Framework for BizTalk (BTDF). Great tool, but has also it’s limitations, maybe the biggest one being the dependency to the BizTalk application structure. You really can’t have multiple logical solutions within same application without having them in a same deployment package.

BTDF itself really doesn’t force you to follow any specific process, you can still send the deployment package via email, if you choose to. BTDF is customizable to some extent, but there are limitations to that too. Therefore your process needs to be able to adapt to the way BTDF has designed to work.

You can also automate most of the deployment tasks using powershell. There is the Biztalkfactory powershell provider, that is now also shipping with the latest versions of BizTalk Server. Or you can use powershell to execute BTSTask jobs or use WMI classes.

Using powershell scripts don’t force you to any specific process any more than BTDF. You can still mess around, but in my opinion using powershell gives the needed flexibility so that it can complement your process - not the other way round. It probably takes a bit more time to get the initial scripts done (maybe topic for my next post), but once you have figured the baseline for your deployment activities, it takes no more than five minutes to create a full deployment script for a new integration solution. And you can still use the excellent tools from BTDF for example to manage binding files or SSO-settings.

To implement the Poor man’s CI process, the deployment scripts should be implemented so that they support the versioning style described. In most cases you don’t need to change the deployment script when you change the version of the solution, and that is one of the key points also. Deployment scripts should have at least the environment and version as parameters. Or you can create a dialog that asks for that information. Or you could have the latest version of the integration stored to some metadata location and just have -latest as a parameter for your script. What ever works for you. The idea is to just run the same script no matter what environment you are deploying to or to what version you are aiming. With this approach the deployment is still not automatic, but the manual step only involves giving the parameters. And that can be done practically by anyone without knowledge about the solution (or BizTalk).

Some points that needs to be taken into account:

• Scripts should have some failsafe logic, for example terminating suspended instances before trying to deploy
• Removing existing deployment enables moving from one version to another more reliably. This also makes sure that your deployment package contains all the needed artifacts if you first remove the old ones.
• If your script needs to be run on certain machine (for example you should only be able to import PROD bindings to PROD server), check the host when the script is executed
• You need to take care of passwords manually

Using scripts for deployments enables you to add more visibility to your deployments. For example you can add auditing log as part of your scrips: what version was deployed? who did the deployment? when exactly was the deployment performed? what was the outcome of the deployment? We have even added Slack notifications to the scripts in my latest project.

Summary

In this post I have described one way to tackle some of the challenges related to managing deployment process when using BizTalk Server. These methods cover only some parts of a real CI process and most of them still involve manual steps. Therefore I call it Poor man’s CI process. Even with it’s limitations, I think that with few simple steps it is possible to improve the quality of your deployment process quite a bit. So:

• Make sure that everything is checked in and builds
• Create a system for versioning all the integration solutions and use change logs
• Create a system for storing complete deployment packages in one central location
• Implement scripts for deployments so that you can be sure that deployments are always done the same way. This also makes life lot smoother as you have lot less manual steps to execute

Future improvements? Lots. Even though I don’t dream about fully automated deployments just yet, I think build server that would automatically generate the binaries from version control to the deployment folders would improve the process quite a lot. The possibilies for adding more “fun” logic to the scripts are endles: how about some flashing light when deployment is completed? I have even done some research about controlling the deployment scripts through a website - that would not be very far from doing the deployments just by a click of a button any more.

As always I’m more than interested on your thoughts and tips. How have you approached the CI process with BizTalk? How would you improve my methods?