Building a DevOps Assessment tool with VSTS, Azure PaaS and OneShare in minutes.

A few weeks ago we joined a global DevOps hackathon from the Netherlands. Our team, creative named ‘DutchOps’, decided to make a DevOps Assessment tool. The hackathon was mainly about sharing, building and transferring DevOps knowledge. 

The System.

The realized system is cloud only. A DocumentDB (see diagram at the bottom) stores all questionnaires. This makes the system flexible to change question types. An Angular Web App loads the questionnaire based on an questionnaire type and renders the questions.

The respondent answers the questions and submits the filled assessment to an Azure Queue. This decouples the system and makes it easier to switch analysis. An Azure function triggered by the queue puts the filled assessment in the DocumentDB, just as a backup when we decide the analysis needs to change. We simply can rerun all documents via the queue. The Azure function also put the filled assessment through a Service Bus topic. Based on the Questionnaire Type it sends the filled assessment to an Azure Web Job or Azure Function.


The Azure Web Job, Azure Function layer of the system does the heavy lifting, analyzing the filled assessments. When a new questionnaire type drops in a new Azure function can be created for the analyzing. Easy extensibility.

When done analyzing the results are dropped in a storage. For now, SQL Azure, only because PowerBI Embedded can handle only SQL Azure, SQL DWH or HD Insight Spark. PowerBI is attached to the result database for overall monitoring. PowerBI embedded is used to render the reports in the Web App for the respondent. When new reports are created for other questionnaires they easily can be added to the PowerBI Azure Workspace, without the need for redeployment.

This system architecture has several benefits. Due to its loosely coupled behavior the different parts can be created, tuned and evolve separately. Also the system very easy can handle new insights in the assessment. New questionnaires with new analytics and reports are added without the need for new deployments. New question types have only a tight dependency on the frontend, a new view section is only what is needed to handle this.

The DevOps Assessment.

While the designed system can handle multiple types of questionnaires and question types, the main goal was to develop a DevOps assessment system. Our subject matter experts based the questions around the CALMS pillars, Culture, Automation,  Lean, Metrics and Sharing. As written in this paper, a clean balance between these pillars is necessary to be successful in DevOps.

DevOps is so much more than just a set of continuous integration and development tools. Proof of that can be found in the so called ‘CALMS pillars’ within the DevOps community. A keen balance between those pillars is crucial for an effective adoption of the DevOps mindset in every organization. Too much focus on automation will lead to an organization which has a state of the art toolset, but which lacks collaboration, empathy and the capability to improve.



The Team.

The team existed of a blend of people and preferences. For sure Visual Studio Team Services is used for the main team activities like task board, code repository, build and release. But there is more to tell.


Collaboration and tools.

The main and ad-hoc communication went through Slack. People are working ad-hoc, distributed and irregular on the system. A central, async and rich messaging system is a must. The where some in person meetings (couldn’t attend none of them) with the product owners and subject matter experts. And some one on one meetings between team members. So, Slack to the rescue. Several times Skype for Business meetings where planned but due to the ad-hoc character and irregular working hours of team members a synchronize collaboration tool didn’t work out. (in 2009 I co-authored a book about collaboration in the cloud).

Initially documents created by team members where shared via Slack, but they got lost in the messages stream. Although you can pin messages in Slack, it is not a document collaboration tool. For this we started to use a SharePoint site on Office 365. Much more comfortable with a better overview and versioning of the documents.

Also on the SharePoint site we configured Sogeti OneShare SharePoint Add-on. It shows the costs we were making on Azure resources, the state of the Azure resources and makes it easy for team members to create new environments. Later more on this.

VSTS Task board and Trello.

The product owners and subject matter experts where hooked on Trello, and where creating backlog items from there. The development team wanted to use the integrated DevOps tool Visual Studio Team Services, not only for backlog items and tasks but also for code, build and release traceability.

Zapier to the rescue. Zapier an API, Web hooks integration tool, which supports an enormous amount of endpoints. Every time a Product Owner adds a card to the Trello board the VSTS Backlog is updated.


We only enabled a one way update, from Trello to VSTS Backlog. This isn’t ideal, it is actually not ideal at all having half connected product owners. It results in backlog items which are out of sync with the team.

IDE’s, builds and release.

The development team used a nice blend of development environments, Visual Studio, JBoss and Eclipse. With GIT on VSTS as our private code repository this wasn’t a problem. They were working on .NET code, PowerShell Script, Angular JS and Azure Resource Manager JSON, all pushed in the GIT repo.

Pasted image at 2016_05_30 12_06 PM

For the branching structure default GitFlow is used, which works very simple and structured. One mistake with version control was that we only create one GIT repo for the all components. But, because the components are all very loosely coupled it should have been better to make a repo per component. They would be easier to Builds and Release them separately.


Builds are running for all components, except the Azure Function with his beta state has a limited integration. Two build agents are used, one VSTS hosted and one private build agent configured on a Azure VM.

The Angular frontend (build on a JBoss IDE) simply has a NPM install – production build step with a publish artifacts. (we know, no Jasmine or Mocha unit tests executed on the UI.)


The API App and Web Jobs have a NuGet package restore step, a Build Visual Studio Solution step with MSBuild parameters to package the solution as a zip (/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true) . With some unit tests steps and a publish artifacts.


And there is a build for the Azure Resource Manager, the definition of all needed Azure resources, with a single publish artefact build step. This can be extended with some actions or validations we want to do on the JSON. For example, set parameters or combine templates from different resources.


The private build agent is supported by Azure DevTest Labs. The scheduling and create with artifacts capabilities makes it easy to use for the team (see this post: Azure Dev Test Browser Lab for Visual Studio Team Services. #VSTS #Azure).

Releasing all the different components and azure resources is done by VSTS release manger. All builds drop packages, zips or other release artefacts to a drop folder. A release process is triggered by this build drop and makes a new installment of the system.

The angular App is deployed with MSDeploy executed via an Azure Powershell script, with the data extracted from the Azure Web App publishsettings file.

cmd.exe /C $("msdeploy.exe -source:IisApp='$wwwroot' -dest:IisApp='$websiteName',computerName='https://$publishURL/msdeploy.axd',userName='$userName',password='$password',authtype='Basic',includeAcls='False' -verb:sync -enableLink:contentLibExtension -enableRule:DoNotDeleteRule -retryAttempts:2")


The last PowerShell release step is to notify the team members via Slack, with this command.

The Web Job App and API App are simply deplyed via the Azure Deploy Web App release step.


For the Azure Resources updating is done via an update on the ARM Json. First on the Development Integration environment than on Production. The update is done via release management Azure Powershell step.


Experiments, Flags, Environments and Sogeti OneShare for PaaS.

Experimenting with the implementation and used Azure Resources is important for teams. When the team uses a feature flag supported system, not really a feature flag implementation is used but due to he loosely coupled components we can release new capabilities only for testing or validating first, there is no direct need for multiple environments.

There is only a development integration environment and a production environment. Development integration is used for validating the integration of the different components and for validating the updating of the system via ARM. So, we know that goes well before putting the feature with flag in production.

There is no space for experimenting with Azure resources, try new capabilities, new resource types or different configurations. To enable this for the team we are using Sogeti OneShare for PaaS solutions. Next that OneShare shows the costs, also per Resource Group it also has the capability to create new Resource Groups based on a template.


This makes it very easy for team members to create a new Resource Group with all needed Azure Resources for the system.

This easy creation of new Resource with a template of the Release step to deploy the system makes teams able to do experiments in a frictionless way.




No system without monitoring and validating. Lucky we build an Azure PaaS system, which already comes with create monitoring capabilities. But to extend the default monitoring and logging of the Azure App Services we added Application Insights monitoring.

Application Insights monitoring can be set on different levels within the application. The first layer of monitoring is availability checking. Initially this Azure capability only did a URL pings, but now also has the feature to run pre-recorded Visual Studio Web Tests. These Web Tests playback on the HTTP layer a specific scenario and report the results in 5 or 15 minute interval. It gives some good insight in the performance of the system and reports odd behavior and drops in performance.


(The drop in performance of the API services was due to a deployment. Not a load balances incremental updated system.)

The other layers of monitoring are the client side code and the server side code. Easily configured within Visual Studio and JS.


and the last layer of monitoring the dependencies the system has with others. See the Power BI API call.




It was a nice hackathon. All the above we setup and created during a few hours of ‘evening’ work. Having already an end to end DevOps SaaS tool like VSTS saves a lot of setup and configure time. Build a PaaS-only solution also saves a lot of infrastructure configuration time, together with the out of the box monitoring capabilities it is great. Focusing on loosely coupled components with the capability to release to production with a flag also saves a lot of time, only one additional environment is needed for update validation of the Azure resources.


But, as fast as the team could go, collaboration is still the key to success. Special with a distributed and irregular working team like this. O365, Slack and VSTS where the collaboration tools.

Pingbacks and trackbacks (1)+

Add comment

Şarkı Sozleri