Make your team cost aware … cost driven development.
Cost is an interesting thing when developing on and for Azure.
A team now has direct influence on the run cost of the different environments. Not only development and test but also staging and production is in their influence zone.
With the right insight in environment cost a team can make architectural decision based on environment billing information.
For example, within Azure you can put all the code in an Azure VM, which is easy (lazy) and pretty expensive when you have them On all the time. Specially when you have Azure VMs for al stages.
See graph below, a team before 22 November everything on VM’s and always on, after 22 November with some more awareness the costs went down with almost 70%.
Project Azure costs, before and after being aware of the environments costs of their environments.
Other example, moving functionality from VMs to Azure Cloud Services lowers the price a bit (and finally also lower the operational costs). But moving Data from an Azure SQL Server to Table Storages is also a very beneficial move. Or moving a Worker Role to Azure Schedule Jobs or Web Jobs really pays off.
Below the Azure Billing costs of a project which tightly makes architectural decisions based on costs. For the team it is a challenge to constantly lower the daily cost.
With the daily updated Azure Costs insight they can see bad decisions (see spike) and good decisions (the big drop) immediately.
Also the slight rising of daily costs over time is something to pay attention to. Although the teams adds new functionality constantly the costs stays almost the same on a daily base.
Teams can get with Azure much more cost aware. For this to happen teams need daily updated insight in costs.
Read more: http://clemensreijnen.nl/post/2015/01/17/Insight-and-Self-Servicing-needs-for-Dev-Test-Teams.aspx
The DevOps battle field … provision, configure and deploy environments.
The current battle field in the software industry is among the provisioning and configuring of environments, specially cloud environments.
To live the DevOps dream one of the skills a team must have is the capability to be fast and flexible in the provisioning, configuring and usages of environments.
These environments cover not only test environments but also the complete needed ALM infrastructure and the team member workspace. See this post: Cloud usage flavors for Development and Test teams.
An interesting part of the battle field gives this survey Deployments in .NET 2014/2015 Report
although the list of tools is pretty long it isn’t compete and more are coming almost on a daily base.
Which deployment tool and technology to choose is really hard and often depends on the experience of a team member or the tools the operations department used to use.
Infrastructure as Code
While Deployment technologies is one part of the battle field and other part and even more important is the provisioning and configuring of environments. The Infrastructure as Code technologies.
For example this is a small list of tools which automate the provisioning and configuration of environments.
To make all these technologies of provisioning, configuring and deploying valuable it must be done in relation to a release, to customer value.
Release Management tools which support the flow and relate it to functionality, reports and tests are a must for teams. Support of tools which help to integrate the environments and releases in to the complete Application Lifecycle.
Microsoft Visual Studio Release Manager is one of these tools, another one is HP Codar. And probably there are many more….
Investing leaning time (and money) in the moving target of Provisioning and Configuring (Cloud) environments is a must for every team. As with all investments, it can be a waste of money. When a technology or tool doesn’t take off, or the capabilities aren’t what you expected.
It is for teams often hard to get a grip around what to do, which technology, which tools. My advice: start by asking the operations department (the Ops part of DevOps)is a good start to make a decision.
Insight and Self Servicing needs for Dev & Test Teams
Two important aspects must be given to teams who work from and on Azure. Insight and Self Servicing.
Insight in usages of Azure resources (and money) in relation to the team activities. When you are in the automation business, you are lazy by default. Team members will forget to switch off machines. Team members will leave test environments running, even when there aren’t any test runs going on or planned.
When you give team members direct insight how they are consuming Azure resource in relation with their activities, they will pay a lot more attention to it.
With this insight the lower cost, pay per use, scenario will really take off.
Self servicing and some help with it. Team member who are busy for days to setup their work machine isn't from this age any more, even teams who are busy for days setting up (and configuring) the complete test infrastructure or automate build and test environments isn’t accepted by customers anymore.
The creation of machines on Azure is pretty easy, but still additional software installing and configure for example networks takes a lot of specialized knowledge not every team member has.
Making it easy for every team member to create environments, also the ones without PowerShell knowledge or even Azure services knowledge, is key for the success of using the Cloud for your teams.
Dev and Test Teams Portal.
At Sogeti we created the OneShare service and solution for teams. An O365 hosted portal where every team/ project gets a site, a Visual Studio online account and an Azure sub-subscription, all connected with each other. to give insight and self servicing for every team member.
For more details watch this video…
DevOps technical meeting video: Ent DevOps | Cloud Teams | Dutch harbour case | Docker containers for DevOps
Live streamed on 16 Oct.. 2014
Welcome to the Live stream of our DevOps technical meeting!
Program is as follows:
18.00-18.45u Introduction & Enterprise DevOps (Dave van Herpen, Sogeti NL)
18.45-19.30u Cloud Team, OneShare (Clemens Reijnen, Patriek van Dorp, Sogeti NL)
19.45-20.30u DevOps collaboration in the Dutch harbour (Peter Siepel, Schuberg Philis/Loodswezen)
20.30-21.00u Use of Docker containers for DevOps practices (Benoît Wilcox, Laurent Guerin, Sogeti France, via webcast)
3 Challenges when Testing from and on the Cloud.
When adopting the Cloud for your test work it comes with several challenges. Looking at the different usages flavors of the cloud for teams, read: Cloud usage flavors for Development and Test teams, then every flavors has its own challenges.
Challenge 1: Team Workspace in the Cloud – Test Environment On Premise.
The scenario is that the team adopted Team Member Workspace on Azure, but the infrastructure (test environments) are staying on-premise.
The planning, preparation, specification and execution of the test work takes place on the team member workstation which is hosted in the Cloud.
The challenges is that the execution of test are triggered from the public cloud and the test runs on the private data center. There needs to be a connection between the team member workstation and the test environment. The same counts for automatic test runs.
Solution: make a network connection between Azure and the On-Premise datacenter. Often a very long discussion with the internal infrastructure people is needed to get this working.
You can create a Site-to-Site VPN, for connecting datacenters to an Azure site or Point-to-Site VPN, when connecting a single machine to an Azure site.
Challenge 2: Acceptance Test Infrastructure in the Cloud – Production On Premise.
the scenario, all environments (development and test) are on Azure except production, this one still lives on-premise. The challenges is to get the test and acceptance (on Azure) environments as equal as possible to the production (on prem) environment.
Solution: The best solution is the usages of the same installation scripts for production as for acceptance and test. Make PowerShell DSC scripts for and re-use them for every environment.
there is also a need for stubs to make testers need to focus on writing isolated tests, which will make more systems valid. Still not all tests can run on Azure Test Infrastructure (performance and load tests won’t give valid results).
the worst case scenario is that an additional pre-production environment needs to be created on-premise for tests which couldn’t be validate in the cloud.
Challenge 3: ALM Infrastructure in the Cloud – test, acceptance and production on- premise
The scenario: The team consumes their ALM infrastructure from the Cloud and use on premise workstations.
Actually from a technical perspective there isn’t a challenge in this scenario. The only need is reliable https connection between the Test case repository (ALM Infrastructure ) and the test environments.
Interesting in this scenario can be the test data. There is probably a security reason why the environments (test, acc and prod) are in the local datacenter. When the test cases, test data and test results are stored on Azure, the security regulations probably also count for that data.
Often production data is used as test data resulting in a no go for this scenario. A solution is to anonemize the test data.
these where just three challenges for the three different components of Cloud usages for Dev and Test on Azure.
There are many more challenges, maybe in another post.
Cloud usage flavors for Development and Test teams.
The cloud for teams have many forms.
Teams can simply use a virtual machine on Azure to work from, the team member workspace flavor as I call it.
Or, they can use the bug tracking database, requirements management, and scrum board from the Cloud. In either a SaaS way, like Visual Studio Online (www.VisualStudio.com) or as an IaaS solution (see Practical guidance for TFVC and TFS on Azure IaaS). Both are the ALM infrastructure in the cloud flavor of Cloud usages by teams.
Another way teams can use the Cloud for their work is that they use it for their Development and Test Infrastructures. For example, when you develop a system for an On-Premise SharePoint farm team members need to have a SharePoint Farm. Developers need the infrastructure for development, debugging and integration, testers for sure to run tests for the system in development. It is the Development and Test Environments flavor.
When adopting the cloud for your Development and Test work, all three flavors are interesting to look at. You can consume them separately but when using all together you are really riding the cloud wave.
Integrate Azure, Visual Studio Online and Office 365 with VSO Rest APIs.
With the just released Visual Studio Online Rest APIs you can do some wonderful things.
We create a solution to support project teams who use Azure for their development and test activities. Teams who use Visual Studio Online for their ALM infrastructure and Azure VMs for their Development and Test environments (as most Sogeti Teams do :-).
The challenges when running heavy cloud based teams like this, is that you want to let them use the Azure and Visual Studio Online resources as efficient as possible. Give them easy to use features to create Development and Test Environments is key. But, also too often VMs keep running for weeks, while no test runs or any code change / team activity has happen. And from a few days ago, Visual Studio Online licenses you want to monitor unused licenses.
Teams should stay Fast and Flexible, and must have insight in what they do, to be efficient. Via this portal we give Teams insight and tools to use the Azure and VSO resources as efficient as possible.
The portal gives information about VSO Licenses used, build and test runs, Azure IaaS state and overall Azure costs, everything that costs money when running your project.
This all works in a SharePoint O365 Provider Hosted App, with Microsoft Azure Management Libraries and the Visual Studio Online Rest APIs consumed via a WebAPI, all hosted on Azure Cloud Services and secured by Windows Azure Active Directory.
Teams get information like:
Azure Development and Test Environments usages.
Beside Azure VM’s state, start, stop and RDP connect there is also the capability to add usage notifications and a schedule rule engine to the VMs. So team members are notified when a VM is unnecessary on. Mobile apps receive the notifications and have the ability to stop the VM.
Build usages and history.
For teams to see the build usages and to get an indication if build environments are proper used.
Visual Studio Online License information of all the team members.
and Team activities.
And as last and Overall Azure subscription costs.
Every project has its own VSO Account and Azure Subscription (in the Azure Enterprise Subscription). In this subscription they maintain the VSO licenses and their Development and Test Azure VMs. This give the team control over what they need (add licenses and developer machines when they need) and insight in the true costs of the project. We upload the terrible excel output from the Enterprise Azure Portal to an Azure SQL database.
All in Office 365 in a single project portal to enable Fast and Flexible Teams. VSO Rest APIs Rock …
The missing knowledge of testers in modern app development.
The game has changed for testers, they need to extend their basic testing skills when they want to stay valuable. As other team members, testers need to be a full member of the team with the current skill set this isn’t.
While supporting teams for a long time in delivering software I noticed I’m always teaching new ‘test’ team members the same skills. Skills a new team member actually already should have had before entering the team room.
A small list.
Knowledge about agile work in teams is the minimal a test team member should have.
When they fail a basic assessment like the Scrum.org open assessment ( https://www.scrum.org/Assessments/Open-Assessments/Scrum-Open-Assessment ) I silently start to cry.
Agile is that mainstream now a days that I even wonder why conferences like ‘The Agile Testingdays’ exists. The practice of agile test work should be crystal clear at this moment. So, mastering agile work is really the bear minimum.
Team Tools support.
Know your tools. Not only the common testing tool ‘Excel’ but more important the Application Lifecycle Management (ALM) tools the team uses.
ALM Tools like Visual Studio Online and other tools mentioned in the Gartner ALM report are important.
Magic Quadrant for Integrated Software Quality Suites
Gartner’s Magic Quadrant for Application Lifecycle Management
The tester isn’t alone anymore, testing isn’t the activity at the end of the waterfall what everybody skips, it is part of the delivery. Having knowledge how to use integrated team tools is a must to be valuable and not being a blocker of productivity.
Basic programing skills
Test automation is a must in the current fast delivery mindset of Continues Delivery. And this test automation must be, just like all the team work, integrated in the team process.
Standalone test automation tools are useless. Test automation connected with the team sprint tasks, the team build runs, the team delivery and the code base is a must for agile teams in being efficient with a focus on quality.
When test automation is in the same code base, in the same rhythm as the whole team, the code written for this test automation is logical the same programming language as the language the system is developed in, at least it is the most efficient.
Having basic programming skills as a tester is a must to be valuable for the team. As the image above shows, only just a small part is manual testing. So start to learn programming.
(Distributed) Version Control
Every artifact a team creates should be under version control. For the codebase it is normal to use a version control system. Either a centralized system like Visual Studio Online or a distributed system like GIT, everything is under version control.
All team members must understand the basics of version control. How to get or pull artifacts in or out and the pros and cons of the usages of branches.
Testers need to be capable to get the version they want to test from version control them self, they need to understand what artifacts are in the version and what features are delivered at what quality.
Knowledge of version control systems will make testers less depended on the other team members who do know what is in what version.
A good read: Version Control for Multiple Agile Teams
System under Test Deployment models.
As mentioned in the previous needed tester’s skill, testers should be less depended on other team members to get their system under test. Knowing how to deploy the software is a great added value for testers. This is for all kind of systems for example, knowing how to install a Store App on a tablet for testing will make the tester immediately more productive.
For more complex systems, knowing how PowerShell, Chocolaty, Puppet, Chef, Visual Studio Release Manager or other deployments mechanisms work would be great.
But at least understand how to start the deployment as a tester is the minimum. In the same category understand how to setup a test infrastructure for automated testing would be very useful.
Understanding the capabilities of current monitoring systems is a big benefit for testers. A tester who understands the behavior of the system under test is a invaluable benefit for the team. This tester helps to solve bugs as quick as possible by providing valuable usages statics, trace files or intellitrace files.
The last skill for a testers I want to talk about is Cloud. When a tester wants to stay in business, he or she needs to understand and usage the Cloud for testing.
Beside easy onboarding of new team members Dev & Test in the Cloud also can be used for your complete test infrastructure and with the new gained deployment knowledge also the provisioning of test environments should be easy for testers.
See this deck: Azure for Teams
I can imagine you think ‘as a tester’ this isn’t my job role, these are the activities and skills of other. But as James Whittaker mentions in this post
“For the vast majority of app development on this planet, software testing is an activity within the development process …. It’s time to bring quality into the 21st century where testing is such an integral part of software development that you’ll often forget you are doing because it has become so familiar.
Testing is integrated in modern development, a testers must evolve. To stay valuable for teams and keep their job the in this post mentioned skills must be added to the capabilities of the modern tester.
Not only these capabilities count for testers, all other team members should master them too. So there is also some lessons to learn for developers.
DECK: Azure for Teams
More business value with fast and flexible teams.
The deck I used (parts of it) at different conferences.
Things to think of when using Team Member IDE on Azure, networks.
Azure for Teams
Having Team Member IDE’s on Azure provides create benefit for teams in moving fast and staying flexible.
But there are still some items you have to think of when start adopting it, networks being connected is an important one:
You never ever develop or work in isolation. Especially with the heterogenic systems we build now a days. So, being connected with the world from your Azure hosted Visual Studio IDE is a must.
For example, the SQL Server database you are consuming in your App is in the local data center. You need a connection to that datacenter from your Azure environment.
You can create a Site-to-Site VPN, for connecting datacenters to an Azure site or Point-to-Site VPN, when connecting a single machine to an Azure site.
· Configure a Site-to-Site VPN using the Management Portal Wizard http://msdn.microsoft.com/en-us/library/windowsazure/dn133795.aspx
· Configure a Point-to-Site VPN using the Management Portal Wizard http://msdn.microsoft.com/en-us/library/windowsazure/dn133792.aspx
This can be challenging, not from a technical perspective but more form an organizational perspective. From Azure you can setup a Side to Side network connection to the datacenter, but this also requires some changes in the company policy in connecting to Azure. Some IT-departments aren’t ready for that.
A solution can be, put the SQL Database beside the VM’s in the same Virtual Network on Azure.