3 Challenges when Testing from and on the Cloud.
When adopting the Cloud for your test work it comes with several challenges. Looking at the different usages flavors of the cloud for teams, read: Cloud usage flavors for Development and Test teams, then every flavors has its own challenges.
Challenge 1: Team Workspace in the Cloud – Test Environment On Premise.
The scenario is that the team adopted Team Member Workspace on Azure, but the infrastructure (test environments) are staying on-premise.
The planning, preparation, specification and execution of the test work takes place on the team member workstation which is hosted in the Cloud.
The challenges is that the execution of test are triggered from the public cloud and the test runs on the private data center. There needs to be a connection between the team member workstation and the test environment. The same counts for automatic test runs.
Solution: make a network connection between Azure and the On-Premise datacenter. Often a very long discussion with the internal infrastructure people is needed to get this working.
You can create a Site-to-Site VPN, for connecting datacenters to an Azure site or Point-to-Site VPN, when connecting a single machine to an Azure site.
Challenge 2: Acceptance Test Infrastructure in the Cloud – Production On Premise.
the scenario, all environments (development and test) are on Azure except production, this one still lives on-premise. The challenges is to get the test and acceptance (on Azure) environments as equal as possible to the production (on prem) environment.
Solution: The best solution is the usages of the same installation scripts for production as for acceptance and test. Make PowerShell DSC scripts for and re-use them for every environment.
there is also a need for stubs to make testers need to focus on writing isolated tests, which will make more systems valid. Still not all tests can run on Azure Test Infrastructure (performance and load tests won’t give valid results).
the worst case scenario is that an additional pre-production environment needs to be created on-premise for tests which couldn’t be validate in the cloud.
Challenge 3: ALM Infrastructure in the Cloud – test, acceptance and production on- premise
The scenario: The team consumes their ALM infrastructure from the Cloud and use on premise workstations.
Actually from a technical perspective there isn’t a challenge in this scenario. The only need is reliable https connection between the Test case repository (ALM Infrastructure ) and the test environments.
Interesting in this scenario can be the test data. There is probably a security reason why the environments (test, acc and prod) are in the local datacenter. When the test cases, test data and test results are stored on Azure, the security regulations probably also count for that data.
Often production data is used as test data resulting in a no go for this scenario. A solution is to anonemize the test data.
these where just three challenges for the three different components of Cloud usages for Dev and Test on Azure.
There are many more challenges, maybe in another post.
Cloud usage flavors for Development and Test teams.
The cloud for teams have many forms.
Teams can simply use a virtual machine on Azure to work from, the team member workspace flavor as I call it.
Or, they can use the bug tracking database, requirements management, and scrum board from the Cloud. In either a SaaS way, like Visual Studio Online (www.VisualStudio.com) or as an IaaS solution (see Practical guidance for TFVC and TFS on Azure IaaS). Both are the ALM infrastructure in the cloud flavor of Cloud usages by teams.
Another way teams can use the Cloud for their work is that they use it for their Development and Test Infrastructures. For example, when you develop a system for an On-Premise SharePoint farm team members need to have a SharePoint Farm. Developers need the infrastructure for development, debugging and integration, testers for sure to run tests for the system in development. It is the Development and Test Environments flavor.
When adopting the cloud for your Development and Test work, all three flavors are interesting to look at. You can consume them separately but when using all together you are really riding the cloud wave.
Integrate Azure, Visual Studio Online and Office 365 with VSO Rest APIs.
With the just released Visual Studio Online Rest APIs you can do some wonderful things.
We create a solution to support project teams who use Azure for their development and test activities. Teams who use Visual Studio Online for their ALM infrastructure and Azure VMs for their Development and Test environments (as most Sogeti Teams do :-).
The challenges when running heavy cloud based teams like this, is that you want to let them use the Azure and Visual Studio Online resources as efficient as possible. Give them easy to use features to create Development and Test Environments is key. But, also too often VMs keep running for weeks, while no test runs or any code change / team activity has happen. And from a few days ago, Visual Studio Online licenses you want to monitor unused licenses.
Teams should stay Fast and Flexible, and must have insight in what they do, to be efficient. Via this portal we give Teams insight and tools to use the Azure and VSO resources as efficient as possible.
The portal gives information about VSO Licenses used, build and test runs, Azure IaaS state and overall Azure costs, everything that costs money when running your project.
This all works in a SharePoint O365 Provider Hosted App, with Microsoft Azure Management Libraries and the Visual Studio Online Rest APIs consumed via a WebAPI, all hosted on Azure Cloud Services and secured by Windows Azure Active Directory.
Teams get information like:
Azure Development and Test Environments usages.
Beside Azure VM’s state, start, stop and RDP connect there is also the capability to add usage notifications and a schedule rule engine to the VMs. So team members are notified when a VM is unnecessary on. Mobile apps receive the notifications and have the ability to stop the VM.
Build usages and history.
For teams to see the build usages and to get an indication if build environments are proper used.
Visual Studio Online License information of all the team members.
and Team activities.
And as last and Overall Azure subscription costs.
Every project has its own VSO Account and Azure Subscription (in the Azure Enterprise Subscription). In this subscription they maintain the VSO licenses and their Development and Test Azure VMs. This give the team control over what they need (add licenses and developer machines when they need) and insight in the true costs of the project. We upload the terrible excel output from the Enterprise Azure Portal to an Azure SQL database.
All in Office 365 in a single project portal to enable Fast and Flexible Teams. VSO Rest APIs Rock …
The missing knowledge of testers in modern app development.
The game has changed for testers, they need to extend their basic testing skills when they want to stay valuable. As other team members, testers need to be a full member of the team with the current skill set this isn’t.
While supporting teams for a long time in delivering software I noticed I’m always teaching new ‘test’ team members the same skills. Skills a new team member actually already should have had before entering the team room.
A small list.
Knowledge about agile work in teams is the minimal a test team member should have.
When they fail a basic assessment like the Scrum.org open assessment ( https://www.scrum.org/Assessments/Open-Assessments/Scrum-Open-Assessment ) I silently start to cry.
Agile is that mainstream now a days that I even wonder why conferences like ‘The Agile Testingdays’ exists. The practice of agile test work should be crystal clear at this moment. So, mastering agile work is really the bear minimum.
Team Tools support.
Know your tools. Not only the common testing tool ‘Excel’ but more important the Application Lifecycle Management (ALM) tools the team uses.
ALM Tools like Visual Studio Online and other tools mentioned in the Gartner ALM report are important.
Magic Quadrant for Integrated Software Quality Suites
Gartner’s Magic Quadrant for Application Lifecycle Management
The tester isn’t alone anymore, testing isn’t the activity at the end of the waterfall what everybody skips, it is part of the delivery. Having knowledge how to use integrated team tools is a must to be valuable and not being a blocker of productivity.
Basic programing skills
Test automation is a must in the current fast delivery mindset of Continues Delivery. And this test automation must be, just like all the team work, integrated in the team process.
Standalone test automation tools are useless. Test automation connected with the team sprint tasks, the team build runs, the team delivery and the code base is a must for agile teams in being efficient with a focus on quality.
When test automation is in the same code base, in the same rhythm as the whole team, the code written for this test automation is logical the same programming language as the language the system is developed in, at least it is the most efficient.
Having basic programming skills as a tester is a must to be valuable for the team. As the image above shows, only just a small part is manual testing. So start to learn programming.
(Distributed) Version Control
Every artifact a team creates should be under version control. For the codebase it is normal to use a version control system. Either a centralized system like Visual Studio Online or a distributed system like GIT, everything is under version control.
All team members must understand the basics of version control. How to get or pull artifacts in or out and the pros and cons of the usages of branches.
Testers need to be capable to get the version they want to test from version control them self, they need to understand what artifacts are in the version and what features are delivered at what quality.
Knowledge of version control systems will make testers less depended on the other team members who do know what is in what version.
A good read: Version Control for Multiple Agile Teams
System under Test Deployment models.
As mentioned in the previous needed tester’s skill, testers should be less depended on other team members to get their system under test. Knowing how to deploy the software is a great added value for testers. This is for all kind of systems for example, knowing how to install a Store App on a tablet for testing will make the tester immediately more productive.
For more complex systems, knowing how PowerShell, Chocolaty, Puppet, Chef, Visual Studio Release Manager or other deployments mechanisms work would be great.
But at least understand how to start the deployment as a tester is the minimum. In the same category understand how to setup a test infrastructure for automated testing would be very useful.
Understanding the capabilities of current monitoring systems is a big benefit for testers. A tester who understands the behavior of the system under test is a invaluable benefit for the team. This tester helps to solve bugs as quick as possible by providing valuable usages statics, trace files or intellitrace files.
The last skill for a testers I want to talk about is Cloud. When a tester wants to stay in business, he or she needs to understand and usage the Cloud for testing.
Beside easy onboarding of new team members Dev & Test in the Cloud also can be used for your complete test infrastructure and with the new gained deployment knowledge also the provisioning of test environments should be easy for testers.
See this deck: Azure for Teams
I can imagine you think ‘as a tester’ this isn’t my job role, these are the activities and skills of other. But as James Whittaker mentions in this post
“For the vast majority of app development on this planet, software testing is an activity within the development process …. It’s time to bring quality into the 21st century where testing is such an integral part of software development that you’ll often forget you are doing because it has become so familiar.
Testing is integrated in modern development, a testers must evolve. To stay valuable for teams and keep their job the in this post mentioned skills must be added to the capabilities of the modern tester.
Not only these capabilities count for testers, all other team members should master them too. So there is also some lessons to learn for developers.
DECK: Azure for Teams
More business value with fast and flexible teams.
The deck I used (parts of it) at different conferences.
Things to think of when using Team Member IDE on Azure, networks.
Azure for Teams
Having Team Member IDE’s on Azure provides create benefit for teams in moving fast and staying flexible.
But there are still some items you have to think of when start adopting it, networks being connected is an important one:
You never ever develop or work in isolation. Especially with the heterogenic systems we build now a days. So, being connected with the world from your Azure hosted Visual Studio IDE is a must.
For example, the SQL Server database you are consuming in your App is in the local data center. You need a connection to that datacenter from your Azure environment.
You can create a Site-to-Site VPN, for connecting datacenters to an Azure site or Point-to-Site VPN, when connecting a single machine to an Azure site.
· Configure a Site-to-Site VPN using the Management Portal Wizard http://msdn.microsoft.com/en-us/library/windowsazure/dn133795.aspx
· Configure a Point-to-Site VPN using the Management Portal Wizard http://msdn.microsoft.com/en-us/library/windowsazure/dn133792.aspx
This can be challenging, not from a technical perspective but more form an organizational perspective. From Azure you can setup a Side to Side network connection to the datacenter, but this also requires some changes in the company policy in connecting to Azure. Some IT-departments aren’t ready for that.
A solution can be, put the SQL Database beside the VM’s in the same Virtual Network on Azure.
Faster realization of business value, some thoughts on Models and CodeGen.
Model driven development and code generation comes in many different forms, and all have the same promise: faster delivery of business value. Not every form has been even successful in realizing this promise. Where some have died silently in their own complexity, and some are standing still in innovation for years, there are other ‘generation’ solutions who are getting the momentum of success.
Learn about the MDD approach and how you can apply it to achieve the maximum business value and reduce the cost of solution development.
Article from 2006 - Implement model-driven development to increase the business value of your IT system- http://www.ibm.com/developerworks/library/ar-mdd1/
Model driven development is about the changing the abstraction level of the used programming languages to realize business value. When you make the language closer to the business solution than C#, C++, Java or any other programming/ machine language it is easier, and more accurate, to describe the solution. When you can run this, more abstract, language (or model) the same way as program languages are executed, than you have a model driven approach. More high level languages, describing the business can be used by the business to describe their own solutions. And, more low level abstraction languages and models can be used by engineers to describe a more technical solution. With these new languages it is easier to communicate about the specific domain, a domain specific language.
In Application Lifecycle Management * languages and models describes the different domains. A specific languages used to describe the business domain are often technology and platform agnostic. While specific languages used to describe the solution are often implementation and platform independent, which gives less re-use.
* Application Lifecycle Management is about the whole spectrum of software development, the processes that are used, the people who use it, make and maintain the applications and about the tools and technology that are used during the complete lifecycle. Application Lifecycle Management is not only about software development and a grown-up development environment but also about business, operations and the connection between these domains.
In both domains, the business and development, domain (often called the problem and solution domain) new possibilities, innovation and business opportunities are insight. But, for both domains making the decision to adopt model driven development need to be taken careful, a well tradeoffs process needs to take place.
With business domain specific languages the promises to gain benefit of fast realization of business value are the biggest.
It is a language used by the business / domain experts to describe the ‘problems’ [needs], the models can be used to generate code or can be executed on a platform.
Vendors who deliver technologies to create and execute Business DSL’s are for example MetaCase (http://www.metacase.com/) and Mendix (http://www.mendix.com), both mature in the solutions they provide. Microsoft has Domain Specific Languages capabilities in its development IDE (http://msdn.microsoft.com/en-us/library/ee943825.aspx), and Eclipse has one of the most used DSL tools addin (http://www.eclipse.org/modeling/ based on the open source project openArchitectureWare). MDA based on UML is another technology for MDD http://www.omg.org/mda/
There are a few adoption challenges with business domain specific languages. One challenge is the clear description (Meta language / the grammar) of the language. Now day’s business are moving and changing faster than the language can evolve. Microsoft changed to a Services and Devices Company, the language used to describe the business needs to change too. And, creating a specific business specific language isn’t that easy.
Another challenge is the execution of the models (languages) on the platforms. The language must be interpreted by the platform to run, or compile to executable code by the platform (see image above). Creating a compiler or an interpreter is very specialized and technical work. Maintaining these will need some effort.
The last challenge has to do with team work. Probably multiple people will work on the business models. Multiple versions of the models will live in different stages of the development process. Maintaining these model versions, work together on one model and find the differences between versions of models is a challenging task. Current available development and modeling environments aren’t mature enough to capture this.
The above mentioned technology vendors target these challenges, and while the innovation of platforms evolves they also need to evolve to execute the models on these new platforms. This will definitely help to lower the maintenance of software systems. Innovation goes slow and only for specific cases promise of faster business value is reached, when you have such a case it definitely pays off to investigate the possibilities.
Technical domain specific languages are used by the team to describe the solution domain. Where business domain languages have the benefit that the subject matter expert can describe the problem. With technical domain specific languages (and models) you loss this benefit. But there are already a lot of specific languages in place for the solution domain, which getting more and more mature.
There are more technical specific languages for one specific domain. For example SQL is a language for working with data. Another specific domain language is WiX for describing installer packages. On the other side there are also languages which are closer to the business. For example Smart User Cases ( http://www.smartusecase.com ), which uses specific User Case stereotypes to describe the business needs and generates with accelerators code, test cases or any other project artifact when there is an accelerator for. Code is generated and executed on the platform, so it is easy when solution is described to generate code for another platform.
Both most technical and business languages models are having the same adoption challenges. Hard to create the language, stay in sync with the platform innovations and use them in teams. But, technical languages are easier to adopt for multiple reasons.
One reason is that the created models and languages can be reused more often, they aren’t business specific, but can be used for multiple business domains. Which makes them more mature, robust and they are probably maintained by a technology vendor or open source group. And, innovation will move faster with more people using them.
For example Visual Studio LightSwitch is an addition to the Microsoft development platform to create business applications for the Desktop and the Cloud. Visual Studio LightSwitch generates the underlying infrastructure. The database, transaction mechanism and helps you craft the screens, which when you choose to build a HTML Client generates a responsive website which runs on all kind of devices.
Not only code generation (or model interpreters) have to be used for domain specific languages also specific frameworks for specific tasks are enablers for faster (and more reliable) realization of business value. Java is a kind of famous for all the different frameworks created for it. Examples are more technical Frameworks like the Spring Framework, but there also have been implementations like an Open Healthcare Framework.
In the same area as Frameworks you can find API’s, Application Programming Interfaces. API’s are the new enablers of fast business value. For almost every business problem there is an API available. The difference with Frameworks is that you don’t have to incorporate them in to your solution, you just use them.
The website ProgrammableWeb.com has a collection of 9920 APIs. With additions every day on all kind of categories, ready for business use with as an interface, a language to help with a specific technical or business domain problem. Use them in your solutions for fast enablers of specific functionality or stich several API’s together with additional business functionality and great a valuable mashup.
On the same technical side as API’s, as fast business value enablers, are Cloud Services. Cloud Services provide specific functionality for systems. The biggest difference with API’s is that Cloud Services provide more the technical infrastructure where API’s provide more business functionality. When your App needs to work with data, authentication or push than cloud service are a fast enabler.
A good example are the Microsoft Azure Mobile services and Media Services. Services that provide configurable functionality with generated code for your systems. It is possible to create it from scratch, but instead of hours it will take days to realize it.
In the area of Cloud Services innovation really moves fast. Services to enable faster business value are added almost on a daily base.
Other Cloud Services where innovation goes fast are services which helps development teams to do their job. Examples are Test and Development Platforms as a Service preconfigured and ready to use environments to quickly start working. Others examples are Load Testing from the Cloud and automated test execution on the Cloud.
Generate systems in the browser
One last innovation which takes off, are the web-based / do it yourself system generation sites. Where cloud services like Media Service from Azure provide generate code which you can incorporate in your code base. These generators are providing ready to use systems from the Web or Apps. A very successful is Windows Phone App Studio. Windows Phone App Studio is a website to generate a full working Windows Phone App in minutes, published to the store.
Another one from Microsoft is Microsoft “Project Siena”.
A very interesting innovation these kind of ready to use generators will available more and more, delivering the promise of code generation. But the generated solutions aren’t business ready, not unique enough, and need defiantly customization afterwards. I actually think it is create for trials, before doing the realy.
Insight on code generation innovation.
Code generation and model driven development are technologies with a promise to realize faster business value, but they are not always as flexible as the business need these days.
Specific to the situation it fulfills this promise. Where the generation from models (domain languages) on the business side are even more specific and should be analyzed careful be for used. But, when the business models are well implemented they will bring the most and fastest business value. Innovations on this area goes slow.
On the technical domain, code generation from models or specific languages (Lightswitch, Entity Framework) are well adopted by teams. The same for API’s, which add easy business knowledge to the systems. Both can benefit from and should use each other. Innovations goes faster in this technical domain, just due to the more frequent use than on the domain modeling side.
Fast innovation takes place in the Cloud Services stack, configurable services with ready to use code in your systems. More and more services will get available in an increasing pace to deliver that business value fast for your business users.
First meaningful activity.
Azure for Teams
When onboarding a developer the time between entering the team room and the first meaningful check-in, is waste.
In the same category the time between a testers enters the team room the first meaningful test specification or test execution is waste.
As everybody knows from Lean, we should aim to identify and eliminate waste in our projects. Mean Time to Valuable Activity is waiting time, time where a team member is idle and is not adding any value to the finished good or service. Creativity is lost, motivation flows away and talent isn’t used, what is another waste type in the 8 waste types defined in Lean.
Onboarding team members is hard. Multiple challenges needs to be handled: understanding the system, knowing the team work ethics, get the necessary tools, get access rights, etcetera.
Team Influence zone.
Most activities are in the teams influence zone. For example, time to understand the system been build can significantly be reduced by lowering the technical complexity. Another good reason to focus on Technical Dept as a team, the same for the methodology used. Activities which are in the hands of the team are adjustable.
Getting the necessary tools, equipment and access rights is often not an activity teams are allowed to handle. For usages rights the internal infrastructure department needs to give approval, for workstations and licenses some / multiple managers needs to sign documents. Sometimes it takes weeks before everything is agreed up on and in the hands of the new team member, ready to start the first meaningful activity.
Team Desktops on Azure
Time the new team member is waiting for equipment is waste, creativity is lost, motivation flows away and talent isn’t used. To remove this waste, is to give control to the team of every activity necessary to onboard a new team member. Technical Dept was already in their hands, getting the necessary tools is the one to get focused on.
With development and test environments in the Cloud (Team desktops on Azure). A team can get much more self-organized. Provisioning new environments for new team members on the fly without any interference of other internal departments, ie people outside the team.
A Team desktops on Azure is nothing more than a Visual Studio or Test Manager environment running on Azure VM’s (IaaS). With the Azure capability of creating your own discs teams can pre-configure the preferred IDE.
When having a MSDN subscription start using Team Desktops on Azure very easily can be done, most often within the boundaries of the subscription benefits.
When getting more mature in using Team Desktops on Azure, an Azure Enterprise Subscription is necessary. A good practice is to create a sub-subscription per project at start of the project. For sure with a team member as co-administrator of that subscription.
When teams can self-service the onboarding of new team members waste is minimized and the new team member is delivering faster more motivated business value to the project. Reducing the time to first meaningful check-in or test specification or test run is a good metric.
WinRT 8.1 Build Error … Task 'GenerateAppxSymbolPackage' failed. Could not find a part of the path …
Suddenly builds start to fail on a Win8.1 project, with the interesting message “Could not find a part of the path ”.
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\AppxPackage\Microsoft.AppXPackage.Targets (1512): Task 'GenerateAppxSymbolPackage' failed. Could not find a part of the path 'D:\Work\drops\projectName_184.108.40.206_Debug_Test'.
This messages is due to the fact that a team member runs the Package Creation Wizard from Visual Studio, sets the Bundle drop location and put the solution in the source repository.
The build server wants to use the same location as the value set by the developer. it is not saved in the Package mannifest, but in the csproj.
the value is saved in the csproj <AppxPackageDir> key, setting it during the build would be better. Setting it to a drop location known by the Build Server solves the error.
Fast and Flexible with Dev and Test Platforms as a Service.
cross post from labs.sogeti.com
Teams need to move fast, every action which results in wait time must be minimized to zero. Teams need to move flexible, context changes must be easy adoptable by the team and the system they realize.
Two very clear, but hard to accomplish aims. One challenges teams face today in reaching these goals are the availability of environments.
Development environments are hard to setup with mostly big hardware requirements. For example setting up a SharePoint 2013 development machine requires a lot of knowledge and time on an very powerful machine.
This will probably slow down the onboarding of a new team member. Requesting the proper hardware, read the book J install, configure and setup the environment.
The same for the environments that are used by testers. Every developer needs his own machine to develop on the solution, also testers need environments to plan, specify and execute tests. This means not only developers environments need to be available also for the tester a separate environment to execute their tests on needs to be available.
For just a normal development team it will work like explained in this video and diagram:
It gets interesting when the team runs sprints of one week and deliver every week a new increment. An increment which needs to be installed on a clean environment. With for this example a clean SharePoint environment, ready for validation. And, let’s say the team releases every four sprints a release for validation by customers an environment, which has the same characteristics as the production environment must be available.
Asking the supporting operations department first every week a new integration and test server and every month a clean ‘production like’ environment will make them crazy. Probably operational rules will even make this an not supported scenario and slows down the team in delivering value.
Two activities; onboarding of team members (Development), and the availability of environments (Test and Acceptance) are slowing down the team. Not only the complexity, but primarily the company dependencies (approval) which come with these activities bring teams value delivery to a full stop.
Dev and Test Platforms as a Service.
Virtualization is getting mainstream, even IaaS (Infrastructure as a Service) is wildly adopted, teams are using them, the benefit can get higher. Ready-to-use platforms easy to spin up for team members. Microsoft already delivers a Platform (Azure VM Template) preinstalled with Visual Studio 2013. http://www.windowsazure.com/en-us/solutions/dev-test/ (see screenshot)
Not only the pre-installed and configured VM’s from Microsoft can be used for this scenario, also the community started to make VM’s http://www.hanselman.com/blog/Over400VirtualMachineImagesOfOpenSourceSoftwareStacksInTheVMDepotAzureGallery.aspx
The other dependencies.
The tools are there, team members are powered with the capabilities they need to have. Now we get to the biggest problem, company policy and restrictions. Who is going to pay the bill?
Probably a complete sign off mechanism starts when a team needs something. A request for a new laptop is formalized in a full workflow with signatures on multiple documents, only getting approved with a proper business case it will probably take a few weeks, two when its fast but when someone is on vacation six till eight weeks. This same request – approval process is used for cloud possibilities. Cloud possibilities which are needed to move teams fast and flexible. This is definitely counterproductive and this dependency need to be solved fast.
Give the power to create environments in the hands of the people who need them. Then teams are ready to get along with the current development (and test) practices to move fast, flexible and deliver business value add a cadence the business needs.