Visual Studio Team System 2010 – Episode 3: The Lifecycle
Previous episodes:
In the last episode Rob talked about the main challenges of testing, and not only for the acceptance tester but all the roles who are responsible for the quality of the product and that's actually everybody.
No Risk- No Test… what needs to be tested, when there is no business risk there is no reason to test, no reason to make unit tests by the developer, no reason to make regression tests and no reason to do acceptance testing. But when there is a risk every role has the goal of making that risk as small as possible. There is a joined effort in execution of tests to prove that the risk is minimized or solved. The development testers, system integration testers and acceptance testers together have to make a strategy on how to test.
Application Lifecycle Management
When talking about Application Lifecycle Management [ALM] terms like accountability, governance and compliance are used. All of them refer back to “working together”, how do we work together during the application lifecycle. ALM is not about tools, it’s about working together. Working together seamless and flexible while staying in control, being measurable and responsible. All the roles in the application lifecycle have a part in this collaboration effort. Tools can help but isn’t’ the core driver.
There are lots of examples of wrong interpreted business requirements, miss communication between development en test [Rob describes a very real example], applications which won’t run in production, operations who don’t understand the applications. All of them resulting in more work, more faults, more costs or even only costs and no application because the project was unplugged. Most of these projects faults, these extra costs are a slip in communication.
Having a strategy how people have to collaborate within the lifecycle is one important piece of Application Lifecycle Management. Most organizational parts have already some kind of process / methodology in place. Having an approach how to share information and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments.
Tools can help with this goal. Having gear in place which supports and stimulates collaboration is a driver for a successful Application Lifecycle Management. But without a plan, how people should collaborate and communicate, tools are useless.
Before ALM…
In the early ages, the waterfall ages, nobody talked about Application Lifecycle Management. Software development was still in his youth, experimenting with proven processes from the industrial age. The era where Taylor educated managers controlled the workforce. Business people writing down what they want [not knowing what they need] and throwing their needs over the wall to the workforce.
There was no communication between the guy who needed the software and the guy who implemented the software, a massive brick wall separated them. All the roles in the Application Lifecycle did their job between walls. People where used to this formal way of working, not arguing, not discussing about the solution, be doing their job as optimal as possible.
It doesn’t need any explanation that this Taylor way of work didn’t work for software development. Some software projects succeeded but many more failed to reach the goal. The main reason why all those projects fail: there was no sharing of ideas, ideas how to achieve business value. Everybody was working in his own domain with their own specific tools, as shown in the picture below, everybody on his own island with his own specific tools, practices and methodologies.
Software development is way different than producing products [Taylor’s theory is about optimizing the work force, something Ford did very well]. Software development needs a shared effort, a shared effort by all the roles in the lifecycle. From the people who have the business ideas and knowledge to the people who have to implement, test and maintain the final solution. Everybody in the lifecycle needs to collaborate with each other. Everybody needs to add their specific knowledge to the solution, that's not the same as adding parts to a car.
It’s interesting to notice that when there are no guidelines / structure in place, and nothing is done to stimulate collaboration. People and groups of same interest starting building their own walls, building their own kingdoms, protecting their environment. So, if we want to change this cubicle way of working, management buy-in needs to be in place, groups aren’t going to collaborate automatically.
Now
Nowadays, everybody understands that communication is king. Roles in the lifecycle need to communicate, need to listen to each other and share ideas about their view on the solution. The Agile methodology gives guidelines and structure , with valuable principles like:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Business people and developers must work together daily throughout the project.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- The best architectures, requirements, and designs emerge from self-organizing teams.
Tooling starts to support these principles [and the other 17] adopting a new way of communicative work around the Application Lifecycle. In 2005 Visual Studio Team System starts to support these principles. The integrated development environment got supported by Team Foundation Server with workitems, process templates and reports, helping to get roles in the lifecycle to communicate in a more efficient manner. Integrated in one tool people can share their workload and workprocesses, giving the project leader and for sure the other roles the ability to have more control over the process of software development, by knowing what the other one is doing.
Telling each other what you are doing is challenging. It’s challenging within a group of people with the same interest, it’s even more challenging between groups of different interests. It’s a culture challenge to get it working. Everybody knows it’s valuable, but most don’t want to give someone else a look in his kitchen. It’s valuable because it give transparency, agreements are known, even if someone is working according to this agreement and people can more easily get in sync. Everybody gets the focus on the business target.
[A dependency diagram of RUP’s process components.]
That people need to work together on a process and workitem level isn’t that strange when we look at the above image. This is a visualization of the degree of coupling between different process elements in the RUP methodology. …A Rational Unified Process (RUP) Plug-in To Support Requirements Quality Assurance [pdf]
The figure shows, that there is a very high coupling for approximately one third of all process components. 21 process components have a coupling less than 25%, 8 process components have a coupling of more than 25% and 3 process components more than 40%.
It’s interesting to notice that the adoption of workitems and processtemplates, although they are valuable, within companies fail. The main reason is the culture of the organization and management which doesn’t fully supports communication. This cultural aspect of communication between different roles, different social groups, should have a main focus while implementing Visual Studio Team System, otherwise success isn’t guaranteed.
This is also recognized by agile gurus like Scott Ambler who is writing in his blog:
Your existing culture and organization can really hinder your ability to scale agile approaches,
In his book “Agile Database techniques” he talks about The Cultural Impedance Mismatch between Data Professionals and Application Developers, discussing the cultural problems according to the communication between database people and code ‘object oriented’ people.
Overcoming the cultural impedance mismatch is much more difficult than overcoming the technical impedance mismatch. [The Cultural Impedance Mismatch Between Data Professionals and Application Developers]
The collaboration starts
While VSTS 2005 and 2008 support communication at a workitem level, the 2010 version of Visual Studio is going to support collaboration at artifact level. Different roles in the lifecycle work together on artifacts. Each of them adds their knowledge, vision and ideas to the solution from their view point. These artifacts are accessible by every role in every phase of the project, adding value throughout the lifecycle.
People are enabled to collaborate, making applications together, and not only by telling what they are doing but most important by working seamless together on the application.
While working together with all the roles and having these ‘ALM’ artifacts accessible for every role, collaboration in the application lifecycle will rise to the next level. People with different social and cultural characters start discussing solutions, trigger each other to improve and being innovative with their view on the application and all within the constraints of their role.
VSTS 2010 enables people to collaborate on artifacts in the Application Lifecycle. While the 2005 and 2008 versions mainly focus on communicating about work [workitems , processes and reports]. With VSTS 2010 the focus moves on to collaboration on artifacts. The main reason this is going to be possible is the investment by Microsoft in the Architecture Edition and Test Edition, widening the scope from developer to more roles in the lifecycle.
Not only development artifacts like source code can be saved to a repository but also test artifacts like test cases and test results. they can also be saved to the same repository, creating the possibility to work together on these artifacts from the same repository.
Testing in the Lifecycle
To figure out how this all works with testing we need a deep dive in the different test roles, what they do, what way they work together with the other team members and what kind of capabilities they need from the tools they use. Rob already talked about the different test roles “developers who are testing, specialist testers during system testing or generalist testers during the final acceptance test”. How do all these test roles work together with for example the business to capture risks of the system?
In the next episodes we will discuss these different test practices what they patterns are and the capabilities the need from tools.
Testing in the Lifecycle [ALM]... a focus on automation
When looking at all the available Testing Tools [for automation] and Test-Management Tools [for control]. You can conclude that there is a lot of help for automation and monitoring of testing, but not really helpful in the ALM.
ALM and Automation.
One of the important parts of ALM is the connection between the different lifecycle tools. For example, design artifacts can / must work together with coding tools, they must give the developer a kind of skeleton and give boundaries within they can start building the solution [ the application diagram together with the service factory for example].
Forrester describes ALM in this paper [The Changing Face Of Application Life-Cycle Management]:
ALM doesn’t support specific life-cycle activities; rather, it keeps them all in sync.
A development effort can still fail miserably even if analysts document business requirements
perfectly, architects build flawless models, developers write defect-free code, and testers execute thousands of tests. ALM ensures the coordination of these activities, which keeps practitioners’ efforts directed at delivering applications that meet business needs.
Traceability of relationships between artifacts.
Some more information about ALM, can be found in this post "ALM Definitions", also a good start to get comfortable with ALM is this presentation "ALM foundational concepts" by Clementino de Mendonça, Senior Development Consultant from Microsoft Services.
He also talks about automation in the slide "Why Automate ALM" and also refers to the Forrester paper.
So, for the tools which needs to support the Application Life-Cycle, integration with other ALM tools is one of the main capabilities it must provide. Next to this capability are reporting and traceability, something most test suites already deliver. But without the integration with other ALM tooling and without a good approach testing stays a manual process, the tools just offer repeatability by automation.
Available Testing Tools.
I divided the tools in "automation" testing and "managing" testing. You can further divide these automation tools in "GUI / Functional, Load and performance tools" and managing in "Bugtracking and Testcase managing tools".
Below a small [not complete] list of test tools current out there.
GUI Test Tools / Functional test Automation
[a must read, if you are planning to select a GUI test tool, is this paper [PDF] from Elisabeth Hendrickson]
- WinRunner by Mercury
- Astra Quick Test and QuickTest Pro by Mercury
- SilkTest (formerly QAPartner) by Segue Software
- QARun by Compuware
- e-Test Suite by Empirix
- QA Wizard by Seapine Software
- Eggplant by Redstone Software
- ...
Load Test / Performance Tools
- LoadRunner by Mercury
- SilkPerformer by Segue Software [now Borland]
- QALoad by Compuware
- e-Load by Empirix
- WebLoad by RadView Software
- WebFT by Radview
- OpenSta
- ...
Bugtracking and Test Case management Tools
- Bugzilla by YOU [open source]
- TestDirector by Mercury
- Jira
- ...
Test Suites
- Visual Studio Test Edition by Microsoft [see table below]
- SQA Suite by Rational
- TestComplete by AutomatedQA
- ...
And there are many many more look-a-like tools.
But most of them... do the same. There are some differences in details but most of them solve the same problem. The GUI / Load test tools are helping to be repeatable and the reporting / management tools are helping to monitor the state of the project. Helpful in the ALM but not that exciting.
New Testing Features in VSTS 2008, Test Types. [PPT]
Test Maturity Levels
The four IO levels of Microsoft's Infrastructure Optimization Model can also be used for testing. It's about the maturity according to "the outcome of testing/ reporting" not HOW to do it.
For the different levels I used the TPI ®Model [ Test Process Improvement ].
The Test Process Improvement (TPI) model has been developed based on the practical
knowledge and experiences of test process development. TPI offers a viewpoint in
the maturity of the test processes within the organization. Based on this understanding
the model helps to define gradual and controllable test process improvement steps.
Source: http://www.cs.helsinki.fi/u/paakki/Andersin.pdf
When comparing these levels with the capabilities of test-tools you can conclude that the GUI, Functional and load tooling offer the "basic" level. They automate the test, defects are found and reported. Those tools don't do reporting about the progress of the tests for the overall system [is everything tested?] and no recommendations according to what should be tested in what way. The user [tester / developer tester] still have to figure out what to do. No integration and no automation of the high-level design / test-case creation process.
The test suites do a better job, they offer more reports and integration. So, I put them under the "standardized" level. Also because the "advanced" level needs to have something with risk and not only documents or work-items but automation..!
These two quote's precisely point to the pain points according to these kind of test-tools:
However, automated execution of tests does not address the problems of costly test development and uncertain coverage of the input domain.
Model-Based Testing in Practice [PDF]
and
How the software [test automation software] can be automated is a technologically interesting problem. But this can lose sight of whether the result meets the testing need.
"Seven Steps to Test Automation Success" by Bret Pettichord
I like this last one, because it points directly to a tricky point according to test tools. They are made by developers [not by testers] and often they aren't focused on the problems that needs to be solved.
Testing Needs
Looking at the ALM Assessment in the "Quality & Test" section, you can take some interesting questions according to what testing really needs.
Are test cases and designs created in line with the design?
Have the areas of greatest risk been identified and tests prioritized accordingly?
Just two, but two important ones. "in line with design" and "identify risks". What actually means: Did you thought about what needs to be tested and in what manner are you going to test it? When you put this in the IO maturity model, you get something like this. [still working on this model]
I like the term "Monkey Testing" everybody immediately understands what it is ;-). Looking at the test-tools, all of them offer only the basic level. No intelligence, not in line with the design, no defined quality and no generation.
They don't stay in sync with other life-cycle activities [see Forrester paper]. Still a lot of hand crafting and manual processes.
Test Case Generation.
There are many papers around test case generation, which focuses on Model Driven Testing with UML.
For example IBM:
Model-driven Testing is a new and promising approach for the automation of software testing. This approach can significantly reduce the most painstaking cycle of all software development efforts—testing. Testing currently comprises between 30% and 70% of all software development projects. This new methodology and toolset will enable software developers and testers to become far more productive and reduce the time-to-market, while maintaining high standards of software quality.
and I'm curious what the status is of SpecExplorer from Microsoft Research.
Spec Explorer is a software development tool for advanced model-based specification and conformance testing. Spec Explorer can help software development teams detect errors in the design, specification and implementation of their systems. The tool is intended to be used by software testers, designers and implementers.
Testing is one of the costliest aspects of commercial software development.
Model-based testing is a promising approach addressing these deficits.
At Microsoft, model-based testing technology developed by the Foundations
of Software Engineering group in Microsoft Research has been used
since 2003. The second generation of this tool set, Spec Explorer, deployed
in 2004, is now used on a daily basis by Microsoft product groups
for testing operating system components, .NET framework components
and other areas. This paper provides a comprehensive survey of the concepts
of the tool and their foundations.
This is a list of papers about test case generation.
- Using UML Collaboration Diagrams for Static Checking and Test Generation [PDF]
Abstract. Software testing can only be formalized and quanti ed when a solid basis for test generation can be de ned. Tests are commonly generated from program source code, graphical models of software (such
as control ow graphs), and speci cations/requirements. UML collaboration diagrams represent a signi cant opportunity for testing because they precisely describe how the functions the software provides are connected in a form that can be easily manipulated by automated means. - Generating Test Sequences from UML Sequence Diagrams and State Diagrams [PDF].
Abstract: UML models offer a lot of information that should not be ignored in testing. By combining different UML components, different views of the program under test are used. The paper concentrates on a technique for generating test cases from a combination of UML sequence and state diagrams. - Test Case Generation from Message Sequence Charts [PDF].
black-box and specific white-box testing for communication protocols and distributed systems. UML models provide scenario descriptions by sequence diagrams respectively MSCs. Thus, the combination
of TTCN- 3, as test description language, and UML by MSC to specify and automatically generate test cases has to be considered.
- Model-Driven Testing with UML 2.0
Abstract. The UML 2.0 Testing Profile provides support for UML 2.0 based model-driven testing. This paper introduces a methodology of how to use the profile in order to transform an existing UML system design
model for tests.
Anyway, with Rosario Team Architect [see CTP 10 Nov'07 download] gets an extra view [dynamic] with the sequence diagram beside the static application diagram and class diagrams. So, it's going to be interesting to take a look if it's possible to generate some "high-level" Test-Cases from those diagrams and I do think, it should be possible [and maybe more easier] to define/ generate test cases for a service factory implementation.
So, maybe we can get to the advanced level with test case generation, Rosario and different models / viewpoints...
More to come...