Testing in the Lifecycle [ALM]... a focus on automation
When looking at all the available Testing Tools [for automation] and Test-Management Tools [for control]. You can conclude that there is a lot of help for automation and monitoring of testing, but not really helpful in the ALM.
ALM and Automation.
One of the important parts of ALM is the connection between the different lifecycle tools. For example, design artifacts can / must work together with coding tools, they must give the developer a kind of skeleton and give boundaries within they can start building the solution [ the application diagram together with the service factory for example].
Forrester describes ALM in this paper [The Changing Face Of Application Life-Cycle Management]:
ALM doesn’t support specific life-cycle activities; rather, it keeps them all in sync.
A development effort can still fail miserably even if analysts document business requirements
perfectly, architects build flawless models, developers write defect-free code, and testers execute thousands of tests. ALM ensures the coordination of these activities, which keeps practitioners’ efforts directed at delivering applications that meet business needs.
Traceability of relationships between artifacts.
Some more information about ALM, can be found in this post "ALM Definitions", also a good start to get comfortable with ALM is this presentation "ALM foundational concepts" by Clementino de Mendonça, Senior Development Consultant from Microsoft Services.
He also talks about automation in the slide "Why Automate ALM" and also refers to the Forrester paper.
So, for the tools which needs to support the Application Life-Cycle, integration with other ALM tools is one of the main capabilities it must provide. Next to this capability are reporting and traceability, something most test suites already deliver. But without the integration with other ALM tooling and without a good approach testing stays a manual process, the tools just offer repeatability by automation.
Available Testing Tools.
I divided the tools in "automation" testing and "managing" testing. You can further divide these automation tools in "GUI / Functional, Load and performance tools" and managing in "Bugtracking and Testcase managing tools".
Below a small [not complete] list of test tools current out there.
GUI Test Tools / Functional test Automation
[a must read, if you are planning to select a GUI test tool, is this paper [PDF] from Elisabeth Hendrickson]
- WinRunner by Mercury
- Astra Quick Test and QuickTest Pro by Mercury
- SilkTest (formerly QAPartner) by Segue Software
- QARun by Compuware
- e-Test Suite by Empirix
- QA Wizard by Seapine Software
- Eggplant by Redstone Software
- ...
Load Test / Performance Tools
- LoadRunner by Mercury
- SilkPerformer by Segue Software [now Borland]
- QALoad by Compuware
- e-Load by Empirix
- WebLoad by RadView Software
- WebFT by Radview
- OpenSta
- ...
Bugtracking and Test Case management Tools
- Bugzilla by YOU [open source]
- TestDirector by Mercury
- Jira
- ...
Test Suites
- Visual Studio Test Edition by Microsoft [see table below]
- SQA Suite by Rational
- TestComplete by AutomatedQA
- ...
And there are many many more look-a-like tools.
But most of them... do the same. There are some differences in details but most of them solve the same problem. The GUI / Load test tools are helping to be repeatable and the reporting / management tools are helping to monitor the state of the project. Helpful in the ALM but not that exciting.
New Testing Features in VSTS 2008, Test Types. [PPT]
Test Maturity Levels
The four IO levels of Microsoft's Infrastructure Optimization Model can also be used for testing. It's about the maturity according to "the outcome of testing/ reporting" not HOW to do it.
For the different levels I used the TPI ®Model [ Test Process Improvement ].
The Test Process Improvement (TPI) model has been developed based on the practical
knowledge and experiences of test process development. TPI offers a viewpoint in
the maturity of the test processes within the organization. Based on this understanding
the model helps to define gradual and controllable test process improvement steps.
Source: http://www.cs.helsinki.fi/u/paakki/Andersin.pdf
When comparing these levels with the capabilities of test-tools you can conclude that the GUI, Functional and load tooling offer the "basic" level. They automate the test, defects are found and reported. Those tools don't do reporting about the progress of the tests for the overall system [is everything tested?] and no recommendations according to what should be tested in what way. The user [tester / developer tester] still have to figure out what to do. No integration and no automation of the high-level design / test-case creation process.
The test suites do a better job, they offer more reports and integration. So, I put them under the "standardized" level. Also because the "advanced" level needs to have something with risk and not only documents or work-items but automation..!
These two quote's precisely point to the pain points according to these kind of test-tools:
However, automated execution of tests does not address the problems of costly test development and uncertain coverage of the input domain.
Model-Based Testing in Practice [PDF]
and
How the software [test automation software] can be automated is a technologically interesting problem. But this can lose sight of whether the result meets the testing need.
"Seven Steps to Test Automation Success" by Bret Pettichord
I like this last one, because it points directly to a tricky point according to test tools. They are made by developers [not by testers] and often they aren't focused on the problems that needs to be solved.
Testing Needs
Looking at the ALM Assessment in the "Quality & Test" section, you can take some interesting questions according to what testing really needs.
Are test cases and designs created in line with the design?
Have the areas of greatest risk been identified and tests prioritized accordingly?
Just two, but two important ones. "in line with design" and "identify risks". What actually means: Did you thought about what needs to be tested and in what manner are you going to test it? When you put this in the IO maturity model, you get something like this. [still working on this model]
I like the term "Monkey Testing" everybody immediately understands what it is ;-). Looking at the test-tools, all of them offer only the basic level. No intelligence, not in line with the design, no defined quality and no generation.
They don't stay in sync with other life-cycle activities [see Forrester paper]. Still a lot of hand crafting and manual processes.
Test Case Generation.
There are many papers around test case generation, which focuses on Model Driven Testing with UML.
For example IBM:
Model-driven Testing is a new and promising approach for the automation of software testing. This approach can significantly reduce the most painstaking cycle of all software development efforts—testing. Testing currently comprises between 30% and 70% of all software development projects. This new methodology and toolset will enable software developers and testers to become far more productive and reduce the time-to-market, while maintaining high standards of software quality.
and I'm curious what the status is of SpecExplorer from Microsoft Research.
Spec Explorer is a software development tool for advanced model-based specification and conformance testing. Spec Explorer can help software development teams detect errors in the design, specification and implementation of their systems. The tool is intended to be used by software testers, designers and implementers.
Testing is one of the costliest aspects of commercial software development.
Model-based testing is a promising approach addressing these deficits.
At Microsoft, model-based testing technology developed by the Foundations
of Software Engineering group in Microsoft Research has been used
since 2003. The second generation of this tool set, Spec Explorer, deployed
in 2004, is now used on a daily basis by Microsoft product groups
for testing operating system components, .NET framework components
and other areas. This paper provides a comprehensive survey of the concepts
of the tool and their foundations.
This is a list of papers about test case generation.
- Using UML Collaboration Diagrams for Static Checking and Test Generation [PDF]
Abstract. Software testing can only be formalized and quanti ed when a solid basis for test generation can be de ned. Tests are commonly generated from program source code, graphical models of software (such
as control ow graphs), and speci cations/requirements. UML collaboration diagrams represent a signi cant opportunity for testing because they precisely describe how the functions the software provides are connected in a form that can be easily manipulated by automated means. - Generating Test Sequences from UML Sequence Diagrams and State Diagrams [PDF].
Abstract: UML models offer a lot of information that should not be ignored in testing. By combining different UML components, different views of the program under test are used. The paper concentrates on a technique for generating test cases from a combination of UML sequence and state diagrams. - Test Case Generation from Message Sequence Charts [PDF].
black-box and specific white-box testing for communication protocols and distributed systems. UML models provide scenario descriptions by sequence diagrams respectively MSCs. Thus, the combination
of TTCN- 3, as test description language, and UML by MSC to specify and automatically generate test cases has to be considered.
- Model-Driven Testing with UML 2.0
Abstract. The UML 2.0 Testing Profile provides support for UML 2.0 based model-driven testing. This paper introduces a methodology of how to use the profile in order to transform an existing UML system design
model for tests.
Anyway, with Rosario Team Architect [see CTP 10 Nov'07 download] gets an extra view [dynamic] with the sequence diagram beside the static application diagram and class diagrams. So, it's going to be interesting to take a look if it's possible to generate some "high-level" Test-Cases from those diagrams and I do think, it should be possible [and maybe more easier] to define/ generate test cases for a service factory implementation.
So, maybe we can get to the advanced level with test case generation, Rosario and different models / viewpoints...
More to come...
How To Collect Data from the Application Diagram to create WSSF Service Models
Post #02 of a collection of posts with some more technical details about the creation of richer implementations with Visual Studio Team Architect.
This is post 1: How To Fire a Guidance Package Recipe from the Implement Application Feature of Team Architect.
[Dynamic System, Lorenz attractor]
First you have to understand System Definition Model (SDM).
SDM is part of the overall Dynamic Systems Initiative (DSI) from Microsoft.
The Dynamic Systems Initiative (DSI) is a commitment from Microsoft and its partners to help IT teams capture and use knowledge to design more manageable systems and automate ongoing operations, resulting in reduced costs and more time to proactively focus on what is most important to the organization. The System Definition Model (SDM) is a key technology component of the DSI product roadmap that provides a common language, or meta-model, that is used to create models that capture the organizational knowledge relevant to entire distributed systems.
Quote from System Definition Model Overview White Paper.
Dynamic Systems Initiative also contains ideas like Application Lifecycle Management, Microsoft Infrastructure Optimization Model and the just released V2 version of the Design For Operations Models [full name: Visual Studio Team System Management Model Designer Power Tool].
Often you here the term Service Modeling Language (SML) in relation with SDM, the SML Insight blog gives some explanation.
The model-based management functionality in Windows Server 2008 is based on Microsoft's System Definition Model (SDM) version3, which provided the basis for the Service Modeling Language (SML) proposal and submission to the World-Wide Web Consortium SML Working Group.
I'm curious how the DSI initiative and SDM / SML are going to evolve with Oslo and with that how Team Architect is going to evolve, a nice subject for another post.
Anyway, SDM is an XML-based meta-language for distributed systems.
The SDM structure behind Team Architect.
You can split the SDM structure behind Team Architect's Distributed System Designers in two parts. One XML part responsible for the loading, structuring and displaying systems, endpoints and resources in the Distributed System Designers, the <DesignData> element [MSDN link]. The other part holds all SDM relevant information, building blocks as SDM objects, relationships, and optional additions to objects such as settings, flows, and constraints, which also has <DesignData> elements.
Best explained with an empty Application Diagram file.
The ROOT Node still named after the codename "Whitehorse", I like that ;-)... next you have a one <Solution> node with a <DesignData> element, contains information about the solution, and one <SdmDocument> with two <DesignData> elements. The first is information for the Deployment diagram the second for the design-surface, this one will contain sub-systems when we add shapes on the design surface. [see XML below]
When we add an WebService shape to the design surface we will get an extra <SdmDocument> element which contains beside <DesignData> all the configuration.
adding an endpoint to the Service will give you this XML.
So, although the structure looks a little bit overwhelming , it's easy to understand and basically you can say that there are models which contains models [The "designsurface"model has "application"models and an "application"model got "endpoints"models which also have types], all the models got <DesignData> and are described in a separate node which contains settings / properties.
Distributed System Designers Reader
Now, you only have to make some kind of Distributed System Designers Reader which collects this information for you. I made a object model [see image] and a factory method [with many XML readers] which loads the model. [started to change that to LINQ]
This last step, actually the complete post, needs to be done when you use VS2005 or VS2008 for Rosario it's much easier [see this post, Rosario Team Architect Exposed, Getting Information Out Off the Designers].
Anyway, next post generate Service Models which uses this object model.
Finally the GAX/GAT February 2008 Final Release
GAX/GAT February 2008 Final Release
The Guidance Automation Extensions (GAX) expands the capabilities of Visual Studio by allowing architects and developers to run guidance packages, such as those included in Software Factories, which automate key development tasks from within the Visual Studio environment.
The Guidance Automation Toolkit (GAT) is a guidance package which allows architects to author rich, integrated user experiences for reusable assets including Software Factories, frameworks, and patterns. The resulting Guidance Packages, composed of templates, wizards and recipes, help developers build solutions in a way consistent with the architecture guidance. In order to use the Guidance Automation Toolkit, you must first install the Guidance Automation Extensions.
New In This Release
The February 2008 Release of the Guidance Automation Extensions and Guidance Automation Toolkit has the following improvements to the earlier release, the July 2007 Community Technology Preview:
− Support for Visual Studio 2005 and/or Visual Studio 2008. This version of GAX will run on either version of Visual Studio. If you don’t have GAX installed, you can install GAX to support Visual Studio 2005 or Visual Studio 2008 or both. The installer will automatically determine which versions of Visual Studio you have installed.
Design for Operations, new release
Just watched Keith Pleas demonstrating the coming release of DFO on dnrTV, "Keith Pleas shows Design for Operations".
Still not available, but will soon be on, http://www.codeplex.com/dfo.
I'm curious if there will be sources available this time, so I can play with this scenario.
Update
Stay tuned for a major update to the designer.
As of 2/3/2008 we are still blocked waiting on the release of Guidance Automation Extensions (GAX) V1.4. No ETA yet.
New features include:
1) Improved model and user experience.
2) Limited support for Enterprise Library Logging.
3) Native support for model export to System Center Operations Manager Management Packs.
4) Support for existing applications through MSIL scanning for Event Log, Performance Counter, WMI and Enterprise Library instrumentation.
Related links:
New Year = Models, Code Discovery and Management Pack Generation
VSTSMMD requirements