@ClemensReijnen

Recent posts

Tags

Categories

Navigation

Pages

Archive

Blogroll

    Disclaimer

    The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway.

    September 18 IASA Netherlands Meeting

    iasa

    Third Tuesday in September is always a special event in The Netherlands. This time especially so since it will host the third Meetup for this year of IASA NL this time at ASML in Veldhoven. We think we have an interesting program with interesting speakers. We hope you will join us!

    Agenda for September 18:

    • 16.00 - ontvangst met koffie
    • 16.15 - welcome, IASA update
    • 16.30 - Viktor Grgic - Agile Architecting, doing more with less
    • 17:45 - Buffet
    • 18:30 - ASML architecture presentation, topic yet to be announced
    • 19:30 - Fishbowl/Open Space
    • 20:30 - Closing and drinks
    • 21:00 – End

    Aanmelden bij Meetup:

    Meetup logo

    Posted: Sep 01 2012, 06:42 by ClemensReijnen | Comments (0) RSS comment feed |
    Filed under: IASA

    Sogeti Labs introduction Video

    An introduction to SogetiLabs telling the story about this initiative connecting our technology leaders to the world.

    http://www.sogeti.com/SogetiLabs

    MTM Coded UI Test automation investment, when to use what.

    All validation (test) activities cost money. So, every activity to test a system should be executed as efficient as possible (see also the ‘testing in sprint’ series on this blog). Adding automation to the execution of these validation saves execution time, which saves money. But, specifying, creating and maintaining these automation cost money. So, the hard question is where is the balance, where the break-even point of test automation.

    Not that easy to answer question but lets give it a try. All quadrants in the ‘Agile Testing quadrant’ have some kind of Visual Studio tool support (No automation with out tool support).

      image

    For the technology focused tests, the ROI of test automation is a kind of clear. Developers create unit tests conform a specific code coverage level, proven the implementation is still correct, which give the team some comfort and quality level. For the load tests its of no discussion that there is a ROI.

    But, when we look at the ‘top - business facing’ tests, ROI of test automation is a bigger challenge. We have to think of how long is the validation member of our project (for example not all functional tests aren’t executed every sprint, only a sub set, only the most important, see this post ‘only meaningful tests’) and how many times is the validation executed (how many times over time but also on different environments). This gives us indications how much effort we must put in our test automation.

    Basic there are three test automation levels:

    1. No Automation
    2. Record and Playback
    3. Test Scripts

    Visual Studio adds two other levels to it.image
    (slide from the testing roadshow)

    All these automation levels have an invest, and a maintainability level. The better you can maintain a test case the longer you can use it for you ever evolving software system. That is the connection between ‘how long’ and ‘how well maintainable’, another connection is the effort it takes to create the automation. the resulting benefit is, you can execute your script over and over again.

    The ideal situation: a test script with very small investment to create, for a test which needs to be executed the whole life of the application which doesn’t change overtime. No investment, no maintainability issues, maximum amount of executions. Result:  maximum ROI.

    Too bad, we’re not living in the ideal world. So some tradeoffs need to make. 

    image No automation.

    No need for maintainable test scripts, no automation investment. I have customers who use MTM for Test Case Management only, and they are happy with it. maintaining thousands of test cases and their execution gathering information about the test coverage of the implemented functionality.  

    In most situations, this is an ideal starting point for adopting MTM and starting to look at test case automation. As a test organization getting used to the benefit of integrated tools the support of different ALM scenario's.

    image
    image Shared Steps with Action Recording | Record Playback parts.

    Collecting an action recording takes some effort. You have to think upfront what you want to do, and often you have to execute the test case several times to get a nice and clean action recording. So there is some investment to create an action recording which you can reuse over and over again. 
    You can’t maintain an action recording. When a application under tests changes or when the test cases changes you have to record every steps again. A fragile solution for automation.
    Using Shared Steps (reusable test steps) with their own action recording solves this a bit.
    image
    Find the test steps which appear in every test case, make a shared step of these steps and add an action recording to it. Optimize this action recoding and reuse the shared step in every test case. This definitely improves the ROI. Now you can fast forward all the boring steps and focus on the real test.

    The good thing, when a shared steps changes, you only have to record that one again.

    Creating multiple shared steps with action recoding and compose a test case is also a good scenario.

    After the zero investment, this is a good next step. You get used to the behavior of action recordings and have the benefit of reusing them throughout the project.
    Action recordings of shared steps keep their value the whole project, there is some effort to create and maintain them but you will execute them for every test case, a good ROI.

    image Test Cases with Action Recordings | Full Test Case Record Playback.

    The same activity as for the Shared Steps action recordings. But, you will use the action recording less and it is harder to maintain (more test steps). So defiantly the ROI is much lower as in the Shared Steps situation.

    The scenario where you create the action recording and execute often, for example on many different environment will bring some benefits. MTM action recordings can be recorded on one environment and playback on another environment.

    Another reason you might want to go with this scenario, is that you want to reuse the action recording for test script generation. See next step.
    image Generate test script from action recording.

    A really nice scenario for quickly creating test automation scripts.
    See this video for a quick how to:


    The maintainability of the generated code is hard. There are some tools in place to react on UI changes, which make it easier. Also challenging are some common development practices like branching, merging and versioning of the test script. See this presentation

    Conclusion, the creation of test scripts generated from action recordings is really fast but hard to maintain. This together with the recording of the action recording (number 2) has its influences on the investment. 
    image Write your own Test Script ( by using the Coded UI Framework ).

    Write the test automation script your self, following all the good coding principles of maintainability and reusability.  like: Separation of concerns, KISS principle, Don't repeat yourself, etcetera.

    Code First API Library is a good starting point.
    image

    This automation scenario is completely opposite of the generated test script (3). This one is hard to create, it will take some effort, but is (if implemented well) very maintainable. and you can follow all the coding practices and versioning strategies. 

    So, MTM with Coded UI support different test automation scenario’s. From fast creation with hard maintainability (2 and 3) to hard to create with good maintainability (4). It is a good to think up front about test automation before random start using the tools.

    My rules of thumb are:

    • Use 2 and 3 in a sprint and maybe in a release timeframe, but not longer. Maintainability will ruin the investment.
    • Use 4 for system lifetime tests.
      They run as long as the system code runs and should be treated and have the same quality as that code. Don’t use it for tests you only run in a sprint, the effort will be to big.
    • Use 0 and 1 always when ever you can. it supports several ALM scenario’s and the shared steps action recording really is a good test record-playback support with some good ROI.

    After a small investigation on a test tool conference I made the conclusion that no tool which supports test script generation (or something similar) all have some kind of workaround to make it maintainable. Only a good test automation framework (for example the Coded UI Framework or Selenium )will make test automation (I’m still talking about business facing tests) code maintainable. Test script generation is nice but you don’t want to maintain it. 

    Het Test Tool Event

    clip_image002

    http://www.sogeti.nl/evenementen/2012/het-test-tool-event

    11. Getting Testing Done in the Sprint - The Customer Test Team

    I don’t mean the acceptance test team, although many think and organize it as if it is the same.

    On a test conference a few weeks ago I heard the story of moving testing out of the sprint, or adding a specific test sprint after a development sprint. Both ‘worksrrounds’ I dislike, you lose the flexibility and feedback loop. It was something like the ‘IndependentParallelTesting’ story, also not my favorite solution.

    During the conference I posted a tweet, and got replies from Sander and Marcel. Lucky me I’m not alone Emoticon met brede lach (I like the last comment from Sander, the process just should support and embrace it, see: http://www.smartusecase.com.)

    image

    image

    The most clear characteristic of the customer test team is that they aren’t part of the scrum team. You could call them early adopters, power users or the customer test team. Often they execute real life scenario’s, play around in the application or when they are early adopters they do some real work with it.

    Beside the question, “do we really need them” is the challenge with these disconnected test teams is, how to get them and their information in a sprint. Threat them as operational findings is a way. But doesn’t solve the challenge of getting their usages data, their findings and reports back on our backlog and in a sprint.

    A nice explanation can be found in this video from ‘Groeten uit Delft’ where Rob van Lanen (www.agilestudio.nl) explains how they work together with operations.

     

    Beside the solution of giving the ‘disconnected’ customer test team a copy of Microsoft Test Manager 2012 and let them execute their test cases in a ‘structured’ exploratory way. http://msdn.microsoft.com/en-us/library/hh191621(v=vs.110).aspx
    Microsoft Test Manager - start exploratory test

    A nice work around to get the findings and knowledge from the customer test team in your teams project, is create a separate backlog for them (read this old post – will update it soon).

    Two other interesting technologies which can help teams with this scenario (findings found outside the team) are:

    1. PreEmptive Analytics for TFS.
      PreEmptive Analytics for TFS CE offers a free, lightweight way to integrate feedback-driven development processes into your development workflow. Your applications will automatically send back exception report data to your PreEmptive Analytics endpoint service as errors occur during their execution


      http://blogs.msdn.com/b/bharry/archive/2012/04/11/preemptive-analytics-in-visual-studio-and-tfs-11.aspx
    2. Collecting IntelliTrace Data Everywhere
      The collector saves IntelliTrace data as a recording that contains details about exceptions, threads, Web requests, modules, and other system information that is collected while your application runs. In most sections of the recording, you can select one of these items and start debugging with IntelliTrace in Visual Studio Ultimate.

     

    Past Tips:
    01. Getting Testing Done in the Sprint - The Team and Activities
    02. Getting Testing Done in the Sprint – Regression Test Sets
    03. Getting Testing Done in the Sprint – Test Automation
    04. Getting Testing Done in the Sprint – Undone Backlog Item
    05. Getting Testing Done in the Sprint – No Double, Triple Testing
    06. Getting Testing Done in the Sprint – PBI Implementation Sequence
    07. Getting Testing Done in the Sprint – Risk and Business driven Tests
    08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
    09. Getting Testing Done in the Sprint – Test Tasks on the Board
    10. Getting Testing Done in the Sprint – the Definition of Done

    Next Tips:
    12. Getting Testing Done in the Sprint – The Test Branch

     

    Posted: Jul 10 2012, 08:02 by ClemensReijnen | Comments (3) RSS comment feed |
    Filed under: ALM | Agile | VS2012

    Sogeti Windows 8 Metro Application Demo

    cross post from: Darren W Baker

    This video is a compilation of Windows 8 Metro Applications that Sogeti consultants are working on world wide. Sogeti was asked to put together a small video to be played at WPC 2012 during one of the Keynotes, and we are all very excited to see our hard work on the big screen.<<read more>>

    Posted: Jul 07 2012, 02:20 by ClemensReijnen | Comments (1) RSS comment feed |
    Filed under: Sogeti | win8

    Test Automation Day Deck - Agile ALM - Getting testing done in a sprint

    Below my http://www.testautomationday.com/ presentation (demo’s will follow next week on my YouTube channel).

    Agile teams find it hard to get the testing effort in sync with the other development activities. Not only development tests are executed during sprints, all other kind of testing activities are part of done. This session will give guidance how Microsoft Visual Studio ALM tools can support agile teams. How to run sprints and get testing done in a sprint.

     

     

    10. Getting Testing Done in the Sprint – the Definition of Done

    Are we done? When this is unclear the team will do too less or too much. Both are bad. The Definition of Done helps to define when you are done, the same as a master test plan helps to define the test levels. A good definition of done specific for your team and project will definitely help knowing when you're ready and to get testing done in a sprint. Mainly because you know when you're done.

    Making a good definition of done is a trivial task and books are written about it.
    A good starting point are these posts:

    One thing I really like to keep in mind when defining a Done List are:

    • Risk Analysis, and together with this the Master Test plan. I should be clear what the risks are and what the impact is on when you are done.
      You can think of Done ListItems, which help understand the level of testing to be done, like:
      • All High Risk BPI’s have a test coverage of ..% and no open bugs
      • All high prio test cases are automated
      • Low Risk, Low Business PBI’s may have 10% open low impact bugs.  
    • Test Levels, balance the unit, functional and acceptance testing and define where what is tested. Definition of Done items like; all unit tests are green, all functional testing is done or all written acceptance tests are executed, are unclear. How many tests should we write to be done? Code Coverage is one for unit testing, but how do you know when your done specifying functional tests and acceptance tests. So, writing down in the Done list that testing is done isn’t enough it also should have a statement about when test writing is Done.
    • When are we done creating the Regression Set and when are we done with Test Automation

    So, when testing is important and you don’t want to create a random amount of test cases for each PBI, should mention it in the Done List, and for sure make the validation automated with a build system.

    image

    Past Tips:
    01. Getting Testing Done in the Sprint - The Team and Activities
    02. Getting Testing Done in the Sprint – Regression Test Sets
    03. Getting Testing Done in the Sprint – Test Automation
    04. Getting Testing Done in the Sprint – Undone Backlog Item
    05. Getting Testing Done in the Sprint – No Double, Triple Testing
    06. Getting Testing Done in the Sprint – PBI Implementation Sequence
    07. Getting Testing Done in the Sprint – Risk and Business driven Tests
    08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
    09. Getting Testing Done in the Sprint – Test Tasks on the Board

    Next Tips:
    11. Getting Testing Done in the Sprint - The Customer Test Team
    12. Getting Testing Done in the Sprint – The Test Branch

    09. Getting Testing Done in the Sprint – Test Tasks on the Board

    It seems obvious that you need to have activities in the sprint for your testing activities. At least when you have something in your Definition Of Done what covers testing, like:

    • Test Complete
    • Integration tested done
    • Functional testing done
    • Regression testing done
    • Performance testing done
    • Acceptance testing done

    See: http://www.scrumalliance.org/articles/106-definition-of-done-a-reference. When there is totally no risk or business value in the system you are creating it can be that you don’t have to test it. (read: Risk and Business driven Tests), for all other systems… you need test tasks on the scrum board. I’ve seen multiple teams working with a scrum board which didn’t had any test activities, testing was a completely separate activity. So, this topic isn’t that strange.

    image

    These test tasks cover all the test activities the team need to execute to create a BPI conform the definition of done. this will be activities like:

    • Create test cases based on this and this test design technique for this PBI.
    • Setup the test infrastructure for load testing
    • Create test data for regression testing.
    • Automate this and this test cases.

    You name it, there are a lot of test tasks which need to be executed for a PBI to get Done. Often people get scared when they see all the test activities that need to be done, and want to minimize them. Bad job, this is a main reason why you can’t finish testing in a sprint.

    A good thing, people (Sogeti) already have been thinking about test activities and they have written a book about it: “TMap Next”.
    An even better thing, I create a TMap 4 TFS app with all the test activities ready to publish to you TFS Scrum board.

    For every phase in the test lifecycle

    a details page with test tasks

    with a detailed description of this task.

    5 2 3

    clip_image002

    Watch on YouTube

    Read the details

    When you don’t have a Windows Phone, you also can use the TMap for VS2010 process template, this one also contains a predefined set of testing tasks. Having predefined tasks isn’t really good, but it is still better as no testing tasks.

    Past Tips:
    01. Getting Testing Done in the Sprint - The Team and Activities
    02. Getting Testing Done in the Sprint – Regression Test Sets
    03. Getting Testing Done in the Sprint – Test Automation
    04. Getting Testing Done in the Sprint – Undone Backlog Item
    05. Getting Testing Done in the Sprint – No Double, Triple Testing
    06. Getting Testing Done in the Sprint – PBI Implementation Sequence
    07. Getting Testing Done in the Sprint – Risk and Business driven Tests
    08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests

    Next Tips:
    10. Getting Testing Done in the Sprint – Done
    11. Getting Testing Done in the Sprint - The Customer Test Team
    12. Getting Testing Done in the Sprint – The Test Branch

    TMap 4 TFS WP7 App – Connect 2 TFS video explanation

    For all the people who don’t own a windows phone and want to see how the TMap for TFS App works… see this video.

    Posted: May 17 2012, 07:41 by ClemensReijnen | Comments (1) RSS comment feed |
    Filed under: ALM | MTM | TMap | VS2010 | VS11 | Testing