06. Getting Testing Done in the Sprint – PBI Implementation Sequence
A challenge in getting testing done in a sprint is the fact that the software (PBI) isn’t ready to be tested till implementation/ coding is done.
How To.
Work on completing each item in the sprint backlog and finish one item after another, see task boards below.
BAD -- Not able to start testing task board, every team member works on a different pbi. Testing can only start at the end of the sprint.
GOOD -- Ready to start testing task board, team members work together on a pbi one after another till it’s done. Testing can start early.
It is a good practice, but not often followed. it really helps getting testing done.
Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities
02. Getting Testing Done in the Sprint – Regression Test Sets
03. Getting Testing Done in the Sprint – Test Automation
04. Getting Testing Done in the Sprint – Undone Backlog Item
05. Getting Testing Done in the Sprint – No Double, Triple Testing
Next Tips:
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…
05. Getting Testing Done in the Sprint – No Double, Triple Testing
One way of getting testing done in a sprint is making it [testing] as efficient as possible, meanwhile this also saves money. One way of getting more efficient is not doing things double or triple. An interesting thing with testing, we do test double or triple. We test it during unit, during unit integration, during functional and during acceptance testing, this can be done better, we could save a lot of time. I’ve seen teams which also automated all these types of tests, testing the same thing... not really efficient, they didn’t manage to get it done in a sprint.
Each phase of the development process has its own test goals and depth of test coverage. The depth particularly depends upon the risks: higher probability of damage requires better test coverage. These test goals are especially aimed at discovering errors that can be found in that phase (“as early as possible”).
- during unit testing the internal logic of the unit is tested,
- the unit integration test needs to show that units ”understand” each other.
- the system test needs to prove/demonstrate that the system complies with the functional requirements that have been agreed to,
- finally, the acceptance test considers how the system fits into the environment in which it will operate.
Test goals and (depth of) test coverage per phase are made compatible with each other using a Master Test Plan (see TMap® www.tmap.net).
Every test must be able to rely on previous tests, only then will the complexity of errors be contained to errors that were introduced during the last step. Setting up a learning cycle makes it possible to correct shortcomings, if any, in the testing process chain.
Each phase of the development process and its corresponding test goal introduces a specific role with specific abilities and tasks. These roles are supported by specific ways of testing, often visualized in the Agile Test Quadrants.
http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
How the quadrants work
Tests on the left-hand quadrants help the team know what code to write, and know when they are done writing it. Tests on the right hand side help the team learn more about the code they’re written, and this learning often translates into new user stories and tests that feed back to the left-hand quadrants. Business stakeholders define quality criteria for the top two quadrants, while the bottom two quadrants relate more to internal quality and criteria.
from: http://readytohelpfortesting.blogspot.com/2012/01/ttesting-quadrants.html
How To.
All these types should work together, and as already mentioned a Master Test Plan (see TMap® www.tmap.net) can help. What also can help; is getting the team together and discus it [what to test where]. This is even more important when the first tip isn’t in place (01. Getting Testing Done in the Sprint - The Team and Activities) or when there is a Customer Test Team (will be explained in tip: 11. Getting Testing Done in the Sprint - The Customer Test Team).
I like the example: where it is very hard to create unit tests for a specific piece off functionality, like login screens. To create unit test you have to use a mocking framework that mimics the user database or AD. This takes some time implement and maintain, while the functional testers will use the login screen (and test it) every time they run a test. A nice piece of functionality to have a good discussion with the tester where to test and automate this.
An other example is a really world scenario I’ve seen; this is where the developers created some great unit tests for input validation, the functional system testers created test scripts to tests the input validation and the acceptance tester tested the input validations... really a scenario where there wasn’t any communication between the testers and people didn’t trust each other (see 01. Getting Testing Done in the Sprint - The Team and Activities)
More reading: the Test Automation Pyramid http://blog.goneopen.com/2010/08/test-automation-pyramid-review/
Tool Support:
All test types are supported by VS2010.
- The development tester:
This tester knows the inner workings of the system and makes use of test-frameworks to automate the tests that he wants to perform. This testing role, has been supported for a while now by technical aids as unit test frameworks in Visual Studio, and is primarily done by developers. VS2010 eases the life of the development tester by intelligent support of Test Driven Development. - The system tester:
The system tester tests the system for completeness, whether requirements have been met. These are not only functional requirements, but also quality requirements like performance and security (see agile test quadrant).
System testers use a lot of scripts to automate their activities, he/she would like to perform the same tests using various infrastructures and configurations. VS2010 has had Load- and Web testing for quite a while in the Test Edition and with the introduction of VS2010 the creation of test scripts is simplified by introduction of the ‘Coded UI’ framework. - The acceptance tester:
The acceptance tester, also called user tester or black box tester, truly is a tester who performs a test of the application and knows nothing about the internal workings of the application. He/she constructs test cases using functional requirements, using various testing techniques.
Acceptance testers, who often still use Excel sheets to perform their tasks, are abundantly well supported in Visual Studio 2010. Test cases are registered in TFS and he/she has access to Microsoft Test Manager on optimal support of, usually, manual testing activities.
There isn’t any support to see if pieces of functionality are tested twice. But with unit test coverage and data diagnostic adapters for functional testing there are some capabilities in place to extract the information from the system. Maybe time to create a custom data diagnostic adapter, visualize it and compare it with the unit test coverage.
Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities
02. Getting Testing Done in the Sprint – Regression Test Sets
03. Getting Testing Done in the Sprint – Test Automation
04. Getting Testing Done in the Sprint – Undone Backlog Item
Next Tips:
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…
04. Getting Testing Done in the Sprint – Undone Backlog Item
A tip straight from the previous Scrum Guide (The guide updated and the undone TIP is removed, read The 2011 Scrum Guide – A Lot of Changes! ). But, the TIP still works, although it isn’t an ‘official’ tip anymore…
End to end testing, is not an agile specific testing practice, it’s a common sense testing practice. Create test cases to detected defects anywhere in the chain. This type of testing has been growing the couple of years enormous due to the rise of Service Oriented Architectures and more recent Cloud Computing. as the TMap end-To-End Testing books describes
“End-to-end testing is a form of testing in which the (business) process is tracked through an interconnected series of systems, which may be based in different organizations, aiming to answer the following questions:
- Does the IT process support the business process adequately? - Are the process and systems integrated with each other in the right way?”
TMap.net End-To-End Testing
End-to-End testing is seen as a challenges in Agile environments, with the most important challenge that systems under development aren’t ready for End-to-End testing. Not every service is finished to test the complete end-to-end business process. Stubs are a solution, but can these be made in the current sprint and are stubs good enough to say something about the quality of the system. And, when the system are ready and a change in a service is needed, how and can we re-run all the end-to-end tests, is there time enough.
All these challenges with end-to-end testing in agile projects can be summarized in two topics:
Scrum solves this problem, this problem of not able to be compliant on the definition of done at sprint finish, by defining ‘Undone work’. the scrum guide mentions this in a TIP and in the final thoughts of the scrum guide
Some organizations are incapable of building a complete increment within one Sprint. They may not yet have the automated testing infrastructure to complete all of the testing. In this case, two categories are created for each increment: the “done” work and the “undone” work. The “undone” work is the portion of each increment that will have to be completed at a later time. The Product Owner knows exactly what he or she is inspecting at the end of the Sprint because the increment meets the definition of “done” and the Product Owner understands the definition. “Undone” work is added to a Product Backlog item named “undone work” so it accumulates and correctly reflects on the Release Burndown graph. This technique creates transparency in progress toward a release. The inspect and adapt in the Sprint Review is as accurate as this transparency.
Within the Visual Studio 2010 Scrum process template this means that you have to link or create tasks to a specific ‘Undone Work’ product backlog item. this ‘Undone Work’ product backlog item accumulates the undone work regarding to End-to-End Testing (and others) over time and can be monitored just as other backlog items.
http://www.slideshare.net/xwarzee/path-to-agility-ken-schwaber
Tool Support
In TFS with the Scrum process template we can create Backlog Items very easy.
So, creating an undone backlog item isn’t that hard, the challenge is take it with you every sprint. for example in my undone backlog item I’ve got some undone tasks from sprint 1 and some from sprint 4. The iteration paths for these tasks still point to the sprint 1 and 4 sprints.
This will result in a wrong sprint backlog, because the tasks who point to other sprints as the current sprint won’t appear in the sprint backlog. So, when moving this Undone backlog item to another sprint because not all tasks are done, you also have to change the iteration path for the corresponding tasks.
Just keep in mind TFS doesn't update the hierarchy, you have to do it manually (or use Excel, my favorite).
Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities
02. Getting Testing Done in the Sprint – Regression Test Sets
03. Getting Testing Done in the Sprint – Test Automation
Next Tips:
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…
03. Getting Testing Done in the Sprint – Test Automation
Automate your testing effort, or better your test execution, is a must to get your testing done during the sprint. Automate your test execution shrinks the time needed to execute the tests, which leaves more time for the team to specify new test cases. On the other hand it also brings some inflexibility in your testing, you need to maintain the test automation code and update it when functionality change, which again cost time which the team better could use to specify and execute test cases.
How to.
So, there need some strategy what to automation and how to automate. The decision what test cases are candidate for automation depend on several decisions. For sure the regression set is a good candidate (see Tip 02. Getting Testing Done in the Sprint – Regression Test Sets) which is a collection of test cases based on risk classifications and business value (see TIP 07. Getting Testing Done in the Sprint – Risk and Business driven Tests).
With test automation you also must take in to account that the test cases is automatable, maintainable and has some ROI. Together with this ROI of test automation also comes the design for testability. It should be easy to create test automations, the design of the application should have this in its design and architecture.
How many, how much effort you have to put in test automation has several investment levels. From zero completely no investment till, a very high investment of changing the architecture for testability and automate s many scenario's as possible.
Tools support.
See this video for the Visual Studio test automation support.
Also Visual Studio test automation support has several investment levels, in the presentation below at slide 38 and 39 these levels are discussed. The rest of the presentation discusses the different tactics of test automation with Coded UI.
Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities
02. Getting Testing Done in the Sprint – Regression Test Sets
Next Tips:
04. Getting Testing Done in the Sprint – Undone Backlog Item
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…
02. Getting Testing Done in the Sprint – Regression Test Sets
Some teams rerun every test every sprint, this is time consuming and isn’t worth the effort. Having a clear understanding what tests to execute during regression testing raises the return of investment of this testing effort and gives more time to specify and execute test cases for the functionality implemented during the current sprint.
How To.
So, collecting a good regression set is important. There area lot of approaches how to get this regressions set, most of them are based on risk classifications and business value (see TIP 07. Getting Testing Done in the Sprint – Risk and Business driven Tests).
From “A good regression test is invaluable.”
The principle is that from each test case a collection of additional data is determined into the test cases for the regression test are ‘classified’. Using these classifications all cross sections along the subsets of test cases can form the total test that are selected.
Automation of this regression set is almost a must (see next TIP 03. Getting Testing Done in the Sprint – Test Automation).
Tool support.
Microsoft Test Manager has an interesting capability to control the amount of regression testing that need to be done during the sprint. the data diagnostic adapter “Test Impact”, it give information about test cases which are impacted by code changes
and see page 63 of the training guide http://www.slideshare.net/clemensreijnen/mtlm-visual-studio-2010-alm-workshop
How to create a regression set in Microsoft Test Manager.
Read:
Guidance for Creating Test Plans and Test Suites and read this http://blogs.msdn.com/b/anutthara/archive/2010/09/22/guidance-for-creating-test-plans-and-test-suites.aspx and this Management, preserving and organization of manual test cases within VSTS 2010.
One thing to keep in mind: In MTM2010 the copying of test cases between test plans copies the reference by default and not the test case. So, a test case can be referenced in multiple test plans and suites, changing a test step will change it on all locations.
Creating regression sets for developer tests and functional system tests are mostly more easy. Unit tests are small and run fast, so no problem to run them all. Unit Integration tests are a bit more challenging, you need to balance a bit more what to run when, but both developer tests and functional system tests, aren’t often the biggest bottlenecks in getting testing done in the sprint. When they are, try to do the same approach as mentioned above… balance what to run when.
Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities
Next Tips:
03. Getting Testing Done in the Sprint – Test Automation
04. Getting Testing Done in the Sprint – Undone Backlog Item
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…
01. Getting Testing Done in the Sprint - The Team and Activities
Every Agile ALM presentation the same question is asked: How do I get testing done in a sprint?
There isn’t a one solution/ practices/ to do/ how to fits all for this. So, I made a small list of tips you could do (or recommended to do, or must do… ), which can help to get testing done in a sprint.
Tip 1: Have testing knowledge in the team.
Actually not a tip, it is a must. This is a kind of obvious but not common and I think the hardest thing to accomplish. It is: get testing knowledge available in your team. When you don't have it, you will fail.
How to
Now, it isn’t that easy to accomplish this collaboration. Test and dev are different, a nice quote from this ‘test’blog:
In the D-world, the world of the Developers, we think Generalist Testers are pencil-pushing, nit-picky quality geeks. Mostly they are beside the point and are easily replaced. They seems to like making much noise about little defects, as if we made those errors deliberately....
In the T-world we don't hate the Developers for their perceptions. We are disappointed about the poor quality of the software.
Bad assumptions on the part of Developers are more to blame for the problems than are software weaknesses.
We never(or seldom) get software what will work right the first time. No, in the T-world we think that developers forget for whom they are building software, it looks like they are building for themselves......
Try to combine these two worlds in one team, you definitely need to come up with a Collaborative Culture:
The three most important concerns are:
- Trust.
- a topic closely associated with trust when it refers to people is Identity.
- Collaborative culture.
- A collaborative culture consists of many things, including:
- Collaborative leadership;
- Shared goals;
- Shared model of the truth; and
- Rules or norms.
- Reward.
- a “reward” for successful collaboration is most often of a non-financial nature.
Show me the value, seems to be the magic word.
Test adds knowledge, knowledge during the grooming the backlog. Helping the product owner with defining proper acceptance criteria. And, help find improper written backlog items with inconsistencies for example. (this is my favorite test activity I didn’t know about before started reading TMap, Assessing the Test Base). In both ways helping the product owner and the team to focus on value.
Tools support, Visual Studio 2010
Tools can help, when using the VS2010 TMap Testing Process template you will notice that the test activities get an important place, helping the tester to get on board.
The ‘Testing’ Tab with test activities, next to the implementation tab.
Still two different worlds, test and dev in this way… but it give a good visual reward of being connected. Probably many teams won’t need an additional visualization of testing effort and can use the scrum process template in combination with their testing methodology, but this will help them to get started.
Interesting topic is the introduction of the manual test tool ‘Test Manager’ with Visual Studio in the current release. It helps teams to get more connected, it shows the pain points where the collaboration isn’t seamless. So, adopting MTM can be a good start for agile teams to get testing aboard. But, be aware the interactions are more important as tools. The tools won’t fix bad collaboration, mismatching identities, lack of trust and won’t give any reward.
Next Tips:
02. Getting Testing Done in the Sprint – Regression Test Sets
03. Getting Testing Done in the Sprint – Test Automation
04. Getting Testing Done in the Sprint – Undone Backlog Item
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…
TFS11 Scrum board update remaining work without opening the work item…
Very nice MSFT added the capability to update the remaining hours without having to open the work item. and even more nice it has some intelligence behind the drop down, because it populates the dropdown list with values in range.
Deck: The agile cloud development scenario in 10 acts…
The presentation from last SDN event … about Hosted TFS
Teams in Visual Studio 11 CTP, The Bug Backlog.
Default are bugs visually in the product backlog. When a tester files a bugs from test manager (or any other client tool), it will be shown in the teams product backlog.
For example the bug found and saved with a test run in Microsoft Test Manager, will show up in the backlog with the state ‘New’.
When you look at the iteration path of this bug, you will see it is saved in the wrong area path. This will happen when you don’t set the area path of the test plan in the test plan properties. Even when you do set the test case iteration path correct.
Recommendation 1: set the test plan iteration properties before you do anything else.
Now the bug will also be seen in the sprint backlog, from where the team can commit to it, add tasks and handle it as a normal backlog item.
Now, not everybody is happy with the solution that every bug is visible in the backlog. many small bugs will blur the visual management capability of the backlog and task board, and maybe very big ones are to big to solve in the same sprint or need some more investigation and product owner collaboration.
The ‘too many too small bugs in the backlog’ situation can easy been solved; make the collaboration between developer en tester better and let them very quickly decide if it can be solved in the same hour, the same day or in the same sprint.
- For the same hour bug fix there is no need to create a work item type (wit) bug for it, just talk and solve the bug.
- For the same sprint bug fix you probably would create a task for so it appears on the task board and you don’t forget it.
So, find the bug – discus – solve or create task. It isn’t possible to create a task (for a bug) linked to a product backlog item directly from Microsoft Test Manager/ Test Runner. So this is going to be a manual activity, just open the task board and add the bug tasks.
so there are two solutions; or you add the bugs to the backlog (A) or you create tasks for it (B). In my point of view having them (the bugs) in the backlog is much better, mainly because of the rich bug information and the traceability (see image below – a bug with tasks, test case, pbi relations and data collector information).
_______________
When you definitely don’t want bugs in your backlog you can use another solution; create a separate bug backlog. It needs some more work but it works. (read this post from Mike Cohn)
The only way to create two backlogs is to work with areas. See below, I Added two areas.
My Team is responsible for the product backlog with just only Product Backlog Items.
This is how the backlog and sprint backlog look now. No more bugs, live is easy again 
To get the bug backlog we have to create an additional team which does the bug triage. I created for this situation a bug triage team with as the one and only member ‘My Team’ with his members.
Make this team responsible for the ‘Bug Backlog’ area and we have a real bug backlog.
There is only one important decision left; how are we going to use the bug backlog, are we going to look at it at the end of the sprint or are we discussing it during the daily standup? And, what do we do with the bugs we are going to solve. It is easy to push them to the ‘real’ sprint backlog by changing the area path. We just prioritized and solved them directly from the bug backlog, and we didn’t used the sprint in the bug backlog.
My favorite solution is a blend of all three solutions:
- For very small one just create tasks.
- For a bit bigger bug but which we can solve in the same sprint, add them to the sprint backlog
- For very big ones which we don’t know add them to a separate bug backlog from where we can do a triage and push them back to the backlog for wider decision making or explanation by the product owner or just solve it and push it to the sprint backlog.
Teams in Visual Studio 11, feature teams and backlogs.
Before continue reading, read the basics - Teams in Visual Studio 11, the basics -
Sometimes a project is bigger than one team can handle (with the recommended team size, see the scrum guide). A solution for this big project problem is the creation of multiple teams and organize scrum of scrums meetings. With the scrum of scrums meetings we hope we will solve the possible integration and reuse problems.
According to Mike Cohn, author of Agile Estimating and Planning, The Scrum of Scrums (SoS) meeting "is an important technique in scaling Scrum to large project teams. These meetings allow clusters of teams to discuss their work, focusing especially on areas of overlap and integration."
Scrum of Scrums - Issues and Value
From a backlog perspective we also have to make a decision, are we going to use one backlog for multiple teams or one backlog per team.
One backlog for multiple teams
In TFS vNext this can be done by assigning the same area to the different teams. in this way they will get the same backlog, and can commit to backlog items for a sprint.
This solution will get more challenging during the sprint planning meeting. The teams also have the same sprint backlog.
I do think teams will create a mess out of this, asking questions like “did we committed to this backlog item or did the other team do it?” and “how much do we burn each sprint?”.
Both teams also share the same task board in TFs vNext when configuring them for the same area, which doesn’t make everything more clear.
I would recommend don’t use the one backlog for multiple teams configuration. So, make teams responsible for their own area in TFS vNext (actually not only in TFS but always).
One backlog per team
One Team, One Backlog is easy configure in TFS vNext, just use the area’s. So, first we must add the necessary area’s for our teams.
Just add new child's to the project (don’t make it too complex). One important thing is that you define the area’s as functional slices, not horizontal slice. For example data layer, business layer or cross cutting concerns aren’t allowed (teams make functionality end users can use!).
Now we have our areas we can set some restrictions (not really necessary when teams trust each other, but it will help making mistakes).
Add the team
Default a team which is added to an area inherits the rights from contributors.
and in the contributors TFS group are all the teams.
So we need to set in the area that other teams aren’t allowed to edit work items. Adding those teams and set their permissions to deny will do the trick. Now the other teams can’t accidentally play with each others backlogs anymore.
(I hope Microsoft will make these messages more friendly.)
All teams follow the projects sprint cycle (see also previous post), but with a different area. When a team is active in a sprint can be configured in the team administration section. It’s not preferable to put a team to sleep for one or two sprints but you can do it.
There are many more solution scenario’s how to handle big project, the presentation download on this page “Scaling Scrum” gives a nice overview of the possible solutions for big agile projects. I do think a backlog per team would be sufficient in 80% of the big project scenario’s.