Running Automated Tests on Physical Environments, the different flavors…

There are multiple ways to execute your test automation effort, you can run them from and on Microsoft Test Manager, from and on Visual Studio, during the build and on the Build Server or during the build on Test Agents configured with Test Controllers or … many flavors. so, where should you execute your automated tests, and how should you configure your test infrastructure.

omgevingen

Where to run your automated tests, and how to configure the test infrastructure?

 

The different flavors

Flavor A: Execution from VS2010…

 
SNAGHTMLdbc748

Purpose: Real developer test / test automation dry run. 
A valid scenario is when Developers create a codedUI test from a test case which found a bug. To reproduce the bug, developers generate the CodedUI add the assertion and verify the result when they fixed the bug.

Triggered by: right mouse click – run tests, in test project, or Test List Editor, run checked tests.

Information: http://msdn.microsoft.com/en-us/library/dd286580.aspx

Pro
Easy setup
Debug-able test automation
No additional configuration needed*

Con
No collection of test results in TFS/
Test Manager 
R
un on developer environment

* Data Diagnostic Settings in the Local.Testsettings file are configurable.

 

Flavor B: Execution during build with Build controller

 
SNAGHTMLeae534

Purpose: Part of Build Verification Tests (its strange to run UI tests on a build and have a build service running in interactive mode)

Triggered by: build

Information:
Set the Build Service to run in interactive mode.
http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
Configure the build to run the tests
http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType

Pro
Easy setup
Test results in build result

Con
No collection of test results in Test Manager (no reporting on test points)
Build Service needs to run in interactive mode (what happens after a restart)
Tests are executed on build environment

   

Flavor C: Execution during build with Test controller

 
SNAGHTMLf3aa28

Purpose: Part of Build Verification Tests, distribute tests over multiple test agents. (Preferred configuration above flavor B)

Triggered by: build

Information:
Configure Test Controller (don’t register it with a project collection )
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers
Configure
Test Agents on clients (interactive mode)
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents 
Configure *.Testsettings file in solution to use Test Controller and Test Agents
Configure the build to run the tests (see B)

Pro
Tests run distributed over multiple environments
Test Results in Build result

Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments
Test Settings from VS

   

Flavor D: Execution from Microsoft Test Manager

 
SNAGHTML11b96c8

Purpose: Part of Regression Tests (other type of test than BVT).

Triggered by: MTM user, right mouse click on test case, run

Information:
Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller and create test ‘agent’ physical environment.
http://msdn.microsoft.com/en-us/library/ee390842.aspx
http://msdn.microsoft.com/en-us/library/dd293551.aspx
Associate CodedUI test with WI Test Case from VS.
http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html

Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM
Full control by the tester

Con
Test Controller needs to be configured with a Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure
Hard to see which test case is automated

   

Flavor E: Execution from MTM during Build…

 
SNAGHTMLfdabcd

Purpose: Part of BVT. Preferred configuration above flavor C. Flavor D and E can be configured together.

Triggered by: Build

Information:
Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
Associate CodedUI test with WI Test Case from VS.
Create Build task to run TCM or MSTEST task for Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx
How to: Run Test Cases with Automation from the Command Line Using Tcm
http://msdn.microsoft.com/en-us/library/dd465192.aspx

Pro
Test run distributed over test environments.
Tests can be configured to run on different configured environments 
Test Result in MTM and TFS
Triggered by build
Test Settings from MTM

Con
Hard to configure
maintenance of TCM commands in build

Enough flavors to choose from… the pro and cons I wrote down can differ per situation. My preferred flavors are E and D configured together. The reason, test settings and test results are where the belong, in the testers hands. And, the TCM command line give much more flexibility what to run where during the build.

Lab Management for virtual environments have different configurations, pros and cons… will make a post for this later…

<UPDATE>
A
n update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution from Colin Dembovsky it’s a XAML build template which lets you easily configure the Build-Deploy-Test workflow for physical environments. one drawback from flavor E will be minimized when using this template, the maintenance of the TCM commands. 

Add comment


Şarkı Sozleri