Did you know you can manage and run CodedUI tests without Visual Studio? In an interview I recently did with .Net Rocks, one topic that came up was how to manage and run CodedUI tests on demand or as part of a nightly test run without using Visual Studio. You can accomplish this using Microsoft Test Manager.

ALM Test Infrastructure

One of the things that isn't well known about the Microsoft ALM Test Infrastructure, which is part of the Visual Studio ALM tool suite, is the ability to manage your automated tests using the Microsoft Test Manager (MTM). With MTM you can manage your regression suites and run the regression suites straight from the tool. To give you some insight on how this is done, let's walk through the steps involved to set this up.

First let's assume you've built a larger set of CodedUI tests that exercise your application. Since the test set has become quite large, running all of the tests might take several hours to complete. How can you run the tests required to validate a change without running all of the tests every time? With MTM you have the ability to build a "test plan." A test plan contains a set of test suites that can be static suites (you pick the tests that are part of the suite), requirement-based suites (you select the requirement that links to the test cases), or query-based suites (you created queries to select the tests that are part of the suite). With query-based suites you can build up suites that target a specific area of your application.

Here's a scenario I like to use to explain how things play out. Imagine we work at an insurance company that has a web-based application it bought from a vendor. It contains different insurance products and business processes which are managed with the product. Let's say it has policies for personal liability, car insurance and home insurance. It has different processes such as policy affirmation, claim settlement and adding a policy holder. Now let's assume the vendor made a change to the claim settlement process. In this case you would like to run a regression set across all products for that particular process to check if something is broken.

Setting up the MTM test cases that relate to the CodedUI tests

Our goal is to create test suites which are tied to the CodedUI tests, and use these sub-selections of tests to complete a test run that validates a specific change made to the system. For this to work there are a couple of things you need to get in place:

1. You need to have a way to differentiate the tests from each other based on the focus areas you have in your application.
2. You need to import all the automated tests as MTM test cases so you can manage and run them from MTM.

So let's look at step one first. You need to name your tests in such a way that you can build regression suites for specific purposes. How can we make this work? Write your CodedUI tests using the page object pattern so you have a good maintainable set of UI tests (you can find more info on this pattern in my course on CodedUI and on my blog here).

Next you need to import those tests into MTM so you can then build out the regression sets.

When you build the test scenarios based on the page objects, you need to ensure you can query on the tests in such a way that you can group tests together based on process or product type. So you need a naming convention. In this particular case it would make sense to say that you have a naming convention where you name the tests as follows: <ProcessName>_<ProductName>_TestScenario. So this would result in names like: PolicyAffirmation_CarInsurance_PersonAgeAbove60 and PolicyAffirmation_CarInsurance_PersonAgeBelow20.

This way you can immediately tell which scenario the tests are validating, and at a later stage, build test suites that group tests based on parts of the name.

Once we have written the tests, the next step is to import the tests in MTM. You can do this manually by creating the test cases in MTM, then going over to Visual Studio to link them to the actual CodedUI test, but it's a very cumbersome process. Luckily there's an alternative! You can import the tests based on the assembly that is built, then generate the test cases in MTM. For this you need to use the tool called TCM.exe, which is part of the standard Visual Studio installation.

Let's assume the assembly that contains the CodedUI tests is called InsuranceRegressionTests.dll. In order to import the tests in Microsoft Test Manager, or better stated, in Team Foundation Server so they show up in Microsoft Test Manager, you need to run the following command line:

>tcm.exe testcase /import /collection:<your TFS Server collection name>  /teamproject:<your team project>  /storage: InsuranceRegressionTests.dll

Optionally, you can filter the test cases that are imported by specifying the /category option, and that will match only tests that have the category attribute specified on top of the test methods or class.

After running this command, you can now run a query in MTM and look at the test cases that were imported. This will look something like shown in the screenshot below:

Structuring your tests

So now that we've imported all of our tests into MTM, let's structure the test now in such a way that we can select a group of tests to validate either a process or product. To do this you first need to create a new test plan:

Now we have a clean new plan that we can start organizing. Add two static test suites at the top level of the plan that dissect the groups created in process and product as shown here below:

The next step is to fill up the process suite with additional query-based suites. Here we are defining the set of tests based on the name in the test containing either the name of the process we need or the product we need. To define a suite that shows all test cases that test the process “Policy Affirmation” we create a query that looks at the title of the test and requires the text "PolicyAffirmation" to be in the title as show below:

And we can do the same for particular products as show below for product, car insurance:

After we have done this for all processes and products, the full regression suite looks like this:

Setting up the Lab Management machines

Now that we have the test plan complete and created nice buckets that focus on specific processes or products, we can now run the tests from MTM. This requires one additional step and that is to set up a lab environment. A lab environment contains the machines that will function as the web client machine running the actual tests. To create the environment, select the "Lab Canagement" tab in MTM.

Now that you've created a new environment, select a test machine you have available in your network. In my case the name of the test machine is “TestClient.”

Running your tests from MTM

Now that we have a lab machine available, we can run the tests from MTM. To do this go to the "Test" tab and there you can select the option to run the test.

Now you will get the dialog asking where you would like to run the tests.

This is where you need specify the build the tests are part of (the build containing the CodedUI tests) and the test environment you want to use. In this case, it's the VSALM test environment we just created. The final thing to note is that the test setting specifies what the diagnostic data collectors will capture during the test run. This is where you can specify to use test settings that contain Intellitrace log capture, video capture and even audio capture on the target machine.

After you start a run, you will get a results view that shows the progress:

And when the run is finished you'll get a summary of the run and a overall status report on the test plan.

Running the regression suites as part of the daily build process

One thing haven't mentioned yet is the fact that we can use the existing test plan we created together with a lab management build. This is a build defined in Visual Studio where we can specify the lab environment we want to use (VSALM), the test plane we want to execute, and the build we will use that contains the test assemblies. You can schedule this build to run every night on the latest build of the CodedUI tests, and this way each morning when you get into the office you'll already see a full summary like one shown above, based on a fully automated regression test that is scheduled and executed.

So with these steps you can manage your CodedUI tests from Microsoft Test Manager. You can create nice functional buckets of test suites by using the query based suites, can run your tests on existing machines just by adding them to the lab environment, and any functional tester can run the regression suites without ever knowing about Visual Studio.

To learn more, check out Test Automation with CodedUI in the Pluralsight library.

Ready to test your skills in Visual Studio? See how they stack up with this assessment from Smarterer. Start this Visual Studio test now