We have arrived at the end of this little series on migrating unit tests from MSTest to XUnit (specifically, XUnit 2). While earlier posts concentrated on writing the tests and the XUnit counterparts to MSTest concepts, this post will briefly look at some non-code aspects to XUnit; most importantly, we will look at getting the tests to run.
Do not depend on test order
One thing to watch out for after migrating your tests is the order in which tests are run. MSTest allowed us to abuse testing by assuming that tests would run in a specific order. XUnit does away with this and will run tests in a random order. This can help to find some obscure bugs, but it can also make migration a little tougher, especially when tests share a data store or some other test fixture. Watch out for that.
Visual Studio
Running tests inside Visual Studio is really simple. Following in the footsteps of web development trends, rather than requiring an extension to the development environment, XUnit uses package management1. All you need to do is add the Visual Studio XUnit test runner package to your project and Visual Studio will be able to detect and run your XUnit tests just like your old MSTests.
Of course, if you're like me and absolutely loathe the built-in Visual Studio test explorer, you can use Resharper (or dotCover), which has built-in support for XUnit.
Command Line
More often than not, our continuous integration setups are scripted and we're unlikely to be running our unit tests via the development tool, such as Visual Studio. For situations like this, you can add the XUnit command line test runner package to your project. This provides a command line utility for running your tests with arguments to control exactly what tests and how2.
Once the package has been added, you can browse to where Nuget is storing packages for your particular project. In the tools folder of the console runner package folder, you will find xunit.console.exe. If you run this with the -? argument, you will get some helpful information.
xunit.console.exe -?
The three options I use the most are -trait, -notrait, and -parallel.
Traits
The two trait options control what tests you are running based on metadata attached to those tests. This is really useful if you have some tests that are resource heavy and only used in certain circumstances, such as stress testing. In MSTest, you could attach arbitrary metadata onto test methods using the TestProperty attribute. XUnit provides a similar feature using Trait. For example, [Trait("Category", "ManualOnly")] could be used to put a method into the ManualOnly category. You could then use the following line to execute tests that lack this trait.
xunit.console.exe -notrait "Category=ManualOnly"
If you wanted to only run tests with a specific trait, you do the same thing but with -trait instead. Both of these options are very useful when combined with -parallel.
Parallelism
XUnit can run tests in parallel, but tests within the same collection are never run in parallel with each other. By default, each test class is its own collection. Test classes can be combined into collections using the Collection attribute. Tests within the same collection will be executed randomly, but never in parallel.
The -parallel option provides four options: no parallelism, running tests from different assemblies in parallel, running tests from different collections in parallel, or running tests from different assemblies and different collections in parallel.
The difference between assembly parallelism, collection parallelism, and both together
Let's assume you have two assemblies, A and B. Assembly A has three collections; 1, 2, and 3. Assembly B has three collections; 4, 5, and 6.
No Parallelism
-parallel noneXUnit will run each collection in one of the assemblies, one at a time, then run each collection in the other assembly one at a time.
Collections 1, 2, 3, 4, 5, and 6 will never execute at the same time as each other.
Parallel Assemblies
-parallel assembliesXUnit will run each collection in assembly A, one at a time, at the same time as running each collection in assembly B, one at a time.
Collections 1, 2, and 3 will not execute at the same time as each other; Collections 4, 5, and 6 will not execute at the same time as each other; but collections 1, 2, and 3 will execute in parallel with 4, 5, and 6, as the 1, 2, and 3 are in a different assembly to 4, 5, and 6.
Parallel Collections
-parallel collectionsXUnit will run each collection within an assembly in parallel, but only one assembly at a time.
Collections 1, 2 and 3 will execute parallel; collections 4, 5, and 6 will execute in parallel; but, 1, 2, and 3 will not run at the same time as 4, 5, and 6.
Parallel Everything
-parallel allXUnit will run each collection in parallel, regardless of its assembly.
Collections 1, 2, 3, 4, 5, and 6 will run in parallel, each potentially running at the same time as any other.
Beware running tests in parallel when first migrating from MSTest. It is a surefire way of finding some heinous test fixture dependencies and you risk thinks like deadlocking on resources. Usually, running assemblies in parallel is a lot safer than running collections in parallel, assuming that tests are collocated in assemblies based on their purpose and the resources they interact with.
In Conclusion…
That brings us to the end of the series. I have focused primarily on migrating from MSTest, leaving out a lot of the nuances to XUnit. I highly recommend continuing your education with the XUnit documentation and through experimentation; having personally migrated several projects, I know you won't regret it.
- I love this approach to augmenting the development environment. It requires no additional tooling setup to get your dev environment working. The source itself controls the tooling versions and installation. Wonderful [↩]
- You can control how tests run under MSBuild too by using various properties. This is discussed more on the XUnit site [↩]
