Running XUnit Tests, Using Traits, and Leveraging Parallelism

We have arrived at the end of this little series on migrating unit tests from MSTest to XUnit (specifically, XUnit 2). While earlier posts concentrated on writing the tests and the XUnit counterparts to MSTest concepts, this post will briefly look at some non-code aspects to XUnit; most importantly, we will look at getting the tests to run.

Do not depend on test order
One thing to watch out for after migrating your tests is the order in which tests are run. MSTest allowed us to abuse testing by assuming that tests would run in a specific order. XUnit does away with this and will run tests in a random order. This can help to find some obscure bugs, but it can also make migration a little tougher, especially when tests share a data store or some other test fixture. Watch out for that.

Visual Studio

Running tests inside Visual Studio is really simple. Following in the footsteps of web development trends, rather than requiring an extension to the development environment, XUnit uses package management1. All you need to do is add the Visual Studio XUnit test runner package to your project and Visual Studio will be able to detect and run your XUnit tests just like your old MSTests.

Of course, if you're like me and absolutely loathe the built-in Visual Studio test explorer, you can use Resharper (or dotCover), which has built-in support for XUnit.

Command Line

More often than not, our continuous integration setups are scripted and we're unlikely to be running our unit tests via the development tool, such as Visual Studio. For situations like this, you can add the XUnit command line test runner package to your project. This provides a command line utility for running your tests with arguments to control exactly what tests and how2.

Once the package has been added, you can browse to where Nuget is storing packages for your particular project. In the tools folder of the console runner package folder, you will find xunit.console.exe. If you run this with the -? argument, you will get some helpful information.

The three options I use the most are -trait, -notrait, and -parallel.

Traits

The two trait options control what tests you are running based on metadata attached to those tests. This is really useful if you have some tests that are resource heavy and only used in certain circumstances, such as stress testing. In MSTest, you could attach arbitrary metadata onto test methods using the TestProperty attribute. XUnit provides a similar feature using Trait. For example, [Trait("Category", "ManualOnly")] could be used to put a method into the ManualOnly category. You could then use the following line to execute tests that lack this trait.

If you wanted to only run tests with a specific trait, you do the same thing but with -trait instead. Both of these options are very useful when combined with -parallel.

Parallelism

XUnit can run tests in parallel, but tests within the same collection are never run in parallel with each other. By default, each test class is its own collection. Test classes can be combined into collections using the Collection attribute. Tests within the same collection will be executed randomly, but never in parallel.

The -parallel option provides four options: no parallelism, running tests from different assemblies in parallel, running tests from different collections in parallel, or running tests from different assemblies and different collections in parallel.

The difference between assembly parallelism, collection parallelism, and both together

Let's assume you have two assemblies, A and B. Assembly A has three collections; 1, 2, and 3. Assembly B has three collections; 4, 5, and 6.

No Parallelism
-parallel none

XUnit will run each collection in one of the assemblies, one at a time, then run each collection in the other assembly one at a time.

Collections 1, 2, 3, 4, 5, and 6 will never execute at the same time as each other.

Parallel Assemblies
-parallel assemblies

XUnit will run each collection in assembly A, one at a time, at the same time as running each collection in assembly B, one at a time.

Collections 1, 2, and 3 will not execute at the same time as each other; Collections 4, 5, and 6 will not execute at the same time as each other; but collections 1, 2, and 3 will execute in parallel with 4, 5, and 6, as the 1, 2, and 3 are in a different assembly to 4, 5, and 6.

Parallel Collections
-parallel collections

XUnit will run each collection within an assembly in parallel, but only one assembly at a time.

Collections 1, 2 and 3 will execute parallel; collections 4, 5, and 6 will execute in parallel; but, 1, 2, and 3 will not run at the same time as 4, 5, and 6.

Parallel Everything
-parallel all

XUnit will run each collection in parallel, regardless of its assembly.

Collections 1, 2, 3, 4, 5, and 6 will run in parallel, each potentially running at the same time as any other.

Beware running tests in parallel when first migrating from MSTest. It is a surefire way of finding some heinous test fixture dependencies and you risk thinks like deadlocking on resources. Usually, running assemblies in parallel is a lot safer than running collections in parallel, assuming that tests are collocated in assemblies based on their purpose and the resources they interact with.

In Conclusion…

That brings us to the end of the series. I have focused primarily on migrating from MSTest, leaving out a lot of the nuances to XUnit. I highly recommend continuing your education with the XUnit documentation and through experimentation; having personally migrated several projects, I know you won't regret it.

 


  1. I love this approach to augmenting the development environment. It requires no additional tooling setup to get your dev environment working. The source itself controls the tooling versions and installation. Wonderful 

  2. You can control how tests run under MSBuild too by using various properties. This is discussed more on the XUnit site 

Some of my favourite tools

Update: This post has been updated to recognise that CodeLineage is now maintained by Hippo Camp Software and not Red Gate Software as was originally stated.

If you know me, you might well suspect this post is about some of the idiots I know, but it is not, this is entirely about some of the tools I use in day-to-day development. This is by no means an exhaustive list, nor is it presented in any particular order. However, assuming you are even a little bit like me as a developer, you will see a whole bunch of things you already use, but hopefully there is at least one item that is new to you. If you do find something new and useful here, or you have some suggestions of your own, please feel free to post a comment.

OzCode

OzCode is an add-in for Visual Studio that provides some debugging super powers like collection searching, adding computed properties to objects, pinning properties so that you don't have to go hunting in the object tree, simpler tracepoint creation, and a bunch more. I first tried this during beta and was quickly sold on its value. Give the 30-day trial a chance and see if it works for you.

Resharper

This seems to be a staple for most C# developers. I was a late-comer to using this tool and I am not sure I like it for the same reasons as everyone else. I actually love Resharper for its test runner, which is a more performant alternative to Visual Studio's built-in Test Explorer, and the ability to quickly change file names to match the type they contain. However, it has a lot of features, so while this is not free, give the trial a chance and see if it fits.

Web Essentials

Another staple for many Visual Studio developers, Web Essentials provides lots of support for web-related development including enhanced support for JavaScript, CSS, CoffeeScript, LESS, SASS, MarkDown, and much more. If you do any kind of web development, this is essential1.

LinqPad

I was late to the LinqPad party, but gave it a shot during Ann Arbor Give Camp 2013 and within my first hour or two of using it, dropped some cash on the premium version (it is very inexpensive for what you get). Since then, whether it is hacking code or hacking databases, I have been using LinqPad as my standard tool for hacking.

For code, it does not have the overhead of creating projects and command line, WinForms or WPF wrapper tools that you would have to do in Visual Studio. For databases, LinqPad gives you the freedom to use SQL, C#, F# or VB for querying and manipulating your database as well as support for many different data sources beyound just SQL Server, providing an excellent alternative to SQL Management Studio.

LinqPad is free, but you get some cool features if you go premium, and considering the sub-$100 price, it is totally worth it.

JustDecompile

When Red Gate stopped providing Reflector for free, JetBrains and Telerik stepped up with their own free decompilers for poking around inside .NET code. These are often invaluable when tracking down obscure bugs or wanting to learn more about the code that is running when you did not write it. While JetBrains' dotPeek is useful, I have found that JustDecompile from Telerik has a better feature set (including showing MSIL, which I could not find in dotPeek).

Chutzpah

Chutzpah is a test runner for JavaScript unit tests and is available as a Nuget package. It supports tests written for Jasmine, Mocha, and QUnit, as well as a variety of languages including CoffeeScript and TypeScript. There are also two Visual Studio extensions to provide Test Explorer integration and a handy context menu. I find the context menu most useful out of these.

Chutzpah is a great option when you cannot leverage a NodeJS-based tool-chain like Grunt or Gulp, or some other non-Visual Studio build process.

CodeLineage

CodeLineage is a free Visual Studio extension from Hippo Camp Software2. Regardless of your source control provider, CodeLineage provides you with a simple interface for comparing different points in the history of a given file. The simple interface makes it easy to select which versions to compare. I do not use this tool often, but when I need it, it is fantastic.

FileNesting

This Visual Studio extension from the developer of Web Essentials makes nesting files under one another a breeze. You can set up automated nesting rules or perform nesting manually.

I like to keep types separated by file when developing in C#. Files are cheap and it helps discovery when navigating code. However, this sometimes means using partial classes to keep nested types separate, so to keep my solution explorer tidy, I edit the project files and nest source code files. I also find this useful for Angular directives, allowing me to apply the familiar pattern  of organizing code-behind under presentation by nesting JavaScript files under the template HTML.

Whether you have your own nesting guidelines or want to ensure generated code is nested under its corresponding definition (such as JavaScript generated from CoffeeScript), this extension is brilliant.

Switch Startup Project

Ever hit F5 to debug only to find out you tried to start a non-executable project and have to hunt for the right project in the Solution Explorer? This used to happen to me a lot, but not since this handy extension, which adds a drop down to the toolbar where I can select the project I want to be my startup project. A valuable time saver.

MultiEditing

Multi-line editing has been a valuable improvement in recent releases of Visual Studio, but it has a limitation in that you can only edit contiguous lines at the same column location. Sometimes, you want to edit multiple lines in a variety of locations and with this handy extension, you can. Just hold ALT and click the locations you want to multi-edit, then type away.

Productivity Power Tools

Productivity Power Tools for Visual Studio have been a staple extension since at least Visual Studio 2008. Often the test bed of features that eventually appear as first class citizens in the Visual Studio suite, Productivity Power Tools enhances the overall Visual Studio experience.

The current version for Visual Studio 2013 provides support for colour printing, custom document tabs, copying as HTML, error visualization in the Solution Explorer, time stamps in the debug output margin, double-click to maximize and dock windows, and much more. This is a must-have for any Visual Studio user.


  1. yes, I went there 

  2. though it was maintained by Red Gate when I first started using it