Running XUnit Tests, Using Traits, and Leveraging Parallelism

We have arrived at the end of this little series on migrating unit tests from MSTest to XUnit (specifically, XUnit 2). While earlier posts concentrated on writing the tests and the XUnit counterparts to MSTest concepts, this post will briefly look at some non-code aspects to XUnit; most importantly, we will look at getting the tests to run.

Do not depend on test order
One thing to watch out for after migrating your tests is the order in which tests are run. MSTest allowed us to abuse testing by assuming that tests would run in a specific order. XUnit does away with this and will run tests in a random order. This can help to find some obscure bugs, but it can also make migration a little tougher, especially when tests share a data store or some other test fixture. Watch out for that.

Visual Studio

Running tests inside Visual Studio is really simple. Following in the footsteps of web development trends, rather than requiring an extension to the development environment, XUnit uses package management1. All you need to do is add the Visual Studio XUnit test runner package to your project and Visual Studio will be able to detect and run your XUnit tests just like your old MSTests.

Of course, if you're like me and absolutely loathe the built-in Visual Studio test explorer, you can use Resharper (or dotCover), which has built-in support for XUnit.

Command Line

More often than not, our continuous integration setups are scripted and we're unlikely to be running our unit tests via the development tool, such as Visual Studio. For situations like this, you can add the XUnit command line test runner package to your project. This provides a command line utility for running your tests with arguments to control exactly what tests and how2.

Once the package has been added, you can browse to where Nuget is storing packages for your particular project. In the tools folder of the console runner package folder, you will find xunit.console.exe. If you run this with the -? argument, you will get some helpful information.

The three options I use the most are -trait, -notrait, and -parallel.

Traits

The two trait options control what tests you are running based on metadata attached to those tests. This is really useful if you have some tests that are resource heavy and only used in certain circumstances, such as stress testing. In MSTest, you could attach arbitrary metadata onto test methods using the TestProperty attribute. XUnit provides a similar feature using Trait. For example, [Trait("Category", "ManualOnly")] could be used to put a method into the ManualOnly category. You could then use the following line to execute tests that lack this trait.

If you wanted to only run tests with a specific trait, you do the same thing but with -trait instead. Both of these options are very useful when combined with -parallel.

Parallelism

XUnit can run tests in parallel, but tests within the same collection are never run in parallel with each other. By default, each test class is its own collection. Test classes can be combined into collections using the Collection attribute. Tests within the same collection will be executed randomly, but never in parallel.

The -parallel option provides four options: no parallelism, running tests from different assemblies in parallel, running tests from different collections in parallel, or running tests from different assemblies and different collections in parallel.

The difference between assembly parallelism, collection parallelism, and both together

Let's assume you have two assemblies, A and B. Assembly A has three collections; 1, 2, and 3. Assembly B has three collections; 4, 5, and 6.

No Parallelism
-parallel none

XUnit will run each collection in one of the assemblies, one at a time, then run each collection in the other assembly one at a time.

Collections 1, 2, 3, 4, 5, and 6 will never execute at the same time as each other.

Parallel Assemblies
-parallel assemblies

XUnit will run each collection in assembly A, one at a time, at the same time as running each collection in assembly B, one at a time.

Collections 1, 2, and 3 will not execute at the same time as each other; Collections 4, 5, and 6 will not execute at the same time as each other; but collections 1, 2, and 3 will execute in parallel with 4, 5, and 6, as the 1, 2, and 3 are in a different assembly to 4, 5, and 6.

Parallel Collections
-parallel collections

XUnit will run each collection within an assembly in parallel, but only one assembly at a time.

Collections 1, 2 and 3 will execute parallel; collections 4, 5, and 6 will execute in parallel; but, 1, 2, and 3 will not run at the same time as 4, 5, and 6.

Parallel Everything
-parallel all

XUnit will run each collection in parallel, regardless of its assembly.

Collections 1, 2, 3, 4, 5, and 6 will run in parallel, each potentially running at the same time as any other.

Beware running tests in parallel when first migrating from MSTest. It is a surefire way of finding some heinous test fixture dependencies and you risk thinks like deadlocking on resources. Usually, running assemblies in parallel is a lot safer than running collections in parallel, assuming that tests are collocated in assemblies based on their purpose and the resources they interact with.

In Conclusion…

That brings us to the end of the series. I have focused primarily on migrating from MSTest, leaving out a lot of the nuances to XUnit. I highly recommend continuing your education with the XUnit documentation and through experimentation; having personally migrated several projects, I know you won't regret it.

 


  1. I love this approach to augmenting the development environment. It requires no additional tooling setup to get your dev environment working. The source itself controls the tooling versions and installation. Wonderful 

  2. You can control how tests run under MSBuild too by using various properties. This is discussed more on the XUnit site 

LINQ: Understanding Your Query Chain

This is part of a short series on the basics of LINQ:

This is the third part in my small series on LINQ and it covers what I feel is the most important thing to understand when using LINQ, query chains. We are going to build on the deferred execution concepts discussed in the last entry and look at why it is important to know your query operations.

Each method in a LINQ query is either immediately executed or deferred. When deferred, a method is either lazily evaluated one element at a time or eagerly evaluated as the entire collection. Usually, you can determine which is which from the documentation or, if that fails, a little experimentation. Why does it matter? This question from StackOverflow provides us with an example:

For those that did not read it or do not understand the problem, let me summarize. The original poster had a problem where values they had obtained from a LINQ query result, when passed into the Except() method on that same query, did not actually exclude anything. It was as if they had taken the sequence 1,2,3,4, called Exclude(2), and that had returned 1,2,3,4 instead of the expected 1,3,4. On the surface, the code looked like it should work, so what was going on? To explain, we need a little more detail.

The example code has a class that described a user. An XML file contained user details and this is loaded into a sequence of User instances using LINQ-to-XML.

As noted in the commentary, the poster understood that at this point, the query is not yet evaluated. With their query ready to be iterated, they use it to determine which users should be excluded using a different query.

And then using those results, the originally loaded list of users is filtered.

Now, it is clear this code is not perfect and we could rewrite it to function without so many LINQ queries (I will give an example of that later in this post), but we do not care about the elegance of the solution; we are using this code as an example of why it is important to understand what a LINQ query is doing.

As noted in the commentary, when the line declaring the initial users query is executed, the query it defines has not. The query does not actually become a real list of users until it gets consumed1. Where does that happen? Go on, guess.

If you guessed GetMatchingUsers, you are wrong. All that method does is build an additional level of querying off the initial query and return that new query. If you guessed the Except() method, that's wrong too, because Except() is also deferred. In fact, the example only implies that something eventually looks at the results of Except() and as such, the query is never evaluated. So, for us to continue, let's assume that after the excludes variable (containing yet another unexecuted query), we have some code like this to consume the results of the query:

By iterating over excludes,  the query is executed and gives us some results. Now that we are looking at the query results, what happens?

First, the Except() method takes the very first element from the users query, which in turn, takes the very first User element from the XML document and turns it into a User instance. This instance is then cast to IUser using OfType2.

Next, the Except() method takes each of the elements in the matches query result and compares it to the item just retrieved from the users collection. This means the entire matches query is turned into a concrete list. This causes the users query to be reevaluated to extract the matched users. The instances of User created from the matches query are compared with each instance from the users query and the ones that do not match are returned for us to output to the console.

It seems like it should work, but it does not, and the key to why is in how queries and, more importantly, deferred execution work.

Each evaluation of a deferred query is unique. It is very important to remember this when using LINQ. If there is one thing to take away from reading my blog today, it is this. In fact, it's so important, I'll repeat it:

Each evaluation of a deferred query is unique.

It is important because it means that each evaluation of a deferred query (in most cases) results in an entirely new sequence with entirely new items. Consider the following iterator method:

It returns an enumerable that upon being consumed will produce a single Object instance. If we had a variable of the result of GetObject(), such as var obj  = GetObject() and then iterated obj several times, each time would give us a different Object instance. They would not match because on each iteration, the deferred execution is reevaluated.

If we go back to the question from StackOverflow armed with this knowledge, we can identify that users is evaluated twice by the Except() call. One time to get the list of exceptions out of the matches query and another to process the list that is being filtered. It is the equivalent of this:

From this code, we would never expect objects to contain nothing since the two calls to the immediately executed GetObjects would return collections of completely different instances. When execution is deferred, we get the same effect; each evaluation of a query is as if it were a separate method call.

To fix this problem, we need to make sure our query is executed once to make the results "concrete", then use those concrete results to do the rest of the work. This is not only important to ensure that the objects being manipulated are the same in all uses of the queried data, but also to ensure that we don't do work more than once3. To make the query concrete, we call an immediately executed method such as ToList(), evaluating the query and capturing its results in a collection.

This is our solution, as the original poster of our StackOverflow question indicated. If we change the original users query to be evaluated and stored, everything works as it should. With the help of some investigation and knowledge of how LINQ works, we now also know why.

Now that we understand a little more about LINQ we can consider how we might rewrite the original poster's example code. For example, we really should not to iterate the users list twice at all; we should see the failure of Except() as a code smell that we are iterating the collection too often. Though making it concrete with ToList() fixes the bug, it does not fix this inefficiency.

To do that, we can rewrite it to something like this:

This update only iterates over each user once, resulting in a collection that excludes the users we don't want4.

In conclusion…

My intention here was to show why it is fundamental to know which methods are immediately executed, which ones are deferred, and whether those deferred methods are lazily or eagerly evaluated. At the end of this post are some examples of each kind of LINQ method, but a good rule of thumb is that if the method returns a type other than IEnumerable or IQueryable (e.g. int or List), it is immediately executed; all other cases are probably using deferred execution. If a method does use deferred execution, it is also helpful to know which ones iterate the entire collection every time and which ones stop iterating when they have their answer, but for this you will need to consult documentation and possibly experiment with some code.

Just knowing these different types of methods can be a part of your query will often be enough to help you write better LINQ and debug queries faster.  By knowing your LINQ methods, you can improve performance and reduce memory overhead, especially when working with large data sets and slow network resources. Without this knowledge, you are likely to evaluate queries and iterate sequences too often, and instantiate objects too many times.

Hopefully you were able to follow this post and it has helped you get a better grasp on LINQ. In the final post of this series, I will ease up on the deep code analysis, and look at query syntax versus dot notation (aka fluent notation). In the meantime, if you have any comments, I'd love to read them.

Examples of LINQ Method Varieties

Immediate Execution

Count(), Last(), ToList(), ToDictionary(), Max(), Aggregate()
Immediate iterate the entire collection every time

Any(), All(), First(), Single()
Iterate the collection until a condition is or is not met

Deferred Execution, Eager Evaluation

Distinct(), OrderBy(), GroupBy()
Iterate the entire collection but only when the query is evaluated

Deferred Execution, Lazy Evaluation

Select(),Where()
Iterate the entire collection

Take(),Skip()
Iterate until the specified count is reached


  1. "consumed" is often used as an alternative to "iterated" 

  2. Cast() should have been used here since all the objects loaded are of the same type 

  3. This is something that becomes very important when working with large queries that can be time or resource consuming to run 

  4. with more effort, I am certain there are more things that could be done to improve this code, but we're not here for that, so I'll leave is as an exercise for you; I'm generous like that 

WeakReferenceDictionary

Further to the last post on my implementation of a strongly-typed variant of WeakReferemce, I thought it would be apropos to post another utility class surrounding weak references. This one really is more academic than useful considering that one could just stick WeakReference<T> into a regular Dictionary<TKey, TValue> and get almost the same functionality. All this class really does is hide away how we're managing the reference inside the dictionary.

So there you have it, WeakReferenceDictionary<TKey, TValue>. It's not a complete replacement for a regular dictionary as it doesn't support all the same interfaces such as the ICollection or ISerialization. Also, I haven't fully commented everything as well as perhaps it should be. I'll leave all that as an exercise for you.