Tag: xunit2

Running XUnit Tests, Using Traits, and Leveraging Parallelism

We have arrived at the end of this little series on migrating unit tests from MSTest to XUnit (specifically, XUnit 2). While earlier posts concentrated on writing the tests and the XUnit counterparts to MSTest concepts, this post will briefly look at some non-code aspects to XUnit; most importantly, we will look at getting the tests to run.

Do not depend on test order
One thing to watch out for after migrating your tests is the order in which tests are run. MSTest allowed us to abuse testing by assuming that tests would run in a specific order. XUnit does away with this and will run tests in a random order. This can help to find some obscure bugs, but it can also make migration a little tougher, especially when tests share a data store or some other test fixture. Watch out for that.

Visual Studio

Running tests inside Visual Studio is really simple. Following in the footsteps of web development trends, rather than requiring an extension to the development environment, XUnit uses package management1. All you need to do is add the Visual Studio XUnit test runner package to your project and Visual Studio will be able to detect and run your XUnit tests just like your old MSTests.

Of course, if you're like me and absolutely loathe the built-in Visual Studio test explorer, you can use Resharper (or dotCover), which has built-in support for XUnit.

Command Line

More often than not, our continuous integration setups are scripted and we're unlikely to be running our unit tests via the development tool, such as Visual Studio. For situations like this, you can add the XUnit command line test runner package to your project. This provides a command line utility for running your tests with arguments to control exactly what tests and how2.

Once the package has been added, you can browse to where Nuget is storing packages for your particular project. In the tools folder of the console runner package folder, you will find xunit.console.exe. If you run this with the -? argument, you will get some helpful information.

xunit.console.exe -?

The three options I use the most are -trait, -notrait, and -parallel.

Traits

The two trait options control what tests you are running based on metadata attached to those tests. This is really useful if you have some tests that are resource heavy and only used in certain circumstances, such as stress testing. In MSTest, you could attach arbitrary metadata onto test methods using the TestProperty attribute. XUnit provides a similar feature using Trait. For example, [Trait("Category", "ManualOnly")] could be used to put a method into the ManualOnly category. You could then use the following line to execute tests that lack this trait.

xunit.console.exe -notrait "Category=ManualOnly"

If you wanted to only run tests with a specific trait, you do the same thing but with -trait instead. Both of these options are very useful when combined with -parallel.

Parallelism

XUnit can run tests in parallel, but tests within the same collection are never run in parallel with each other. By default, each test class is its own collection. Test classes can be combined into collections using the Collection attribute. Tests within the same collection will be executed randomly, but never in parallel.

The -parallel option provides four options: no parallelism, running tests from different assemblies in parallel, running tests from different collections in parallel, or running tests from different assemblies and different collections in parallel.

The difference between assembly parallelism, collection parallelism, and both together

Let's assume you have two assemblies, A and B. Assembly A has three collections; 1, 2, and 3. Assembly B has three collections; 4, 5, and 6.

No Parallelism
-parallel none

XUnit will run each collection in one of the assemblies, one at a time, then run each collection in the other assembly one at a time.

Collections 1, 2, 3, 4, 5, and 6 will never execute at the same time as each other.

Parallel Assemblies
-parallel assemblies

XUnit will run each collection in assembly A, one at a time, at the same time as running each collection in assembly B, one at a time.

Collections 1, 2, and 3 will not execute at the same time as each other; Collections 4, 5, and 6 will not execute at the same time as each other; but collections 1, 2, and 3 will execute in parallel with 4, 5, and 6, as the 1, 2, and 3 are in a different assembly to 4, 5, and 6.

Parallel Collections
-parallel collections

XUnit will run each collection within an assembly in parallel, but only one assembly at a time.

Collections 1, 2 and 3 will execute parallel; collections 4, 5, and 6 will execute in parallel; but, 1, 2, and 3 will not run at the same time as 4, 5, and 6.

Parallel Everything
-parallel all

XUnit will run each collection in parallel, regardless of its assembly.

Collections 1, 2, 3, 4, 5, and 6 will run in parallel, each potentially running at the same time as any other.

Beware running tests in parallel when first migrating from MSTest. It is a surefire way of finding some heinous test fixture dependencies and you risk thinks like deadlocking on resources. Usually, running assemblies in parallel is a lot safer than running collections in parallel, assuming that tests are collocated in assemblies based on their purpose and the resources they interact with.

In Conclusion…

That brings us to the end of the series. I have focused primarily on migrating from MSTest, leaving out a lot of the nuances to XUnit. I highly recommend continuing your education with the XUnit documentation and through experimentation; having personally migrated several projects, I know you won't regret it.

 

  1. I love this approach to augmenting the development environment. It requires no additional tooling setup to get your dev environment working. The source itself controls the tooling versions and installation. Wonderful [↩]
  2. You can control how tests run under MSBuild too by using various properties. This is discussed more on the XUnit site [↩]

AssemblyInitialize, AssemblyCleanup and Sharing State Between Test Classes in XUnit

We have covered quite a bit in this series on migrating from MSTest to XUnit and we have not even got to the coolest bit yet; data-driven theories. If that is what you are waiting for, you will have to wait a little longer. Before we get there, I want to cover one last piece of test initialization as provided in MSTest by the AssemblyInitialize and AssemblyCleanup attributes.

As we saw in previous posts, we can use the test class constructor and Dispose() for TestInitialize and TestCleanup, and IClassFixture and fixture classes for ClassInitialize and ClassCleanup.  For the assembly equivalents, we use collections and the ICollectionFixture interface.

A collection is defined by a set of test classes and a collection definition. A test class is designated as being part of a specific collection by decorating the class with the Collection attribute, which provides the collection name. A corresponding class decorated with the CollectionDefinition attribute should also exist as this is where any collection fixtures are defined. All classes that share the same collection name will share the collection fixtures from which the definition class derives.

public class MyFirstFixture
{
    public MyFirstFixture()
    {
        // Some initialization
    }
}

public class MySecondFixture : IDisposable
{
    public MySecondFixture()
    {
        // Some initialization
    }

    public void Dispose()
    {
        // Some cleanup
    }
}

[CollectionDefinition("MyCollection")]
public class MyCollectionDefinition : ICollectionFixture<MyFirstFixture>, ICollectionFixture<MySecondFixture>
{
    //Nothing needed here
}

[Collection("MyCollection")]
public class TestClass1
{
    [Fact]
    public void TestMethod1()
    {
        // Do some testing
    }
}

[Collection("MyCollection")]
public class TestClass2
{
    [Fact]
    public void TestMethod1()
    {
        // Do some more testing
    }
}

The example code above shows a collection definition with two fixtures and two test classes defined as part of that collection. Note how the fixtures control initialization and cleanup using constructors and IDisposable 1 . We can modify those classes to reference the collection fixtures just as we did with class-level fixtures; by referencing the fixture in the constructor arguments as shown here.

[Collection("MyCollection")]
public class TestClass1
{
    public TestClass1(MySecondFixture fixture)
    {
        // Do something with the collection fixture
    }

    [Fact]
    public void TestMethod1()
    {
        // Do some testing
    }
}

I really like this approach over the attributed static methods of MSTest. This seems to more easily support code reuse and makes intentions much clearer, separating the concerns of tests (defined in the test class) from fixtures (defined by the fixture types). The downside is that fixture types do not get access to the ITestOutputHelper interface so if you want your fixtures to output diagnostic information, you should consider a logging library like Common.Logging. Also, your fixture types must be in the same assembly as your tests. Of course, that doesn't preclude the fixtures from using types outside of the assembly, so you can always put shared implementation between test assemblies in some other class library.

And that brings our migration of shared initialization to a close. You can find more information on sharing context across tests on the xunit site. Next up, we will look at data-driven tests. Thank you for your time. If you have any questions, please leave a comment.

  1. A fixture type can used with IClassFixture<T> or ICollectionFixture<T> [↩]

ClassInitialize, ClassCleanup, and Sharing Data Across Tests in XUnit2

So far in this series on migrating from MSTest to XUnit, we have looked at:

In this post, we will look at how we can share setup and cleanup code across tests in a test class in XUnit. MSTest allows you to define shared setup and cleanup code for an entire test class by using methods decorated with the ClassInitialize and ClassCleanup attributes. Unlike their counterparts, TestInitialize and TestCleanup, methods decorated with these class-level attributes are executed just once per class, rather than once per test in the class. Using these class-level attributes, we can execute code, generate fixture objects, and load test data that can be used across all tests in the class without having the overhead of repeating this for every test in the class. This is useful when that initialization or cleanup is expensive, such as creating a database connection, or loading several data files.

As we have seen so far, XUnit is light on decorating non-test methods with attributes, instead relying on language syntax that mirrors the purpose of the code. In the case of TestInitialize and TestCleanup, XUnit uses the test class constructor and IDisposable. It should come as no surprise that this pattern is also used when it comes to class-level initialization and cleanup.

IClassFixture<T>

There are two parts to shared initialization and cleanup in XUnit: declaring what shared items a test class uses, and referencing them within test methods.

To declare specific setup is required, a test class must be derived from IClassFixture for each shared setup/cleanup. The T in IClassFixture is the actual type responsible for the initialization and cleanup via its constructor and IDisposable implementation.

public class MyFixture : IDisposable
{
    public MyFixture()
    {
        // Setup here
    }

    public void Dispose()
    {
        // Cleanup here
    }
}
public class MyTests : IClassFixture<MyFixture>
{
    [Fact]
    public void Test()
    {
        // Do something knowing that MyFixture was instantiated
    }
}

The XUnit test runner sees that your test class is deriving from IClassFixture and ensures that an instance of MyFixture is created before your tests are run and disposed of when all the tests are completed. I really like this approach over the MSTest equivalent, as it moves the setup and initialization from being about the test class to being about the test fixture, the thing being setup. You can even have more than one fixture, so if you use two databases in your tests, you can have one fixture for each database and explicitly specify the use of each. It also means that you can set things that are supposed to be immutable for the duration of tests to be readonly and enforce that immutability. This is even clearer when referencing fixtures in tests.

public class MyTests : IClassFixture<MyFixture>
{
    private readonly MyFixture _fixture;

    public MyTests(MyFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public void Test()
    {
        // Do something with _fixture
    }
}

As shown in the preceding example, to reference a test fixture in your test class methods, you just need to add a corresponding argument to the constructor and XUnit will inject the fixture. You can then use the fixture, and assign it or something obtained from it to a member variable of your class. Not only that, but you can mark that member as readonly and be explicit about what tests can and cannot do to your test state. Personally, this approach to shared initialization and cleanup feels much more intuitive. I can easily reuse my initialization and setup code without cluttering my test classes unnecessarily, and I can be explicit about the immutability of any shared state or setup.

And that is it; now you not only know how to share repeatable setup across tests (as provided by TestInitialize and TestCleanup in MSTest), but also how to do the same for setup across the whole test class (as MSTest does with ClassIntialize and ClassSetup).

But, what of AssemblyInitialize and AssemblyCleanup? Well, that's probably a good place to start in the next post. As always, you are welcome to leave a comment letting me know how you are liking this series on migrating to XUnit, or perhaps bringing up something that you'd like me to cover.

TestContext Equivalence in XUnit2, or How Do I Write Output in XUnit2?

So far, in discussing the migration of MSTest to XUnit2, I have only touched on the basics. First, I covered using the XUnitConverter to make the transition a little easier, then I provided an overview of how the basic MSTest concepts tests, test initialization, and test cleanup were supported. In this post I want to look at the confusing dumping group that is TestContext. My goal with these posts is to provide useful search results for those looking for information on XUnit, especially when migrating from MSTest. For a moment though, let me rant about TestContext.

I am sure that there are people out there who totally get all the values and functions TestContext provides; however, other than writing log messages, I have found the TestContext properties wildly confusing and unhelpful. I somehow doubt I am alone. Not only that, but the inconsistent ways a method gets access to the TestContext dependent on whether it is an instance method or a static method makes things confusing1.

TestContext Properties

TestContext Properties

TestContext Methods
TestContext Methods

Looking at the documentation from MSDN (shown above), it should be clear to anyone that TestContext is trying to do a lot of different things. There are methods in there for outputting things to the test trace messages, adding results files, and running timers, and properties for getting all sorts of information from directories for results, to the name of the test, to things related to data driven tests. Overall, it is convoluted and confused; do you know the difference between ResultsDirectory, TestResultsDirectory, and TestRunResultsDirectory? Even after working with MSTest for nearly 10 years, I could not begin to tell you. In my opinion, TestContext does too much and is almost always unintuitive to use, so how does XUnit tackle these things?

Well, first of all, XUnit does not even begin to replace the timer stuff because, well, there are plenty of timer options available in the .NET framework, so just use one of them.

Second, XUnit manipulates the current directory when running tests, so the location retrieved from Environment.CurrentDirectory will be where the tests are running and where test data will live. We will focus on data-driven testing in another post, but suffice to say that you do not need to decipher pages of documentation or debug tests to understand where to find your test data or where to output your test results (if you have something to output, of course).

Finally, if you want to write to the test output, XUnit provides an interface that you can inject into your constructor for doing just that and only that; ITestOutputHelper.

ITestOutputHelper

Since the XUnit documentation is very helpful on how to use this interface, I won't go into great detail here. However, for those looking to replace TestContext.WriteLine, this is what you need to do.

using Xunit;
using Xunit.Abstractions;

public class MyTests
{
    private readonly ITestOutputHelper _output;

    public MyTests(ITestOutputHelper output)
    {
        _output = output;
    }

    [Fact]
    public void Test1()
    {
        var stringValue = "a thing to output";
        _output.WriteLine($"My string value is \"{stringValue}\"");
    }
}

The test runner observes from the signature of the class constructor that it requires the ITestOutputHelper interface and injects it, making it available throughout the test execution, including during the Dispose method, if present.

Next time, we will take a look at how XUnit tackles sharing initialization across multiple tests. In the meantime, if you are finding these posts useful or have any questions, please leave a comment.

  1. a property called TestContext with public getter and setter for instance methods; or passed as a parameter for static methods [↩]

TestMethod, TestInitialize, and TestCleanup in XUnit2

In the last post, I briefly described how to automatically migrate your MSTest tests to XUnit by using the XUnitConverter utility. Of course, nothing is ever that simple; MSTest has some concepts that XUnit expresses very differently1 like how to share code between tests whether that is setup, fixtures, cleanup, or data. Some of these concepts are implemented differently enough that automating the migration from one to the other would be very difficult if not impossible. However, some of it really is that simple. Before we look at the difficult examples, I thought it would be useful to illustrate how some of the simple concepts map from MSTest to XUnit using an example2.

So, let's look at an MSTest example (contrived, of course):

[TestClass]
public class MyTests
{
    private MemoryStream _stream;

    [TestInitialize]
    public void Initialize()
    {
        _stream = new MemoryStream();
    }

    [TestCleanup]
    public void Cleanup()
    {
        _stream.Dispose();
    }

    [TestMethod]
    public void Test1()
    {
        //Arrange
        //Act
        //Assert
    }

    [TestMethod]
    public void Test2()
    {
        //Arrange
        //Act
        //Assert
    }
}

Clearly, I cheated by not actually making the tests do anything, but the content of the test methods is mostly irrelevant; you set some stuff up, you do something, and you assert a result–it's all the same regardless of the test framework. However, this is a simple example of a test class written for the MSTest framework. There are attributes to tell the framework that the class is a test class, which methods inside of it are test methods, and which methods should be called before and after each test. In this case, our test initialization creates a stream, which is then disposed of in the cleanup method; each test method would get sandwiched in the middle.

After converting to XUnit with the converter tool, the same class will look something like this:

public class MyTests : IDisposable
{
    private MemoryStream _stream;

    public MyTests()
    {
        _stream = new MemoryStream();
    }

    public void Dispose()
    {
        _stream.Dispose();
    }

    [Fact]
    public void Test1()
    {
        //Arrange
        //Act
        //Assert
    }

    [Fact]
    public void Test2()
    {
        //Arrange
        //Act
        //Assert
    }
}

There are a few things that happened.

  1. The class no longer has an attribute. XUnit knows the class is a test class because it contains tests3.
  2. The tests are decorated with a [Fact] attribute, which is equivalent to [TestMethod].
  3. The [TestInitialize] and [TestCleanup] attributes are gone. Instead, the class constructor is used for test initialization and the Dispose method along with deriving from IDisposable indicates that there is test cleanup code.

Overall, I love how the XUnit syntax works with C# syntax and .NET idioms in declaring tests. Not only does this reduce the ceremony around defining tests by reducing the various decorators, but it also allows for cleaner coding practices. For example, we can now correctly mark our memory stream member variable as readonly.

private MemoryStream _stream;

By relying on C# syntax and standard interfaces in this way, the lifecycle of a test is clearer too; XUnit will construct and dispose the class for each test in the class, making it easy to see how each test will run. This idiomatic way of declaring tests allows for separation of concerns, keeping test classes light and focused. This will be illustrated when we later look at other concepts in MSTest like [ClassInitialize] and [ClassCleanup], TestContext, and [DeploymentItem], and how XUnit tackles the problems these concepts solved.

  1. and for good reasons, IMHO [↩]
  2. XUnit documentation has a handy table but I don't think it's as illustrative as it could be [↩]
  3. why MSTest did not make assumptions like this, I do not know [↩]