DataSource and Data-driven Testing Using XUnit

If you are anything like me, you avoided data-driven tests in MSTest because they were such a pain to write and maintain. However, I know not everyone is like me and I also know that even though we try to avoid things, we do not always succeed. So, in this entry in the series on migrating from MSTest to XUnit, we will look at migrating your data-driven tests, but before we get into the details, let's briefly recap on what MSTest provides for data-driving tests; the DataSource attribute.

The DataSource attribute specifies a data source from which data points are loaded. The test can then reference the specific data row in the data source from the TestContext's DataRow property. You may recall that we touched on the jack-of-all-trades, master-of-none TestContext back in the entry on outputting from our tests. As MSDN explains, given a table of data rows like this:

FirstNumberSecondNumberSum
011
112
2-3-1

DataSource is used like this:

I do not recall using this attribute in earnest, and though perhaps others think of it more fondly, I found it a frustration to use. Thankfully, XUnit is much more inline with my intuition1.

Data-driven test methods in XUnit are called theories and are adorned with the Theory attribute2. The Theory attribute is always accompanied by at least one data attribute which tells the test runner where to find data for the theory. There are three built-in attributes for providing data: InlineData, MemberData, and ClassData. Third-party options are also available (check out AutoFixture and its AutoData attribute) and you can create your own if you find it necessary.

InlineData provides a simple way to describe a single test point for your theory; MemberData takes the name of a static member (method, field, or property) and any arguments it might need to generate your data; and ClassData takes a type that can be instantiated to provide the data. A single test point is provided as an array of type object, and while XUnit provides the TheoryData types to allow strongly-typed declaration of test data points, fundamentally, every data source  is IEnumerable<object[]>. Finally, rather than the obscure TestContext.DataRow property, data points are provided to a theory test method via the test method's arguments.

So, given all this information, the above example from MSDN could be expressed as follows:

I took the liberty of using the arrange/act/assert test layout as part of this rewrite as I think it enhances test readability. However, you can see from this example how much easier data-driven testing is under XUnit when compared with traditional MSTest3.

Of course, I totally skipped the fact that the MSTest example used a database for the test data source. That was deliberate to simplify the example, but if we still wanted to use a database to obtain our data points (or some other data source), we can leverage the MemberData or ClassData attributes, or even roll our own.

That brings us to the end of this post on migrating data-driven tests from MSTest to XUnit. This also brings us almost to the end of the whole series on migrating from MSTest to XUnit; if you think I missed something important, please leave a comment. Next time, we will finish up with a look at some of the bits around running XUnit tests such as parallel execution, test runners, and the like.


  1. in case you hadn't noticed, I like XUnit 

  2. instead of [Fact] as on non-data-driven tests 

  3. This approach makes so much sense that MSTest introduced it for phone and WinRT app tests and is bringing it to everyone with MSTest version 2 

TestContext Equivalence in XUnit2, or How Do I Write Output in XUnit2?

So far, in discussing the migration of MSTest to XUnit2, I have only touched on the basics. First, I covered using the XUnitConverter to make the transition a little easier, then I provided an overview of how the basic MSTest concepts tests, test initialization, and test cleanup were supported. In this post I want to look at the confusing dumping group that is TestContext. My goal with these posts is to provide useful search results for those looking for information on XUnit, especially when migrating from MSTest. For a moment though, let me rant about TestContext.

I am sure that there are people out there who totally get all the values and functions TestContext provides; however, other than writing log messages, I have found the TestContext properties wildly confusing and unhelpful. I somehow doubt I am alone. Not only that, but the inconsistent ways a method gets access to the TestContext dependent on whether it is an instance method or a static method makes things confusing1.

TestContext Properties

TestContext Properties

TestContext Methods
TestContext Methods

Looking at the documentation from MSDN (shown above), it should be clear to anyone that TestContext is trying to do a lot of different things. There are methods in there for outputting things to the test trace messages, adding results files, and running timers, and properties for getting all sorts of information from directories for results, to the name of the test, to things related to data driven tests. Overall, it is convoluted and confused; do you know the difference between ResultsDirectory, TestResultsDirectory, and TestRunResultsDirectory? Even after working with MSTest for nearly 10 years, I could not begin to tell you. In my opinion, TestContext does too much and is almost always unintuitive to use, so how does XUnit tackle these things?

Well, first of all, XUnit does not even begin to replace the timer stuff because, well, there are plenty of timer options available in the .NET framework, so just use one of them.

Second, XUnit manipulates the current directory when running tests, so the location retrieved from Environment.CurrentDirectory will be where the tests are running and where test data will live. We will focus on data-driven testing in another post, but suffice to say that you do not need to decipher pages of documentation or debug tests to understand where to find your test data or where to output your test results (if you have something to output, of course).

Finally, if you want to write to the test output, XUnit provides an interface that you can inject into your constructor for doing just that and only that; ITestOutputHelper.

ITestOutputHelper

Since the XUnit documentation is very helpful on how to use this interface, I won't go into great detail here. However, for those looking to replace TestContext.WriteLine, this is what you need to do.

The test runner observes from the signature of the class constructor that it requires the ITestOutputHelper interface and injects it, making it available throughout the test execution, including during the Dispose method, if present.

Next time, we will take a look at how XUnit tackles sharing initialization across multiple tests. In the meantime, if you are finding these posts useful or have any questions, please leave a comment.


  1. a property called TestContext with public getter and setter for instance methods; or passed as a parameter for static methods 

Migrating from MSTest to XUnit 2

We recently migrated most of our testing from the MSTest framework1 to XUnit 2 (from here on in, I will be referring to this as just XUnit). This was not a change taken lightly since it touched a lot of files, but we were motivated by a number of XUnit features, including reduced need to attribute test classes, easier data-driven tests, and parallel test execution.

Sadly, if you try this you may discover as we did that the XUnit documentation is equal parts super helpful and woefully lacking, depending on what you are trying to do. After hearing yet another colleague lament how hard it was to find information on some feature or other of XUnit, I thought it might be a good idea to document some of the things I have learned and hopefully, introduce yet another helpful XUnit resource to the Internet2.

For those not familiar with XUnit, the basics are pretty easy. In fact the existing XUnit documentation includes a handy table mapping concepts in other test frameworks to their XUnit equivalents. You can check that table out for details, but basically, through the use of attributes, constructors, IDisposable, and other interfaces, XUnit uses what I would describe as a more natural approach than other frameworks to concepts like tests, test initialization and cleanup, and test fixtures. Of course, this means that migrating from one framework to XUnit involves a bunch of file editing, but fear not for there is help.

XUnitConverter

The bulk of the migration was made a lot easier by using the XUnitConverter, a tool available in the dotnet/codeformatter GitHub repository. Although it does not take care of everything (beware if you have multiple test classes per file) and, depending on your preferred code format, can mess your formatting up a bit, but it does make the migration a lot easier.

The XUnitConverter runs against a csproj file. You can use PowerShell to recurse your solution and process all your projects like this:

Once the converter has done its thing, it is easy to identify further changes by using the compiler (things don't like to build if something did not work right). Although most things get converted with ease— [TestMethod] becomes [Fact], [TestInitialize]  becomes a constructor, complex tests will need a little more assistance to fully migrate. For example, XUnit uses interfaces and fixture classes to replace the kind of shared initialization and cleanup that MSTest provides via the [ClassInitialize] and [ClassCleanup]. We will start tackling these issues next time.


  1. Version 1 of MSTest, not the new and improved MSTest version 2 

  2. I would like to document my anecdotal information before I even consider tackling something a little more structured like contributing to the official documentation