Running XUnit Tests, Using Traits, and Leveraging Parallelism

We have arrived at the end of this little series on migrating unit tests from MSTest to XUnit (specifically, XUnit 2). While earlier posts concentrated on writing the tests and the XUnit counterparts to MSTest concepts, this post will briefly look at some non-code aspects to XUnit; most importantly, we will look at getting the tests to run.

Do not depend on test order
One thing to watch out for after migrating your tests is the order in which tests are run. MSTest allowed us to abuse testing by assuming that tests would run in a specific order. XUnit does away with this and will run tests in a random order. This can help to find some obscure bugs, but it can also make migration a little tougher, especially when tests share a data store or some other test fixture. Watch out for that.

Visual Studio

Running tests inside Visual Studio is really simple. Following in the footsteps of web development trends, rather than requiring an extension to the development environment, XUnit uses package management¹. All you need to do is add the Visual Studio XUnit test runner package to your project and Visual Studio will be able to detect and run your XUnit tests just like your old MSTests.

Of course, if you're like me and absolutely loathe the built-in Visual Studio test explorer, you can use Resharper (or dotCover), which has built-in support for XUnit.

Command Line

More often than not, our continuous integration setups are scripted and we're unlikely to be running our unit tests via the development tool, such as Visual Studio. For situations like this, you can add the XUnit command line test runner package to your project. This provides a command line utility for running your tests with arguments to control exactly what tests and how².

Once the package has been added, you can browse to where Nuget is storing packages for your particular project. In the tools folder of the console runner package folder, you will find xunit.console.exe. If you run this with the -? argument, you will get some helpful information.

xunit.console.exe -?

The three options I use the most are -trait, -notrait, and -parallel.

Traits

The two trait options control what tests you are running based on metadata attached to those tests. This is really useful if you have some tests that are resource heavy and only used in certain circumstances, such as stress testing. In MSTest, you could attach arbitrary metadata onto test methods using the TestProperty attribute. XUnit provides a similar feature using Trait. For example, [Trait("Category", "ManualOnly")] could be used to put a method into the ManualOnly category. You could then use the following line to execute tests that lack this trait.

xunit.console.exe -notrait "Category=ManualOnly"

If you wanted to only run tests with a specific trait, you do the same thing but with -trait instead. Both of these options are very useful when combined with -parallel.

Parallelism

XUnit can run tests in parallel, but tests within the same collection are never run in parallel with each other. By default, each test class is its own collection. Test classes can be combined into collections using the Collection attribute. Tests within the same collection will be executed randomly, but never in parallel.

The -parallel option provides four options: no parallelism, running tests from different assemblies in parallel, running tests from different collections in parallel, or running tests from different assemblies and different collections in parallel.

The difference between assembly parallelism, collection parallelism, and both together

Let's assume you have two assemblies, A and B. Assembly A has three collections; 1, 2, and 3. Assembly B has three collections; 4, 5, and 6.

No Parallelism
-parallel none

XUnit will run each collection in one of the assemblies, one at a time, then run each collection in the other assembly one at a time.

Collections 1, 2, 3, 4, 5, and 6 will never execute at the same time as each other.

Parallel Assemblies
-parallel assemblies

XUnit will run each collection in assembly A, one at a time, at the same time as running each collection in assembly B, one at a time.

Collections 1, 2, and 3 will not execute at the same time as each other; Collections 4, 5, and 6 will not execute at the same time as each other; but collections 1, 2, and 3 will execute in parallel with 4, 5, and 6, as the 1, 2, and 3 are in a different assembly to 4, 5, and 6.

Parallel Collections
-parallel collections

XUnit will run each collection within an assembly in parallel, but only one assembly at a time.

Collections 1, 2 and 3 will execute parallel; collections 4, 5, and 6 will execute in parallel; but, 1, 2, and 3 will not run at the same time as 4, 5, and 6.

Parallel Everything
-parallel all

XUnit will run each collection in parallel, regardless of its assembly.

Collections 1, 2, 3, 4, 5, and 6 will run in parallel, each potentially running at the same time as any other.

Beware running tests in parallel when first migrating from MSTest. It is a surefire way of finding some heinous test fixture dependencies and you risk thinks like deadlocking on resources. Usually, running assemblies in parallel is a lot safer than running collections in parallel, assuming that tests are collocated in assemblies based on their purpose and the resources they interact with.

In Conclusion…

That brings us to the end of the series. I have focused primarily on migrating from MSTest, leaving out a lot of the nuances to XUnit. I highly recommend continuing your education with the XUnit documentation and through experimentation; having personally migrated several projects, I know you won't regret it.

I love this approach to augmenting the development environment. It requires no additional tooling setup to get your dev environment working. The source itself controls the tooling versions and installation. Wonderful [↩]
You can control how tests run under MSBuild too by using various properties. This is discussed more on the XUnit site [↩]

Testing AngularJS: inject, spies and $provide

Testing is an important part of software development. To a software developer, automating that testing is an important part of software development, that is because to a software developer (at least one like me) testing is boring…unless we can make it seem like software development.

In my last few posts on AngularJS we looked at a way to monitor HTTP activity and guard against page navigation when requests were pending. However, we didn't validate that the code actually worked. Fortunately, the great ways to encapsulate client-side business logic and isolate it from the user experience that AngularJS provides, coupled with excellent support from angular-mocks, make testing AngularJS easy. In this post, we will take a glimpse at how.

Jasmine, CoffeeScript and Chutzpah

There are a few approaches to JavaScript testing, but they usually involve the same general components; a test framework, a test runner, and a test language. Thankfully (for me and you), this post is not an exhaustive discussion of testing options or their pros and cons. Instead, I will be stating what I use and assuming that they are the best choice¹. A great place to start is the JavaScript Testing Tactics presentation from Justin Searls, which can be found here along with other talks he has given.

Based on Justin's testing tactics, which I saw at SEMjs, I write all my tests using Jasmine, CoffeeScript and jasmine-given. The outcome is a terse testing DSL² that is low on ceremony and high on readability³.

Chutzpah

While it is common on greenfield projects to use a test runner such as Karma launched by Gulp or Grunt, I started my work on a legacy project where the build process was maintained using Visual Studio and MSBuild. Chutzpah is a suite of tools that fits this development process nicely, including a test runner as a NuGet package and some simple Visual Studio integration via extensions.

In addition, Chutzpah supports multiple testing frameworks⁴, multiple languages⁵ and code coverage metrics using Blanket.js. Basically, Chutzpah is awesome; fact.

Testing an Angular Factory

With our test framework, test runner and test language selected, we can look at our first test. We are going to test saHttpActivityInterceptor and the very first thing we should test is that saHttpActivityInterceptor actually exists. The following test does exactly that.

###
## <reference path="../angular-sa.js"/>
###

describe 'saHttpActivityInterceptor', ->
  Given -> module 'somewhatabstract'
  describe 'exists', ->
    When => inject (saHttpActivityInterceptor) =>
      @saHttpActivityInterceptor = saHttpActivityInterceptor
    Then => expect(@saHttpActivityInterceptor).toBeDefined()

Just as with AngularJS implementations, AngularJS tests start with some setup: a reference path to the file under test (other files such as the Jasmine framework, AngularJS and angular-mocks are included via the Chutzpah configuration file), a describe call under which to group all tests for saHttpActivityInterceptor, and a Given call that ensures the somewhatabstract module is loaded at the start of each child test (the module method is provided by Angular Mocks).

The actual unit test is declared starting at line 7 with describe 'exists', ->. This test is very simple; it states that when we try to inject our factory, we should get something other than undefined. The => syntax in CoffeeScript (also known as "fat arrow" syntax⁶) ensures that the When and Then calls share the same this context so that the @saHttpActivityInterceptor variable is shared between them (the @ symbol preceeding a variable in CoffeeScript indicates a context-level variable). The value stored in the @saHttpActivityInterceptor variable is obtained by asking AngularJS to inject it using the inject function, a helpful utility from angular-mocks.

This test works, you can verify it easily by commenting out the interceptor declaration in the JavaScript file we referenced⁷, but it is not a great test. If saNavigationGuard does not exist, this test will fail, yet our interceptor still exists. What we have done is create a simple integration test instead of a unit test; we need to isolate the thing under test, saHttpActivityInterceptor, from its dependency, saNavigationGuard.

inject and $provide

To isolate our item under test, we need to provide our own version of saNavigationGuard . We can do this using a fake; an object that pretends to be the real thing. We will use a Jasmine spy as a fake to represent saNavigationGuard and then provide it to AngularJS using the $provide service⁸. Because AngularJS uses the most recent definition when injecting dependencies and because our newly created spy is the most recent definition of saNavigationGuard, it is that spy which ultimately gets injected into saHttpActivityInterceptor when the test runs.

###
## <reference path="../angular-sa.js"/>
###

describe 'saHttpActivityInterceptor', ->
  Given -> module 'somewhatabstract'
  Given -> module ($provide) ->
    fakeNavigationGuard = jasmine.createSpyObj 'saNavigationGuard', ['registerGuardian']
    $provide.value 'saNavigationGuard', fakeNavigationGuard;return
      
  describe 'exists', ->
    When => inject (saHttpActivityInterceptor) =>
      @saHttpActivityInterceptor = saHttpActivityInterceptor
    Then => expect(@saHttpActivityInterceptor).toBeDefined()

Not only are we now isolating saHttpActivityInterceptor , but because our fake is being used in place of the real saNavigationGuard, we can check that any guardian registered with our fake works properly⁹.

###
## <reference path="../angular-sa.js"/>
###

describe 'saHttpActivityInterceptor', ->
  Given -> module 'somewhatabstract'
  Given -> module ($provide) ->
    fakeNavigationGuard = jasmine.createSpyObj 'saNavigationGuard', ['registerGuardian']
    $provide.value 'saNavigationGuard', fakeNavigationGuard;return
      
  describe 'exists', ->
    When => inject (saHttpActivityInterceptor) =>
      @saHttpActivityInterceptor = saHttpActivityInterceptor
    Then => expect(@saHttpActivityInterceptor).toBeDefined()
      
  describe '#guardian', ->
    describe 'is registered', ->
      Given => inject (saNavigationGuard) =>
        @saNavigationGuard = saNavigationGuard
      When => inject (saHttpActivityInterceptor) =>
      Then => expect(@saNavigationGuard.registerGuardian).toHaveBeenCalledWith jasmine.any(Function)
      
    describe 'returns undefined when there are no pending requests', ->
      Given => inject (saNavigationGuard) =>
        saNavigationGuard.registerGuardian.and.callFake (guardian) => @guardian = guardian
        inject (saHttpActivityInterceptor) =>
      When => @actual = @guardian()
      Then => expect(@actual).toBeUndefined()
    
    describe 'returns message when there are pending requests', ->
      Given => inject (saNavigationGuard) =>
        saNavigationGuard.registerGuardian.and.callFake (guardian) => @guardian = guardian
        inject (saHttpActivityInterceptor) => saHttpActivityInterceptor.request { method: "POST" }
      When => @actual = @guardian()
      Then => expect(@actual).toBeDefined()

And there we have it, a little test suite that validates the saHttpActivityInterceptor, almost. You may note that to test the guardian, we had to actually use one of the factory functions, request, but how do we know that the request function works if we haven't tested it? We should add some tests, but since we can only check the functionality of request, response, and responseError via the guardian call and we can only check the guardian call via the functionality of those other methods, we don't have a good way to gain high confidence in the functionality of any of these methods. Therefore, if we really wanted to test this effectively, we need to refactor the count functionality into its own factory. That way we can inject and validate the count state independently of the thing under test. For now, that's an exercise for another time.

Finally…

In this post, I have shown how we can test a simple AngularJS factory using Jasmine, Jasmine-Given and the built-in testing support of the AngularJS framework via Angular Mocks; specifically, module, inject and $provide.

While not exhaustive, I hope this look at testing Angular-based code encourages you to begin testing your own applications. Code discussed in this and earlier related posts can be found in a repository on GitHub. The repository includes a Visual Studio solution and project to run the tests, including NuGet and Bower restore to get the appropriate packages for running the tests. I intend to expand the code in GitHub as I write more blogs on AngularJS and AngularJS testing.

In the next post, we will take a look at how I structure directives to simplify testing and what that testing looks like. We might even see some of the cooler testing tricks for AngularJS that enable us to synchronously test asynchronous operations and validate web requests. Until then, feel free to ask questions in the comments and carry on coding.

I recommend that you investigate for yourself before choosing what works best for you and your development processes [↩]
Domain-specific Language [↩]
IMHO YMMV [↩]
Jasmine, QUnit and Mocha [↩]
CoffeeScript, TypeScript or plain old JavaScript [↩]
Be careful when using this "fat arrow" syntax – sharing context across unit tests can cause side-effects including false pass and fail results [↩]
Go on, try it. The code is on GitHub [↩]
Make sure to use $provide before any calls to inject [↩]
Though we could also have done this by spying on the real saNavigationGuard.registerGuardian function using Jasmine's spyOn function, such an approach assumes the remainder of saNavigationGuard has no unwanted side-effects, which is not necessarily true [↩]