Using PowerShell to make BrowserStack local testing easier with Microsoft Edge

BrowserStack has been an incredibly useful resource for tracking down bugs and testing fixes when I am working on websites. This often requires accessing locally deployed sites or sites accessible over a VPN connection, and to do that, BrowserStack needs some local code running to be able to route the traffic accordingly.

BrowserStack logoUp until recently, my browser of choice has been Google Chrome, for which BrowserStack provides a handy extension to add support for local sites. However, since the Windows Creators Update, I have been giving Microsoft Edge a shot1 and no such extension exists. Instead, BrowserStack provides a download, BrowserStackLocal.exe , and a secret with which to run it. This works great, but there are a couple of annoyances.

  1. I have to remember to run it.
  2. It is a blocking process.

There are a variety of ways this problem can be solved. I decided to take the opportunity to expand my PowerShell fu and put together some cmdlets to run the BrowserStackLocal process in the background. Specifically, I wanted to compare PowerShell jobs with plain old processes for this specific purpose.

First: Jobs

Since the running the command is a blocking operation, I decided to try wrapping it in a PowerShell job so that it would sit in the background. This is useful since the job gets terminated when the PowerShell session ends, which makes it less likely for me to forget. The downside is that each PowerShell session could have its own job, but only the one that started BrowserStackLocal will actually end it, but I was certain I could work with that.

Getting started

The first cmdlet for starting BrowserStackLocal is cunningly named Start-BrowserStackLocal , shown here:

This has basic room for improvement, like having the secret and the path be parameters to the cmdlet, or environment variables; I happened to stop tinkering once it worked for me, so feel free to expand on it.

At the start, we check to see if we already have a job for BrowserStackLocal since we only need one. If we do not, then we get on with making sure BrowserStackLocal can be found where we expect it. If everything looks good, then the job gets started.

To tackle the chance that my script may fail due to the BrowserStackLocal command either getting an incorrect key or discovering it is already running, I added a Wait-Job call. The nice thing here is that since normally BrowserStackLocal blocks, we can assume that if the job did not reach a completion state, then the executable command is running. I take advantage of that fact, so if the Wait-Job returns, we can assume things went wrong and dump the details of the problem back to the console.

Stopping the job

Once the job is running, we need to be able to terminate it.

This is a much simpler cmdlet than the one to start the job. It has two main tasks:

  1. See if the job is actually running
  2. If it is, stop it

I added some helpful output so we could see it working and that was that.

Output from jobs-based cmdlets
Output from jobs-based cmdlets

Problems with jobs

This solution using jobs works great but it is not ideal. Each PowerShell session has its own jobs, so you have to know which session actually started BrowserStackLocal in order to stop it. Not only that, but if PowerShell did not start it at all, you cannot stop it from there with these commands at all. Jobs are great but they are not really the right tool for this…er…job.

Second: Processes

The wise man would have probably started here. I did not because I wanted to learn about jobs. Now that I have, I am wiser and so, I thought I would recreate my success but this time using the Xxx-Process  cmdlets of PowerShell.

Getting started again

Using processes, the start cmdlet looks like this:

Since the BrowserStackLocal executable starts more than one process, I added some loops to output information about those processes. Now if we try to start the command and it is already running, we will get the same feedback, regardless of where the command was started.

Stopping the process

Switching to processes makes the stop code a little more complicated, but only because I wanted to provide some additional detail (we could have just called Stop-Process BrowserStackLocal and it would stop all matching processes).

Output from process-based cmdlets
Output from process-based cmdlets

Helpful aliases

Finally, to make the task of starting and stopping a little less arduous, I added some aliases (inspired by the helpful sasv and spsv aliases of Start-Service and Stop-Service).

Conclusion

TL;DR: Use processes to start processes in the background2.

The rest

I am pretty pleased with how this little PowerShell project worked out. I get to keep using Microsoft Edge with minimal effort beyond what I had when using Google Chrome for my BrowserStack testing, enabling me to take advantage of the performance and battery-life improvements Edge has over Chrome. Not only that, but I got to learn some new things about PowerShell.

  1. You don't get closures entirely for free in PowerShell. I suspected this, but I learned the hard way. However…
  2. We can pass local variables into script blocks using the $using:<variable> syntax instead of passing an argument list and adding parameters to our script.
  3. Debugging jobs can be a pain until you learn the value of Receive-Job for getting error information.
  4. Use Wait-Job with a little time out to give your job chance to fail so that you can spit out some error information.
  5. You have to stop a job before you can remove it.
  6. Don't use jobs to control background processes; use processes instead

I have not gone so far yet as to start the BrowserStackLocal service automatically, but I can see value in doing so, especially if I did a lot of BrowserStack testing on local sites every day (of course, I'd probably want to redirect the output to $null in that scenario rather than see feedback on the running processes with every shell I opened).

What are your thoughts? Do you use PowerShell jobs? Do you use BrowserStack? Will you make use of these cmdlets? Fire off in the comments.


  1. Yes, the battery life is noticeably better than when using Chrome; yes, I am frustrated that I cannot clear cookies for a specific site 

  2. Well, duh. But if I had taken that attitude, I would not have learned about jobs. 

Running XUnit Tests, Using Traits, and Leveraging Parallelism

We have arrived at the end of this little series on migrating unit tests from MSTest to XUnit (specifically, XUnit 2). While earlier posts concentrated on writing the tests and the XUnit counterparts to MSTest concepts, this post will briefly look at some non-code aspects to XUnit; most importantly, we will look at getting the tests to run.

Do not depend on test order
One thing to watch out for after migrating your tests is the order in which tests are run. MSTest allowed us to abuse testing by assuming that tests would run in a specific order. XUnit does away with this and will run tests in a random order. This can help to find some obscure bugs, but it can also make migration a little tougher, especially when tests share a data store or some other test fixture. Watch out for that.

Visual Studio

Running tests inside Visual Studio is really simple. Following in the footsteps of web development trends, rather than requiring an extension to the development environment, XUnit uses package management1. All you need to do is add the Visual Studio XUnit test runner package to your project and Visual Studio will be able to detect and run your XUnit tests just like your old MSTests.

Of course, if you're like me and absolutely loathe the built-in Visual Studio test explorer, you can use Resharper (or dotCover), which has built-in support for XUnit.

Command Line

More often than not, our continuous integration setups are scripted and we're unlikely to be running our unit tests via the development tool, such as Visual Studio. For situations like this, you can add the XUnit command line test runner package to your project. This provides a command line utility for running your tests with arguments to control exactly what tests and how2.

Once the package has been added, you can browse to where Nuget is storing packages for your particular project. In the tools folder of the console runner package folder, you will find xunit.console.exe. If you run this with the -? argument, you will get some helpful information.

The three options I use the most are -trait, -notrait, and -parallel.

Traits

The two trait options control what tests you are running based on metadata attached to those tests. This is really useful if you have some tests that are resource heavy and only used in certain circumstances, such as stress testing. In MSTest, you could attach arbitrary metadata onto test methods using the TestProperty attribute. XUnit provides a similar feature using Trait. For example, [Trait("Category", "ManualOnly")] could be used to put a method into the ManualOnly category. You could then use the following line to execute tests that lack this trait.

If you wanted to only run tests with a specific trait, you do the same thing but with -trait instead. Both of these options are very useful when combined with -parallel.

Parallelism

XUnit can run tests in parallel, but tests within the same collection are never run in parallel with each other. By default, each test class is its own collection. Test classes can be combined into collections using the Collection attribute. Tests within the same collection will be executed randomly, but never in parallel.

The -parallel option provides four options: no parallelism, running tests from different assemblies in parallel, running tests from different collections in parallel, or running tests from different assemblies and different collections in parallel.

The difference between assembly parallelism, collection parallelism, and both together

Let's assume you have two assemblies, A and B. Assembly A has three collections; 1, 2, and 3. Assembly B has three collections; 4, 5, and 6.

No Parallelism
-parallel none

XUnit will run each collection in one of the assemblies, one at a time, then run each collection in the other assembly one at a time.

Collections 1, 2, 3, 4, 5, and 6 will never execute at the same time as each other.

Parallel Assemblies
-parallel assemblies

XUnit will run each collection in assembly A, one at a time, at the same time as running each collection in assembly B, one at a time.

Collections 1, 2, and 3 will not execute at the same time as each other; Collections 4, 5, and 6 will not execute at the same time as each other; but collections 1, 2, and 3 will execute in parallel with 4, 5, and 6, as the 1, 2, and 3 are in a different assembly to 4, 5, and 6.

Parallel Collections
-parallel collections

XUnit will run each collection within an assembly in parallel, but only one assembly at a time.

Collections 1, 2 and 3 will execute parallel; collections 4, 5, and 6 will execute in parallel; but, 1, 2, and 3 will not run at the same time as 4, 5, and 6.

Parallel Everything
-parallel all

XUnit will run each collection in parallel, regardless of its assembly.

Collections 1, 2, 3, 4, 5, and 6 will run in parallel, each potentially running at the same time as any other.

Beware running tests in parallel when first migrating from MSTest. It is a surefire way of finding some heinous test fixture dependencies and you risk thinks like deadlocking on resources. Usually, running assemblies in parallel is a lot safer than running collections in parallel, assuming that tests are collocated in assemblies based on their purpose and the resources they interact with.

In Conclusion…

That brings us to the end of the series. I have focused primarily on migrating from MSTest, leaving out a lot of the nuances to XUnit. I highly recommend continuing your education with the XUnit documentation and through experimentation; having personally migrated several projects, I know you won't regret it.

 


  1. I love this approach to augmenting the development environment. It requires no additional tooling setup to get your dev environment working. The source itself controls the tooling versions and installation. Wonderful 

  2. You can control how tests run under MSBuild too by using various properties. This is discussed more on the XUnit site 

DataSource and Data-driven Testing Using XUnit

If you are anything like me, you avoided data-driven tests in MSTest because they were such a pain to write and maintain. However, I know not everyone is like me and I also know that even though we try to avoid things, we do not always succeed. So, in this entry in the series on migrating from MSTest to XUnit, we will look at migrating your data-driven tests, but before we get into the details, let's briefly recap on what MSTest provides for data-driving tests; the DataSource attribute.

The DataSource attribute specifies a data source from which data points are loaded. The test can then reference the specific data row in the data source from the TestContext's DataRow property. You may recall that we touched on the jack-of-all-trades, master-of-none TestContext back in the entry on outputting from our tests. As MSDN explains, given a table of data rows like this:

FirstNumberSecondNumberSum
011
112
2-3-1

DataSource is used like this:

I do not recall using this attribute in earnest, and though perhaps others think of it more fondly, I found it a frustration to use. Thankfully, XUnit is much more inline with my intuition1.

Data-driven test methods in XUnit are called theories and are adorned with the Theory attribute2. The Theory attribute is always accompanied by at least one data attribute which tells the test runner where to find data for the theory. There are three built-in attributes for providing data: InlineData, MemberData, and ClassData. Third-party options are also available (check out AutoFixture and its AutoData attribute) and you can create your own if you find it necessary.

InlineData provides a simple way to describe a single test point for your theory; MemberData takes the name of a static member (method, field, or property) and any arguments it might need to generate your data; and ClassData takes a type that can be instantiated to provide the data. A single test point is provided as an array of type object, and while XUnit provides the TheoryData types to allow strongly-typed declaration of test data points, fundamentally, every data source  is IEnumerable<object[]>. Finally, rather than the obscure TestContext.DataRow property, data points are provided to a theory test method via the test method's arguments.

So, given all this information, the above example from MSDN could be expressed as follows:

I took the liberty of using the arrange/act/assert test layout as part of this rewrite as I think it enhances test readability. However, you can see from this example how much easier data-driven testing is under XUnit when compared with traditional MSTest3.

Of course, I totally skipped the fact that the MSTest example used a database for the test data source. That was deliberate to simplify the example, but if we still wanted to use a database to obtain our data points (or some other data source), we can leverage the MemberData or ClassData attributes, or even roll our own.

That brings us to the end of this post on migrating data-driven tests from MSTest to XUnit. This also brings us almost to the end of the whole series on migrating from MSTest to XUnit; if you think I missed something important, please leave a comment. Next time, we will finish up with a look at some of the bits around running XUnit tests such as parallel execution, test runners, and the like.


  1. in case you hadn't noticed, I like XUnit 

  2. instead of [Fact] as on non-data-driven tests 

  3. This approach makes so much sense that MSTest introduced it for phone and WinRT app tests and is bringing it to everyone with MSTest version 2 

AssemblyInitialize, AssemblyCleanup and Sharing State Between Test Classes in XUnit

We have covered quite a bit in this series on migrating from MSTest to XUnit and we have not even got to the coolest bit yet; data-driven theories. If that is what you are waiting for, you will have to wait a little longer. Before we get there, I want to cover one last piece of test initialization as provided in MSTest by the AssemblyInitialize and AssemblyCleanup attributes.

As we saw in previous posts, we can use the test class constructor and Dispose() for TestInitialize and TestCleanup, and IClassFixture<T> and fixture classes for ClassInitialize and ClassCleanup.  For the assembly equivalents, we use collections and the ICollectionFixture<T> interface.

A collection is defined by a set of test classes and a collection definition. A test class is designated as being part of a specific collection by decorating the class with the Collection attribute, which provides the collection name. A corresponding class decorated with the CollectionDefinition attribute should also exist as this is where any collection fixtures are defined. All classes that share the same collection name will share the collection fixtures from which the definition class derives.

The example code above shows a collection definition with two fixtures and two test classes defined as part of that collection. Note how the fixtures control initialization and cleanup using constructors and IDisposable 1 . We can modify those classes to reference the collection fixtures just as we did with class-level fixtures; by referencing the fixture in the constructor arguments as shown here.

I really like this approach over the attributed static methods of MSTest. This seems to more easily support code reuse and makes intentions much clearer, separating the concerns of tests (defined in the test class) from fixtures (defined by the fixture types). The downside is that fixture types do not get access to the ITestOutputHelper interface so if you want your fixtures to output diagnostic information, you should consider a logging library like Common.Logging. Also, your fixture types must be in the same assembly as your tests. Of course, that doesn't preclude the fixtures from using types outside of the assembly, so you can always put shared implementation between test assemblies in some other class library.

And that brings our migration of shared initialization to a close. You can find more information on sharing context across tests on the xunit site. Next up, we will look at data-driven tests. Thank you for your time. If you have any questions, please leave a comment.


  1. A fixture type can used with IClassFixture<T> or ICollectionFixture<T>  

ClassInitialize, ClassCleanup, and Sharing Data Across Tests in XUnit2

So far in this series on migrating from MSTest to XUnit, we have looked at:

In this post, we will look at how we can share setup and cleanup code across tests in a test class in XUnit. MSTest allows you to define shared setup and cleanup code for an entire test class by using methods decorated with the ClassInitialize and ClassCleanup attributes. Unlike their counterparts, TestInitialize and TestCleanup, methods decorated with these class-level attributes are executed just once per class, rather than once per test in the class. Using these class-level attributes, we can execute code, generate fixture objects, and load test data that can be used across all tests in the class without having the overhead of repeating this for every test in the class. This is useful when that initialization or cleanup is expensive, such as creating a database connection, or loading several data files.

As we have seen so far, XUnit is light on decorating non-test methods with attributes, instead relying on language syntax that mirrors the purpose of the code. In the case of TestInitialize and TestCleanup, XUnit uses the test class constructor and IDisposable. It should come as no surprise that this pattern is also used when it comes to class-level initialization and cleanup.

IClassFixture<T>

There are two parts to shared initialization and cleanup in XUnit: declaring what shared items a test class uses, and referencing them within test methods.

To declare specific setup is required, a test class must be derived from IClassFixture<T> for each shared setup/cleanup. The T in IClassFixture<T> is the actual type responsible for the initialization and cleanup via its constructor and IDisposable implementation.

The XUnit test runner sees that your test class is deriving from IClassFixture<MyFixture> and ensures that an instance of MyFixture is created before your tests are run and disposed of when all the tests are completed. I really like this approach over the MSTest equivalent, as it moves the setup and initialization from being about the test class to being about the test fixture, the thing being setup. You can even have more than one fixture, so if you use two databases in your tests, you can have one fixture for each database and explicitly specify the use of each. It also means that you can set things that are supposed to be immutable for the duration of tests to be readonly and enforce that immutability. This is even clearer when referencing fixtures in tests.

As shown in the preceding example, to reference a test fixture in your test class methods, you just need to add a corresponding argument to the constructor and XUnit will inject the fixture. You can then use the fixture, and assign it or something obtained from it to a member variable of your class. Not only that, but you can mark that member as readonly and be explicit about what tests can and cannot do to your test state. Personally, this approach to shared initialization and cleanup feels much more intuitive. I can easily reuse my initialization and setup code without cluttering my test classes unnecessarily, and I can be explicit about the immutability of any shared state or setup.

And that is it; now you not only know how to share repeatable setup across tests (as provided by TestInitialize and TestCleanup in MSTest), but also how to do the same for setup across the whole test class (as MSTest does with ClassIntialize and ClassSetup).

But, what of AssemblyInitialize and AssemblyCleanup? Well, that's probably a good place to start in the next post. As always, you are welcome to leave a comment letting me know how you are liking this series on migrating to XUnit, or perhaps bringing up something that you'd like me to cover.