Drop the BOM: A Case Study of JSON Corruption in WordPress

GiveCampIn September, I attended Ann Arbor Give Camp, a local event that connects non-profits with the local developer community to fulfill technological goals. As part of the project I was working on, I installed a plugin called CiviCRM into a WordPress deployment that was running on an IIS-based server.

It turned out that WordPress integration for CiviCRM was relatively new and a problem unique to IIS-based deployments existed after installation. This led to a white screen when I tried to access CiviCRM. I spent some time troubleshooting and eventually found the issue after I edited two files to track it down. The fix was quickly implemented. Unfortunately, I then discovered that some other features were not working properly.

The primary places this new issue surfaced were in displaying dialog windows within CiviCRM. It turned out that these dialogs obtained their UI via an AJAX call that returned some JSON and for some reason, jQuery was indicating that the call failed. Investigating further, I saw that the API call was successful (it returned a 200 status result) and the JSON appeared completely fine. How strange.

JSON in binary editor of Visual Studio
JSON in binary editor of Visual Studio

I made some debug changes to the JavaScript using the Google Chrome development tools and looked at the failure method jQuery was calling. In doing so, I discovered jQuery was reporting a parsing error for the JSON result. This seemed bizarre, after all, the JSON looked fine to me. I decided to verify it by copying and pasting it into Sublime. Still, the JSON looked just fine. Being tenacious, I saved the JSON to a text file and then opened it in Visual Studio's binary editor and there, the problem appeared. There were two characters at the start of the file before the first brace: byte order marks.

Corrupted JSON in Google Chrome developer tools
Corrupted JSON in Google Chrome developer tools

A byte order mark (often referred to as a BOM) is a Unicode character used to indicate the endianness (byte order) of a text file or stream1. JSON is not supposed to include them at all. In hindsight, I could have seen this issue much sooner if I had paid closer attention to the JSON response in the Network tab of Chrome's developer tools. This view had shown two red dots (see above) before the opening brace, each dot corresponding to a BOM that Chrome knew shouldn't be there. Of course, I had no idea what they meant and so I promptly ignored them. Lesson learned.

So, armed with the knowledge of why the JSON was causing parser errors, I had to find out what was causing this malformation and fix it. After reading about how a BOM in an incorrectly formatted PHP file2 could cause BOMs to be prepended in the PHP output, I started looking at each PHP file that would be touched when generating the API response. Alas, nothing initially stood out. I was getting frustrated when I had an epiphany; I had edited exactly two files in trying to fix the installation issue and there were exactly two BOMs. Coincidence?

I went to the two files that I had edited, downloaded them and discovered they both had BOMs. I re-saved them, this time without a BOM and uploaded them back to the site, which fixed the JSON corruption and got the CiviCRM plug-in in to working order.

In tracking down and fixing this self-made issue, I learned a few valuable lessons:

  1. Learn to use my developer tools
  2. Never assume it is not my fault
  3. It pays to understand how things work

Hopefully, my misfortune in this one incident will help someone track down their own issue with corrupted JSON in WordPress. If so, please share in the comments. Together, our mistakes can be someone else's salvation.

  1. Wikipedia – http://en.wikipedia.org/wiki/Byte_order_mark []
  2. one saved as Unicode with byte order mark []

Testing AngularJS: $resource

This is the fourth post taking a look at testing various aspects of AngularJS. Previously, I covered:

In this installment we'll take a look at what I do to isolate components from their use of the $resource service and why.

$resource

The $resource service provides a simple way to define RESTful API endpoints in AngularJS and get updated data without lots of promise handling unnecessarily obfuscating your code. If you have created a resource for a specific route, you can get the returned data into your scope as easily as this:

$scope.data = myResource.get();

The $resource magic ensures that once the request returns, the data is updated. It's clever stuff and incredibly useful.

However, when testing components that use resources, I want to isolate the components from those resources. While I could use $httpBackend or a cache to manipulate what results the resource returns, these can be cumbersome to setup and adds unnecessary complexity and churn to unit tests1. To avoid this complexity, I use a fake that can be substituted for $resource.

spyResource

My fake $resource is called spyResource. It is not quite a 1-1 replacement, but it does support the more common situations one might want (and it could be extended to support more). Here it is.

spyResource = function (name) {
	var resourceSpy = jasmine.createSpy(name + ' resource constructor').and.callFake(function () { angular.copy({}, this); });

	resourceSpy['get'] = resourceSpy.prototype['get'] = jasmine.createSpy('get');
	resourceSpy['$save'] = resourceSpy.prototype['$save'] = jasmine.createSpy('$save');
	resourceSpy['$delete'] = resourceSpy.prototype['$delete'] = jasmine.createSpy('$delete');

	return resourceSpy;
};

First of all, it is just a function. Since it is part of my testing framework, there is no need to wrap it in some fancy AngularJS factory, though we certainly could if we wanted.

Second, it mimics the $resource service by returning a function that ultimately copies itself. This is useful because you do not necessarily have access to the instances of a resource that are created in your code before posting an object update to your RESTful API. By copying itself, you can see if the $save() call is made directly from the main spyResource definition, even if it was actually called on an instance returned by it because they share the same spies.

To use this in testing, the $provide service can be used to replace a specific use of $resource with a spyResource. For example, if you defined a resource called someResource, you might have:

describe 'testing something that uses a resource', ->
  Given => module ($provide) =>
    $provide.value 'someResource', spyResource('someResource');return
  ...

Now, the fake resource will be injected instead of the real one, allowing us to not only spy on it, but to also ensure there are no side-effects that we have not explicitly set up.

Finally…

I have covered a very simple technique I use for isolating components from and spying on their usage of AngularJS resources. The simple fake resource I provided for this purpose can be easily tailored to cater to more complex scenarios. For example, if the code under test needs data from the get() method or the $promise property is expected in get() return result, the spy can be updated to return that data.

Using this fake resource instead of $httpBackend or a cache to manipulate the behavior of a real AngularJS resource not only simplifies the testing in general, but also reduces code churn by isolating the tests from the API routes that can often change during development.

As always, please leave a comment if you find this useful or have other feedback.

 

 

  1. API routes can often change during development, which would lead to updating `$httpBackend` test code so that it matches []

Track pending web requests with AngularJS

Navigation Guard

In the previous two posts (here and here), I covered the AngularJS factory, saNavigationGuard that I had created. This factory provides a simple way to guard against inconvenient page navigation in our Angular applications. However, up until now, I have not demonstrated one of the coolest uses for this. Before we look at this awesome use of saNavigationGuard, let us take a short trip into late last year.

I was getting started on my first big project with AngularJS and I had added a busy indicator to my user experience. To control when the indicator was active, I used a counter. This counter was incremented and decremented before and after using Angular resources. It was a nest of promises and error handlers, just to make sure the counter worked properly. I was not entirely happy with it but it was the best I could work out from my knowledge of Angular and JavaScript, so I submitted a pull request. My colleague reviewed the work and, looking at the busy indicator and the code to control the counter, stated, "there must be a better way."

I did not know at the time, but he was right, there is a better way and it uses an Angular feature called "interceptors".

Interceptors

Interceptors provide hooks into web requests, responses and their errors allowing us to modify or handle them in different ways. They are provided via an AngularJS provider such as a factory or service and injected into the $httpProvider using config as follows.

angular.module('somewhatabstract').config(['$httpProvider', function($httpProvider) {
	$httpProvider.interceptors.push('saHttpActivityInterceptor');
}]);

In this snippet, the name of the interceptor, saHttpActivityInterceptor, is added to the array of interceptors on $httpProvider.

The interceptor itself is a little more complex.

angular.module('somewhatabstract').factory('saHttpActivityInterceptor', ['$q', 'saNavigationGuard', function($q, saNavigationGuard) {
	var pendingRequestsCounter = 0;

	var updateCounter = function(method, delta) {
		if (method != 'POST' && method != 'PUT' && method != 'DELETE') {
		    return false;
		}
		pendingRequestsCounter += delta;
		return true;
	};

	saNavigationGuard.registerGuardian(function() {
		return pendingRequestsCounter > 0 ? 'There are changes pending.' : undefined;
	});

	return {
		request: function(request) {
			request.saTracked = updateCounter(request.method, 1);
			return request;
		},

		response: function(response) {
			if (response.config && response.config.saTracked) {
				updateCounter(response.config.method, -1);
			}
			return response;
		},

		responseError: function(rejection) {
			if (rejection.config && rejection.config.saTracked) {
				updateCounter(response.config.method, -1);
			}
			return $q.reject(rejection);
		}
	};
}]);

The interceptor factory returns an object that, in this case, has three methods: request, response and responseError. A fourth method, requestError, can also be included in interceptors if needed. Before returning our interceptor, the interceptor factory registers a guardian with saNavigationGuard that will guard against navigation if the pendingRequestsCounter is greater than zero.

The interceptor monitors requests and responses. On each request, the request method is checked, if it is POST, PUT or DELETE, the pendingRequestsCounter variable is incremented by one and the request is tagged to indicate it is being tracked. We flag it so that we know to pay attention when the corresponding response or response error occurs. In the response and response error handlers, we decrement our counter based on the tracking flag and the method.

Finally…

The outcome of using this interceptor is that if the user tries to navigate away from the page after a request has been made but before its response has been received, they will see a message asking them to consider postponing the navigation.

In the next post, we will look at testing both this interceptor and the saNavigationGuard functionality using my preferred combination of jasmine, CoffeeScript and jasmine-given.

As always, please consider leaving a comment if you have found this post useful or have any alternatives.

Unit testing attribute driven late-binding

I've been working on a RESTful API using ASP WebAPI. It has been a great experience so far. Behind the API is a custom framework that involves some late-binding. I decorate certain types with an attribute that associates the decorated type with another type1. The class orchestrating the late-binding takes a collection of IDecorated instances. It uses reflection to look at their attributes to determine the type they are decorated with and then instantiates that type.

It's not terribly complicated. At least it wasn't until I tried to test it. As part of my development I have been using TDD, so I wanted unit tests for my late-binding code, but I soon hit a hurdle. In mocking IDecorated, how do I make sure the mocked concrete type has the appropriate attribute?

var mockedObject = new Mock();

// TODO: Add attribute

binder.DoSpecialThing( mockedObject.Object ).Should().BeAwesome();

I am using Moq for my mocking framework accompanied by FluentAssertions for my asserts2. Up until this point, Moq seemed to have everything covered, yet try as I might I couldn't resolve this problem of decorating the generated type. After some searching around I eventually found a helpful Stack Overflow question and answer that directed me to TypeDescriptor.AddAttribute, a .NET-framework method that provides one with the means to add attributes at run-time!

var mockedObject = new Mock();

TypeDescriptor.AddAttribute(
    mockedObject.Object.GetType(),
    new MyDecoratorAttribute( typeof(SuperCoolThing) );

binder.DoSpecialThing( new [] { mockedObject.Object } )
    .Should()
    .BeAwesome();

Yes! Runtime modification of type decoration. Brilliant.

So, why didn't it work?

My binding class that I was testing looked a little like this:

public IEnumerable<Blah> DoSpecialThing( IEnumerable<IDecorated> decoratedThings )
{
    return from thing in decoratedThings
           let converter = GetBlahConverter( d.GetType() )
           where d != null
           select converter.Convert( d );
}

private IConverter GetBlahConverter( Type type )
{
    var blahConverterAttribute = Attribute
        .GetCustomAttributes( type, true )
        .Cast<BlahConverterAttribute>()
        .FirstOrDefault();

    if ( blahConverterAttribute != null )
    {
        return blahConverterAttribute.ConverterType;
    }

    return null;
}

Looks fine, right? Yet when I ran it in the debugger and took a look, the result of GetCustomAttributes was an empty array. I was stumped.

After more time trying different things that didn't work than I'd care to admit, I returned to the StackOverflow question and started reading the comments; why was the answer accepted answer when it clearly didn't work? Lurking in the comments was the missing detail; if you use TypeDescriptor.AddAttributes to modify the attributes then you have to use TypeDescriptor.GetAttributes to retrieve them.

I promptly refactored my code with this detail in mind.

public IEnumerable<Blah> DoSpecialThing( IEnumerable<IDecorated> decoratedThings )
{
    return from thing in decoratedThings
           let converter = GetBlahConverter( d.GetType() )
           where d != null
           select converter.Convert( d );
}

private IConverter GetBlahConverter( Type type )
{
    var blahConverterAttribute = TypeDescriptor
        .GetAttributes( type )
        .OfType<BlahConverterAttribute>()
        .FirstOrDefault();

    if ( blahConverterAttribute != null )
    {
        return blahConverterAttribute.ConverterType;
    }

    return null;
}

Voila! My test passed and my code worked. This was one of those things that had me stumped for longer than it should have. I am sharing it in the hopes of making sure there are more hits when someone else goes Internet fishing for help. Now, I'm off to update Stack Overflow so that this is clearer there too.

  1. Something similar to TypeConverterAttribute usage in the BCL []
  2. Though I totally made up the BeAwesome() assertion in this blog post []