Drop the BOM: A Case Study of JSON Corruption in WordPress

GiveCampIn September, I attended Ann Arbor Give Camp, a local event that connects non-profits with the local developer community to fulfill technological goals. As part of the project I was working on, I installed a plugin called CiviCRM into a WordPress deployment that was running on an IIS-based server.

It turned out that WordPress integration for CiviCRM was relatively new and a problem unique to IIS-based deployments existed after installation. This led to a white screen when I tried to access CiviCRM. I spent some time troubleshooting and eventually found the issue after I edited two files to track it down. The fix was quickly implemented. Unfortunately, I then discovered that some other features were not working properly.

The primary places this new issue surfaced were in displaying dialog windows within CiviCRM. It turned out that these dialogs obtained their UI via an AJAX call that returned some JSON and for some reason, jQuery was indicating that the call failed. Investigating further, I saw that the API call was successful (it returned a 200 status result) and the JSON appeared completely fine. How strange.

JSON in binary editor of Visual Studio
JSON in binary editor of Visual Studio

I made some debug changes to the JavaScript using the Google Chrome development tools and looked at the failure method jQuery was calling. In doing so, I discovered jQuery was reporting a parsing error for the JSON result. This seemed bizarre, after all, the JSON looked fine to me. I decided to verify it by copying and pasting it into Sublime. Still, the JSON looked just fine. Being tenacious, I saved the JSON to a text file and then opened it in Visual Studio's binary editor and there, the problem appeared. There were two characters at the start of the file before the first brace: byte order marks.

Corrupted JSON in Google Chrome developer tools
Corrupted JSON in Google Chrome developer tools

A byte order mark (often referred to as a BOM) is a Unicode character used to indicate the endianness (byte order) of a text file or stream1. JSON is not supposed to include them at all. In hindsight, I could have seen this issue much sooner if I had paid closer attention to the JSON response in the Network tab of Chrome's developer tools. This view had shown two red dots (see above) before the opening brace, each dot corresponding to a BOM that Chrome knew shouldn't be there. Of course, I had no idea what they meant and so I promptly ignored them. Lesson learned.

So, armed with the knowledge of why the JSON was causing parser errors, I had to find out what was causing this malformation and fix it. After reading about how a BOM in an incorrectly formatted PHP file2 could cause BOMs to be prepended in the PHP output, I started looking at each PHP file that would be touched when generating the API response. Alas, nothing initially stood out. I was getting frustrated when I had an epiphany; I had edited exactly two files in trying to fix the installation issue and there were exactly two BOMs. Coincidence?

I went to the two files that I had edited, downloaded them and discovered they both had BOMs. I re-saved them, this time without a BOM and uploaded them back to the site, which fixed the JSON corruption and got the CiviCRM plug-in in to working order.

In tracking down and fixing this self-made issue, I learned a few valuable lessons:

  1. Learn to use my developer tools
  2. Never assume it is not my fault
  3. It pays to understand how things work

Hopefully, my misfortune in this one incident will help someone track down their own issue with corrupted JSON in WordPress. If so, please share in the comments. Together, our mistakes can be someone else's salvation.

  1. Wikipedia – http://en.wikipedia.org/wiki/Byte_order_mark []
  2. one saved as Unicode with byte order mark []

jQuery validation of select elements where multiple="multiple"

I was recently working on a simple feature in our web client at work. Part of the feature required that the user be able to select one to many items from a multiple select list. My first stab at this worked great; I added the required class to the select element and voila, the user was limited to at least one item. However, I thought it would be nice to give a custom error message with a little more context than just "this field is required" and perhaps provide the ability to limit the selection to a variable minimum and maximum length.

I quickly discovered that I don't understand the jQuery Validation plug-in and the documentation seems to be written by someone who thinks I might know way more than I actually do1. My next step was to find examples but they were all related to comparing element values on more standard input elements, which just served to confuse me further, so I embarked on a quest to find out how to solve my problem and write a blog about it.

Now, I'm sure that what I am about to document will seem like child's play to the jQuery ninjas out there, but for every ninja equipped with the skills to silently dispatch every henchman in the room, there's someone like me who just wants to get out of the room alive. So, please bear that in mind as you read on and whether a ninja or just a survivor, please comment.

<!DOCTYPE html>

<html lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta charset="utf-8" />
    <title>My jQuery real good fun time yeah page</title>
    <script src="http://code.jquery.com/jquery-1.9.1.js" type="text/javascript"></script>
    <script src="http://ajax.aspnetcdn.com/ajax/jquery.validate/1.11.1/jquery.validate.js" type="text/javascript"></script>
    <script type="text/javascript">
        /* Placeholder for my amazing coding skills. */
    </script>
</head>
<body>
    <form id="mySuperCoolForm">
        <fieldset>
            <legend>Select some stuff</legend>
            <select name="things" multiple="multiple">
                <option>thing 1</option>
                <option>thing 2</option>
                <option>thing 3</option>
                <option>thing 4</option>
                <option>thing 5</option>
            </select>
            <input type="submit" />
        </fieldset>
    </form>
</body>
</html>

This is the simple HTML page that I'm going to work with. There's nothing complex here, just a form with a select and a submit input. I have also added some script imports for jQuery and the jQuery validate plug-in and an empty script element for our validation definitions to live.

To this, we want to add some validation to make sure that at least one element in the list is selected when the form is submitted. First, just to check things are wired up correctly, we will just specify the select named "things" as being required. Rather than do this by just adding the required class, let's use some scripting, that way we can manipulate the script as we go.

$(function () {
    $('#mySuperCoolForm').validate({
        rules:{
            things:{
                required:true
            }
        }
    });
});

This code is not doing much. It is telling the jQuery validate plug-in to attach its validation to our form, mySuperCoolForm with the rule that our field, things is required. If you try this out and click the submit button without selecting anything, you'll get a message stating, "This field is required." Not a very descriptive message, so let's fix that by adding a message for our "things are required" rule.

$(function () {
    $('#mySuperCoolForm').validate({
        rules: {
            things: {
                required: true
            }
        },
        messages: {
            things: {
                required: 'Please select at least one thing.'
            }
        }
    });
});

Again, not a very complex change. I have added a messages field to the JSON object being passed to the validate method. This field provides messages that should be displayed when validation rules fail. In this case, we are stating we want at least one thing to be selected whenever our required rule fails. You should note the correlation here between the name of our field, things and the rules and messages that are attached to it. This is important to understanding how to manipulate the jQuery validation. When a rule fails to validate for a named field, the message under the same named field with the matching name to that failed rule will be displayed.

That works nicely so it's job done, right? Not quite. You see, while this works for the cases where one to many items need selecting, it feels a little hacky. I'd rather say explicitly "I want x items selected" rather than rely on the required rule working the way it does for multiple select scenarios. I like being explicit. So, let's get explicit2.

Validating with minlength

It just so happens that the jQuery validation plug-in has another rule, minlength that appears to be exactly what we want. Thinking this, I naively wrote some code to require at least two items. I also included some string formatting for the error message so that the number of things in the message matched the number required by the rule.

$(function () {
    $('#mySuperCoolForm').validate({
        rules: {
            things: {
                minlength: 2
            }
        },
        messages: {
            things: {
                minlength: $.format('Please select at least {0} things.')
            }
        }
    });
});

Now, if we select two things, we can submit just fine, but if we select just one thing, we get told we need at least two. So far so good. However, if we select nothing, the form passes validation and submits fine! What? We stated a minimum of two and yet zero passes. That makes no sense. It made even less sense when I set my minlength to 1 and it didn't appear to do any validation at all. Everything was feeling so intuitive, what happened? By looking at the code, it becomes clearer.

The minlength rule looks like this:

minlength: function( value, element, param ) {
			var length = $.isArray( value ) ? value.length : this.getLength($.trim(value), element);
			return this.optional(element) || length >= param;
		}

If you put a breakpoint here, you'll discover that when nothing is selected, value is null, but that's okay because length becomes `0` in that circumstance anyway. The problem is on the second line of the function, which I have highlighted. It checks to see if the element is optional or not and if it is optional, the element passes validation regardless. A quick look at the optional function shows that it calls the required rule and returns the inverse of its result. When we have no items selected, required returns false which optional turns to true which incorrectly says our data is valid. However, when only one item is selected, required returns true which means the `length >= param` check occurs and our validation fails as we would like. While this behaviour is probably intuitive for say, string input fields where no value and values of a certain minimum length make sense, this is confusing when using minlength with select fields.

To get our validation working as we would like, we have to apply both the required and minlength rules as follows:

$(function () {
    $('#mySuperCoolForm').validate({
        rules: {
            things: {
                required: true,
                minlength: 2
            }
        },
        messages: {
            things: {
                required: 'Please select at least 2 things.'
                minlength: $.format('Please select at least {0} things.')
            }
        }
    });
});

Unfortunately, now we've lost the nice dynamic nature of our error messages. If we change the minlength rule to 3, we have to remember to edit the message for required. It seems we have gone from "feels hacky" to "feels hackier".

What I really want is for required to behave differently when I have the minlength rule applied to a select element. When using minlength on a select element, I want required to default to true but my error message to come from the minlength message format. This would feel far more intuitive for this use case than the current behaviour does. So, given that the plug-in does not do this, what can we do to fix it?

Implementing an intuitive minlength

The jQuery Validation plug-in provides the ability to add custom validation methods. Not only does this allow the addition of new methods, but it also allows for the replacement of existing methods. This gives us two options:

  1. Create a new minlengthofselection or similar that follows the rules we want.
  2. Override the existing minlength that does things the way we want.

While the first option is probably safest for those already familiar with the behaviour as it stands, option two is more fun. Guess which one I did3.

$(function () {
    $.validator.addMethod('minlength', function (value, element, param) {
        var length = $.isArray(value) ? value.length : this.getLength($.trim(value), element);

        if (element.nodeName.toLowerCase() === 'select' && this.settings.rules[$(element).attr('name')].required !== false) {
            // could be an array for select-multiple or a string, both are fine this way
            return length >= param;
        }

        return this.optional(element) || length >= param;
    }, $.format('Please select at least {0} things.'));

    $('#mySuperCoolForm').validate({
        rules: {
            things: {
                minlength: 2
            }
        },
        messages: {
            things: {
                minlength: $.format('Please select at least {0} things.')
            }
        }
    });
});

In this final update to the code, I provided a new implementation to the validator for minlength. This new version is almost identical to the default except that when checking a select element, it looks to see if the field is explicitly not required, thereby assuming that we want something to be selected. Finally, we have a minlength validation check that works intuitively for our scenario as well as the more common scenarios surround string lengths and we don't have to hard code numbers into the error message.

Conclusion

In conclusion, I ran into what is probably a corner case when it comes to using jQuery Validation where things were not as intuitive as I had hoped. In doing so, I was able to learn more about the plug-in and eventually, resolve my specific issue.

  • Have you had a similar problem?
  • Do you have improvements to suggest?
  • Should I submit this minor change for inclusion in the jQuery Validation plug-in?

Please, comment and let me know.

Acknowledgements

I would like to thank, alen, the author of this tutorial which gave me a foundation to work from when investigating and resolving this issue.

  1. Turns out the real reason was my expectations of what should work and the reality of what did work did not match, making the documentation hard to interpret. []
  2. Not in that way! []
  3. Hint: It's not the first one. []