One of the biggest challenges of software development is doing things at scale. A simple website with few pages can be built from scratch and be online in a few hours. A website that serves hundreds of thousands of pages to millions of visitors every day is a much more complicated challenge.
This is a story I wrote several years ago about some work I did on GOV.UK. It's about architecture, technical debt and planning ahead. It's also about some very specific JavaScript problems that I had to solve.
A bit of background
GOV.UK is a large and complicated site. I work on GOV.UK Publishing, which is any part of GOV.UK that begins with the address www.gov.uk. Most of the pages we serve are non interactive - that is, text content. We provide information. If you need to do something that involves a transaction like applying for a visa or renewing your passport, that's handled by a separate service managed by another department.
GOV.UK Publishing isn't a single thing. It consists of a number of separate applications that are each responsible for serving a different part of the site. For example, the homepage is handled by one application, while pages for government departments are handled by another. To prevent repetition of code, particularly frontend code, we have an application called Static that assembles the page layout for every application. This is a slightly old approach that we're trying to change.
Each application also has dependencies in the form of packages both from inside and outside of GOV.UK. One of the most significant of these is govuk_publishing_components, our gem that provides frontend components such as buttons and form elements and other shared code. This is our (relatively) new way of sharing frontend code across applications, intended to replace much of what Static does.
GOV.UK Accounts
In 2020 I joined a team that was building an account system on GOV.UK. Most of our effort went into building a new, separate application that would allow users to sign into an account on GOV.UK. This application served the account pages, including the sign in and out pages and other pages where you could manage your account. This was a new application and we tried to make it as standalone as possible. This meant that unlike our other applications it did not rely on Static.
Signing into an account was a new thing and we were keen to understand it as much as possible. One of the best tools for that is analytics, so we needed to get some tracking included to understand how users interact with accounts.
Analytics
Like many websites we use analytics to understand user behaviour. Specifically, we use Universal Analytics, a tool from Google that stores anonymous information about visitor behaviour and allows performance analysts to analyse this data to help improve the site.
It reports lots of useful things, like the most visited pages and what browsers are used, so we can ensure that any changes we make work properly for everyone. We only store this data if visitors consent to being tracked and we don't store any personal information or anything that could identify individual visitors.
Adding basic Universal Analytics to a website involves only a few lines of code, but we have a lot of extra custom code to provide more specific tracking. There are also code tests in place to ensure all of this works correctly, as well as some documentation. All of this amounts to nearly five thousand lines of code.
The analytics code was originally provided by GOVUK Frontend Toolkit, a now deprecated dependency. A few years ago I migrated the analytics code from Toolkit into Static, so that changes could be made. My original intention at the time was to migrate it into the components gem, as we're slowly moving Static towards retirement. Unfortunately I encountered some unexpected technical hurdles, so only migrated it to Static as an interim step.
Since the GOV.UK Accounts application didn't use Static, we needed to find another way of getting analytics into the application.
Getting analytics into Accounts
There were a number of options for getting analytics into the GOV.UK Accounts application - using Toolkit, using a copy of the code from Static, writing new analytics code. I decided that the best option was to complete what I'd started a few years ago and finish migrating the analytics code into the components gem.
This represented not only a significant piece of work but also a potential risk for the existing analytics. If not handled properly, it could break all tracking across the site. To reduce this risk I started by making a copy of the analytics code from Static in the components gem that only Accounts would use, so that if it didn't work for some reason at least nothing else would be affected by that change.
This was also useful because it turned out I needed to modify the analytics code. The version in Static used hard coded values for the GA property - the unique code that every site needs in order to connect to Universal Analytics - but the accounts application needed to use a different one. I rewrote the initialisation of the analytics code so that the GA property could be passed to it, rather than being hard coded.
Now we had two versions of the analytics code - one in Static, and one in the components gem with this minor modification. The plan had worked and the GOV.UK Accounts application was launched on time with analytics applied.
Having duplicate code isn't great, and having five thousand lines of duplicate code is even worse. I knew this and the risks involved from the start, but it was still surprising that only a few weeks later two changes were made to the original analytics code in Static by other developers. Now we had two diverging branches of code that did the same thing. It was time to deal with this tech debt and finish the migration.
Problems, problems, problems
The first thing to solve was a problem I haven't even mentioned yet. When I copied the analytics code into the components gem it didn't work right away. The test suite reported many failures, not only failures with the new tests but also some failures in the existing ones. Because this was a copy of the analytics code and I knew we weren't using the failing parts of it for Accounts, I'd disabled these tests, but now I needed to re-enable and fix them.
The failures in the existing tests were caused by the sudden existence of the analytics code objects and functions, which the component gem tests weren't expecting. A lot of our frontend components make some kind of call to our analytics code, so those tests were mocking the required analytics objects to check they were being called at the right time, then cleaning up after themselves by deleting those empty objects. Since the tests are always run in a random order to avoid race conditions and the real analytics code was now being initialised by each of the analytics tests, errors occurred. Imagine playing pass the parcel where someone randomly eats the parcel.
The fix was to remove all of the mocked analytics objects from tests and instead use the real ones, add checks where appropriate for the existence of various objects before creating them, reset calls to the analytics code at the end of tests that used it, and move the initialisation of the analytics code to the very start of the tests, using a beforeAll function. This would also more accurately simulate the production environment.
JavaScript test failures both confusing and intermittent
Unfortunately, that didn't solve all the problems. It turned out that some of the existing tests were still not cleaning up after themselves properly, leaving changes in the URL or stray elements in the page, which were them causing other tests to fail.
Fixing these problems still didn't solve all of the failing tests. Some of the analytics tests were failing intermittently, which suggested that other tests still weren't cleaning up after themselves properly. Making this more difficult was the frequency of the failure - sometimes the tests had to be run twenty times or more for the right (or rather wrong) sequence to occur. Weirder still, even repeating that exact failing sequence of tests didn't always produce the error.
The first intermittent failure was in the analytics ecommerce tests. The cause of this turned out to be another test not cleaning up after itself properly - a stray element was left in place after the test had run, which affected the ecommerce tests. The thing that was interesting about this was that the stray element was being removed before the start of the other tests, but not after, which meant the test was cleaning up for itself, but not for any tests that followed it - hence the intermittent fail.
The analytics external link tracker test was also failing intermittently. This was a mystery. The external link tracker didn't do much - it fired an analytics event when a link was clicked, using pattern matching to first determine if the link was to a different site.
I added some console output and the problem got even weirder - it seemed like certain events were firing more than once, sometimes dozens of times. The first time the event fired the results were as expected, but following events were incorrect, causing the test to fail. The problem turned out to be annoyingly simple. The external link tracker was checking for link clicks by setting a listener on the body element, delegated from link elements that matched a regex for external links. In production, this listener was being initialised once, but in testing it was being initialised multiple times by other tests.
This meant that the listener started by the external link tracker test was working correctly, but all of the other listeners created after it were not. Every time a test that used this was run the listener was created and persisted - imagine clicking on a link and finding that it opened a new website dozens of times. Removing that listener before each of the external link tracker tests allowed the tests to pass.
One more failure
One more test failed because the part of the expected analytics output included the automatically detected device pixel ratio of the screen. This passed when the test was run from the command line but failed when run in a browser, because the expected output was hard coded as 1 but my screen's pixel ratio was different.
The solution was to add a check to get the actual device pixel ratio. It's possible this problem had been present for a long time but never noticed, because most of the time these tests are run using a headless version of Chrome.
Final cleanup and migration
At last, all of the failing tests were fixed. The task ahead was to switch the main site from using the analytics code in Static to using the new version in the components gem. There were some last minute tasks to do including copying across the changes made to the Static analytics code since the code diverged, and updating how Static initialised the analytics code to match the recent changes. Then everything was ready.
The switch over went very smoothly. For all the effort that went into making this change, the outcome was relatively disappointing - nothing really changed. There was no user impact, no big announcement. Everything worked as expected. We did the hard work to make it simple.
So what difference has all this made and what's next? A large chunk of code has moved from one place to another. It's now a bit more robust from the process and we understand it a bit better. A thing has been removed from Static, which makes it smaller and slightly easier to understand, as we work towards simplifying and possibly retiring it.
There's still much to do - a lot of this code is old and undocumented, and the various moves over the years mean that it's not structured as well as it could be - we have an ongoing task to 'de-spaghetti-fy' it. But it's now in a place where we can look to do that, and that work will continue.
Things to learn, probably
This has been a lengthy read, but it still doesn't convey the hours, days and fistfuls of hair that I expended on this work. It felt like I'd climbed a mountain, and fallen off it several times during the process. Test failures can be baffling, but intermittent failures are the worst.
So after all this rambling narrative, what can we take away?
- Anything involving code gets harder at a larger scale.
- Code testing is essential.
- Combining two large codebases can cause unexpected problems, including new problems that never occurred before.
- Sometimes a code test can pass even if problems are still present, like cleanup not happening entirely correctly. This might not be a problem now, but it could be later.
- Sometimes intermittent test failures only happen very infrequently.
- Despite your best efforts, the environment in which you test your code might not be exactly the same as the production environment where it runs.
- JavaScript can be really, really complicated sometimes.
- I need a holiday.