How we simplified our frontend build

By: Dave Martin

In the run up to Christmas, as things were winding down for the year, we had a bit of free time here at UVD. So of course, we did what anyone would do — start re-writing our frontend typescript build system. “How hard can it be?” we said.

At the time, we were using JSPM, and had a number of problems with it:

• It was very slow. Each page refresh effectively builds the whole site, and was taking about 15 seconds.
• It’s not trivial to learn — anyone new to JSPM has to learn about the different registries, how SystemJS loads your files, the loader config, the builder…
• Our app bundle was pretty big, at 1.5MB (no judging please). We were sure we could reduce this.
• The latest version, 0.17, has been in beta since January 2016 (a year at the time of writing), and still has some bugs.
• Karma was tied into JSPM via the karma-jspm plugin. Ideally, our test runner wouldn’t know anything about our build system.
• We were using NPM for dependencies of our Gulp tasks and NPM scripts, and JSPM for all other dependencies. We wanted our dependencies all in one package manager.

The main problem, and the cause of some of the issues above, was that JSPM is too complex (i.e. “consisting of many different and connected parts”). The way we were using it, it encompassed a package manager (with multiple registries), module loader, bundler, minifier, transpiler… the list goes on. This means that when you need to add another step to your build system (e.g. Typescript compilation), you need to add a JSPM plugin rather than use a completely separate tool. These plugins are of varying quality and, by their very nature, entwine JSPM with another tool. This makes the build system more complex and error prone as a result.

To give an example of why this is bad, previously we tried to add sourcemaps to our unit tests, and found ourselves changing our SystemJS loader config to get plugin-typescript to work with karma-sourcemap-loader. It proved more hassle than it was worth, and we were forced to give up with it.

Migrating from JSPM

So we knew we wanted to use something different, but what? Webpack is pretty much the de-facto standard in the JavaScript community, and has some nice features like bundle splitting and hot module reloading. However, it has the same basic problem as JSPM — it’s very complex. You can in theory use it as just a simple module bundler, but it’s very rarely used like this in practice. It’s more commonly used as an entire build system (see here and here), and IMO the Webpack community reflects this.

Instead of a monolithic build system like Webpack, we wanted to use a collection of simple (i.e. “Composed of a single element; not compound”) tools, which each have one job. This would allow us to far better understand our build system, and also give us much more flexibility to alter it if needed. We hoped it would also prevent situations where we have to alter one bit of our build system to get another bit working, like changing the SystemJS loader config to get unit tests passing.

We decomposed our overall build system into 5 distinct parts: package management, transpilation, bundling, minification, and testing. We chose NPM as the package manager, because we were already using it alongside JSPM. The Typescript compiler handles all our transpilation from ES6/typescript to ES5 (no babel required). For minification, UglifyJS seemed like the natural choice, and it has a nice command line interface.

We chose Rollup as the module bundler because it uses the ES6 module standard, it’s very simple by default, and also it can perform tree shaking to reduce the bundle size. We did have to add a few plugins (six) which proved a pain point, but unfortunately this was unavoidable due to the way Rollup is architected.

That leaves us with testing. We were originally using Karma, and our first instinct was to stick with it. However, we soon realised that Karma required knowledge about the package manager, transpiler and bundler and we wanted to avoid this. Mocha seemed like a better solution — it wouldn’t have to know about Rollup or NPM (directly), as Node would handle the dependency resolution. It would still need to transpile the Typescript to CommonJS modules, for which we used ts-node, which effectively makes Node a Typescript runtime.

Since our app uses AngularJS, which depends heavily on the DOM, we had to mock the browser APIs. JSDOM is perfect for this, and gives us the added benefit of having to explicitly specify any browser APIs the app uses — with Karma you implicitly depend on the APIs of the browser you run your tests in. Moving to Mocha was very straightforward, with very little configuration needed, although there were a few issues with CommonJS imports (more on that later). A full test run was also much faster, since we no longer had to wait for the Karma server to start up, and for it to open Chrome.

What went wrong?

Unsurprisingly for such a big undertaking, we did encounter several issues along the way:

• We initially tried to use Gulp to compile typescript, and pipe it into Rollup. This would break apart transpilation and module bundling. However, we found that the gulp-typescript plugin was loading files from disk rather than from the Gulp stream, so we abandoned this idea. Instead, we used the Rollup CLI the rollup-typescript plugin, which isn’t ideal but gets the job done.
• The biggest pain was how ES6 and CommonJS modules interact. We’re not alone in this — see here, here and here. We found that some imports worked correctly in our app, but not in our tests, or vice versa. The problem was due to the different ways Rollup and the Typescript compiler handle default exports. The solution was to add module.exports.default to the CommonJS module, as in this PR.. This obviously isn’t ideal, and is something we hope is fixed in future versions of Rollup.
• Rollup sometimes can’t figure out the exports of a CommonJS module, so you have to explicitly specify them. This is a bit of a pain for libraries like Lodash, but hopefully will become less of an issue as more libraries support ES6 modules.
• When we first ran Rollup, it was trying to import lots of node builtins, such as fs and http. We eventually figured out it was socket.io that was importing them. Basically, the socket.io-client module contains the code:

var WebSocket = BrowserWebSocket;
if (!WebSocket && typeof window === 'undefined') {
  try {
WebSocket = require('ws');
  } catch (e) { }
}

• Rollup doesn’t realise that typeof window === 'undefined' will always be false in browsers, and so imports the ws node library, which in turn imports lots of Node builtins. The (slightly hacky) solution we found was to use rollup-plugin-replace to rewrite the if condition to if (false), which allows Rollup to figure out that ws should not be imported. Hopefully this issue will disappear as Rollup’s static analysis improves.
• We spent a fair amount of time shuffling around the order of Rollup plugins to get them to work together. The Rollup contributors are aware of this issue, and it should be alleviated in future versions.

Benefits

• Test runs are faster — down from 55s to 40s
• The bundle size is down from over 1.5mb to under 1.3mb (about 15% smaller)
• Faster builds — down from around 15 seconds to under 10 seconds using rollup-watch
• Dramatically less configuration needed
• We now have a more modular build process
• The overall build process is easier to understand

Summary

We gained some nice benefits from moving away from JSPM. Personally, I think the biggest one is that our build system is now much simpler, and much easier to understand and reason about. It’s still not perfect, and there were a few speed bumps along the way, but overall it was definitely worth it. Also, my psychiatrist says the damage isn’t permanent.

If you’re currently using a monolithic build system like JSPM or Webpack, I’d highly recommend you think about how you can make your build system simpler. It may take some work, but it’s worth it in the long run.