How we approached automated testing for an Angular 4 app

We recently started writing a new app, an Angular 4 single page web app backed by a REST API. I’m going to be discussing our approach to testing the app, in general terms. I’ll talk about the different types of tests we use, and why we use them. I’m not going to go into detail about how to test Angular apps specifically, as this has already been done to death, but I will touch on it occasionally.

I’ll start with a bit of a disclaimer: testing is a controversial subject in software development. There are lots of blog posts out there, each with a different opinion on how best to test your software — and each one claims that the others are all wrong. This post is no different, so if you’re looking for an unbiased guide look elsewhere . You may disagree with our approach, but hopefully it will at least give you some food for thought.

So, without further ado, let’s kick things off by talking about unit tests:

Unit testing

The term “unit test” is a bit vague — a google search will give you several different definitions. The best one I could find is:

“A unit test is a way of testing a unit — the smallest piece of code that can be logically isolated in a system”

In our app, this “smallest piece of code” is usually an Angular component or a pure “utility” function.

Testing isolated units of code, as opposed to the entire system, gives you far greater code coverage. To explain why we’ll need to do a bit of maths (just a tiny bit). Let’s imagine we’re testing a system of n units, each with m possible states. Ideally, we want a test for each possible state of the system under test, which will give us 100% code coverage (by this definition). If we test each unit in isolation (i.e the unit is our system under test), we write m tests for each unit, giving us n*m tests in total. However, the system as a whole has m^n states – the combination of all states of each individual component. Therefore, even for small values of n and m, you get far more code coverage by writing unit tests. This is the fundamental reason why unit testing works better than system level testing.

There are other very good reasons to use unit tests: they’re fast, easy to write (especially if you’re testing pure functions), and allow you to easily pinpoint the source of any failures. Also, since by definition each test only depends on one unit, they’re far less brittle than system tests. For these reasons, we heavily prioritised unit tests over other types of test. We’re aiming to build a Testing Pyramid with a very wide base.

So we’ve established that unit testing is the way to go. But how do we actually write the tests? Example based tests are the most common way — if you’re testing a function, you just call the function with some example arguments, and check that the result is what you expect. For example, let’s imagine we have a function add which just adds 2 numbers. We might write the following test:

describe('add', () => { it('should add 2 numbers', () => { expect(add(1, 2)).toBe(3); }) )

However, this misses lots of edge cases, because it’s only testing one possible state. Each argument is a 64 bit number, therefore the function has (2^64)^2 = 3.40e38 possible combinations of valid inputs. Suddenly, writing a single test seems hopelessly inadequate, and indeed in this case we would have missed overflow errors. This brings us neatly on to:

Generative testing

Generative testing effectively allows you to automatically generate a large number of tests. It works like this: you generate some test data conforming to constraints that you specify, use that data to test your class/function, then validate that the result also conforms to some constraint. For example, say you want to test the function sort, which takes an array of numbers and returns the sorted array. We would generate a large number (say 1000) of random arrays, then use each one as the input to sort. We would then verify each output of sort by checking that the output array is sorted, the input and output arrays have the same length, each element of the input array is in the output array, and so on.

We used generative testing because it gives you orders of magnitude more test coverage, and so it’s great for finding edge cases. We decided to use the TestCheck.js library, a JavaScript port of Clojure’s Test.Check. TestCheck uses “generators”, functions that return random values conforming to some spec, to create the test data. You use these generators like this:

const { check, gen, property } = require('testcheck'); // Generator that generates integers const generator = gen.int; const result = check( property( generator, // predicate function to verify funcToTest x => funcToTest(x) === x + 1 ) )

We had to write some of our own generators for data structures returned from the backend API, but this was fairly painless.

Generative testing works best with pure functions, so we couldn’t use it to test every part of the app — unfortunately Angular relies too much on classes and mutable state. However, we did try to separate out as much logic into pure “utility” functions as possible (always a good idea), which we then wrote generative tests for. Also, if you squint hard enough, some Angular components can be thought of as pure functions, if they render based solely on their inputs and don’t rely on, or mutate, state (i.e. they are a “dumb” component). We also wrote generative tests for these components, which was not quite as easy as for pure functions, but still fairly straightforward. Generatively testing all our utility functions and dumb components allowed us to catch several bugs that almost certainly would’ve been undetected otherwise.

E2E testing

E2E tests are extremely slow and brittle, and require a huge amount of time to maintain. Also, when you get a failing test it’s often very difficult to pinpoint exactly what’s going wrong, and often the error is in the tests themselves as opposed to your code. However, they do allow you to check that all the different components/units in your app work together properly — they test the “glue” in between your components (e.g. Angular module configuration, routing, etc.).

Bearing all this in mind, we intentionally tried to keep the amount of E2E tests to an absolute minimum. There may be bugs that we miss because of this, perhaps due to browser “quirks”, but the effort required to catch those bugs is better spent elsewhere.

Our app relies on an API, which we needed to mock for our E2E tests. There are mock servers that have lots of functionality, like specifying a sequence of responses, but they necessarily add a lot of complexity — you have to keep the configuration, and state, of the mock server in sync with the actual tests. We’re not aiming to test every possible sequence of responses, so we used a static (a.k.a. stateless) mock server.

We initially ran our E2E tests in Chrome only, and manually tested in other browsers. This was very time consuming, so we looked into a way of running our E2E tests in multiple browsers. The main options were to run them on our CI system, or use a third party service. To set up all the different browsers on our CI system would’ve required a huge amount of work, and slowed down our build, so we decided to go with a 3rd party service. We went with BrowserStack Automate — it was very easy to switch over, just a few tweaks to our Protractor config file. Although we don’t have many E2E tests, running them in multiple browsers does give us far more confidence that the app will work in any browser.

Next steps

Although we think the app is already pretty robustly tested, there are a few potential improvements we may do in future:

• As mentioned above, we had to manually write the Testcheck.js generators for API response data structures. It would be nice to be able to automatically generate these, perhaps from the Swagger definition we’re using.
• RxJS observables are difficult to test generatively, due to their asynchronous nature. Luckily, our app doesn’t have much complex logic for dealing with observables at the moment. In future, we may separate any complex mapping/filtering/sorting logic into transducers, which can be tested independently of RxJS. This will also help make the code more generic, and re-usable.
• The more eagle-eyed readers among you will have noticed that I haven’t talked about integration tests at all. We currently don’t have any, mainly because they aren’t set up by default in projects generated using the Angular CLI. They do share many of the same disadvantages as E2E tests, albeit to a lesser extent, but nevertheless we will probably add some as our application grows.

I hope you’ve enjoyed this blog post, and it’s given you something to think about. If you’d like to explain to me why I’m wrong, please leave a comment below.

Originally published at www.uvd.co.uk on June 26, 2017.