How we do generative testing in a JavaScript application

In my previous blog post, I briefly talked about generative testing (also known as property based testing). In this post, I’ll go into more detail about how generative tests work, and how to implement it in JavaScript. But first, a quick refresher:

What is generative testing?

Say you’re testing a function, squareRoot, which computes the square root of a number. The standard way of testing it would be to write something like this:

describe('squareRoot', () => { it('should return 2 when given 4', () => { expect(squareRoot(4)).toEqual(2); }); it('should return 3 when given 9', () => { expect(squareRoot(9)).toEqual(3); }); it('should return 0 when given 0', () => { expect(squareRoot(0)).toEqual(0); }); });

These tests all pass, and look pretty solid at first glance — we’ve checked that the function works as expected, and even tested the possible edge case of squareRoot(0). This method of testing is known as "example based testing". You specify examples of test data (in this case 4, 9, and 0), and check that the result of applying the function to that data is what you expect.

However, one problem with this approach is that you can miss edge cases. In the example above, we haven’t tested our squareRoot function for negative numbers, Infinity, NaN, or numbers with a non-integer square root. Also, that’s assuming it will only ever be given a number – what happens when it’s given a string? You can try to think of each possible edge case, but there are so many in JavaScript that chances are you’ll miss a few.

Generative tests don’t rely on example data, instead you specify constraints that your test data must obey, and then you verify that the output of the function obeys some other constraints. For example, in our squareRoot function, we have the constraint that the output squared should equal the input (i.e. squareRoot(x) ^ 2 = x), given that the input is a positive number. The test framework generates hundreds of possible inputs for the function, then verifies that the each output is correct.

Generative tests give you really good test coverage, and are much more effective at discovering bugs and edge cases than example based tests. They also force you to think more deeply about how your function should behave — for example, what should squareRoot return for negative numbers: undefined, null, NaN, or something else? Or is it the responsibility of the caller to only give it positive numbers? You need to think about these things before you can get your tests passing.

How does it work?

I’ll use Lee Byron’s TestCheck.js library as a guide to explain exactly how generative testing works. It’s a port of a Clojure library, test.check, so obviously it’s much higher quality than most js libraries (deal with it).

TestCheck.js has a check function, that you basically need to provide 2 things: one or more "generators" (basically functions that return random test data), and a predicate function which takes the generated values and returns a boolean. check will generate test data and run the predicate function a specified number of times. If the predicate returns true each time, then the test has passed. If it returns false even once, then the test has failed. Without further ado, here’s some code:

const squareRoot = Math.sqrt; const result = check( property( gen.number, x => { const sqrt = squareRoot(x); return sqrt * sqrt === x; } ), { numTests: 1337 } // options )

This will run a generative test that verifies the constraint we discussed above (squareRoot(x) ^ 2 = x). However, this test will actually fail, and result will be something like:

{ result: false, seed: 1510848781067, fail: [ -0.5 ], shrunk: { depth: 1, result: false, smallest: [ -1 ], totalNodesVisited: 2 }, failingSize: 0, numTests: 1 } }

This is has alerted us to the fact that our squareRoot function doesn’t work with negative numbers. There are a couple of interesting things about this result object:

• It gives you a seed value, so that you can easily re-create a failing test run if the test fails intermittently. You can pass this seed to check in the options map.
• If there is a failure, check will start a "shrinking" process – basically it will try to find the minimal set of test data that will cause the test to fail, to aid with debugging. This is where the smallest value comes from.

To fix the test, we have to think about how our function works i.e. what should it return when given a negative number? Because I’m lazy, let’s just make it the caller’s responsibility to only give our function positive numbers, and update our test to only generate (finite) positive numbers:

const result = check( property( // only positive numbers, excluding NaN and Infinity gen.posNumber.suchThat(x => Number.isFinite(x)), x => { const sqrt = squareRoot(x); return sqrt * sqrt === x; } ), { numTests: 1337 } // options )

However, this will still fail, due to floating point errors. This is unfortunately just a fact of life when dealing with floating point numbers in generative tests. One way around it is to change the predicate to something like:

x => { const sqrt = squareRoot(x); return Math.abs(sqrt * sqrt - x) <= x * 0.000001; }

We now have our first passing generative test!

Custom generators

There are lots of built in generators for generating strings, numbers, objects, etc.. These are enough for many situations, but often you need to be more specific about the shape of your test data. You can easily create custom generators by combining the built in generators, and by using .then or .suchThat (RTFM). One common need is to generate objects with specific keys, each with a specific type. You can easily do that like so:

const myCustomGenerator = gen.object({ id: gen.posNumber, name: gen.string, age: gen.posNumber.suchThat(age => age < 150) })

You could also use custom generators for things like address strings, UUID strings, and URLs.

I’m going to take this opportunity to shamelessly promote my own library swagger-testcheck which you can use to automatically create generators from a swagger definition. This cuts out a lot of boring work manually writing generators, and also helps you keep your generators up to date as your api changes.

Drawbacks of generative tests

Although I’m sure you’re all convinced that generative testing is the future, they do have a few drawbacks:

• Rounding errors — as we saw before, floating point rounding errors are a real pain when using generative tests.
• Maintaining custom generators — your custom generators can easily get out of sync with your code, and can be a headache to maintain. For example, if you have a function that takes an object with keys a and b, if you want to add a third key c to this object you will need to update the generator as well. If you fail to do this, it often causes failing tests, but not always.
• Execution time — generative tests take far longer than example based tests to execute, which can be a problem if you’re testing something computationally expensive. You can lower the number of test runs to compensate for this, but in doing so you increase the risk of missing bugs.
• Effort — generative tests are generally harder to write than example based tests, because they force you to really understand how the system under test should/does work. This is usually effort very well spent IMO, but not always.

Further reading

Generative tests give you much better test coverage than example based tests, and more importantly really force you to think about how the system under test should behave. They also force you to explicitly encode any assumptions you make about your function’s inputs/outputs, which means that reading generative tests gives you much more information than reading example based tests. They are not without their disadvantages though, so be careful.

Originally published at www.uvd.co.uk on November 20, 2017.