Levels of Testing

Every automated test answers a question, and the useful way to sort tests is by the size of the question they answer. A unit test asks does this function do what it says. An end-to-end test asks can a person actually buy something. Both are worth having. They are not substitutes, and the most expensive mistake in test automation is treating them as if they were.

#The trade every level makes

Move up a level and your test covers more, in exchange for three things:

• Speed. A unit test runs in microseconds. An end-to-end test runs in seconds, and a suite of them in tens of minutes.
• Determinism. Every real component you include is a component that can be slow, unavailable, or in a state you did not arrange.
• Diagnosis. A failing unit test names the function. A failing end-to-end test tells you that checkout is broken, and then you go and find out why.

Move down a level and you buy all three back, in exchange for the only thing that actually matters to a user: the confidence that the pieces work together. Nothing at the unit level can tell you that the ORM emits the query you think it does, that two services agree on a field name, or that the button is reachable.

#The levels, in order

• Unit testing — one unit, no I/O
• Unit testing with mocks — the solitarist position, and where it goes wrong
• Unit testing without mocks — the sociable position, and where it goes wrong
• Test doubles — mock, stub, fake, spy and dummy, used precisely
• Integration testing — your code against a real dependency
• Integration testing with stubs — your code against a dependency you control
• Contract testing — proving two services still agree without deploying both
• Component testing — one deployable, in isolation, through its real interface
• End-to-end testing — the whole system, driven the way a user drives it
• The testing pyramid — how many of each

#How to decide where something belongs

Ask what would have to be wrong for this test to fail, and whether that is the smallest thing that could be wrong. A test that fails because a discount was calculated incorrectly belongs at the unit level. A test that fails because the discount column is DECIMAL(4,2) and the number does not fit belongs at the integration level. A test that fails because the discount is correct everywhere but never rendered belongs at the end.

Most teams that feel their suite is unreliable have tests sitting one or two levels higher than the risk they were written for. The fix is almost never a better tool; see the testing pyramid and flaky tests.