Mutation Testing

Coverage tells you a line executed. Mutation testing tells you whether anybody was watching.

The technique: make a small change to the production code — flip a comparison, remove a statement, swap an operator — and run the tests. If a test fails, the mutant is killed. If every test still passes, the mutant survived, and you have found a fault your suite cannot detect.

#Why it finds what coverage cannot

export function shippingCents(order: Order): number {
  if (order.subtotalCents >= 5_000) return 0;      // ← free over £50
  return 395;
}

it('is free over the threshold', () => {
  expect(shippingCents(anOrder({ subtotalCents: 10_000 }))).toBe(0);
});
it('is charged below it', () => {
  expect(shippingCents(anOrder({ subtotalCents: 1_000 }))).toBe(395);
});

100% line coverage. 100% branch coverage. And the mutation >= → > survives: neither test uses a subtotal of exactly 5,000, so nothing notices when the boundary moves.

That surviving mutant is the missing test:

it('is free at exactly the threshold', () => {
  expect(shippingCents(anOrder({ subtotalCents: 5_000 }))).toBe(0);   // kills it
});

Off-by-one at a boundary is among the most common real defects there is, and this is the only automated technique that reliably points at it.

#The tools

// stryker.config.mjs
export default {
  packageManager: 'npm',
  testRunner: 'jest',
  reporters: ['html', 'clear-text', 'progress'],
  coverageAnalysis: 'perTest',          // only run tests that touch the mutant
  mutate: ['src/pricing/**/*.ts', '!src/**/*.test.ts'],
  thresholds: { high: 85, low: 70, break: 60 },
  // Practical: mutate only what changed, so the run fits in a pipeline.
  since: { enabled: true, ignoreStatic: true }
};

npx stryker run

<!-- PITest, Maven -->
<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <configuration>
    <targetClasses><param>com.example.pricing.*</param></targetClasses>
    <mutationThreshold>70</mutationThreshold>
    <withHistory>true</withHistory>          <!-- incremental, much faster -->
  </configuration>
</plugin>

#Reading the report

src/pricing/shipping.ts
  Line 12  ConditionalExpression   >= → >      SURVIVED
  Line 12  EqualityOperator        >= → <      killed
  Line 13  BlockStatement          removed     killed
  Line 15  ArithmeticOperator      * → /       SURVIVED

Mutation score: 50% (2/4)

Two findings:

• Line 12, >= → > survived. The boundary is untested. Real gap.
• Line 15, * → / survived. Something arithmetic is not asserted at all — often a value the test computes rather than states literally.

A surviving mutant is either a missing test, a missing assertion, or code that does not matter. All three are worth knowing.

#Equivalent mutants

Some mutations do not change behaviour and therefore cannot be killed:

for (let i = 0; i < items.length; i++)     // mutated to i != items.length

Behaviourally identical for this loop, so it survives forever. Equivalent mutants are the main source of noise, detecting them is undecidable in general, and the practical response is to ignore a small number of known ones rather than chase a 100% score.

This is why a mutation score of 100% is not a sensible target. 70–85% on the code that matters is an excellent result.

#Making it affordable

Mutation testing runs your suite once per mutant. A 500-mutant run against a 30-second suite is four hours if done naively.

What makes it practical:

1. Mutate only what changed. Stryker's since, PITest's withHistory. This turns an overnight job into a two-minute one.

# GitHub Actions — on pull requests, changed files only
- run: npx stryker run --since=origin/main

2. Per-test coverage analysis. Run only the tests that execute the mutated line. coverageAnalysis: 'perTest' typically gives a 5–20x speedup.

3. Scope it to code where correctness is expensive. Pricing, permissions, tax, state machines, anything financial. Not the UI, not glue, not configuration.

4. Run it on a schedule for the full codebase, weekly, and act on the report.

#What a low score means

A module with 90% coverage and a 40% mutation score is a common and alarming discovery. It usually means one of:

• Tests that call code and assert nothing meaningful
• Assertions on the wrong thing — toBeDefined() where a value was intended
• Tests that assert on mock interactions rather than outcomes
• Boundaries tested at one side only

Each is a real weakness, and none of them is visible in a coverage report.

#When it is worth it

Yes: financial calculations, tax, pricing, permissions, state machines, parsers, anything where a silent wrong answer is expensive. Also excellent as a one-off audit of a suite you have inherited and do not trust.

No: UI rendering, glue code, configuration, generated code, and any codebase where the unit suite is so slow that a mutation run is measured in days. Fix the suite speed first.

Used well it is the only practical answer to "are these tests any good", which is a question coverage is routinely and wrongly asked.