Flaky Tests

A flaky test fails without the code changing. Its real cost is not the minutes lost to reruns — it is that it teaches the team that red does not mean broken. Once that is learned, every genuine failure is one more thing to rerun, and the suite has stopped being a gate.

#The six causes

#1. Waiting for the wrong thing

Far and away the most common. The test asserts before the application has finished, and whether it passes depends on machine load.

// Flaky: a guess about how long something takes.
await page.click('#save');
await page.waitForTimeout(1000);
expect(await page.locator('.toast').textContent()).toBe('Saved');

// Stable: an assertion that retries until the condition holds or it times out.
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByRole('status')).toHaveText('Saved');

The rule: wait for a condition, never for a duration. Every modern tool supports this — Playwright's web-first assertions, Cypress's retry-ability, Selenium's WebDriverWait with an expected condition, waitFor in Testing Library. A sleep in a test is a bug report waiting to be filed.

The subtle version of this bug is asserting on a value read once:

// Still flaky: isVisible() reads the DOM one time, right now.
expect(await page.locator('.toast').isVisible()).toBe(true);

// Stable: the locator is re-evaluated until it is true or the timeout hits.
await expect(page.locator('.toast')).toBeVisible();

#2. Test interdependence

Test B passes only because test A ran first and left something behind. Invisible in sequence, fatal under parallelism or sharding, and the failure appears in the wrong test.

Detect it by shuffling the order and by running the suite twice in the same container.

#3. Shared state and resource contention

The same account, the same row, the same port, the same file. Covered in parallel test execution and test data management; the cure is uniqueness by construction.

#4. Time

Tests that break at midnight, on the last day of a month, during a daylight-saving transition, or on a leap day. Tests that assume "today plus one day" is 24 hours later. Tests that compare a timestamp generated twice.

// Flaky twice a year and on 31 January.
var nextMonth = DateTime.Now.AddMonths(1);

// Deterministic: inject the clock and pick the date deliberately.
var clock = new FakeClock(new DateTime(2026, 1, 31, 12, 0, 0, DateTimeKind.Utc));

See dependency injection.

#5. Real networks and third parties

A sandbox API that is slow at 09:00 UTC, a CDN that occasionally 503s, a DNS lookup that takes four seconds. Anything outside your process is a source of non-determinism.

Cure: do not touch third parties in the functional suite. Use stubs. Keep a separate, small, non-blocking suite that exercises the real integration on a schedule, so you still learn when a provider changes something.

#6. Genuine race conditions in the application

The rarest cause and the most valuable finding. The test is not flaky; the product is, and the test is the only thing that noticed.

Before dismissing a flaky test, check whether it is telling you about a real concurrency bug. Double submits, optimistic-concurrency conflicts and duplicate message processing are all real defects that first show up as "a test that occasionally fails".

#Detecting flakiness on purpose

Do not wait for it to annoy someone. Look for it.

# Playwright: run the suite ten times and report anything that is not stable.
npx playwright test --repeat-each=10 --workers=4

# Just the suspect, many times:
npx playwright test checkout.spec.ts --repeat-each=50

# pytest
pytest --count=20 tests/integration        # pytest-repeat

# JUnit 5
@RepeatedTest(20)

A nightly job that runs the suite five times and reports any test that did not pass all five is a couple of hours of work and finds flakiness before your colleagues do.

#Measure it

If you cannot count flakiness you cannot manage it. Configure the runner to distinguish passed from passed after retry:

// playwright.config.ts — retried-then-passed is reported as "flaky"
export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  reporter: [['html'], ['json', { outputFile: 'results.json' }]]
});

Then track one number: flaky runs as a percentage of all runs. Above about 1% the team stops trusting red. Publish it where people see it.

#Quarantine, with an expiry date

Sometimes a test cannot be fixed today. Quarantine is better than a retry loop, provided it has the two things a skip usually lacks:

// An owner and a date. Without both, quarantine is deletion with extra steps.
test.fixme(
  'exports a PDF of the monthly statement',   // flaky: PDF worker race
  { annotation: [{ type: 'issue', description: 'ENG-4412, owner: @sam, review 2026-10-15' }] },
  async ({ page }) => { /* ... */ }
);

Fail the build if a quarantined test is older than 30 days. The point is to make quarantine a decision with a clock on it rather than a wastebasket.

#Diagnosing the one you have

1. Reproduce it       --repeat-each=50, ideally under load
2. Capture it         trace on-first-retry; screenshot on failure
3. Read the trace     what was on screen at the moment it gave up?
4. Classify it        which of the six causes above?
5. Fix the cause      not the symptom — a longer timeout is not a fix

Step 3 is where a trace is worth more than every other artefact combined: it shows the DOM, the network and the console at the exact moment of failure, which is usually enough to see that the button was present but disabled, or that a request 500'd and the UI never recovered.

A longer timeout deserves its own warning. It converts a fast failure into a slow one and moves the flake further into the future. Occasionally the timeout really was too short for a loaded CI machine; far more often it is hiding one of the six causes.