Evidence & Diagnostics

The test failed on a machine you cannot reach, in a browser that no longer exists, eleven minutes into a run that took twenty. What have you got?

If the answer is a stack trace and a timeout message, you are about to spend an afternoon trying to reproduce it locally — and if it is flaky, you will fail. The gap between "the build is red" and "I know why" is almost entirely a function of what the run recorded while it was still happening.

#What to record, and when

Always, on failure: a screenshot at the moment of failure and the full page HTML. Both are nearly free and both answer the most common question, which is "what was actually on screen".

On failure, for anything non-trivial: a trace — Playwright's is the best of these by a distance. A trace is a recording of every action, every network request, every console message and a DOM snapshot at each step, and it turns "it timed out waiting for the button" into "the button was there, disabled, because the previous request 500'd".

On retry: a screen recording. Video is large and mostly redundant with a trace, but it is the only artefact that shows timing and animation, and it is the one you can put in front of somebody who does not use the test tooling.

Always: a test report in a format the CI platform understands, so failures appear in the pull request rather than inside a log file somebody has to open.

#The rule

Configure artefacts so that a failure is diagnosable on the first look and a success costs nothing. Every modern runner supports exactly this — screenshot: 'only-on-failure', trace: 'on-first-retry', video: 'retain-on-failure' — and the default configuration of most projects does not use it.

This section is short because the topic is simple and almost universally under-done. The cost of getting it right is one afternoon; the saving is every debugging session after that.