Test Reporting

A test result that nobody sees does not change behaviour. The gap between a suite that runs and a suite that matters is mostly a reporting problem.

#JUnit XML, for machines

Every CI platform understands it, whatever produced it.

// Playwright
reporter: [
  ['junit', { outputFile: 'test-results/junit.xml' }],
  ['html', { open: 'never' }],
  ['github']                     // inline annotations on the PR diff
]

// Jest
reporters: [
  'default',
  ['jest-junit', {
    outputDirectory: 'test-results',
    outputName: 'junit.xml',
    classNameTemplate: '{classname}',
    titleTemplate: '{title}',
    addFileAttribute: 'true'     // lets the CI link a failure to a source line
  }]
]

pytest --junitxml=test-results/junit.xml
dotnet test --logger "junit;LogFilePath=test-results/junit.xml"
./gradlew test    # writes build/test-results/test/*.xml by default

addFileAttribute and its equivalents matter more than they look: without a file and line, the CI platform can only tell you a test failed, not where.

#Getting it in front of people

The most valuable single change: annotations on the pull request diff.

# GitHub Actions
- name: Publish results
  uses: dorny/test-reporter@v1
  if: always()
  with:
    name: Tests
    path: 'test-results/junit.xml'
    reporter: java-junit
    fail-on-error: true

# Playwright's own GitHub reporter does this with no extra action
reporter: process.env.CI ? [['github'], ['junit', { outputFile: 'test-results/junit.xml' }]] : 'list'

The difference between "the check is red, open the run, expand the job, read the log" and "there is a red marker on line 42 of the file you changed" is the difference between a failure fixed in ten minutes and one fixed tomorrow.

#Rich reports, for humans

# Playwright's HTML report — traces, screenshots and videos linked inline
npx playwright show-report

# Allure — language-agnostic, with history and trends
allure generate allure-results --clean -o allure-report

Allure is worth knowing about because it works across ecosystems: a Java service, a Python service and a TypeScript front-end can publish into one report with one vocabulary. Its history view — pass rate and duration per test across builds — is the feature to use it for.

// Allure annotations carry through to the report
@Epic("Checkout")
@Feature("Payment")
@Severity(SeverityLevel.CRITICAL)
@Link(name = "ENG-4412", url = "https://tracker/ENG-4412")
@Test
void aDeclinedCardShowsAnError() { }

#The three numbers worth tracking

Per test, over time — not per suite, per run.

Pass rate. A test at 100% for six months that has never failed may be proving nothing. A test at 80% is either finding a real intermittent bug or is flaky; both need attention.

Flaky rate. Passed-after-retry, as a proportion of runs. Above about 1% the team stops trusting red. This is the single most important test-suite health metric and most teams do not measure it.

Duration. The slowest ten tests usually account for a surprising share of the pipeline, and are usually fixable — a sign-in through the UI, a waitForTimeout, a fixture that rebuilds something per test.

# The quickest version of all three, from the JUnit XML
xmllint --xpath '//testcase[@time > 5]/@name' test-results/junit.xml

Platforms that give you this without work: Azure DevOps Test Analytics and TeamCity test history. Elsewhere, push JUnit results into a small database and chart them — a hundred lines of script that pays for itself within a quarter.

#Making flakiness fail the build

Most runners report a retried-then-passed test as a pass, which is how flakiness becomes invisible. Playwright distinguishes it; make the pipeline act on that:

- name: Fail if any test only passed on retry
  run: |
    flaky=$(jq '[.suites[].specs[]? | select(.tests[].status == "flaky")] | length' report.json)
    echo "flaky tests: $flaky"
    test "$flaky" -eq 0

Start with a threshold rather than zero if you have a backlog, and ratchet it down. A number that can only go down is a much easier thing to agree on than a rewrite.

#Notifications

Notify on transitions, not on every run. A message on every green build is noise that trains people to filter the channel — including the red ones.

// Jenkins: "unstable" means tests failed; "failure" means the pipeline broke.
post {
    unstable { slackSend color: 'warning', message: "Tests failed: ${env.BUILD_URL}" }
    fixed    { slackSend color: 'good',    message: "Back to green: ${env.BUILD_URL}" }
}

fixed is the underused one. Knowing that main is healthy again is worth a message; knowing it was healthy for the 340th consecutive build is not.