Load, Stress and Soak Testing

Performance testing asks how fast the system is. Load testing asks what happens when you push it — and specifically, what breaks first.

That answer is more useful than a latency number, because it tells you what to fix and what to buy.

#Four tests, four questions

Most teams run the first and call it done. The second and fourth find the incidents.

#Load profiles

// k6 — four scenarios in one file, run one at a time by name.
export const options = {
  scenarios: {
    // 1. Load: the traffic we expect, held steady.
    load: {
      executor: 'constant-arrival-rate',
      rate: 200, timeUnit: '1s', duration: '15m',
      preAllocatedVUs: 200, maxVUs: 1000
    },

    // 2. Stress: climb until something gives.
    stress: {
      executor: 'ramping-arrival-rate',
      startRate: 50, timeUnit: '1s',
      preAllocatedVUs: 200, maxVUs: 3000,
      stages: [
        { target: 200, duration: '5m' },
        { target: 500, duration: '5m' },
        { target: 1000, duration: '5m' },
        { target: 2000, duration: '5m' }
      ]
    },

    // 3. Spike: what a marketing email looks like.
    spike: {
      executor: 'ramping-arrival-rate',
      startRate: 50, timeUnit: '1s',
      preAllocatedVUs: 100, maxVUs: 3000,
      stages: [
        { target: 50, duration: '2m' },
        { target: 1500, duration: '30s' },   // the email goes out
        { target: 1500, duration: '3m' },
        { target: 50, duration: '2m' }       // does it recover?
      ]
    },

    // 4. Soak: modest load, long enough for slow problems to appear.
    soak: {
      executor: 'constant-arrival-rate',
      rate: 100, timeUnit: '1s', duration: '8h',
      preAllocatedVUs: 150, maxVUs: 400
    }
  }
};

k6 run --env SCENARIO=stress script.js

#Coordinated omission

The most important measurement error in load testing, and the reason many homegrown tests are optimistic by a factor of ten.

With a fixed pool of virtual users, each waits for its response before sending the next request. When the system slows down, the test slows down with it — so the offered load falls exactly when it should be rising, and the recorded latencies exclude all the requests that would have been sent during the slow period.

// Wrong: 100 VUs in a loop. Offered load drops as the system degrades.
export const options = { vus: 100, duration: '10m' };

// Right: requests are issued at a fixed rate regardless of response time.
export const options = {
  scenarios: { steady: {
    executor: 'constant-arrival-rate',
    rate: 200, timeUnit: '1s', duration: '10m',
    preAllocatedVUs: 200, maxVUs: 2000
  }}
};

If the runtime reports that it could not allocate enough VUs to maintain the rate, that is not a configuration warning — it is the finding. The system could not keep up.

#What to watch while it runs

The client-side numbers tell you that it broke. The server-side ones tell you why.

• Saturation: CPU, memory, disk I/O
• Connection pools: the most common bottleneck in web applications, and the one that produces the sharpest cliff
• Database: active connections, lock waits, slow query log, replication lag
• Queue depth, if anything is queued
• GC pauses, on a managed runtime
• Errors by type — timeouts, 5xx, connection refused — which usually identify the failing component directly

The shape of the failure is the diagnosis:

Latency climbs linearly, errors flat        → a saturated resource, gracefully queueing
Latency flat, then a vertical cliff         → a pool or a limit was exhausted
Errors spike, latency stays low             → something is failing fast (circuit breaker, or a crash)
Latency climbs over hours at constant load  → a leak. Soak test found it.

#Soak testing

The one most often skipped, and the one that finds the problems that wake people up:

• memory leaks, which need hours to become visible
• connection pool exhaustion from a path that leaks one connection per thousand requests
• unbounded caches
• log files filling a disk
• token or session expiry not handled on a long-lived client
• a scheduled job that interacts badly with sustained load

Run it for eight hours against a production-like environment, overnight, weekly. Plot memory and connection counts over the whole window — the shape of the line is the result, not the final number.

#Test data volume

A load test against an empty database measures the wrong system. The query that is 4ms against 1,000 rows and 4 seconds against 10 million is the single most common production performance incident, and no amount of concurrency against a small dataset will reveal it.

Generate production-like volume and distribution — never copy production data, for the reasons in test data management — and include the skew: the one customer with 50,000 orders is the one whose page times out.

#Environments

Load testing needs an environment shaped like production. A test against a single container on a laptop tells you about the laptop.

Terraform for test infrastructure covers provisioning one on demand and destroying it afterwards, which is the only way this stays affordable.

#In production, carefully

Increasingly common, and reasonable with guard rails:

• Shadow traffic. Mirror real requests to a new version; discard the responses. Real load, real distribution, no user impact.
• A small synthetic percentage, clearly marked, with isolated data, and an automatic stop when error rates rise.
• Progressive rollout with real traffic — a canary — which is load testing with the safest possible profile.

A full stress test in production, without a stop condition, is how you create the incident you were trying to prevent.