You cannot load-test a real-time system the way you load-test an API

TL;DR: You cannot load-test a real-time system with a request-per-second tool, because there are no requests — there are N streams running continuously, and the interesting question is how many the box holds before something gives. Three things make it unlike API load testing: the spike that kills you is a simultaneous cold start, not peak traffic; the failure often builds over hours, so a thirty-minute ramp cannot see it; and the memory limit you set yourself fires before true demand, hiding the exact number you were trying to measure.

Somebody asks how many streams the server holds, and the reflex is to reach for the tool that answers that question everywhere else: generate load, ramp it, find the knee. It is the right instinct applied to the wrong shape of system, and it produces a number that is confidently wrong.

This is what we learned building the answer for an always-on video fleet after a capacity incident — what to generate, what to watch, and which of the failures only appear if you deliberately arrange for them.

Why doesn't a request-per-second tool work here?#

Because the load is not made of requests. An API's load is a rate: arrivals per second, each one short. A real-time system's load is a population: N streams, each running continuously, each holding a decoder, an encoder, a file handle, a socket and a slice of memory for as long as it exists.

That changes what a test is. There is no throughput to ramp — there is a fleet size to increase and hold. The knee you are looking for is not "at what rate do latencies rise", it is "at what N does something die, and which thing dies first". Requests per second is not a smaller version of that question; it is a different question.

Two practical consequences follow immediately. Steady state is cheap and misleading — a fleet at rest looks comfortable, which is exactly what the dashboard showed before our incident. And the services that matter are the ones that scale linearly with the fleet: one persistent process per stream is the pattern to count, along with the transient processes that fire per event on top of it. Anything that does not grow with N is co-load, not capacity — and watch for the things added later that quietly do grow with it, such as a per-stream transcription sidecar.

Per-stream cost is the input to all of this, and it is worth measuring separately before you test a fleet at all — what one stream actually costs, on CPU and against a hardware ceiling that has no gradient.

What is the spike that actually kills you?#

A simultaneous cold start — and it is the single most under-tested state in this class of system.

Configure N streams, then restart the service or reboot the machine. Every pipeline launches at once, and the transient goes far above steady state: N processes allocating, N connections negotiating, N decoders warming, all in the same few seconds. In our own incident this was the spike behind a fleet-wide outage, and the reason it went unnoticed for so long is specific and worth stating:

A gradual ramp does not reproduce it. If your test adds streams one at a time until it finds the knee, you have measured the steady-state ceiling and completely skipped the transient one — which is lower, and which happens in production every time anything restarts. Deployments cause it. Crashes cause it. A host reboot causes it. It is not an edge case; it is Tuesday.

So the test has to include the shape explicitly: bring up N at once, at several values of N, and watch the peak rather than the plateau. If your steady-state ceiling is 30 and your cold-start ceiling is 18, the honest capacity of that machine is 18.

Why does an overnight run find what a thirty-minute run cannot?#

Because the failure mode that gets you is often a slope, not a level.

Our production incident did not arrive instantly. Resident memory crept — a slow accumulation in a long-lived process, invisible at any single moment and fatal across a day. A twenty- or thirty-minute ramp step samples a level; it cannot see a gradient that needs hours to become a number.

So a capacity test needs at least one fixed-N soak measured in hours, and what you read from it is not the peak but the slope: memory against time, disk against time. A flat slope at a high level is a machine you can size. A gentle upward slope at a comfortable level is a machine that will fail on a schedule you have not calculated yet.

The same applies to disk. Continuous recording fills storage at a rate that is trivially predictable and routinely un-modelled, and cleanup jobs are themselves periodic spikes. A soak shows both; a ramp shows neither.

The measurement your own limits prevent#

This is the trap that cost us the most, and it is beautifully circular: you cannot measure a container's true demand from inside its own memory limit.

Set a limit, run the fleet up, and the kernel kills the container when its cgroup reaches that limit (cgroup v2 memory controller) — which tells you that the limit exists, not what the workload actually wanted. The number you are trying to discover is the peak demand, and the cap fires strictly before the workload reaches it. Every run produces the same answer: the limit. The workload's real appetite is never observed.

There is no clever way around it. To measure demand you have to raise or remove the cap for the measurement, on a stand rather than in production, and let the workload show you its peak. Then you set the limit from the measurement rather than from a guess — ours is measured peak × ~1.3, with alerting at about 80% of that, so the alert fires while there is still somewhere to go.

The general principle is worth carrying to any capped resource: a guardrail you set to a guessed value will report that guess back to you forever. If a limit was never derived from a measurement, every incident it causes will look like a workload problem rather than a configuration one.