Skip to content

// Media & Streaming × Real-Time Ops

Monitoring and incident response for an always-on camera fleet

What to watch on an always-on video fleet: the encoder session count rather than GPU utilisation, the degradation that precedes failure, and the cold-start ceiling a ramp test never finds.

Start with a Sprint

// The problem

Why this is hard

A camera fleet does not go down so much as it goes wrong and keeps running. That is what makes it expensive: the same failure on a request-shaped system pages somebody within a minute, and here it bills quietly for weeks.

The reason is that the reassuring numbers are the ones that do not bind. A hardware encoder's ceiling is a session count, not a load, and the two are unrelated — a box can be out of sessions while its GPU reports single-digit utilisation, which is exactly the reading that says there is room for ten times more. Cross that ceiling and the work quietly moves to the CPU: no error, no log line, no change in what viewers see. The only thing that moves is the bill.

And the spike that actually takes the fleet down is not peak viewership. It is every camera reconnecting in the same second — after a deploy, a switch reboot or a power event on the site — so the fleet meets its own transient ceiling rather than its steady-state one. A test that adds streams one at a time finds the steady-state ceiling and skips the transient one entirely, which is the lower of the two and the one production meets on an ordinary Tuesday.

// How we do it

The approach for this fit

Alert on the session count, not on utilisation

`nvidia-smi encodersessions` lists what is actually open; the cap is per system rather than per card on consumer hardware. That count is the alert. Utilisation stays on the dashboard, but it never fires anything - on our own box it reads 8% with a single encode live, and nothing about that dial tells you seven sessions are all you have left.

Catch the fallback the hour it happens

A stream that drops to software encoding produces identical output and costs 2.8x the CPU. Nothing errors, so the tell has to be watched for directly: the session count falling while the stream count does not, and cores-per-stream stepping up.

Alert on degradation, not on a threshold

On a fleet that records as well as publishes, the disk gives you the most warning of anything on the box: write latency climbs while throughput still looks fine, and page cache starts losing the pages the encoder is about to want. That is minutes of notice before anything errors, and it is worth more than any level set on the resource itself.

Re-measure the cold-start ceiling

Bring N streams up at once, at several values of N, and watch the peak rather than the plateau. That transient ceiling is lower than the steady-state one, it is what a deploy meets, and it moves every time a per-stream service is added.

// Proof

Measured on our own hardware

CPU cost of a silent fallback

2.8×

on an i9-13900HX with ffmpeg 7.1.1, 2026-08-04: swapping only the encoder, NVENC out and libx264 in, decode left on the CPU in both arms — 2.74-2.84x across five interleaved A/B pairs at 1080p60. A ratio, not an absolute - the absolutes drifted ~10% with package temperature and are not published.

GPU utilisation, one live encode

8%

on our own production box, re-verified 2026-08-01, with one live 1080p59 encode running and seven sessions still free. An alert wired to this number cannot fire in time: it is not a load climbing toward a limit, it is a dial that barely moves while a separate counter is spent. Alert on the counter.

of video, queue bounded in frames

50 ms

at 59.94 fps; the same setting holds 100 ms at 30. A buffer tuned at development framerate runs too shallow in production and drops video while audio keeps flowing - a degradation no resource threshold reports.

Not a client outcome and not a vendor figure — numbers we took, on hardware we can name, with the method published in full. The write-ups: How many cameras fit on one box — and the ceiling that hides until you cross it, Video not publishing, audio fine — the bug that only appears at production framerate.

// FAQ

Common questions

The counters that reach a ceiling: open encoder sessions against the cap, cores and memory per stream against a measured peak, and cost per minute. Plus the early signals - write stalls and rising latency - because on this class of system they arrive minutes before the failure does. Utilisation is recorded and never alerted on.

By watching for it, because nothing reports it. The output is identical and no error is raised; what changes is that an encoder session closes while the stream keeps running, and that stream's CPU cost multiplies by about 2.8x. Both are visible if something is looking at the session count and at cores-per-stream, and invisible otherwise.

Read access to the metrics and the host or container the media processes run on, which is enough to establish which counters bind on your system. The measurements that need a raised memory limit or a cold start at several fleet sizes run on a stand rather than in production - a stand that has to match the shape (same services, same per-stream process model, same storage class) but not the absolute hardware, because the ratios transfer and the absolutes do not. Production is touched only to read.

// Part of

Where this sits

// Let's build

Building Monitoring and incident response for an always-on camera fleet?

Tell us where you are. We reply within a day with a concrete next step.