// Media & Streaming × Real-Time Ops
Monitoring and incident response for an always-on camera fleet
What to watch on an always-on video fleet: the encoder session count rather than GPU utilisation, the degradation that precedes failure, and the cold-start ceiling a ramp test never finds.
// The problem
Why this is hard
A camera fleet does not go down so much as it goes wrong and keeps running. That is what makes it expensive: the same failure on a request-shaped system pages somebody within a minute, and here it bills quietly for weeks.
The reason is that the reassuring numbers are the ones that do not bind. A hardware encoder's ceiling is a session count, not a load, and the two are unrelated — a box can be out of sessions while its GPU reports single-digit utilisation, which is exactly the reading that says there is room for ten times more. Cross that ceiling and the encode can fall through to software with no log line, no error and no change in output. Only the CPU bill moves.
And the spike that actually takes the fleet down is not peak viewership. It is a simultaneous cold start — every pipeline launching at once after a deploy, a crash or a host reboot. A test that adds streams one at a time finds the steady-state ceiling and skips the transient one entirely, which is the lower of the two and the one production meets on an ordinary Tuesday.
// How we do it
The approach for this fit
Alert on the session count, not on utilisation
`nvidia-smi encodersessions` lists what is actually open; the cap is per system rather than per card on consumer hardware. That count is the alert. Utilisation stays on the dashboard, but it never fires anything - it reads 8% on a box with no sessions left.
Catch the fallback the hour it happens
A stream that drops to software encoding produces identical output and costs 2.8x the CPU. Nothing errors, so the tell has to be watched for directly: the session count falling while the stream count does not, and cores-per-stream stepping up.
Alert on degradation, not on a threshold
Write stalls, page-cache eviction and rising latency usually precede a hard failure by minutes. Where they do, they are a better alert than any level on the resource itself, because they arrive while there is still time to act.
Re-measure the cold-start ceiling
Bring N streams up at once, at several values of N, and watch the peak rather than the plateau. That transient ceiling is lower than the steady-state one, it is what a deploy meets, and it moves every time a per-stream service is added.
// Proof
Measured on our own hardware
CPU cost of a silent fallback
swapping only the encoder, NVENC out and libx264 in, decode left on the CPU in both arms: 2.74-2.84x across five interleaved A/B pairs at 1080p60. A ratio, not an absolute - the absolutes drifted ~10% with package temperature and are not published.
GPU utilisation at the session ceiling
on our own production box, re-verified 2026-08-01, while a live 1080p59 encode ran. An alert wired to this number cannot fire: it is not a load that climbs toward a limit, it is a dial that reads calm while a separate counter runs out. Alert on the counter.
of video in a queue bounded by frames
at 59.94 fps; the same setting holds 100 ms at 30. A buffer tuned at development framerate runs too shallow in production and drops video while audio keeps flowing - a degradation no resource threshold reports.
Not a client outcome and not a vendor figure — numbers we took, on hardware we can name, with the method published in full. The write-up: Video not publishing, audio fine — the bug that only appears at production framerate.
// FAQ
Common questions
// Part of
// Let's build
Building Monitoring and incident response for an always-on camera fleet?
Tell us where you are. We reply within a day with a concrete next step.