Skip to content

// Logistics × Real-Time Ops

Operating a fleet's telemetry after it ships

Running freight telemetry in production: alerting on the age of the last fix rather than the health of the socket, and surviving the deploy that reconnects every driver at once.

Start with a Sprint

// The problem

Why this is hard

A fleet's telemetry does not fail loudly. The socket reports connected, the dashboard stays green, and the thing that broke — a subscription that did not come back, a position an hour old, a deploy that put every driver in the same queue — produces no error anywhere. Request-shaped monitoring cannot see any of it, because none of it is a request. So the operating problem is not building alerts; it is choosing the counter that binds, and it is almost never the one that looks like health.

// How we do it

The approach for this fit

Alert on the age of the last fix, not the state of the socket

Per vehicle, not per page: a fleet-level 'last updated' keeps refreshing cheerfully while one device goes silent. The counter that binds is elapsed time since capture, and it needs the capture time to exist first.

Instrument the connect path, because nothing else does

Reconnect volume, connect-handler latency and re-subscription rate — the three quantities a deploy moves and no HTTP dashboard records. Sized against what your authorisation path can actually sustain.

Drain on shutdown, with a window sized to the fleet

Stop accepting, tell clients when to return across a spread the server chose, then exit on a deadline inside the orchestrator's. The library's default spread is real and sized for browser tabs.

Re-measure as the fleet grows

Every one of these thresholds is a division with fleet size on top. A stagger tuned by hand today is wrong the first time the customer doubles, so the number is derived and re-derived rather than set.

// Proof

Measured on our own hardware

median reconnect wait, 10k fleet

10.5 s

On the reconnection library's default jitter, against a connect path good for 500/s. Nothing failed and nothing errored in any arm — every client was simply waiting, which is why no request metric shows it.

of replays leave the wrong fix

99%

Every one of those writes succeeded. A buffered device flushes, the last request to land wins, and the live position ends on a fix from the middle of the buffer — with no error, no retry and no failed request for anything to alert on.

loss on a link told to lose none

0.55%

Why the figure on the dashboard is not the thing to alert on: the same link's loss rate moved more than twentyfold with queue depth alone, and every discard was the sender's own doing.

Not a client outcome and not a vendor figure — numbers we took, on hardware we can name, with the method to be published alongside them.

// FAQ

Common questions

The alerts, the thresholds they were derived from, and the arithmetic that derived them — so the next person can redo it when the fleet changes instead of inheriting constants nobody can defend. Concretely: per-vehicle freshness alerting, instrumentation on the connect path, a drain sequence on shutdown, and a written statement of what each threshold assumes. We operate and instrument the live path; we do not rewrite it.

Because most fleet alerting fires on the wrong axis. A socket that reports connected, a page that reports fetched, a queue that reports drained — all of them are true and none of them is the question. The thresholds here come from a division you can check: fleet size over the rate your connect path sustains, elapsed time against a per-vehicle-class expectation. An alert whose number has an argument behind it is one somebody will still trust in six months.

Read access to the running system and somewhere to put dashboards and alerts. The work is instrumentation and operating procedure, not a code change to your tracking layer — where we do find something that has to change in the application (a payload that cannot carry a capture time, a connect handler doing too much), it is written up and handed over rather than shipped inside an ops engagement.

// Part of

Where this sits

// Let's build

Building Operating a fleet's telemetry after it ships?

Tell us where you are. We reply within a day with a concrete next step.