The deploy that logs every driver back in at once
For a request/response API a missing shutdown handler costs a handful of failed requests. For a socket system it costs a synchronised reconnect of every connected client, each one re-running an authorisation query — and none of it shows up in the metrics you already have.
On this page
TL;DR: Graceful shutdown for websockets is a different problem from graceful shutdown for an HTTP API, and the difference is not one of degree. A request/response service that exits without draining loses the handful of requests in flight. A socket service that exits without draining disconnects every client it has, simultaneously — and each of them reconnects, and each reconnect re-runs whatever work you do on connect. The reconnection library's defaults spread that stampede across about one second, which looks like a solution and is not.
Three facts about a system, each of them individually unremarkable.
The process has no signal handling: no SIGTERM handler, no SIGINT handler, nothing that runs
between "the orchestrator wants this container gone" and the process ending. The deploy replaces the
container outright rather than draining it. And every client that connects runs an authorisation
query on connect, to work out which orders that driver is allowed to watch.
Each of those is ordinary. Composed, they mean that every deploy is a synchronised, fleet-wide re-authorisation, and that nothing in your monitoring is shaped to show it to you.
Why a socket service is a different problem
The instinct that makes this easy to miss is a correct instinct about a different architecture.
For a request/response API, a missing shutdown handler has a bounded and well-understood cost. The requests in flight at the moment of exit fail; there are usually a few dozen; the client retries or the user refreshes; your error-rate graph shows a spike one scrape wide and returns to baseline. Everyone has seen that spike. It is annoying rather than serious, and adding the handler is on somebody's list.
A socket service does not have "requests in flight". It has state that exists because the connection exists. Killing the process does not fail some work — it invalidates every relationship the server was holding, all at once, for every client. And every one of those clients is written to notice and reconnect, because that is what makes a socket system usable in the first place.
| Request/response API | Socket service | |
|---|---|---|
| What is lost on exit | the requests in flight | every connection, and the state each implied |
| How many clients notice | those mid-request — a few | all of them, simultaneously |
| What the client does next | retries one request, or nothing | reconnects, and re-establishes everything |
| What the reconnect costs the server | one request's work | a connect handshake plus whatever runs on connect |
| Where it shows in your metrics | the HTTP error rate | nowhere, unless you built it |
That last row is the one that keeps this invisible. Nothing about a websocket reconnection storm passes through the HTTP request pipeline your dashboards are built on. There is no 5xx. There is no elevated request latency, because there are no requests. The work is real — every reconnecting client runs an authorisation query — and it lands in a shape your instruments were not made to see.
The defaults look like they solved this
Here is the part I find genuinely interesting, because it is not an absence of care.
The client sets no reconnection options at all, so it inherits the library's defaults. It would be
easy to write that up as "nothing is staggered", and it would be wrong. The
Socket.IO client defaults to a
reconnectionDelay of 1000 ms with a randomizationFactor of 0.5, which means each client waits a
value drawn from a window around that base:
// the FIRST reconnect attempt, per client
const delay = 1000 * (1 + (Math.random() - 0.5)) // 500-1500 msLater attempts back off — the documented ladder runs 500–1500 ms, then 1000–3000, then 2000–5000, to a five-second ceiling. Which sounds like it spreads things out, and after a deploy it does not: the server is back within seconds, so essentially every client lands on attempt one and never reaches the wider rungs. The narrow window is the only one that runs.
So there is jitter. It is deliberate, it is on by default, and it was put there by people who understood exactly this failure. It is simply sized for the problem it was written for: keeping a handful of browser tabs from retrying in lockstep after a blip.
Point it at a fleet and the arithmetic stops working. Every client that was connected wakes up inside the same one-second window, which for a system with a device per vehicle means the entire fleet re-authorises inside a single second. A one-second spread across a fleet is not a stagger; it is a rounding error on simultaneity. The defaults are the reason nobody looks: the option exists, it has a sensible value, and the box is ticked.
How wide does the window need to be?
Wide enough that the peak is a rate your authorisation path can serve without queueing, which makes it a division rather than a preference.
Take the number of connected clients, divide by the connect-time work rate you can actually sustain, and that quotient is your minimum window. A fleet of a few hundred devices against an authorisation query you would rather not run more than a few dozen times a second is already a window of tens of seconds, not one. And it scales with the fleet, which is the property that matters: a stagger tuned by hand today is wrong the first time the customer grows.
Two corollaries fall out of that, and both are more important than the number:
The window belongs to the server, not to the client's defaults. The server knows how many clients it had and what its own connect path costs; the client knows neither. A client-side constant is a guess made in the wrong place, by someone without the inputs.
Reconnect delay and the work on connect are one problem. Halving the connect cost and doubling the window buy the same relief. It is worth checking whether the authorisation query needs to run on every connect at all, or whether its answer can outlive the connection it was computed for — because the cheapest stampede to survive is the one that does less work per arrival.
Configured jitter is worse than none when it is the wrong size
What does the narrow window actually cost?
The division above is arithmetic, and arithmetic is easy to nod at and not act on. So I staged the storm.
A bounded connect path — ten concurrent authorisations of 20 ms each, which is a connection pool in front of an indexed query, and comes to 500 connects per second sustainable. Then a fleet of real sockets, each client waking at its own draw from the window and connecting for real. The jitter shape is the library's, not an invention. The work on connect is a calibrated CPU burn rather than a real query, because what the handler does changes none of the queueing; the service time is published with the numbers so you can recompute them against your own.
And it is one machine talking to itself. Every one of those sockets is a loopback connection from the same process, so there is no network latency, no TLS handshake, no load balancer deciding where each client lands, and no connection that fails and has to try again. This measures queueing at the accept path and nothing else — which is the quantity the argument turns on, and also the reason to read every number below as a floor. A real fleet arrives over links that add all four.
| Fleet | Window | p50 | p95 | Slowest | Whole fleet back |
|---|---|---|---|---|---|
| 500 | library default | 57 ms | 124 ms | 134 ms | 1.6 s |
| 500 | 5 s | 19 ms | 23 ms | 34 ms | 7.5 s |
| 2 000 | library default | 1.65 s | 3.2 s | 3.3 s | 4.8 s |
| 2 000 | 5 s | 24 ms | 43 ms | 61 ms | 7.5 s |
| 10 000 | library default | 10.5 s | 20.4 s | 21.5 s | 23.0 s |
| 10 000 | 5 s | 9.1 s | 17.5 s | 18.5 s | 26.0 s |
| 10 000 | 30 s | 23 ms | 34 ms | 56 ms | 45.0 s |
| 10 000 | 120 s | 19 ms | 26 ms | 40 ms | 180.0 s |
The division holds, and it holds sharply. Five hundred clients into a one-second window is 500 arrivals per second against 500 of capacity — right at the line, and the p50 is already close to three times the service time. Two thousand is four times over, and the median client waits 1.65 seconds for work that takes twenty milliseconds. Ten thousand is twenty times over and the median waits ten and a half seconds, with the unlucky tail past twenty.
Then look at the last two rows. The predicted minimum window for ten thousand clients is 10 000 ÷ 500 = twenty seconds — and thirty seconds brings the median back to 23 ms, essentially the service time, while five seconds barely helps at all. The prediction was written before the run; the run put the transition exactly where the division said it would be.
Nothing here is the network's fault and nothing appears in a request metric. The harness counts failed connections in its own column and that column reads zero in every arm — nothing errored, nothing was refused, nothing timed out. Every one of those clients was simply waiting, and the only place waiting shows up is a metric somebody deliberately built.
The trade is in the last column, and it is the reason this is a decision rather than a default. A wide window flattens latency and stretches full recovery: at 120 seconds no client waits more than 40 ms, and the fleet is not whole for three minutes. A narrow window has everyone back fast, in the sense that they are all back in the queue. Which of those you want depends on what the clients do while they wait — and that is a question about your product, not about your reconnect settings.
- 120 s window180 s
- 30 s window45 s
- 5 s window26 s
- library default (0.5–1.5 s)23 s
What a graceful socket shutdown actually looks like
The missing piece is draining, and it is three steps rather than one.
Stop accepting new connections, but keep serving the ones you have. For Socket.IO this is
io.close(), which closes the underlying server to new connections; the important half is that
existing sockets keep working while you do the rest. On a request API this step is the whole of
graceful shutdown. Here it is the setup.
Tell the clients to go, with a spread you choose. This is the step that has no equivalent in the HTTP world and the one people skip. Emit an application-level "reconnect after N" and let the client schedule against a number the server picked, drawn across a window sized to the fleet. If the client ignores it and falls back to its own defaults, you have lost nothing — but the clients that honour it are the ones that stop being a stampede.
Then exit, on a deadline. Wait for connections to drain, and stop waiting after a bounded time,
because a shutdown that can hang is a deploy that can hang. The orchestrator has its own patience —
Kubernetes sends SIGKILL after terminationGracePeriodSeconds — and your deadline should sit
inside it, otherwise the graceful path you wrote is the one that never finishes.
None of that is much code. What it needs is a signal handler, which is the thing that was missing at the start.
- Stop acceptingexisting sockets keep working
- Tell clients whena window the SERVER sized
- Exit on a deadlineinside the orchestrator's
FAQ
We deploy with a rolling update, so connections move over gradually. Are we fine? Better, but the mechanism is the same one scaled down. A rolling update replaces pods in batches, so each batch disconnects its share of the fleet at once — a smaller stampede, several times over, and possibly onto replicas that are still coming up. What rolling updates fix is total unavailability. What they do not fix is a connect path that is expensive per client, because that cost is paid per reconnection either way.
Would a sticky load balancer or a shared session store help? They solve a different problem — which server a reconnecting client lands on, and whether its state survives the move. Valuable, and orthogonal: even with perfect session affinity and shared state, a process that dies without draining still disconnects everyone at once, and they still all come back inside the same window. Fix the shutdown first; it is smaller work and it is upstream of the rest.
How would we tell whether we have this? Two greps and a read, and it takes about ten minutes. Search your server entry point for a signal handler; if there is none, the container is being killed rather than asked to stop. Search your client for reconnection options; if there are none, you have the library's defaults and you should read what they actually are rather than assuming. Then read the connect handler and ask what it does per connection, because that is the quantity being multiplied by your fleet.
Should we just not re-authorise on connect? Not "just" — but do make it a decision rather than an inheritance. Authorising on connect is the correct default: it is the moment you know who is asking. The question is whether the answer has to be recomputed from scratch every time a connection drops and returns, which on a mobile fleet is often, for reasons that have nothing to do with the driver's permissions changing.
The three things worth taking away
- A missing shutdown handler is a different bug in a socket service. In an HTTP service it costs the requests in flight. Here it costs every connection at once, and the cost lands on the connect path rather than in your request metrics.
- Default jitter is real and sized for browser tabs. Around one second of spread stops a handful of clients retrying in lockstep. For a fleet it is a rounding error on simultaneity — and worse than nothing, because the option looks configured.
- Draining is three steps, and the middle one has no HTTP equivalent: stop accepting, tell the clients when to come back with a spread you chose, then exit on a deadline that sits inside the orchestrator's.
The other half of this system's story — what it does with the positions those reconnecting clients report — is the timestamp that was never on the wire.
The wider shape this belongs to — what a fleet costs you between the field device and the dispatcher's screen — is on freight telemetry that survives a bad link.
If you are running a socket system and have never watched a deploy from the clients' side, that is the kind of thing we come in for.
