Skip to content

// Fintech × Real-Time Ops

Deploy safety and incident response for a live market feed

Keeping a market feed correct through its own rollouts: the replica count nobody wrote down, the overlap a start-first deploy is supposed to have, and what a doubled consumer does quietly.

Start with a Sprint

// The problem

Why this is hard

Most of what goes wrong on a feed goes wrong on the way past, not while it is running. A rollout that overlaps the old process and the new one is doing exactly what it was configured to do, and for that window every consumer written to be the only one of itself is two of itself. Nothing about that looks like an incident: no error, no restart loop, no failed health check — just a stretch of minutes where each message is handled twice by design nobody wrote down. The constraint that makes it safe is a replica count sitting in a deploy file, and it is usually the one number in the system that has no comment, no test and no owner.

// How we do it

The approach for this fit

Write down the replica count you are already depending on

Every consumer that assumes it is alone — a scheduler, a sequence counter, a dedupe window held in memory — is enforcing a maximum, not a minimum, and enforcing it by accident. We find them, then put the constraint where the next person changing capacity will see it rather than in someone's memory.

A start-first rollout is an overlap, and that is what it is for

The instinct after the first duplicate is to remove the overlap, which trades correctness for a gap in coverage. The better answer is to make the window survivable: a consumer that tolerates seeing a message twice can be deployed the way availability wants, and one that cannot has quietly capped your uptime at whatever a single instance gives you.

Give the second delivery somewhere to appear

A doubled side effect is not a failure signal — it is a successful operation that happened twice, and it enters every counter as ordinary volume. What separates it from load is identity: keys carried through the write path, so a repeat is a number you can watch across a deploy rather than something a customer reports afterwards.

Rehearse the rollout against the consumer, not the endpoint

A health check answers whether the process started. It says nothing about whether the thing that process is the only copy of survived being two copies for ninety seconds. The exercise worth running before a release is the overlap itself, on a replica of the topology, with the duplicates counted.

// Proof

Measured on our own hardware

handled twice, or by nobody

12 ↔ 12

Events a single-instance consumer handled twice during one start-first rollout — and, with the policy switched to stop-first, the same count handled by no instance at all. It is a swap, not a fix. Five trials per arm, the replacement ready 250 ms after it starts on a 20 ms feed, one process on loopback with no containers and no orchestrator, so a real image pull and health check only make the overlap longer.

doubled, 50 ms to 1 s readiness

2 → 50

Events handled twice as the replacement's readiness was swept from 50 ms to one second: 2, 5, 12, 25 and 50, against the 2.5, 5, 12.5, 25 and 50 that readiness over a 20 ms publish interval predicts. A faster health check shrinks the count in proportion, and only no overlap at all takes it to zero — which is the coverage a start-first rollout is buying.

Not a client outcome and not a vendor figure — numbers we took, on hardware we can name, with the method to be published alongside them.

// FAQ

Common questions

An inventory of the places your system assumes a single instance, the deploy configuration that currently guarantees it, and the gap between the two written down as something a person changing capacity will read. Then the counters that would have shown the last duplicate, and a rehearsal of a rollout with them running. It is deliberately not a monitoring installation — you almost certainly have one, and it is aimed at the connection rather than at what the connection carries.

Read access to the deploy configuration and the consumer code, and a session capture if the question touches the feed itself. Nothing in production and no credentials — so the review your side has to run is the one for a supplier who never holds your data, which is usually a different form and a much shorter queue.

No, and anyone our size telling you otherwise is describing a rota they do not have. What is on offer is stated hours, a named person, and the work that makes the hours matter less: the counters that catch a duplicate on the deploy rather than on the complaint. Cover outside those hours is a conversation about a second person, not a line in a retainer.

// Part of

Where this sits

// Let's build

Building Deploy safety and incident response for a live market feed?

Tell us where you are. We reply within a day with a concrete next step.