Live captions on a call — where the latency actually comes from
Adding captions to a call sounds like a feature request and behaves like an architecture decision. The latency does not come from the network or the model — it comes from the window the recogniser needs before it will commit to a word, and from where you attached the tap.
On this page
TL;DR: Live transcription does not make a call slower unless you build it into the media path, and you should not. The delay a user can feel does not come from the network or from model inference — it comes from the window the recogniser needs before it will commit to a word. Attach the recogniser as a sidecar on a copy of the audio, segment on voice activity rather than a fixed clock, and the call's own latency is untouched no matter how slow the transcription gets.
"Can we add captions?" arrives as a feature request and behaves like an architecture decision. It sounds cheap because the model is somebody else's problem — an API, a library, a container — and the audio is already flowing.
The trap is that both intuitions about where the time goes are wrong. Teams brace for network cost and model cost, and the number a user actually perceives is set by neither. Worse, the version that does slow the call down is the one that looks tidiest on a diagram.
Two notes on scope before the mechanics. This is about captions on a human-to-human call — the case where transcription is a passenger. It is a different problem from a voice AI agent, where the transcript is on the critical path because a model has to answer it; there the budget is end-to-end and everything below changes. And on provenance: the speech side here is first-hand from our own local push-to-talk speech-to-text work — which records, segments on the VAD, and then transcribes the finished utterance in a batch pass; the media-pipeline side is from operating WebRTC systems. We have not run captions across a production call fleet, and where this article reasons rather than reports, it says so.
Why does transcription add latency at all?
Because a recogniser is not a function from sound to text — it is a function from a window of sound to text, and the window has to fill before anything comes out.
That is the whole mechanism, and it explains the shape of every trade below. A speech model given 200 ms of audio has almost no context: it cannot tell you whether the sound it just heard ends a word or starts a longer one. Given two seconds it does far better, because human speech disambiguates backwards — the end of a phrase changes what its beginning was. So accuracy improves with window length, and the window is latency, paid before inference even starts.
This produces the only genuinely hard trade in the feature:
- Short windows → text appears quickly, is wrong more often, and gets rewritten in front of the reader as later audio arrives.
- Long windows → text is markedly better and lands late enough that it has stopped being a caption and started being a transcript.
Inference time sits on top of that, and it is usually the smaller half. This is why "use a faster model" so often fails to fix a captions complaint: the complaint is about the window, and the window is a product decision rather than a hardware one.
Where should the audio tap go?
On a copy of the audio, off the media path, always. This is the decision that determines whether captions can ever slow the call down, and it is worth making explicitly rather than by default.
There are two shapes:
Inline. Audio flows through the transcription component on its way to the other participant. It is the tidier diagram, it makes "transcribe then forward" trivially ordered, and it is wrong: every millisecond the recogniser spends is now added to the conversation, and every failure it has is now a call failure. A model that stalls for two seconds has just made the call unusable, and a model that crashes has hung up.
Sidecar. The media path is untouched and the recogniser receives a duplicate of the audio. Now transcription latency is decoupled entirely from conversation latency: captions can be five seconds behind and the call is exactly as fast as before. If the recogniser dies, captions stop and nobody stops talking.
| Inline | Sidecar | |
|---|---|---|
| Where the recogniser sits | in the media path | on a copy of the audio |
| Transcription is slow | the call is slow | captions lag, call unaffected |
| Recogniser crashes | the call drops | captions stop, call continues |
| Model you can afford | only a fast one | the best one that fits the budget |
| Verdict | never | always |
The sidecar is not merely safer, it changes what you are allowed to build. Once transcription cannot affect the call, you can use a slower and better model, batch, retry, run a second pass, or fail — none of which are options when you are standing in the media path.
In a server-side media topology this is natural: the server already has the decoded or forwardable audio, so the tap is a subscriber like any other — and on the browser side the platform now exposes the same idea directly, since a media stream track can be read as frames via WebRTC Encoded Transform without standing between the peers. The one thing to check is that the tap does not inherit the call's own quality-of-service assumptions — a recogniser that back-pressures the pipeline when it falls behind has quietly become inline again. The queue in front of it must drop, not block, which is the same lesson a starved buffer teaches on the video side from the opposite direction.
- Audio arrivesthe only path the call uses
- Fan out a copya subscriber like any other
- Segment on speechthe recogniser's own clock
- Transcribe and emitas far behind as it needs to be
Fixed windows, or voice activity detection?
Voice activity detection, in almost every case, and the reason is that it changes what you are waiting for.
A fixed window is a clock: transcribe every N milliseconds regardless of what the audio contains. It is simple, and it spends the budget badly — it cuts words in half at the boundary, it runs the model over silence, and it makes the same latency promise whether somebody is mid-sentence or has stopped talking entirely.
Voice activity detection — Silero VAD is the one we use — segments on speech instead. The recogniser is handed an utterance that ended, which is exactly the unit it is best at, and silence costs nothing because nothing is submitted — and that is the part worth taking, because it is architectural rather than numeric.
We are not going to give you an end-to-end number here, and the reason is the point of this section. Our own work sets latency targets per stage, in a design document, and has not measured the path end to end. A stage target is not a budget and a budget is not a measurement; the three get quoted back as though they were the same thing, and a figure that has crossed two of those boundaries is exactly the kind that ends up in somebody's architecture review as a benchmark. What survives without a number is the shape: segment on speech, submit an utterance that has already ended, and count whatever you do measure from the moment the speaker stopped rather than from an arbitrary tick. What the code fixes is upstream of it — the denoiser we use imposes a 10 ms frame, which is the library's constant rather than a capture rate we chose, and the capture path resamples to 16 kHz before anything downstream sees it.
Two caveats that cost time if you meet them late:
- The segmenter is a latency component too. Detection needs a hangover period to be sure speech ended, and that period is added to every utterance. Tuning it too tight fragments sentences into pieces the model then transcribes worse; too loose and every caption is late by the hangover.
- Long utterances still need cutting. Somebody who talks for a minute without a pause produces one segment and one very late caption, so a maximum length has to force a boundary anyway — the fixed window returns as a backstop rather than as the primary mechanism.
What does the model choice actually cost?
Less than people expect on speed, and more than they expect on everything else.
Running the recogniser locally is realistic now in a way it was not a few years ago: a local implementation such as whisper.cpp puts a competent model on ordinary hardware with no per-request network cost at all.
Local versus hosted is a latency argument before it is a privacy one. A hosted recogniser adds a network round trip per segment and a dependency that can rate-limit you; a local one adds hardware and an operational surface. The round trip is often smaller than the window you already accepted, which means the honest comparison is rarely about speed — it is about where the audio is allowed to go and who is on the hook when it stops working.
Two tricks worth knowing, both from our own local recogniser work:
- Detect the language once, not per segment. Language detection is a separate inference pass, and running it on every utterance pays for it repeatedly to answer a question whose answer almost never changes mid-call. We run it on the first chunk only and carry the result.
- Post-processing is a second latency stage, and it belongs after the caption appears. Cleaning punctuation or fixing terminology with a language model is genuinely valuable, and it is a second inference pass — making the caption wait for it adds that pass to every utterance. We have not timed it, so no multiplier appears here. Emit the raw caption, then replace it — the reader sees text quickly and sees it improve, which is also how they already expect captions to behave.
What to measure
Three numbers, and they must be measured separately or the wrong one gets optimised.
- Segment latency — from the end of an utterance to text on screen. This is what a user calls "the captions are slow", and it is the sum of the detection hangover, the window, inference and transport.
- Call latency, with captions on and off. If these differ at all, the tap is not a sidecar, and no amount of model tuning will fix what is actually an architecture problem.
- Correction rate — how often displayed text is rewritten. It is the honest cost of a short window, it is invisible in every latency metric, and past a certain rate a reader finds fast, flickering captions worse than slower stable ones.
The third is the one teams do not instrument, and it is the one that decides whether shortening the window actually improved anything.
A blocking queue turns a sidecar back into a stage
FAQ
Will adding captions make our calls laggier? Not if the recogniser is on a copy of the audio. Sidecar the tap, make the queue in front of it drop rather than block, and call latency is unchanged by construction — that is the property to verify with a measurement, not an assumption, because a back-pressuring queue silently makes the tap inline.
Is a faster model the fix for slow captions? Usually not. Inference is typically the smaller half of what the user feels; the larger half is the window the recogniser waits to fill plus the pause needed to decide speech has ended. Measure the split before buying hardware — if inference is 20% of segment latency, halving it moves nothing anybody notices.
Is this the same as building a voice AI agent? No, and the difference is where the transcript sits. Captions are a passenger: late text is a degraded feature. In a voice agent the transcript is on the critical path to a spoken reply, so the budget is end-to-end and every trade in this article is made differently. Most published material on "real-time transcription" is about the second case.
Do we need per-speaker attribution? If the media topology gives you a separate audio stream per participant, you get it for free by tapping each stream — which is a strong argument for tapping before any mixing. Recovering who spoke from a single mixed track is a different and much harder problem, and it is worth avoiding by architecture rather than solving.
The four things worth taking away
- The window is the latency. A recogniser needs context before it commits, and that wait is paid before inference starts — which is why a faster model so often changes nothing a user notices.
- Tap a copy, never the path. A sidecar decouples transcription latency from conversation latency completely, and it is what lets you choose a slower, better model without a penalty.
- Segment on speech, not on a clock — with a maximum length as the backstop, and with the detection hangover counted as part of your latency budget, because it is.
- Instrument correction rate alongside latency. Shortening the window always improves the number people watch and always degrades the one they do not, and only one of those is what the reader experiences.
If captions are on the roadmap, the decision worth making early is the tap, not the model. The model can be swapped in an afternoon; a recogniser that has been standing in the media path since the first prototype is a rewrite.
Designing this into a call product that already exists — where the tap goes, what the budget really is, and what it costs to run — is the kind of question an Architecture Sprint answers with measurements rather than opinions.
The real-time media side of this — the transport the tap hangs off, and what it costs to run — is on real-time video for streaming platforms.
