Mobile viewers stall while desktop is fine — why your SFU can't make them a lower-quality layer
Everyone assumes the media server adapts quality. It selects quality — among the encodings the publisher already produced. Here is what that changes for a fleet that publishes one stream: the four fixes that don't work, the three that do, and the second job the transcoder was doing for you until you switched it off.
On this page
- What does an SFU actually do when it "adapts quality"?
- Why can't the server just make a smaller copy?
- Four things people try first, and what each one really does
- adaptiveStream and dynacast — selectors, not generators
- Temporal layers — real, and codec-dependent
- The camera's "smart" encoding mode — one stream, fewer bits
- SVC via VP9 or AV1 — right idea, wrong fleet
- How do you tell which situation you are in?
- What actually works, in order of cost
- What was the transcoder quietly doing for you?
- FAQ
- The four things worth taking away
TL;DR: An SFU never creates a video layer. It forwards the ones your publisher already encoded and picks between them per subscriber. So if a camera or an encoder publishes a single 1080p stream, no server-side setting will hand a weak mobile connection something smaller — adaptiveStream and dynacast select among layers, they do not generate them. Simulcast without transcoding means the publisher sends two or three encodings. If it sends one, you have exactly three options and two of them are work.
The symptom arrives sorted by device, which is what makes it look like a client bug. Desktop viewers on office wifi are fine. Phones on mobile data stall, buffer, or show a frozen first frame on the same stream. Someone checks the server, finds that adaptive streaming is switched on, and concludes the setting is broken.
It is not broken. It is doing exactly what it says, and what it says is narrower than what everyone hears. The gap between "the SFU adapts quality" and what an SFU can actually do is the whole article, and it decides whether your fix is a config change, a week of engineering, or a per-minute invoice.
What does an SFU actually do when it "adapts quality"?
It chooses. That is the entire mechanism, and the standards are explicit about it. RFC 7667 describes the selective forwarding middlebox as providing per-receiver adaptation "without any signal processing at the middlebox" — the box "selects which parts of a scalable bitstream (or which bitstream, in the case of simulcasting) to forward to each of the receiving endpoints" (RFC 7667 §3.7). RFC 8853 puts the production side just as plainly: it is the sending endpoint that creates "multiple differently encoded versions of the same media source in different RTP streams", and the middlebox's job is to "select which RTP stream(s) to forward" (RFC 8853).
LiveKit's own description of its SFU says the same thing from the implementation side: "the SFU receives all three layers for the track and makes a decision on which layer to forward to each subscriber" (An introduction to WebRTC Simulcast).
Three layers arrive. One goes out. Nothing new is made anywhere in that sentence. If one layer arrives, one layer goes out — to everybody, at the resolution and bitrate the publisher chose, including the phone that cannot carry it.
- Cameraone H.264 encoding
- Publisherforwards what arrived
- SFUselects among what arrived
- Every subscribersame layer, weak links included
Why can't the server just make a smaller copy?
Because making one means decoding the incoming video and encoding it again, and that is a different machine. An SFU forwards packets: it is cheap, it is fast, and it scales, precisely because it performs no signal processing on the media. Ask it to produce a 360p version of your 1080p stream and you have asked it to become a transcoder — a CPU or GPU workload per stream, not a packet-routing workload per stream.
That is why every managed platform bills transcoding as its own line item, separate from bandwidth and connections. It is not a surcharge on a feature; it is a different product running on different hardware. The trade you are actually looking at is not "why won't the server do this for free" but "who pays the encode: your publisher, or the vendor's transcoder?" Both are real answers. The one answer that does not exist is nobody.
Four things people try first, and what each one really does
Each of these is proposed in almost every thread on this problem, and each is a different way of hoping the layer already exists.
| What people reach for | What it actually does | Why it cannot help here |
|---|---|---|
| adaptiveStream / dynacast | picks which layer a subscriber receives; stops sending one nobody watches | both operate on a set of layers that already exists |
| "the SFU can just drop frames" | real, where the forwarder can see which frames are droppable | VP8 marks that in the RTP payload; H.264 as specified has no field for it |
| The camera's "smart" encoding mode | lowers the bitrate of the one stream it already sends | one stream, decided for every viewer at once |
| SVC via VP9 or AV1 | genuinely layered — one bitstream a forwarder can strip | a camera emits H.264, so producing VP9 means transcoding again |
Each row in detail, because the reason is what stops you re-proposing it in three months.
adaptiveStream and dynacast — selectors, not generators
Both are real features and both are worth having. Neither makes a layer. LiveKit's documentation describes adaptiveStream as managing "quality of subscribed video tracks to optimize for bandwidth and CPU", choosing "an appropriate resolution based on the size of largest video element it's attached to", and dynacast as automatically pausing "video layer publications when they aren't being consumed by subscribers" (LiveKit — codecs and more).
Read those two definitions with the question "does this create anything?" in mind. One picks a layer to subscribe to; the other stops sending a layer nobody watches. Both operate on a set that already exists. Turn both on against a single-layer publisher and you have a selector with one item to pick and a pauser with nothing to pause.
Temporal layers — real, and codec-dependent
Dropping frames rather than resolution is a genuine adaptation path, and it is the one people are half-remembering when they insist the SFU can degrade a stream on its own. It works when the forwarder can see which frames are droppable without decoding anything. For VP8 it can: the payload descriptor carries a two-bit temporal-layer index, and the RFC states that "media-aware network elements MAY use the information in the VP8 payload descriptor … to identify non-reference frames and discard them" (RFC 7741). LiveKit notes that VP8 temporal scalability is enabled automatically by browsers when simulcast is used.
For H.264 there is nowhere to put that information. The payload header defined by
RFC 6184 is one octet — a forbidden bit, nal_ref_idc
and nal_unit_type — and it carries no temporal id. A forwarder looking at an H.264 packet cannot
tell a droppable frame from a required one without parsing the bitstream, which is the thing it
exists not to do. There is a generic escape hatch: the Dependency Descriptor RTP header extension
carries temporal_id and spatial_id for any codec, and its spec says that while designed for AV1
"it may prove useful for other codecs as well"
(AV1 RTP spec). It is not what a camera or a browser
negotiates for H.264 today, so plan around its absence rather than its existence.
The camera's "smart" encoding mode — one stream, fewer bits
Zipstream, H.264+, Smart Codec: every camera vendor ships one, and the name suggests it is producing something adaptive. It is producing something smaller. Axis documents Zipstream as reducing "the average bit rate by removing unnecessary data", via a dynamic GOP in which "unnecessary I-frames are removed in order to further reduce the bit rate" and a dynamic FPS mode "triggered by the motion in the scene" (Axis — Zipstream technology).
That is rate control applied to one encoded stream, decided by the camera for all viewers at once. A quieter stream is not a second stream. It reduces what every subscriber receives, including the desktop viewer who had bandwidth to spare, and it gives the SFU nothing extra to choose between.
SVC via VP9 or AV1 — right idea, wrong fleet
Scalable coding genuinely solves the layer problem: one bitstream that contains its own lower
layers, so a forwarder can strip them. LiveKit activates it automatically for VP9 and AV1 with the
L3T3_KEY scalability mode. It is the correct answer for a browser-to-browser product and the wrong
one for a camera fleet, for a reason that has nothing to do with the codec's merits: a surveillance
camera does not emit VP9 or AV1. It emits H.264, H.265 or MJPEG. Getting SVC out of it means
decoding and re-encoding — transcoding, which is the cost you were trying to remove. You have not
avoided the encode; you have moved it and changed its name.
There is a second cost worth checking before betting on it: SVC only helps subscribers who can
decode that codec, and the clients most in need of a smaller layer are usually the ones with the
narrowest codec support. WebKit's own release notes are the place to check that, not a
compatibility table: announcing the change in Safari 17.4, Apple noted that until then "support on
iOS and iPadOS was limited to VP8 in WebRTC" (WebKit features in Safari
17.4). That is recent enough to
matter for a fleet whose viewers are on whatever phone they already own. Call
RTCRtpSender.getCapabilities("video") on the oldest device you support and read the answer instead
of assuming it.
How do you tell which situation you are in?
Two minutes, on the publisher, before you change anything on the server.
1 · Count the encodings in the offer. A publisher that is genuinely simulcasting says so in SDP —
an a=simulcast line and one rid per layer, per RFC 8853. This is the shape you are looking for:
a=rid:h send
a=rid:m send
a=rid:l send
a=simulcast:send h;m;lOne rid, or no simulcast attribute at all, means one layer regardless of what any server setting
claims.
2 · Count the outbound streams in the stats. Three encodings produce three outbound-rtp
entries, each with its own rid and its own frame size. One entry is the finding:
// on the PUBLISHING peer connection
const stats = await pc.getStats();
for (const s of stats.values()) {
if (s.type === "outbound-rtp" && s.kind === "video") {
console.log(s.rid ?? "(no rid)", s.frameWidth + "x" + s.frameHeight, s.bytesSent);
}
}3 · Then look at what the subscriber got. If the failing client's inbound-rtp reports the same
frame size as the healthy one, nothing was adapted, because there was nothing to adapt to.
The tell that separates this from every other "video is bad on mobile" cause: it correlates with the publisher's configuration and not with the subscriber's network. A genuine bandwidth-adaptation failure shows different subscribers receiving different layers, some badly. This shows every subscriber receiving the same layer, and some of them drowning in it.
What actually works, in order of cost
Three routes, and the right one depends on which is scarcer for you — engineering time or per-minute spend.
| Route | What it costs | What it buys | When it wins |
|---|---|---|---|
| Use the source's existing sub-stream | nothing to encode; some publisher plumbing | a second layer today | the source already emits one and it is small enough to help |
| Publisher-side simulcast | your CPU (~0.30 cores/stream, measured) + ~17% publisher bandwidth | per-subscriber adaptation, no transcode line | the fleet is large enough that per-minute transcode hurts |
| Vendor transcoding | a per-minute line that scales with publish time | you stop caring what the source emits | engineering time is scarcer than money, or the fleet is small |
Check whether your source already produces a second stream. Almost every IP camera does, and it
costs nothing to use: Hikvision addresses the main stream and the sub-stream as separate RTSP
channels — /Streaming/channels/101 and /102 (Hikvision
support).
The camera encoded both already. One command tells you what you have:
ffprobe -v error -rtsp_transport tcp -select_streams v:0 \
-show_entries stream=profile,level,width,height,pix_fmt \
-of default=noprint_wrappers=1 "rtsp://<camera>/<sub-path>"Run it before you plan around it, because the sub-stream is a second encoding and not necessarily
a small one. On a real camera we probed for this article, sampling each stream for 10 seconds over
RTSP/TCP with -c copy (no decode, so nothing is re-measured through our own encoder):
| resolution | declared level | bitrate over 10 s | |
|---|---|---|---|
| main | 1920×1080 | 4.0 | ~431 kbit/s |
| sub | 1280×720 | 3.1 | ~247 kbit/s |
The sub carries 44% of the pixels for 57% of the bits — it saves 43%, not the 75% the resolution drop suggests, because the camera did not scale its bitrate with its pixel count. Still worth having and still free; just not the rescue layer a 360p encoding would be. One camera, one 10-second sample, and bitrate on a surveillance scene moves with motion — measure yours rather than inheriting these.
The catch is on the publishing side: putting the two on one track as simulcast layers needs a publisher that can express multiple encodings, whereas publishing them as two separate tracks always works and moves the switching decision to the client.
Publish real simulcast from a participant-style publisher. Decode once, pass the camera's high layer through untouched, encode one small layer, publish both. Per-subscriber adaptation comes back with no vendor transcode line at all, and you pay encode only for the layer that is cheap to encode. In a browser this is one argument, and the layers are the browser's own problem:
pc.addTransceiver(track, {
direction: "sendonly",
sendEncodings: [
{ rid: "h", maxBitrate: 2_500_000 },
{ rid: "m", maxBitrate: 700_000, scaleResolutionDownBy: 2 },
{ rid: "l", maxBitrate: 200_000, scaleResolutionDownBy: 4 },
],
});Do not read that snippet as the general answer. scaleResolutionDownBy is the browser running the
extra encodes for you; a camera, a GStreamer pipeline or a server-side compositor has no such
handler, and has to produce every layer itself. The API shape is the easy half.
The honest costs: simulcast raises publisher bandwidth — LiveKit measured about 17% for all three layers in 2021 — a participant path may be billed differently from an ingress path on a managed platform, and the encode is yours to host.
We benchmarked that last cost rather than estimating it. Transcoding a synthetic 1080p25 H.264 source on an idle-ish i9-14900, measured as CPU-seconds per second of video: decode alone 0.13 cores, decode plus one 640×360 layer 0.30 cores, a full 1080p re-encode 0.79 cores. So the simulcast path — decode once, pass the high layer through, encode one small one — costs roughly a third of re-encoding everything, which is the entire economic case for building it.
One correction while we are here, because an earlier version of this article inherited it: decode does not dominate that budget. It is 44% of the simulcast path and 17% of a full re-encode — real, but the encoder is the bigger half in both. Synthetic content and one CPU, so treat it as an order of magnitude and bench your own before sizing a box.
Or turn transcoding on and pay for it — or leave it on, if the decision to switch it off has not been made yet and you would like the full account of that trade first. This is a legitimate answer and it is the one to pick when engineering time is scarcer than money. What it buys is that you stop caring what the source emits. What it costs is a per-minute line that scales with publish time rather than viewing time — which is why it is unremarkable for a meeting product and brutal for a fleet that publishes 24/7 whether anyone is watching or not.
If the honest answer is that you need a publisher built rather than a setting changed, that is a scoping question before it is an engineering one — our estimator puts a range on it in a couple of minutes, starting from real-time video as the domain.
- Decode oncethe only unavoidable cost
- High layerpassed through untouched
- Low layerencoded small
- Publish bothas simulcast encodings
- SFUnow has something to choose
What was the transcoder quietly doing for you?
One thing worth knowing before you pick a route, because it is the cost of the two that remove transcoding. The transcoder was also normalising the codec profile: it decoded whatever the camera sent and re-encoded it on conservative, browser-friendly terms. Passthrough hands the camera's own bitstream to the browser instead, and some browsers are stricter than others about what they accept.
We met this the day transcoding went off in one of our own systems: live video went black on iOS
Safari only, with Android and desktop unaffected and every transport counter healthy. The cause is a
single number the camera declares in its SPS, the tell is bytesReceived climbing while
framesReceived stays at zero, and the fix costs no CPU at all — it is a bitstream rewrite rather
than an encoder.
It is a distinct problem with a distinct diagnosis, so it has its own article: video is black on iPhones and fine everywhere else. Read it before you switch transcoding off, not after.
Before you conclude the server is at fault
outbound-rtp entries on the publishing peer connection first.FAQ
Isn't "adaptive bitrate" supposed to handle this automatically? It handles the selection automatically. It cannot handle the production. Adaptive bitrate in an SFU means each subscriber gets whichever of the publisher's encodings suits its connection right now, switched without a renegotiation. With one encoding published there is nothing to switch to, and the feature reports itself as working because, strictly, it is.
Can the SFU at least drop frames to save the weak subscriber? Only if it can identify droppable frames without decoding, which depends on the codec. VP8 exposes a temporal-layer index in the RTP payload descriptor, so a forwarder can strip a layer. H.264's payload header has no such field, so the same forwarder is blind unless a generic dependency descriptor is negotiated — which is not what cameras and browsers do for H.264 today.
Does any of this apply to a normal browser-based product? Barely, and that is why the problem is so confusing when you meet it. Browser SDKs enable simulcast by default, so a web publisher hands the SFU three layers without anyone thinking about it. The failure belongs to publishers that are not browsers: cameras, GStreamer or FFmpeg pipelines, WHIP ingress — which brings failure modes of its own — and server-side compositors, all of which publish exactly what you told them to publish.
We enabled simulcast and it still doesn't adapt. What now?
Verify it rather than trusting the flag — "simulcast not working" and "the SFU is not switching
quality" are usually the same finding underneath. A publisher that cannot encode multiple layers,
because of the encoder, the hardware or the codec configuration, can still negotiate happily and send
one. Count outbound-rtp entries in getStats() on the publisher. If there is one, the setting did
not take effect and the server was never the problem.
Video went black on iPhones after we turned transcoding off. Is that this?
It is the other half of it. Nothing is wrong with the layers; the profile is. The transcoder was
normalising the camera's H.264 into something iOS negotiates, and passthrough stopped doing that.
Check framesReceived against bytesReceived on the iPhone: data arriving and no frames accepted
means the stream is being rejected, not lost. The section above has the mechanism and a fix that
costs no CPU.
The four things worth taking away
- An SFU selects; it never generates. The layer a weak subscriber needs has to exist before the server can choose it, and the only place it can come from is the publisher or a transcoder.
- Check the publisher, not the server. One
ridand oneoutbound-rtpentry is the whole diagnosis, and it takes two minutes. Every server-side setting in this space is downstream of it. - The encode does not disappear, it relocates. Publisher-side simulcast, vendor transcoding and SVC are three different places to pay for the same work — and for a source that only speaks H.264, SVC quietly turns back into transcoding.
- Transcoding was doing a second job you were not billed for separately. It normalised the codec profile. Turn it off and the camera's own bitstream reaches the browser, where a declared level above what iOS negotiates is dropped in silence. The fix is a bitstream rewrite, not an encoder — but you have to know to look.
If your fleet publishes one stream and your viewers are not on one network, that is not a tuning problem and no amount of server configuration will make it one. Decide where the second encoding is going to be made.
Sizing this decision for a real fleet — what the publisher would cost to build, against what the transcode line costs to keep — is exactly what an Architecture Sprint is for: measured, costed, and yours to act on with or without us.
The rest of the cost-and-capacity picture for an always-on fleet is on real-time video for streaming platforms.
