NVIDIA published new encode-prefill-decode (EPD) disaggregation benchmarks on September 9, 2026, showing that a multimodal serving stack can cut time to first token by as much as 93% in media-heavy tests and improve same-SLO goodput by separating vision encoding from language-model prefill and decode. The important word is can. The same design regressed end-to-end latency by 2.5% in one light-image, long-output case.
That makes EPD an architecture decision, not a universal speed switch. Before adding encoder workers, measure how much request time is actually spent fetching media, decoding it, running the vision transformer, prefilling the language model, and decoding output. Split the encoder only when that isolated stage is large enough to repay queueing, transfer, and operational overhead.
Why Multimodal Requests Block More Than Themselves
A text request can begin prefill as soon as tokenization and routing are complete. An image request adds media download, decompression, model-specific preprocessing, and a vision-transformer forward pass before the language model can consume visual embeddings.
In a simple aggregated deployment, one worker schedules vision encoding, prefill, and decode. That keeps the topology understandable and avoids transferring embeddings between workers. It also creates one queue for very different jobs. A batch containing a heavy image request can delay text-only traffic even though those text requests never need the vision encoder.
The NVIDIA EPD benchmark measured this head-of-line blocking under a sustained 50:50 mix of text and image traffic. Separating the encoder reduced mean text time to first token from 92.3 ms to 53.3 ms, a 42.2% reduction. Image-request TTFT fell from 289.9 ms to 200.6 ms, or 30.8%.
Those numbers do not mean every multimodal service should be split. They show what becomes possible when encoder work is a meaningful part of the critical path and the scheduler can isolate it.
Measure a Stage Budget Before Changing Topology
Start with traces from production-shaped requests, not one synthetic image prompt. For every request, record five spans:
- Media fetch and decode
- Model-specific image preprocessing
- Vision encoder forward pass
- Language-model prefill
- Token decode
Group the results by image count, image resolution or visual-token budget, output length, model, precision, and whether traffic mixes text with media. The useful metric is the encoder share of pre-decode latency:
encoder_share = (media_preprocess_ms + vision_forward_ms) / ttft_ms
Do not use this ratio alone. Track p50 and p95 TTFT, end-to-end latency, throughput under a service-level objective, GPU utilization by worker role, queue time, and embedding-transfer bytes. The architecture has paid for itself only if it improves the metric your users feel without shifting the bottleneck into networking or decode.
NVIDIA's published sweep makes the dependency visible. With five images held constant and output length rising from 128 to 2,048 tokens, heterogeneous EPD's end-to-end advantage narrowed from 20.3% to 5.2%. The colocated version moved from an 11.8% gain to a 2.5% regression. Decode eventually dominated, so optimizing the earlier stage mattered less.
Choose Between Three Placements
There are three useful starting topologies.
| Topology | Best first use | Main risk |
|---|---|---|
| Aggregated | Light media, long outputs, small deployment | Head-of-line blocking as media load grows |
| Colocated EPD | Homogeneous GPU fleet with meaningful encoder load | Encoder workers contend with prefill/decode |
| Separate encoder tier | Mixed GPU fleet and heavy media input | Transfer and orchestration complexity |
Colocated EPD separates queues and batching while keeping encoder and language-model workers on the same GPUs. It avoids reserving an expensive accelerator exclusively for a relatively small vision model. This is usually the first experiment for a homogeneous cluster.
A separate tier becomes attractive when lower-cost GPUs can handle vision encoding while premium GPUs remain dedicated to prefill and decode. NVIDIA's test used RTX 6000D GPUs for encoders and GB200 GPUs for the language-model stages, moving embeddings through NIXL. That is a useful pattern, but it assumes the network and transfer path can stay below the capacity saved on the main tier.
The reproducible Dynamo experiment exposes image count, visual-token budget, output length, model, backend, and topology as sweep dimensions. Copy that shape, then replace its inputs and SLO with your own. A benchmark that cannot represent your request mix is a demo, not a capacity plan.
Test the Break-Even Surface, Not One Headline Number
EPD benefits grow when visual work expands relative to language-model work. More images, higher visual-token budgets, shorter outputs, smaller language models, sparse mixture-of-experts models, and lower language-model precision all move the balance toward the encoder.
In NVIDIA's model-size ablation, colocated EPD delivered 2.62 times the aggregated goodput for a 4B model, 1.50 times for 9B, and only 0.65 times for 27B. The larger dense model made prefill and decode heavy enough that encoder separation no longer paid. Quantizing active language-model weights from BF16 to NVFP4 moved the goodput ratio from 1.78x to 2.64x while the vision encoder stayed in BF16, again increasing the encoder's relative share.
Build a small matrix around the variables you can actually route on:
- image count or total visual tokens;
- expected output-length bucket;
- model and precision;
- repeated versus unique media;
- text-only versus mixed traffic;
- latency SLO and current queue depth.
For each cell, run aggregated and EPD variants at increasing request rates. Mark the point where EPD improves same-SLO throughput after including the encoder tier and network cost. The result should be a routing policy, not a blanket deployment flag.
Optimize the Work Before You Disaggregate It
Splitting a slow stage is not the same as fixing it. NVIDIA's parallel media decoding documentation moves image fetching and decompression into a Rust frontend CPU pool, then transfers decoded pixels to the backend. It can remove serial download and decode time, but it does not eliminate the vision encoder.
The SGLang project's EPD architecture and performance RFC reports a similar production lesson: parallel image download, pipelining, GPU-side preprocessing, batching, and reduced redundant work doubled encoder QPS in its standalone benchmark while keeping average latency roughly comparable. This is independent evidence that the encoder path itself needs profiling and optimization before topology alone can help.
Repeated media creates another lever. Dynamo's embedding cache stores vision-encoder outputs in a CPU-side LRU cache. Product catalogs, shared diagrams, and multi-turn conversations can skip re-encoding on a hit. Entirely unique images receive no benefit, so measure working-set size and hit rate before allocating host memory.
For fleets with multiple backends, multimodal KV routing gives images stable identities and balances cache overlap against projected load. Exact URL bytes are the default identity; equivalent images at different URLs may miss unless frontend decoding hashes the decoded content. That detail can decide whether a cache benchmark survives real CDNs and signed URLs.
Roll Out With a Reversible Gate
Deploy EPD behind workload-aware routing. Start with the cells that showed a clear margin in your sweep, such as multi-image requests with short or medium outputs. Keep light-image, long-output traffic aggregated until data proves otherwise.
Use a canary that records both user-facing and infrastructure outcomes. A practical promotion gate could require:
- p95 TTFT improvement of at least 20%;
- no more than 3% end-to-end regression in any protected workload bucket;
- higher throughput at the same SLO after counting all encoder GPUs;
- stable error rate and no material embedding-transfer retries;
- a rollback path that returns routing to aggregated workers in minutes.
Open-source support is still moving. The vLLM EPD tracker lists active work across connectors, metadata propagation, metrics, video support, tests, and failure handling. Pin framework and container versions, run an end-to-end correctness suite for images and video, and do not assume a roadmap item is production-ready because the architecture exists elsewhere.
Limitations and Tradeoffs
The headline NVIDIA results use specific models, precisions, hardware, networking, token budgets, and SLOs. They are evidence of a pattern, not a forecast for another cluster. Transfer overhead can erase gains on slower networks, and reserving a same-class GPU for a small encoder can reduce overall utilization.
More worker roles also mean more queues, autoscaling policies, health checks, traces, and failure modes. A remote encoder that succeeds after the downstream request times out still burns capacity. Cancellation, backpressure, retry ownership, and partial failure need explicit tests.
Caching changes both economics and privacy posture. Vision embeddings remain derived from user media. Define retention, eviction, tenancy isolation, and observability rules before enabling a shared cache.
Finally, TTFT is not the whole user experience. Long responses can remain decode-bound, and media preprocessing can still dominate before the vision transformer starts. Optimize the largest measured span first.
The Bottom Line
EPD disaggregation is valuable when media encoding is large enough to block prefill, especially for multi-image inputs, short-to-medium outputs, quantized or sparse models, and mixed text-plus-image traffic. It can also lose when decode dominates or coordination overhead exceeds the isolated work.
Trace the five serving stages, reproduce both aggregated and split topologies against your own workload matrix, and promote only the buckets that clear a reversible SLO and cost gate. The right question is not whether encoder disaggregation is faster. It is where your break-even surface begins.
