Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

A new push-based KV transfer approach for vLLM's prefill-decode disaggregation moves coordination off the critical path, yielding up to 3x better TTFT and 30% better TPOT in AWS benchmarks.

By AIWI EditorialReviewed by Chayanika DivekarAI-assisted summary · human-reviewed1 min readPublished Oct 19, 2026

Single source1 source

Reality Check

Availability: Availability unclear
Hype-o-meter: Mostly substance
Who it's for: ML engineers · Enterprises · Developers
Do this now: Review the connector design if you run vLLM disaggregated inference on AWS EC2 P5en with EFA.
Claims check: Performance figures are stated as benchmark results on specific AWS instances; broader generalizability is not addressed in the input.
Corroboration: Single source · 1 source

Key Takeaways

vLLM's prefill-decode disaggregation requires transferring the KV cache between engine instances
A push-based NIXL connector moves transfer coordination off the critical path
Benchmarks report up to 3x better time-to-first-token (TTFT)
Benchmarks also show 30% better time-per-output-token (TPOT) on EC2 P5en with EFA

The AWS Machine Learning Blog describes a push-based KV transfer method for vLLM, which splits large language model inference into separate prefill and decode stages running on different engine instances. Because this disaggregation requires moving the key-value (KV) cache between instances, transfer efficiency directly affects latency. The post introduces a push-based NIXL connector designed to move transfer coordination off the critical path.

In benchmarks run on Amazon EC2 P5en instances using AWS Elastic Fabric Adapter (EFA), the approach reportedly delivers up to 3x better time-to-first-token (TTFT) and 30% better time-per-output-token (TPOT).

Why it matters

TTFT and TPOT are core latency metrics for LLM serving. Reducing the overhead of KV cache transfer in disaggregated setups can improve responsiveness without changing the underlying model.

Who should care

Teams deploying vLLM for disaggregated inference, particularly those running on AWS EC2 P5en instances with EFA, may find the connector relevant to their serving performance.

Was this helpful?

Sources & Further Reading

Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM — AWS Machine Learning Blog · Oct 19, 2026

Frequently Asked Questions

What performance gains are reported?

Up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.

What problem does the connector address?

It moves KV cache transfer coordination off the critical path in vLLM's prefill-decode disaggregation.

How LLM Inference Works: Tokens, KV Cache, and Why It Costs What It Costs — Jul 13, 2026
Jev is Vercel’s fastest-adopted AI model ever — Oct 16, 2026
Meta’s Muse Spark Open now available in Amazon SageMaker JumpStart — Oct 15, 2026
How NVIDIA GPUs Help Accelerate OpenAI’s GPT-6 Astra Ultrafast — Oct 1, 2026

#inference #llm #benchmarks #cloud #gpu