When you send a prompt to an LLM, a lot happens between your request and the streamed reply. Understanding it demystifies three things that otherwise seem arbitrary: why you pay per token, why long contexts get expensive, and why some models feel snappy while others lag. This pairs with our guide on how transformers work.

Two phases: prefill and decode

LLM inference splits into two very different phases.

Prefill processes your entire prompt at once. Because all the prompt’s tokens are known up front, the model handles them in parallel in a single efficient pass. This is compute-heavy but fast — it’s why a long prompt adds less latency than you’d expect. Prefill determines your time-to-first-token.

Decode generates the answer one token at a time. The model produces a token, appends it to the sequence, and runs again to produce the next — autoregressively, in a loop. Each step is a full pass over the model’s weights to produce just one token. This phase is memory-heavy and sets your tokens-per-second.

The KV cache: what makes decode affordable

Here’s the key optimization. During decode, each new token needs to “attend” to all previous tokens. Naively, you’d recompute the key and value vectors for the entire sequence every single step — enormous wasted work.

The KV cache fixes this: it stores the keys and values already computed for past tokens, so each new step only computes them for the one new token and appends to the cache. That turns per-step work from quadratic to linear and is the main reason generation is practical.

The trade-off is memory. The cache grows with context length and batch size, and for long contexts it often becomes the dominant memory consumer — more than the model weights. That memory pressure is the hidden cost behind long-context features.

Why generation is memory-bound

A crucial, counterintuitive fact: decode is usually limited by memory bandwidth, not raw compute. To generate each token, the GPU must read the model’s weights and the KV cache from memory. For a single request, the chip spends more time moving data than doing math — so generation speed is capped by how fast memory can be read, not by the GPU’s peak FLOPS. This is why simply having a faster processor doesn’t linearly speed up generation.

How serving systems make it fast

Because a single request underuses the GPU, the big lever is batching — processing many users’ requests together so the expensive weight-reads are shared. Modern serving stacks add more:

  • FlashAttention (arXiv:2205.14135) computes attention without materializing the full attention matrix, cutting memory use and speeding things up — same math, far less overhead.
  • PagedAttention / vLLM (arXiv:2309.06180) manages the KV cache like virtual memory in pages, eliminating waste and allowing far more requests to be batched — a large throughput win.
  • Quantization stores weights (and sometimes the KV cache) at lower precision, shrinking memory and speeding reads — see LoRA & efficient fine-tuning for the related quantization ideas.

Why this explains your bill and your latency

Everything above shows up in how models are priced and how they feel:

  • Per-token pricing reflects real per-token work. Output tokens cost more than input because decode (one-at-a-time) is pricier than prefill (parallel). Compare rates on our model pricing cheat sheet.
  • Long contexts cost more because the KV cache and attention scale with length.
  • The metrics that matter are time-to-first-token (prefill) and tokens-per-second (decode) — not a single “speed” number.

The honest bottom line

LLM inference isn’t magic pricing — it’s physics. Prefill is a fast parallel pass; decode is a memory-bound loop where the KV cache is both the key optimization and the main cost. Once you see that, per-token pricing, long-context surcharges, and latency all make sense — and you can design prompts and systems that spend those tokens wisely. For the interview version, see our LLMs & transformers questions.