mediumLLMs & TransformersReviewed Jul 24, 2026

How do top-k and top-p (nucleus) sampling control LLM text generation?

Both truncate the probability distribution over the next token before sampling, to avoid picking absurd low-probability tokens while keeping variety. Top-k keeps only the k highest-probability tokens, renormalizes, and samples from them; k is fixed, so in a very confident context you may still include unlikely options, and in a flat distribution you may exclude good ones. Top-p (nucleus) instead keeps the smallest set of tokens whose cumulative probability exceeds p (e.g. 0.9), so the candidate pool grows when the model is uncertain and shrinks when it's confident, making it more adaptive. They're often combined with temperature, which reshapes the distribution first. Lower p/k yields safer, more deterministic text; higher values yield more diverse, creative, or risky output. Setting k=1 is equivalent to greedy decoding.

samplingdecodingtop-ptop-k

More LLMs & Transformers questions

See all LLMs & Transformers questions →