This AWS Machine Learning Blog post describes building a tiered KV cache for large language model inference on Amazon SageMaker HyperPod using Curvine. According to the post, running LLM inference at scale forces a trade-off between provisioning oversized GPU instances or accepting slow time-to-first-token. The described method extends the KV cache into a shared, distributed NVMe pool, allowing multiple replicas to reuse cache at near-local-disk speeds while running on cost-efficient instances.
Why it matters
KV cache management is a key constraint when serving large models. Offloading the cache into a shared NVMe tier is presented as a way to reduce reliance on expensive GPU instances without incurring the latency penalty of cold caches.
Who should care
Teams deploying large LLMs on Amazon SageMaker HyperPod and engineers concerned with inference cost and time-to-first-token may find the approach relevant.