How do you approach capacity planning for an AI system?

Work from expected load to resources. Estimate peak requests per second and average input/output tokens per request — tokens, not just request count, drive both cost and GPU/throughput needs. If self-hosting, capacity is bounded by GPU memory (model weights plus the KV cache, which grows with context length and batch size) and by throughput under your latency target; benchmark tokens-per-second at realistic batch sizes. If using an API, capacity is really the provider's rate limits plus your budget, so plan quota and fallbacks. Add headroom for spikes, use autoscaling with warm capacity to hide cold starts, and batch non-urgent work. Re-plan as traffic and context lengths grow — long-context features quietly multiply memory and cost.