Why would you put an AI gateway or proxy in front of LLM providers?

A gateway is a single internal service that all app code calls instead of hitting provider APIs directly. It centralises the concerns you'd otherwise reimplement everywhere: authentication and key management, routing (choose model by cost/latency/task), fallback to alternate providers, rate limiting and per-team budgets, caching, logging, and usage analytics. It also gives you a stable interface so you can swap or add providers without touching application code, and a choke point to enforce policy — PII redaction, prompt-injection filtering, and audit trails. The trade-off is one more hop (added latency) and a component that must be highly available, so keep it thin and horizontally scalable.