How do you design rate limiting and cost management for AI APIs?

Because you pay per token and upstream providers impose their own limits, control both request rate and token spend. Enforce per-user and per-tenant quotas with a token-bucket limiter, and track token consumption (not just request count) against a budget. Queue and back-pressure bursts rather than dropping them, and respect provider 429s with exponential backoff plus jitter. Cut cost structurally: route to cheaper models where quality allows, cache aggressively, trim context and output length, and use batch APIs for non-urgent jobs. Add spend alerts and a hard kill-switch per tenant so one runaway loop can't produce a surprise bill. Expose usage back to users so the incentives are visible.