How do you handle failover and fallback strategies for AI systems?

Assume the model API will fail, time out, or rate-limit, and design layers. Retry transient errors with capped exponential backoff and jitter. On persistent failure, fail over to an alternate provider or a secondary model behind the same interface — this is a strong argument for a provider-agnostic gateway. If all models are down, degrade gracefully: serve a cached answer, return retrieved sources without a generated summary, or show an honest 'try again shortly' rather than an error page. Use timeouts and circuit breakers so a slow provider doesn't exhaust your threads and cascade. Make retries idempotent to avoid duplicate side effects. Log every fallback so you can see how often the primary path fails.