How do you design for the latency-versus-quality trade-off in AI systems?

Start from the user's real latency budget — a chat UI can stream tokens and feel fast even at multi-second totals, while an autocomplete needs sub-100ms. Levers to trade quality for speed: route simple requests to a smaller/cheaper model and hard ones to a frontier model (a classifier or heuristic decides), stream output so time-to-first-token is low, cache repeated or semantically similar queries, shrink retrieved context, and cap output length. For agents, parallelise independent tool calls. Reserve the slow, high-quality path for cases that need it. Measure both p50 and p95 latency and quality on a fixed eval set, then tune the routing threshold — the goal is 'good enough, fast' for most traffic and 'best' where it matters.