AI System Design: Interview Questions
The system-design round for AI engineers — how to architect RAG, agents, and LLM services that stay fast, cheap, and reliable at scale. Covers caching, fallbacks, rate limiting, multi-tenancy, and the latency-vs-quality trade-offs interviewers actually probe.
easyHow do you approach an open-ended AI system design question?
Treat it like any system-design round, then add the AI-specific layers. First clarify requirements: who uses it, expected traffic, latency budget, accuracy bar, and cost ceiling. Sketch the data flow — ingestion, retrieval or context building, the model call, and post-processing. Then reason explicitly about the three levers that dominate AI systems: quality (which model, grounding, evaluation), latency (caching, streaming, smaller models), and cost (token usage, batching, routing cheap-vs-expensive models). Call out failure modes — the model API can be slow, rate-limited, or wrong — and design fallbacks. Finish with how you'd measure success and catch regressions. Interviewers reward stated assumptions and trade-off reasoning far more than a 'perfect' diagram.
mediumHow would you design a RAG system to chat with a company's documents?
Two pipelines. Offline ingestion: load documents, chunk them (respecting structure — headings, tables), embed each chunk, and store vectors plus metadata in a vector database, re-running on updates. Online query: embed the user's question, retrieve the top-k relevant chunks (often hybrid — vector plus keyword — then re-rank), assemble a grounded prompt with citations, and call the LLM. Stream the answer back. Key design choices: chunk size and overlap, how many chunks fit the context budget, access control so users only retrieve permitted documents, and caching of embeddings and frequent queries. Add evaluation on a golden question set and a 'no relevant context found' path so the model says it doesn't know rather than hallucinating.
mediumHow do you design for the latency-versus-quality trade-off in AI systems?
Start from the user's real latency budget — a chat UI can stream tokens and feel fast even at multi-second totals, while an autocomplete needs sub-100ms. Levers to trade quality for speed: route simple requests to a smaller/cheaper model and hard ones to a frontier model (a classifier or heuristic decides), stream output so time-to-first-token is low, cache repeated or semantically similar queries, shrink retrieved context, and cap output length. For agents, parallelise independent tool calls. Reserve the slow, high-quality path for cases that need it. Measure both p50 and p95 latency and quality on a fixed eval set, then tune the routing threshold — the goal is 'good enough, fast' for most traffic and 'best' where it matters.
mediumHow do you implement caching strategies for LLM applications?
Several layers, from cheapest to smartest. Exact-match cache: hash the full prompt and store the response — trivial and safe for deterministic (low-temperature) calls. Semantic cache: embed the query and return a stored answer when a past query is similar enough (above a tuned threshold) — powerful but risky, since near-duplicates can have different correct answers, so set the threshold conservatively. Prompt/prefix caching: many providers cache a shared system prompt or document prefix so you only pay to process the changing suffix — big savings for RAG and long instructions. Also cache embeddings and retrieval results. Always add TTLs and cache-busting for content that changes, and never cache personalised or permission-sensitive responses across users.
mediumHow do you design rate limiting and cost management for AI APIs?
Because you pay per token and upstream providers impose their own limits, control both request rate and token spend. Enforce per-user and per-tenant quotas with a token-bucket limiter, and track token consumption (not just request count) against a budget. Queue and back-pressure bursts rather than dropping them, and respect provider 429s with exponential backoff plus jitter. Cut cost structurally: route to cheaper models where quality allows, cache aggressively, trim context and output length, and use batch APIs for non-urgent jobs. Add spend alerts and a hard kill-switch per tenant so one runaway loop can't produce a surprise bill. Expose usage back to users so the incentives are visible.
hardHow do you handle failover and fallback strategies for AI systems?
Assume the model API will fail, time out, or rate-limit, and design layers. Retry transient errors with capped exponential backoff and jitter. On persistent failure, fail over to an alternate provider or a secondary model behind the same interface — this is a strong argument for a provider-agnostic gateway. If all models are down, degrade gracefully: serve a cached answer, return retrieved sources without a generated summary, or show an honest 'try again shortly' rather than an error page. Use timeouts and circuit breakers so a slow provider doesn't exhaust your threads and cascade. Make retries idempotent to avoid duplicate side effects. Log every fallback so you can see how often the primary path fails.
mediumHow do you design an AI system that gracefully degrades when the model is unavailable?
Decide in advance what a 'reduced but useful' experience looks like, since AI is often one feature inside a larger product. Tiers of degradation: swap the frontier model for a faster/cheaper one, then a cached or templated response, then a non-AI path — for search, fall back to keyword results; for a RAG chatbot, return the retrieved source passages with citations even if you can't generate a summary. Keep the rest of the app working so one dependency doesn't take the whole product down. Communicate honestly in the UI ('summaries are temporarily unavailable') rather than failing silently or hallucinating. Circuit breakers trigger the downgrade automatically and recover when the model is healthy again.
mediumHow would you design a conversational AI system with memory across sessions?
Separate short-term from long-term memory. Short-term is the current conversation, kept in the context window — summarise or truncate older turns when it fills. Long-term memory persists across sessions: after each conversation, extract durable facts (preferences, key details) and store them, either as structured records or as embedded snippets in a vector store keyed by user. At the start of a session, retrieve the relevant memories and inject them into the prompt — retrieval, not stuffing everything in. Design choices: what's worth remembering (avoid hoarding noise), how to update or contradict stale facts, and privacy — users must be able to view and delete their memory. Guard against prompt-injected 'remember this' instructions poisoning long-term state.
hardHow would you design a multi-tenant AI platform where each business gets its own chatbot?
Isolation is the core problem. Each tenant needs their own knowledge base, configuration (prompt, branding, allowed tools), and — critically — data isolation so one business can never retrieve another's documents. Partition vector storage by tenant (separate namespaces/collections or a mandatory tenant filter on every query) and enforce it server-side, never trusting the client. Share the serving infrastructure for efficiency but meter usage and apply per-tenant rate limits and budgets so a noisy tenant can't degrade others. Provide per-tenant evaluation and analytics. Watch for cross-tenant leakage via caching (never share a semantic cache across tenants) and prompt injection. The pattern is shared compute, strictly isolated data and config.
mediumWhy would you put an AI gateway or proxy in front of LLM providers?
A gateway is a single internal service that all app code calls instead of hitting provider APIs directly. It centralises the concerns you'd otherwise reimplement everywhere: authentication and key management, routing (choose model by cost/latency/task), fallback to alternate providers, rate limiting and per-team budgets, caching, logging, and usage analytics. It also gives you a stable interface so you can swap or add providers without touching application code, and a choke point to enforce policy — PII redaction, prompt-injection filtering, and audit trails. The trade-off is one more hop (added latency) and a component that must be highly available, so keep it thin and horizontally scalable.
hardHow do you design a RAG system that handles conflicting information across sources?
First detect the conflict, then decide policy. At retrieval, attach metadata to every chunk — source, publish date, authority level. When retrieved chunks disagree, you have options: prefer the most recent or most authoritative source via re-ranking; or surface the disagreement, having the model present both positions with citations rather than silently picking one. Which is right depends on the domain — for policy docs, recency and authority win; for research, showing the debate is more honest. Instruct the model to ground every claim in a cited chunk and to flag when sources conflict instead of blending them into a false consensus. Log conflicts so owners can fix the underlying data. Never let the model average two contradictory facts into a made-up middle.
hardHow do you approach capacity planning for an AI system?
Work from expected load to resources. Estimate peak requests per second and average input/output tokens per request — tokens, not just request count, drive both cost and GPU/throughput needs. If self-hosting, capacity is bounded by GPU memory (model weights plus the KV cache, which grows with context length and batch size) and by throughput under your latency target; benchmark tokens-per-second at realistic batch sizes. If using an API, capacity is really the provider's rate limits plus your budget, so plan quota and fallbacks. Add headroom for spikes, use autoscaling with warm capacity to hide cold starts, and batch non-urgent work. Re-plan as traffic and context lengths grow — long-context features quietly multiply memory and cost.
hardHow would you design a multimodal search system over text, images, and video?
Use a shared embedding space so a query in one modality can retrieve another. A model like CLIP maps images and text into the same vector space; for video, sample keyframes (and optionally transcribe audio to text) and embed those. Offline: extract, embed, and index all content with metadata in a vector database. Online: embed the query — text, image, or both — and run nearest-neighbour search, then re-rank. Design choices: how finely to sample video (cost vs recall), combining modalities in one query, and hybrid search that also uses captions/transcripts as text signals so keyword matches aren't lost. Watch storage and compute — video embeddings are large — and evaluate cross-modal retrieval quality on a labelled set, since shared-space alignment is imperfect.
mediumWhat metrics would you use to evaluate the performance of an AI system?
To evaluate the performance of an AI system, you should consider various metrics depending on the nature of the task. For classification tasks, metrics like accuracy, precision, recall, and F1-score are essential. For regression tasks, mean absolute error (MAE), mean squared error (MSE), and R-squared are commonly used. Additionally, in natural language processing (NLP) systems, perplexity and BLEU scores can be used for evaluating language models. It's also crucial to evaluate user satisfaction and system efficiency metrics such as latency and throughput.
hardHow do you ensure scalability in an AI system's architecture?
To ensure scalability in an AI system's architecture, design with modular components that can be independently scaled. Leverage microservices architecture to separate different functionalities, allowing each component to scale horizontally as needed. Use cloud services to take advantage of elastic compute resources. Implement load balancing to distribute requests evenly and consider using asynchronous processing for tasks that do not require immediate feedback. Data storage should also be scalable; opt for distributed databases or data lakes that can handle growing datasets efficiently.
No questions match your filter.