If you’ve read what RAG is, you know the idea: give a language model the right documents at the moment you ask, so it answers from real sources instead of memory. This guide is the next step — how to actually design one that works in production. It’s the classic answer to a system-design interview question (see our AI system design questions), and the good news is that most of RAG is a search problem with an LLM on the end.

The two pipelines

Every RAG system is really two pipelines that meet at a vector index.

Offline — ingestion (runs when documents change):

  1. Load documents from their sources (files, databases, web, APIs).
  2. Chunk each into passages.
  3. Embed each chunk into a vector.
  4. Index the vectors plus metadata in a store.

Online — query (runs on every question):

  1. Retrieve the most relevant chunks for the question.
  2. Augment the prompt with those chunks as context.
  3. Generate an answer, grounded in that context, with citations.

Design each stage deliberately — the defaults are rarely optimal.

Chunking: the underrated decision

Chunking quietly determines your ceiling. Too large and you stuff irrelevant text into the context (and pay for it); too small and you shatter the meaning across fragments the retriever can’t connect.

  • Respect structure. Split on natural boundaries — headings, paragraphs, sections, table rows — not blindly every N characters. A chunk that straddles two topics retrieves poorly.
  • Size to your budget. Pick a chunk size that lets several relevant chunks fit comfortably in the model’s context window alongside the question and answer.
  • Overlap a little. A small overlap between adjacent chunks preserves context that would otherwise be cut mid-thought.
  • Keep metadata. Store the source, section, date, and permissions with each chunk — you’ll need them for filtering, citations, and access control.

Embeddings and the vector index

Each chunk is turned into an embedding — a vector where semantically similar text sits close together (see what are embeddings). Choose an embedding model that fits your domain and languages, and remember: if you change the embedding model, you must re-embed everything, since vectors from different models aren’t comparable.

Store the vectors in an index that supports fast approximate nearest-neighbor search — most use a graph-based method like HNSW (arXiv:1603.09320) under the hood. For a few thousand docs, an in-memory index (e.g. FAISS) is fine; a managed vector database pays off at scale or when you need metadata filtering, high availability, and frequent updates.

Retrieval: where RAG lives or dies

Naive “embed the question, take the top-k by cosine similarity” is the version that disappoints. Two upgrades matter most:

  • Hybrid search. Combine dense vector search (semantic) with keyword/BM25 search (exact terms). Vectors miss exact matches like error codes, product SKUs, and names; keywords miss paraphrases. Together they’re far more robust.
  • Re-ranking. Retrieve a larger candidate set (say top-20), then use a cross-encoder re-ranker to score each candidate against the query and keep the best few. Re-ranking is often the single biggest quality jump in a RAG system.

Also apply metadata filters (date, source, permissions) so retrieval only considers chunks the user is allowed to see and that are still current.

The prompt and generation

Assemble the retrieved chunks into a grounded prompt: instruct the model to answer using only the provided context, to cite which chunks it used, and to say it doesn’t know when the context doesn’t contain the answer. Stream the response so time-to-first-token feels fast. Keep the model’s temperature low for factual grounding. If you’re choosing the generation model, our AI tools comparison ranks the options.

Evaluate the two stages separately

The most important debugging principle in RAG: a wrong answer is either a retrieval miss or a generation failure, and only stage-wise metrics tell you which.

  • Retrieval: on a labelled set of questions with known relevant passages, measure whether the right chunk lands in the top-k (hit rate, MRR/NDCG).
  • Generation given context: measure faithfulness (is every claim grounded in the retrieved chunks?) and answer relevance.

Our full guide on how to evaluate an LLM app covers the tooling. Build this eval early — it’s the feedback loop that turns RAG tuning from guesswork into progress.

Production concerns

The core pipeline is only half the system. In production you also design for:

  • Freshness. Re-ingest changed documents on a schedule or via events, and delete stale vectors. Knowledge updates by editing documents, not retraining — that’s a core RAG advantage, but only if ingestion keeps up.
  • Access control. Enforce per-user/tenant permissions at retrieval time, server-side, so a user can never retrieve a document they shouldn’t see. Never trust the client, and never share a cache across permission boundaries.
  • Caching. Cache embeddings and frequent query results; provider prompt-caching cuts the cost of a shared instruction/context prefix.
  • Latency and cost. Trim the number and size of retrieved chunks, cap output length, and route simple queries to a smaller model. Watch token spend — long contexts get expensive fast.
  • Handling “no answer.” When retrieval finds nothing relevant, return that honestly instead of letting the model hallucinate.

The honest bottom line

A good RAG system is 80% search engineering — chunking, hybrid retrieval, re-ranking, and evaluation — and 20% prompting. Teams that treat it as “just call an LLM with some documents” ship demos that fall apart on the sixth question; teams that invest in retrieval quality and stage-wise evaluation ship systems that hold up. Start simple, measure retrieval and generation separately, and add infrastructure only when you hit a real limit.