RAG in One Lesson (Answer From Your Own Data)

Last reviewed Jul 7, 2026Building with AI · intermediate

What you'll learn

Describe the RAG loop: chunk, embed, retrieve, generate
Explain why retrieval quality decides answer quality
Know when RAG beats fine-tuning

The problem RAG solves

A plain LLM answers from its training — which is frozen at a cutoff date, doesn’t include your private documents, and blurs details. Retrieval-augmented generation (RAG) fixes this by fetching the right source text at question time and handing it to the model as context. The model then answers from real, current, specific material instead of fuzzy memory.

This is how you build a chatbot over your company handbook, product docs, or knowledge base.

The four-step loop

Chunk — split your documents into retrievable pieces, ideally along structure (a heading and its section), keeping titles attached so each chunk makes sense alone.
Embed — convert each chunk into a vector (the embeddings lesson) and store it.
Retrieve — embed the user’s question, find the closest chunks, and (for best results) rerank them.
Generate — give those chunks to the LLM with a firm instruction: answer only from this context; if it’s not here, say so; cite what you used.

The one thing that makes or breaks it

Say it out loud: most RAG failures are retrieval failures. If step 3 doesn’t surface the right chunk, no amount of prompt polish in step 4 will produce a correct answer — the model simply never saw the fact. So when a RAG system is wrong, check retrieval first: was the correct chunk even in the set you passed in? Teams that measure retrieval (did we fetch the right passage?) separately from generation (did the model use it well?) fix problems in days; teams that treat it as one black box flail.

RAG vs fine-tuning

Reach for RAG when answers must be grounded in factual, changing, or private information — you update the documents and the system stays current, no retraining. Reach for fine-tuning when you need a consistent style or format the base model can’t hold. They’re not rivals; production systems often use RAG for knowledge and light fine-tuning for tone.

To take this to production quality — chunking strategy, hybrid retrieval, reranking, and evals — work through the RAG best-practices playbook.

Try it yourself — Simulate grounded answering

Prompt

Answer ONLY using the context below. If the answer isn't in the context,
say "Not found in the provided documents."
Context: """Our refund policy: enterprise plans are refundable within 30
days; monthly plans are non-refundable."""
Question: Can I get a refund on my monthly plan?

Sample output

No — according to the provided policy, monthly plans are non-refundable.
(Enterprise plans are refundable within 30 days, but that doesn't apply to
monthly plans.)

Sample output — AI responses may vary.

Try in Claude →

Common mistakes

Blaming the LLM for wrong answers when retrieval was the problem — the right chunk was never fetched. Measure retrieval separately.
Chunking documents into blind fixed-size windows that cut sentences and tables. Chunk along structure (headings/sections) instead.
Skipping the 'answer only from context, else say not found' instruction, so the model quietly blends in its own (possibly stale) knowledge.

How to check the AI's answer

Log which chunks were retrieved for each answer. If answers are wrong, check whether the right chunk was even in the retrieved set before touching the prompt.
Ask a question whose answer is deliberately NOT in your documents — a good RAG system should say 'not found,' not invent one.

Key terms

RAG (retrieval-augmented generation) — A pattern where relevant documents are retrieved and given to the LLM as context so it answers from them.

Chunk — A piece of a document (ideally a section) that is embedded and retrieved as a unit.

Grounding — Constraining the model to answer from provided sources, with citations, rather than its own memory.