RAG & Vector Databases: Interview Questions
Retrieval-augmented generation is one of the most-asked applied AI topics — chunking, embeddings, vector search, reranking, and how to debug a RAG system that returns wrong answers.
easyWhat is retrieval-augmented generation (RAG)?
RAG is a pattern where relevant documents are retrieved at query time and given to an LLM as context, so it answers from real sources instead of its trained memory. The loop: chunk your documents, embed them into vectors, store them, embed the user's query, retrieve the closest chunks, and generate an answer grounded in them. It's how you build assistants over private or fast-changing knowledge without retraining the model, and it reduces hallucination when done well.
easyWhat is a vector database and why use one for RAG?
A vector database stores embeddings and does fast nearest-neighbour search — finding the vectors closest to a query vector using approximate algorithms (e.g. HNSW) that scale to millions of items. You use one in RAG to retrieve the most semantically similar chunks quickly. Options include Pinecone, Weaviate, Qdrant, Milvus, and pgvector (Postgres). For small corpora a simple in-memory index suffices; a dedicated vector DB matters at scale and with metadata filtering.
mediumHow should you chunk documents for RAG?
Chunk along the document's structure — headings and sections — rather than blind fixed-size windows that cut sentences and split tables. Keep the document title and section path attached to each chunk so it makes sense in isolation. Aim for chunks big enough to hold a complete idea but small enough to be specific; some overlap helps continuity. Good chunking is one of the highest-leverage RAG decisions: a chunk that carries its own context retrieves and reads correctly on its own.
hardWhat is hybrid search and why is it often better than pure vector search?
Hybrid search combines lexical search (BM25/keyword) with vector (semantic) search, then merges the results. Vector search captures meaning but can miss exact tokens — part numbers, error codes, names, rare terms — while keyword search nails those but misses paraphrases. Combining them, often with a reranker on the merged candidates, consistently beats either alone on production QA corpora. It's a common answer to 'my RAG misses obvious exact-match results'.
mediumWhat is reranking in a RAG pipeline?
Reranking is a second stage that reorders retrieved candidates by relevance using a more powerful (but slower) model, typically a cross-encoder that scores each query-chunk pair directly. You retrieve a broad set cheaply (say top 50), rerank to the best few, and pass only those to the LLM. This improves precision and lets you send fewer, better chunks — improving answers and cutting token cost at the same time.
hardYour RAG system returns wrong answers. How do you debug it?
Check retrieval first — most RAG errors are retrieval failures. Log the chunks retrieved for the failing query and ask: was the correct chunk even in the set? If not, the problem is chunking, the embedding model, or missing hybrid search. If the right chunk was retrieved but the answer is still wrong, it's a generation/grounding problem — tighten the prompt to answer only from context and cite sources. Measure retrieval (recall@k) separately from generation so you know which stage to fix.
mediumHow do you keep a RAG system's answers fresh and access-controlled?
Attach metadata to every chunk — source, updated-at, and access permissions — and filter at query time: enforce the user's entitlements and prefer recent versions. Re-embed documents when they change (event-driven or scheduled) rather than re-indexing everything rarely. The two worst RAG incidents in practice — leaking a document to the wrong user and answering from a superseded version — are both metadata-and-pipeline problems, not model problems.
hardWith large context windows, is RAG still needed?
Often yes. Even with big context windows, stuffing everything in is expensive (you pay for all those input tokens every call), can bury key facts in the middle where models under-use them, and doesn't scale to large or changing corpora. RAG retrieves only what's relevant, keeping cost down and precision up, and updates instantly as documents change. Long context and RAG are complementary — long context helps within a request; RAG decides what deserves to be in it.
mediumWhen should you use cosine similarity versus dot product versus Euclidean distance for vector search?
Cosine similarity measures the angle between vectors and ignores magnitude, so it's the default when embeddings encode direction (semantic meaning) but vary in length. Euclidean (L2) distance measures straight-line distance and is sensitive to magnitude. Dot product combines both direction and magnitude; it rewards larger-norm vectors. The key rule: match the metric your embedding model was trained with. Many models (e.g. OpenAI, sentence-transformers) produce normalized or near-normalized vectors, and when vectors are unit-normalized, cosine, dot product, and Euclidean rank neighbors identically, so the choice mainly affects speed and index compatibility. Dot product is common in maximum-inner-product-search (MIPS) setups where norm carries signal. Always normalize consistently at index and query time.
hardWhat are the tradeoffs between HNSW and IVF indexes for approximate nearest neighbor search?
HNSW (Hierarchical Navigable Small World) builds a multi-layer graph and traverses greedily from coarse to fine layers. It gives excellent recall/latency and needs no training, but uses more memory (stores graph edges) and is costlier to build; key params are M (edges per node) and efConstruction/efSearch. IVF (Inverted File) clusters vectors with k-means into nprobe-searchable partitions; you search only the closest lists. IVF trains faster to query, uses less memory, and scales to huge datasets, especially combined with product quantization (IVF-PQ) to compress vectors, but recall depends heavily on nlist/nprobe tuning and it requires a training step on representative data. Rule of thumb: HNSW for high recall at moderate scale in RAM; IVF-PQ for billion-scale or memory-constrained deployments where some recall loss is acceptable.
mediumHow do you choose an embedding model and its dimensionality for a RAG system?
Start from your domain and task: use benchmarks like MTEB as a starting point but validate on your own labeled queries, since domain fit often beats leaderboard rank. Consider retrieval quality, max sequence length (must cover your chunk size), language coverage, and cost/latency. Higher dimensions (e.g. 1536, 3072) can capture more nuance but increase storage, memory, and search cost roughly linearly. Many modern models support Matryoshka representation learning, letting you truncate dimensions (e.g. 3072 to 512) with graceful quality loss to save space. Also weigh open-source self-hosting (no per-call cost, data privacy, but you manage infra) versus API models. Critically, the same model must embed both documents and queries, and re-embedding everything is expensive, so choose deliberately to avoid churn.
mediumHow do you evaluate retrieval quality using recall@k, precision@k, and MRR/NDCG?
Retrieval metrics need a labeled set of queries with known relevant documents. Recall@k = fraction of all relevant docs that appear in the top k; for RAG this is often the most important metric because if the answer isn't retrieved, the LLM can't use it. Precision@k = fraction of the top k that are relevant, which matters when context window or reranking budget is tight. MRR (Mean Reciprocal Rank) rewards putting the first relevant result high; useful for single-answer lookups. NDCG accounts for graded relevance and position, discounting hits lower in the ranking. In practice, tune retrieval for high recall@k (e.g. retrieve 20-50), then use a reranker to improve precision at the smaller k you actually feed the LLM. Always report k explicitly.
mediumWhat are query rewriting and query expansion, and how do they improve retrieval?
User queries are often short, ambiguous, or use different vocabulary than the documents. Query rewriting transforms the query before retrieval: resolving conversational context (turning 'what about its pricing?' into a standalone question), correcting typos, or decomposing a multi-part question into sub-queries retrieved separately. Query expansion adds related terms or generates multiple query variants to widen recall. A popular technique is HyDE (Hypothetical Document Embeddings): the LLM drafts a hypothetical answer, and you embed that instead of the raw question, because the fake answer sits closer to real answer passages in embedding space. Multi-query retrieval generates several paraphrases and unions the results. These methods trade extra LLM latency/cost for better recall, and are especially valuable for underspecified or conversational queries.
hardHow does ColBERT's late-interaction multi-vector retrieval differ from single-vector dense retrieval?
Standard dense retrieval compresses a whole document into one vector, losing token-level detail. ColBERT keeps one vector per token and scores a query-document pair with 'late interaction': for each query token it finds the maximum similarity across all document tokens (MaxSim), then sums those maxima. This preserves fine-grained term matching while staying more efficient than full cross-encoder attention, giving strong out-of-domain generalization. The cost is storage and complexity: you store many vectors per document (partly mitigated by quantization in ColBERTv2), and retrieval needs specialized indexes like PLAID. It's often used as a first-stage retriever or reranker. The tradeoff is higher index size and engineering overhead versus better precision than single-vector search, particularly on queries needing exact term correspondence.
mediumWhy is metadata filtering important in vector search and what are pre- vs post-filtering tradeoffs?
Metadata filtering restricts search to vectors matching structured attributes (date, source, tenant, language, access permissions), which is essential for correctness (multi-tenant isolation, security) and relevance (only search recent or authorized docs). Post-filtering runs ANN search first then drops non-matching results, which is simple but can return too few or zero hits if matches are rare in the top-k. Pre-filtering restricts the candidate set before/during the ANN traversal, guaranteeing enough matching results but potentially breaking the ANN index's efficiency (graph or cluster structure assumes the full set). Modern vector DBs (Qdrant, Weaviate, pgvector, Pinecone) implement filtered ANN that integrates the filter into traversal for a balance. Design metadata schemas and indexes upfront; high-cardinality or selective filters especially need engine-native filtered search to stay both correct and fast.
easyHow do you decide chunk size and overlap when splitting documents for RAG?
Chunk size balances two failure modes: too large dilutes the embedding (multiple topics in one vector, lower precision and wasted context), too small fragments ideas and loses the context needed to answer. A common starting range is 256-512 tokens for dense factual QA, larger for narrative or code. Overlap (often 10-20% of chunk size) repeats text at boundaries so a sentence split across chunks isn't orphaned, preserving continuity for retrieval and generation. Prefer semantic or structural boundaries (paragraphs, headings, sentences) over fixed character counts, and respect the embedding model's max sequence length so nothing is silently truncated. There's no universal answer: tune on your own eval set, and consider storing small chunks for retrieval precision while returning larger parent context (parent-document retrieval) to the LLM.
hardWhat are parent-document retrieval and contextual retrieval, and what problems do they solve?
Small chunks retrieve precisely but often lack the surrounding context an LLM needs to answer well. Parent-document (small-to-big) retrieval indexes small chunks for matching, but when a chunk hits, it returns the larger parent section or document that contains it, giving the model fuller context while keeping retrieval precise. Contextual retrieval, popularized by Anthropic, addresses a different problem: an isolated chunk may be ambiguous ('the company grew 3%' — which company, which year?). Before embedding, an LLM prepends a short chunk-specific context blurb situating it within the whole document, which meaningfully reduces retrieval failures, especially when combined with BM25 and reranking. Both trade extra indexing work or storage for better answer quality; contextual retrieval adds one-time LLM cost per chunk (reducible with prompt caching).
mediumHow do you make a RAG system refuse or say 'I don't know' when the answer isn't in the retrieved context?
Two layers help. First, retrieval gating: use similarity-score thresholds or a lightweight relevance check so that if the top results are weak, you short-circuit to a fallback answer instead of forcing generation. Second, prompt and instruction design: explicitly instruct the model to answer only from the provided context and to say it cannot find the answer otherwise, ideally with a required citation for every claim so unsupported statements are visible. Reranking and query rewriting reduce false 'no answer' cases by improving recall first. Evaluate this behavior directly by including unanswerable questions in your test set and measuring the refusal (abstention) rate and false-answer rate. Faithfulness/groundedness scoring (e.g. via an LLM judge or RAGAS) catches cases where the model answered despite insufficient context. Calibrating the threshold trades coverage against hallucination risk.
No questions match your filter.