How do you make a RAG system refuse or say 'I don't know' when the answer isn't in the retrieved context?

Two layers help. First, retrieval gating: use similarity-score thresholds or a lightweight relevance check so that if the top results are weak, you short-circuit to a fallback answer instead of forcing generation. Second, prompt and instruction design: explicitly instruct the model to answer only from the provided context and to say it cannot find the answer otherwise, ideally with a required citation for every claim so unsupported statements are visible. Reranking and query rewriting reduce false 'no answer' cases by improving recall first. Evaluate this behavior directly by including unanswerable questions in your test set and measuring the refusal (abstention) rate and false-answer rate. Faithfulness/groundedness scoring (e.g. via an LLM judge or RAGAS) catches cases where the model answered despite insufficient context. Calibrating the threshold trades coverage against hallucination risk.