How do you decide chunk size and overlap when splitting documents for RAG?

Chunk size balances two failure modes: too large dilutes the embedding (multiple topics in one vector, lower precision and wasted context), too small fragments ideas and loses the context needed to answer. A common starting range is 256-512 tokens for dense factual QA, larger for narrative or code. Overlap (often 10-20% of chunk size) repeats text at boundaries so a sentence split across chunks isn't orphaned, preserving continuity for retrieval and generation. Prefer semantic or structural boundaries (paragraphs, headings, sentences) over fixed character counts, and respect the embedding model's max sequence length so nothing is silently truncated. There's no universal answer: tune on your own eval set, and consider storing small chunks for retrieval precision while returning larger parent context (parent-document retrieval) to the LLM.