What is multimodal RAG, and how does it differ from text-only RAG?

Multimodal RAG retrieves and grounds on non-text content — images, tables, charts, audio, video — not just text passages. Two common approaches. One: use a shared-embedding model (like CLIP) so you can embed images and the query in the same space and retrieve images directly, then feed them to a vision-language model to answer. Two: convert other modalities into text first — caption images, transcribe audio, extract table text — then run ordinary text RAG over those descriptions. Many systems combine both. The extra challenges over text RAG are embedding heterogeneous content well, chunking things like documents-with-figures without losing the figure, larger storage and compute, and evaluating retrieval quality across modalities. The generator must also be multimodal if you pass raw images rather than captions.