How can a photos app find every picture of “a dog on a beach” when nobody ever tagged those photos? The answer is a 2021 model called CLIP, and the idea behind it — a shared space for text and images — quietly powers a huge amount of modern multimodal AI. Here’s how it works.
The problem: text and images live in different worlds
A computer stores an image as a grid of pixels and text as a sequence of tokens. There’s no obvious way to ask “does this picture match these words?” because the two are completely different kinds of data. To search across them, you need a common language.
CLIP’s idea: one shared embedding space
CLIP (Learning Transferable Visual Models From Natural Language Supervision) trains two encoders at once — one that turns images into vectors, one that turns text into vectors — with a single goal: matching image-caption pairs should land close together in the shared vector space, and mismatched pairs should land far apart.
It learns this through a contrastive objective over hundreds of millions of image-text pairs scraped from the web. For each batch, the model pulls each image toward its true caption and pushes it away from all the other captions. Do that at massive scale and you get a space where the vector for a photo of a beach sits near the vector for the words “a beach.”
What a shared space unlocks
Once images and text live in the same space, comparing them is just measuring distance:
- Cross-modal search. Embed a text query, embed all your images, and return the nearest images — that’s “search my photos by describing them.” It works in reverse too: give an image, find matching captions or documents.
- Zero-shot classification. To classify an image without task-specific training, embed candidate label phrases (“a photo of a cat,” “a photo of a dog”) and pick whichever is closest to the image. CLIP was remarkable precisely because it did this well on categories it was never explicitly trained to label.
Extending it to video and beyond
Video is just images over time plus audio. To make it searchable, you sample keyframes and embed them into the same space, and often transcribe the audio to text and embed that too. Now a text query can retrieve a moment inside an hour of footage. The same recipe generalizes: as long as you can encode a modality into the shared space, you can search across it.
Building a real multimodal search system
In practice, a cross-modal search pipeline looks like this:
- Offline: run all your content — images, video frames, documents — through the right encoders, and store the resulting vectors plus metadata in a vector database.
- Online: embed the incoming query (text, image, or both), run a nearest-neighbour search, then re-rank the top results.
- Hybrid signals: combine the embedding search with keyword matches on captions and transcripts, so exact terms aren’t lost.
This is the same retrieval backbone as RAG, extended across modalities.
The honest caveats
The magic has limits. Cross-modal alignment is imperfect — the image and text spaces are pulled together, not perfectly fused — so retrieval quality varies by domain and you should measure it on labelled query-item pairs (recall@k, MRR), not assume it works. CLIP-style models also inherit biases from their web training data, and they can be fooled by adversarial or out-of-distribution inputs.
Still, the core idea — encode everything into one space and let distance mean similarity — is one of the most useful in applied AI. For the interview-ready version, see our multimodal AI interview questions; for the foundations of the embeddings underneath, read what are embeddings.