If you’ve read what embeddings are, the next question is: where do you put millions of them so you can search them instantly? That’s what a vector database is for. It’s the quiet workhorse behind RAG, semantic search, and AI agent memory.
What it actually does
An embedding turns a chunk of text (or an image, or audio) into a vector — a list of numbers — where semantically similar things sit close together. A vector database stores those vectors and answers one core question extremely fast:
“Given this query vector, which stored vectors are the most similar?”
That’s nearest-neighbor search. Similarity is usually measured by cosine similarity or dot product. The result is the handful of items closest in meaning to your query — the passages to feed an LLM, the products to recommend, the images that match a description.
Why it needs to be clever: approximate search
Comparing your query against every stored vector one by one (exact search) is fine for thousands of vectors but hopeless for millions. So vector databases use Approximate Nearest Neighbor (ANN) search: they trade a tiny bit of accuracy for an enormous speed gain.
The most common method is HNSW (Hierarchical Navigable Small World, arXiv:1603.09320), a layered graph you can traverse to hop toward the nearest neighbors in a few steps instead of scanning everything. It’s why a good vector database can search millions of vectors in single-digit milliseconds.
The feature that really matters: metadata filtering
Raw similarity search is only half the job. In production you almost always need to filter: only search documents this user is allowed to see, only items from the last 90 days, only this product category. A real vector database stores metadata alongside each vector and applies these filters at query time. Getting filtering right (and fast) is often what separates a toy from a production system — and it’s central to access control in RAG.
When you actually need one
Be honest about scale before adding infrastructure:
- A few thousand vectors: an in-memory index or a library like FAISS is simple and fast. No database needed.
- A traditional database you already run: many now offer vector-search extensions — often enough at modest scale, one less system to operate.
- A dedicated vector database earns its keep when: you have large and growing collections, frequent inserts/updates and deletes, you need metadata filtering, high availability, and horizontal scale.
Start with the simplest thing that works and graduate to a managed vector database when you hit a real limit — not before.
Where it fits
Two big jobs rely on it:
- RAG — retrieve the relevant chunks to ground an LLM’s answer (see how to design a RAG system).
- Agent memory — store and recall facts across sessions for AI agents.
The honest bottom line
A vector database is a focused tool: store embeddings, find the nearest ones fast, filter by metadata. That’s it — and that’s enough to power most modern AI retrieval. The main mistake teams make isn’t choosing the wrong one; it’s reaching for heavy infrastructure before their scale justifies it. Match the tool to the problem, and it’s a reliable backbone. For the interview angle, see our RAG & vector database questions.