How do you choose an embedding model and its dimensionality for a RAG system?

Start from your domain and task: use benchmarks like MTEB as a starting point but validate on your own labeled queries, since domain fit often beats leaderboard rank. Consider retrieval quality, max sequence length (must cover your chunk size), language coverage, and cost/latency. Higher dimensions (e.g. 1536, 3072) can capture more nuance but increase storage, memory, and search cost roughly linearly. Many modern models support Matryoshka representation learning, letting you truncate dimensions (e.g. 3072 to 512) with graceful quality loss to save space. Also weigh open-source self-hosting (no per-call cost, data privacy, but you manage infra) versus API models. Critically, the same model must embed both documents and queries, and re-embedding everything is expensive, so choose deliberately to avoid churn.