How do you evaluate multimodal AI systems?

Evaluation depends on the task, and it's harder than text because quality is often visual/subjective. For understanding tasks (VQA, captioning), score against labelled answers — accuracy for VQA, and caption metrics (CIDEr, plus embedding or LLM-judge scoring) that go beyond word overlap. For cross-modal retrieval, use recall@k and MRR on labelled query-item pairs. For generation (image/video/audio), automated metrics like FID (distribution realism) and CLIPScore (prompt-image alignment) give signal but correlate imperfectly with human perception, so human evaluation of fidelity, prompt adherence, and artefacts remains essential. Also test faithfulness — does the model actually use the image rather than answer from text priors? — and red team cross-modal attacks that text-only safety tests miss. As always, evaluate on a representative set, not cherry-picked demos.