How do vision-language models process images alongside text?

A vision-language model (VLM) bolts a vision encoder onto a language model. The image is split into patches and run through an encoder (typically a vision transformer) that outputs visual embeddings. A projection/adapter layer maps those into the language model's token embedding space, so the image becomes a sequence of 'visual tokens' the LLM can attend to alongside the real text tokens. The LLM then generates text conditioned on both. Training aligns the two: a large corpus of image-text pairs teaches the projection and often fine-tunes the model to follow visual instructions. This is why VLMs can caption images, answer questions about them, and read charts — but they can also 'ignore' the image and answer from text priors if alignment is weak, a common failure mode.