What is document understanding, and how do models parse documents with layouts?

Document understanding is extracting meaning from documents where layout carries information — invoices, forms, contracts, scientific papers with tables and figures. Plain text extraction loses structure (which number is the total, which cell belongs to which column), so models combine three signals: the text (via OCR or native text), each token's spatial position on the page, and often the visual image itself. Layout-aware models encode position alongside text so they learn that a value under a 'Total' header is the total. Increasingly, vision-language models read the page image directly, handling tables and figures without separate OCR. Challenges: multi-page documents, complex nested tables, and scanned/noisy scans. A common failure is a VLM that answers single-image questions but falls apart across a long multi-page document — chunking and page-aware retrieval help.