easyLLMs & TransformersReviewed Jul 24, 2026

What is a token in an LLM?

A token is the unit an LLM reads and writes — a chunk of text roughly 4 characters or ¾ of a word in English. Text is split by a tokeniser; common words are one token, rarer words split into several, and punctuation and spaces count too. Models have a maximum token limit (the context window), and APIs bill per token, so tokens matter for both capability and cost.

llmbasics

📘 Learn this properly: Tokens & Context Windows →

More LLMs & Transformers questions

See all LLMs & Transformers questions →