Chain-of-Thought and Reasoning Models, Explained

Letting a model 'think' before answering measurably improves hard reasoning. Here is how chain-of-thought works, how it grew into dedicated reasoning models, and when the extra cost pays off.

By AIWI EditorialEdited by Chayanika Divekar3 min readPublished Jul 13, 2026

Single source1 source

Key Takeaways

Chain-of-thought (CoT) prompting — asking a model to reason step by step before answering — measurably improves accuracy on math, logic, and multi-step problems.
It works because generating intermediate steps lets the model break a problem into pieces instead of guessing an answer in one leap.
Reasoning models take this further: they're trained to produce long internal reasoning ('thinking') and spend extra compute at inference time before answering.
This is 'test-time compute' — trading more tokens, latency, and cost for better answers on genuinely hard tasks.
It's not free or always right: reasoning models are slower and pricier, and step-by-step text can still reach a confident wrong answer, so verification still matters.

One of the most important discoveries in applied AI is almost embarrassingly simple: if you let a model think before it answers, it gets a lot better at hard problems. That idea grew from a prompting trick into a whole class of “reasoning models.” Here’s how it works and when to use it. It builds on our prompt engineering basics.

Chain-of-thought prompting

The foundational finding, from Chain-of-Thought Prompting Elicits Reasoning, is that asking a model to show its reasoning — “let’s think step by step” — substantially improves accuracy on math, logic, and multi-step questions.

Why does it help? An LLM generates one token at a time. Forced to answer immediately, it has to compress all the reasoning into a single leap and often guesses. Given room to write out intermediate steps, it can break the problem into manageable pieces, and each step conditions the next — so the final answer is built on visible working rather than a hunch. The reasoning tokens are, in effect, scratch paper.

From prompting to trained reasoning models

Prompting a model to think is powerful but limited by what the base model learned. The next step was to train models specifically to reason: to produce long internal chains of thought, check their own work, explore alternatives, and only then answer. These “reasoning models” (the o1-style and “thinking” model families) don’t just answer — they deliberate.

Crucially, they spend more compute at inference time doing this. Where a normal model does a fixed amount of work per answer, a reasoning model can “think” for many thousands of tokens before responding — often hidden from you — and that extra thinking is what lifts performance on the hardest benchmarks.

Test-time compute: the core trade-off

This is the key mental model: test-time compute. Historically, we made models smarter by making them bigger and training them longer (train-time compute). Reasoning models add a second dial — spend more compute at inference to think harder about a specific problem.

That’s a genuine trade-off. More thinking means:

Better answers on hard, multi-step tasks.
More tokens, higher latency, higher cost — you’re paying for all that hidden reasoning (see the model pricing cheat sheet).

When to use it — and when not to

Reach for reasoning on genuinely hard problems: competition-level math, complex planning, intricate debugging, careful multi-constraint tasks.
Skip it for simple lookups, summarization, formatting, or casual chat — a standard model is faster, cheaper, and just as accurate there. Using a reasoning model for easy tasks mostly buys you a slower, pricier answer.

The honest limits

Thinking out loud is not a guarantee of truth. A model can produce a fluent, plausible chain of reasoning that still lands on a confident wrong answer — the steps can rationalize a mistake rather than catch it. And the visible “thinking” isn’t always a faithful account of how the model actually reached its answer. So reasoning models raise the ceiling on hard tasks, but they don’t remove the need to verify important outputs — see how to evaluate an LLM app.

The honest bottom line

Chain-of-thought turned “let it think” into one of the highest-leverage ideas in AI, and reasoning models productized it by trading inference compute for smarter answers. Use that power where the problem is genuinely hard, save your money and latency where it isn’t, and keep verifying — thinking longer makes models more capable, not infallible. For the interview version, see our LLMs & transformers questions.

Was this helpful?

Sources & Further Reading

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv:2201.11903) — arXiv

Frequently Asked Questions

What's the difference between chain-of-thought prompting and a reasoning model?

Chain-of-thought is a prompting technique — you ask any model to 'think step by step.' A reasoning model is trained to do that reasoning natively and to spend extra inference compute 'thinking' before it answers, usually producing much longer and more reliable reasoning than prompting alone can. One is how you ask; the other is how the model was built.

Should I always use a reasoning model?

No. Reasoning models are slower and cost more because they generate a lot of hidden thinking tokens. Use them for genuinely hard multi-step problems (math, complex planning, tricky code). For simple lookups, formatting, or chat, a standard model is faster, cheaper, and just as good.

How LLM Inference Works: Tokens, KV Cache, and Why It Costs What It Costs — Jul 13, 2026
How to Actually Evaluate an LLM App (Without Fooling Yourself) — Jul 11, 2026
How Transformers Work: The Architecture Behind Every Modern AI — Jul 11, 2026
LoRA and Efficient Fine-Tuning, Explained — Jul 11, 2026

#llm #prompt-engineering #training #benchmarks #education