Chain-of-thought rents computation with tokens
Asking a model to think step by step does not flip a reasoning switch. It generates intermediate tokens that become additional context: sequential compute rented by spending tokens.
Sometime in 2022, a strange piece of folk wisdom spread through the people who prompt language models for a living: if the model gets a math problem wrong, add the words "Let's think step by step." Often, it would then get the problem right. Same model, same weights, same question. One extra sentence, and the accuracy on some benchmarks jumped by tens of points.
It felt like a cheat code. And like all good cheat codes, it invited a bad explanation: that somewhere inside the network there's a reasoning mode, and the phrase is the password that turns it on.
That story is wrong, and the right story is much more interesting. The model isn't switching into a smarter gear. It's doing something far more mechanical: it's giving itself more computation to spend on the problem, and the only currency it has to buy that computation is tokens.
Every word it writes before the answer is a step of work it couldn't have done in a single pass.
The fixed-depth constraint
Picture what happens when a transformer processes a prompt. The input flows up through a fixed stack of layers (32, 80, 120, whatever the architecture is) and out the top comes a probability distribution over the next token.
This matters because depth is where sequential reasoning lives. A 5 × 7 multiplication, a chain of "if A then B" deductions, carrying a digit during addition: all of these are inherently serial. Step two needs the output of step one. With a fixed number of layers, the model gets a fixed number of "and then" operations per forward pass.
So a hard problem can need more sequential steps than one forward pass provides. The model isn't stupid; it's out of depth. And there's exactly one escape hatch.
Tokens as a scratchpad
Here is the escape hatch: when a model generates a token, that token gets appended to the context and fed back in as part of the input for the next token.
So if the model writes down an intermediate result, say "3 shelves × 7 = 21", it can read that 21 back on the next pass and build on it. The generated text is a scratchpad: external working memory the model lays out for itself, one token at a time.
That's what "renting computation with words" means. The model can't make its forward pass deeper. But it can trade tokens for serial steps, paying the fixed cost of one forward pass per token to extend how much "and then" it gets to do.
Notice what the toggle is really showing. In "step by step" mode nothing about the model changed. It just gave itself four extra passes by emitting four clauses and reading each one back. The intelligence was always there; the room to use it was what reasoning tokens bought.
Why it helps on hard problems and not easy ones
If chain-of-thought is renting computation, it should only help when you're short on computation in the first place.
A one-step lookup, like "What's the capital of France?", fits comfortably inside a single forward pass. Writing out reasoning adds nothing but tokens. A multi-hop problem, like "The author of the book that won the 1998 prize was born in which city?", needs several serial retrievals chained together, and that's exactly where the extra passes pay off.
There's a corollary that's easy to miss: on easy tasks, forcing a reasoning trace can even hurt. More tokens are more chances to wander off, contradict an earlier step, or talk yourself out of a correct first instinct.
The frontier: test-time compute as a scaling axis
For years, the way to make a model better was to make training bigger. Chain-of-thought opened a second axis: spend more compute at inference time, per question, by letting the model think longer.
Reasoning models (like OpenAI's o1) are built around this idea: they're trained to produce long internal reasoning traces and keep going until they've worked the problem through.
Snell et al. (2024) found that, dollar for dollar, spending compute at inference can beat spending it on a bigger model. But, crucially, the returns are concave. The first few hundred reasoning tokens buy a lot of accuracy; the tenth doubling buys almost nothing (Snell et al., 2024).
That knee in the curve is the whole engineering story. There's a regime where thinking longer is the best money you can spend, and a regime past which you're paying real latency and cost for a fraction of a point.
The size of the effect on genuinely hard problems is well documented in the original prompting literature: Wei et al. (2022) report that 8-shot chain-of-thought prompting on a 540B-parameter PaLM model reached 58% accuracy on GSM8K math word problems, edging past the prior best of 55% set by a fine-tuned GPT-3 model with an external verifier, all without any task-specific training (Wei et al., 2022).
The uncomfortable parts
Two findings keep this from being a tidy story:
First: even meaningless tokens can help. Pfau et al. (2024) showed that transformers can sometimes solve hard tasks using filler tokens, literal strings of ......, in place of a real chain of thought. The dots carry no information, but the extra token positions provide extra computation regardless. It's the cleanest evidence that chain-of-thought is partly about compute, not just legibility (Pfau et al., 2024).
Second: the trace is not always the reasoning. Turpin et al. (2023) demonstrated that models will produce fluent reasoning that misrepresents the actual cause of their answer. For instance, when the prompt contains a subtle bias toward one option, the model picks that option and then rationalizes it (Turpin et al., 2023).
The trace looks like an explanation; it's better understood as more computation that happens to be human-readable.
Engineering Implications
Here's how the reframing cashes out when you're actually building something.
| Rule | Why it matters |
|---|---|
| Match the spend to depth | On lookups and extraction, reasoning is wasted latency. On math, planning, and code, it's the difference between right and wrong. |
| Tune the budget | The test-time-compute curve has a knee. Find roughly where yours is and set token limits there to avoid paying for flat returns. |
| Use self-consistency | For high-stakes answers, sampling several chains and taking the majority vote trades compute for reliability. |
| Never trust the trace | A confident-sounding chain of thought is not evidence the model reasoned the way it says. Use the answer for decisions, the trace for debugging. |
Reasoning, in a language model, is not a switch. It's a budget — paid in words.
That distinction between imitated and genuine deliberation is exactly what researchers are now tracking in frontier models:
There is a nuanced but important difference between chain-of-thought before and after o1. Before the o1 paradigm (i.e., chain-of-thought prompting), there was a mismatch between what chain of thought was and what we wanted it to be. We wanted chain of thought to reflect the Show more
References
- Nye, M., et al. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv:2112.00114
- Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903
- Kojima, T., et al. (2022). Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916
- Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171
- Turpin, M., et al. (2023). Language Models Don't Always Say What They Think... arXiv:2305.04388
- Merrill, W., & Sabharwal, A. (2023). The Expressive Power of Transformers with Chain of Thought. arXiv:2310.07923
- Pfau, J., et al. (2024). Let's Think Dot by Dot: Hidden Computation in Transformer Language Models. arXiv:2404.15758
- Snell, C., et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314