Hallucination is architectural, not a bug
A next-token predictor has no native epistemic state. Correctness and confidence are structurally uncoupled, so reliability must come from system architecture, not better prompts.
I once asked a strong model for the page number of a quote in a book I had open on my desk. It gave me a number. Confident, specific, formatted exactly like a real citation. It was wrong. So I asked again, rephrasing, and it gave me a different number, equally confident, equally specific, equally wrong.
There was no tremor in the prose, no "I think," no hedge. The sentence that contained the fabricated page number read exactly like the sentences that contained true facts a moment earlier.
The model was not lying, because lying requires knowing the truth and choosing against it. It was doing the only thing it knows how to do: producing a plausible next token, then another, then another.
We call this hallucination, and we tend to talk about it like a defect: a bug that better engineering will eventually stamp out. I want to argue the opposite. Hallucination is not bolted onto an otherwise-truthful system. It falls directly out of what the model was trained to do.
Once you see that, you stop waiting for the bug fix and start building for the world as it actually is.
The model maximizes plausibility, not truth
Strip a language model down to its training loop and there is exactly one thing happening. The model sees a stretch of text, predicts the probability of the next token, and gets penalized in proportion to how surprised it was by the token that actually came next. Cross-entropy loss, repeated over trillions of tokens.
Notice what is not in that loop. There is no oracle of truth. There is no fact-checker standing beside the loss function whispering "careful, that one's false." The gradient only ever says: the real corpus continued this way, so make that continuation more likely next time. The model is being shaped into a faithful model of the distribution of text, not a model of the state of the world.
Most of the time this distinction doesn't bite, because human text is mostly about a real world. "The capital of France is ___" continues with "Paris" in the corpus because Paris really is the capital of France. Plausible and true ride together.
The trouble is at the boundary, where they come apart. "The 1987 novel The Glass Atrium was written by ___" has a smooth, confident continuation too, even though no such novel exists. The corpus is full of sentences of the form "[Title] was written by [Name]." It fills the blank with the most plausible-sounding author. The model is not malfunctioning. Its job was never to tell you the truth; it was to continue the text the way the corpus would.
Confidence is not correctness
If the model has no truth signal, you might hope it at least has a confidence signal: that when it is about to fabricate, its probabilities go flat.
The right tool for thinking about this is calibration. A model is calibrated when its stated confidence matches its observed accuracy: the things it says with 70% confidence are true about 70% of the time.
A model is calibrated when a stated confidence of 70% means it is right about 70% of the time. Points below the diagonal are overconfident.
A pretrained model often tracks perfect calibration surprisingly well. That is genuinely good news. But then we align the model, and the news gets complicated. OpenAI's GPT-4 technical report shows the pretrained model was well calibrated, and that post-training with RLHF noticeably degraded that calibration (OpenAI, 2023).
The intuition: RLHF optimizes for responses humans prefer, and humans reliably prefer answers that sound confident, decisive, and helpful. So the process gently teaches the model to round its uncertainty up. We are, in effect, training the warning light to stay off because users find the warning light annoying.
Why some prompts reliably hallucinate and others don't
If you collect a model's failures, they aren't random. They cluster in the long tail.
Ask for the capital of France and you'll get Paris every time. Ask for the exact page number of a quote, the precise attendance at a minor 1990s match, or the DOI of a specific paper, and the failure rate climbs sharply. The common thread is training-data frequency.
Facts the model saw millions of times are burned in. Facts it saw once or never are not stored as facts at all, and when you ask, the model does what it always does: produces the most plausible-looking filler.
The rarer a fact is in the training corpus, the less reliably the model recalls it — and the more it fills the gap with a plausible guess.
A practical corollary: you can predict where a model will hallucinate before you run it. If your prompt asks for a rare, specific, verifiable fact, treat the answer as a hypothesis, not a result.
Is hallucination unavoidable?
Two recent lines of work argue that hallucination isn't merely common; it's mathematically baked in.
- The computability argument. Xu et al. show that for any computable LLM, there exist ground-truth functions it cannot match everywhere, so a model used as a general problem solver will inevitably hallucinate on some inputs (Xu et al., 2024).
- The statistical argument. Kalai et al. show that hallucinations arise as ordinary statistical errors. Because most benchmarks grade like multiple-choice tests, awarding points for a correct guess and zero for "I don't know," abstaining is strictly dominated by guessing. We are optimizing models to be confident test-takers (Kalai et al., 2025).
Stop expecting the model to police its own truth, and design the system around it. Retrieval is the most direct counter: Lewis et al. (2020) measured a 10-point exact-match gap on Natural Questions between a retrieval-grounded model (44.5) and a comparably-sized parametric-only model (34.5): the difference between answering from a verifiable source and answering from compressed memory (Lewis et al., 2020).
Engineering Implications
If hallucination is architectural, the engineering goal shifts from eliminate it to contain it, detect it, and route around it.
| Tactic | Mechanism |
|---|---|
| Ground in retrieved facts (RAG) | Moves the question from the long tail of parametric memory to reading off a page in context. |
| Demand verifiable citations | The citation must point at a retrieved span, and the system confirms the span actually supports the claim. |
| Reward abstention | Build abstention into the interface and scoring, so 'I'm not sure' is a first-class, point-scoring answer. |
| Constrain decoding | If the output must be valid JSON or a real database entity, constrain the decoder so invalid tokens can't be sampled. |
| Separate faithfulness from factuality | Measure if the output stays true to the source (faithfulness) independently from whether it is true about the world (factuality). |
Treat every model output as a draft. The model is a fast, fluent first-drafter with no fact-checker attached. So attach one: retrieval, verification, constrained outputs, or another model in a checker role. The architecture won't supply ground truth; your system has to.
That's the right question to keep asking, and it's surfacing more often in public discussion:
Some questions to consider on how much you understand LLMs: 1. Why do they hallucinate? Is hallucination a "bug" or a "feature" of the architecture? 2. Can an LLM reference where they got information it recites from? 3. Why do most LLM vendors limit output to ~4,000 tokens?
References
- Ji, Z., et al. (2022). Survey of Hallucination in Natural Language Generation. arXiv:2202.03629
- Kadavath, S., et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221
- Xu, Z., Jain, S., & Kankanhalli, M. (2024). Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv:2401.11817
- Kalai, A. T., et al. (2025). Why Language Models Hallucinate. arXiv:2509.04664
- Guo, C., et al. (2017). On Calibration of Modern Neural Networks. ICML. arXiv:1706.04599
- OpenAI. (2023). GPT-4 Technical Report (see Figure 8 on calibration). arXiv:2303.08774
- Bender, E. M., et al. (2021). On the Dangers of Stochastic Parrots... FAccT '21. Link
- Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401