Temperature is not a creativity dial
LLM temperature is the same mathematical object as temperature in statistical mechanics. It controls the entropy of the sampling distribution, and that changes how you should debug it.
The "creativity dial" framing for temperature is popular because it is useful. It is also inaccurate in a way that makes production systems harder to debug. Temperature is not a mode switch. It is entropy control, and it obeys the same mathematics as temperature in statistical mechanics.
That is not analogy. It is identity under a change of names.
The equation is the same
In statistical mechanics, the Boltzmann distribution gives the probability of a system being in state i with energy E_i: P(i) ∝ e^(−E_i / kT). In LLM sampling, softmax with temperature τ gives: P(i) = softmax(z_i / τ). Map state to token, energy to negative logit, temperature to temperature. The operations are the same. Dividing logits by τ is heating or cooling the distribution.
At τ → 0, the distribution collapses to the top token: determinism, repetition, mode collapse. At τ → ∞, logit differences vanish and you are sampling noise. Production sits somewhere between, and where you sit determines what breaks.
Same seven candidate tokens and the same logits every time. Only τ changes — this is the entire mechanism.
Holtzman et al. (2019) quantified the low-entropy failure mode directly: greedy decoding produced repeated n-grams in 43% of generated continuations, versus 0.5% for human-written text over the same prefixes. That 43% is what "mode collapse" looks like in a production log, not an abstraction.
Why the same τ feels different across prompts
Entropy is a function of both the logits and the temperature. A prompt that produces peaky logits (the model is confident) will feel over-constrained at τ = 0.7. A prompt with flat logits (the model is unsure) will feel chaotic at the same value. If you want consistent behavior across diverse inputs, you need to monitor effective entropy, not just pin τ and hope.
Separation is the fix, not the right number
| Stage | Goal | Temperature | Pairing |
|---|---|---|---|
| Brainstorm | High variance, diverse candidates | 0.9–1.2 | top-p 0.95, sample N |
| Extraction | Accurate, structured output | 0.1–0.3 | top-k 10, schema validation |
| Code edit | Precise, deterministic | 0.0–0.2 | fixed seed, lint check |
| Rerank/judge | Calibrated scoring | 0.3–0.5 | logprobs if available |
Temperature alone is not enough: it flattens everything including the tail where low-probability junk lives. Pair it with top-p or top-k to control tail risk separately from overall entropy. The rule: temperature sets the spread; nucleus sampling sets the floor.
That reframing (temperature as a diversity knob for ensembles, not a creativity setting) is exactly how practitioners are starting to use it.
Sunday morning read: More Agents is All You Need by Junyou Li et al. ☕️ We've all heard about scaling the number of parameters, but what about scaling the number of agents? From the paper, "LLM performance may likely be improved by a brute-force scaling up of the number of Show more
References: Holtzman et al., The Curious Case of Neural Text Degeneration (2019), Hugging Face