LLM knowledge has three shapes
Parametric knowledge lives in the weights, contextual knowledge lives in the window, retrieved knowledge is fetched at inference. They age, fail, and update differently.
The fastest way to get confused building LLM systems is to treat "knowledge" as one thing. The same meeting will produce "the model knows Python," "it forgot our policy," "just put it in the prompt," "we need RAG for freshness," and "let's fine-tune so it remembers." All of these point at real phenomena. None of them point at the same thing.
There are three distinct kinds of knowledge in an LLM application, and they age, fail, and cost different amounts to update.
Parametric: the weights
Parametric knowledge is what the model absorbed during training: language competence, general facts up to the cutoff, reusable skills. It is distributed across parameters, fast at inference, and extremely slow to update. You cannot easily prove what is in it. You cannot cite it. And the world moves faster than training cycles.
If your product needs correctness guarantees, parametric knowledge is a weak foundation by default.
Contextual: the window
Contextual knowledge is whatever you inject into the prompt: instructions, a short policy excerpt, a table of values. It is as flexible as copy-paste and as fragile as ordering. When a model ignores a policy that was "in the prompt," it is rarely magic; it is usually a context management failure: too long, too noisy, contradicted elsewhere, or buried under the fold.
Retrieved: tools and indexes
Retrieved knowledge is fetched at inference time through RAG, database queries, or API calls. It does not change the model; it changes the facts the model can condition on. Updates are data operations, not training runs. Claims are traceable because the source is in the log. This is why retrieved knowledge is the backbone of any system that needs to stay correct over time.
The original RAG paper put a number on the gap between parametric and retrieved knowledge: Lewis et al. (2020) reported 44.5 exact match on Natural Questions, against 34.5 for a comparable parametric-only T5-11B model: a 10-point gap from giving the model somewhere to look instead of something to remember.
Only parametric knowledge is anchored to a training date — the other two don't care what day it is.
| Type | Update cost | Auditability | Failure mode |
|---|---|---|---|
| Parametric (weights) | Retraining required | None — black box | Stale, uncited, unverifiable |
| Contextual (window) | Edit the prompt | Full — it is in the log | Ignored when too long or noisy |
| Retrieved (RAG/tools) | Reindex the corpus | Full — chunk is visible | Retrieval misses or injects noise |
The decision is not which is best, it is which owns what
Anything with a weekly update cycle should not live in weights.
The shortcut that holds: skills live in weights, rules live in context, facts live in retrieval. Mixing those up is the root cause of most "the model is broken" tickets that turn out to be architecture tickets.
Researchers are now finding direct mechanistic evidence for exactly this shortcut:
From RAG to Rich Parameters Investigates more closely how LLMs utilize external knowledge over parametric information for factual queries. Finds that in a RAG pipeline, LLMs take a “shortcut” and display a strong bias towards utilizing only the context information to answer the Show more
References: RAG (Lewis et al., 2020), Anthropic, OpenAI