Context engineering is memory management
The context window has the same failure modes as working memory: capacity limits, primacy and recency effects, interference. Engineers who understand this write better prompts.
Give someone a list of words and ask them to recall it later, and the same pattern shows up every time: they remember the first few words and the last few, and everything in the middle goes hazy. Psychologists call this the serial position effect. It's also a surprisingly good description of how attention behaves inside a long context window.
LLMs have the same problem. Not because they're trying to imitate humans, but because attention, like memory, is not uniformly distributed across a sequence.
Position is not neutral
When a transformer processes a long prompt, its attention weights are not flat. Tokens near the beginning and near the end exert more influence on the final generation. Tokens buried in the middle of a large context window are statistically less likely to be "heard." This is not a quirk of a particular model. It falls directly out of how self-attention interacts with positional encodings and training distributions: models are trained on text where the most important information is usually at the start or the end.
The practical consequence is brutal: if you put your most critical instruction in the middle of a 50k-token prompt, the model will often ignore it. Not because it's unintelligent. Because you misunderstood the architecture.
Context is not storage. It's working memory.
The distinction matters enormously. Storage is persistent, random-access, and position-independent. Working memory is limited, position-sensitive, and subject to interference. Your context window behaves like the second one.
When you inject 40 documents into a retrieval-augmented prompt, you are not giving the model a filing cabinet. You are giving it a list of things to hold in mind while it works. The model's ability to reason over those documents degrades as the list grows, as documents conflict with each other, and as the instructions drift further from the retrieved content. This is not a retrieval problem. It is a cognitive load problem.
The good engineers I've watched build LLM systems think about context the way good teachers think about a lecture: what absolutely must be said, in what order, with what emphasis? Everything else is noise that makes the important things harder to hear.
The four laws of context
Lead with the contract. Your system instructions (role, constraints, output format) belong at the very top where primacy effects work in your favor. The model's first impression of what it's supposed to do shapes everything that follows.
Land critical constraints at the end. The final instructions before the user's message get the full benefit of recency. If a safety constraint or output requirement absolutely must be obeyed, put it last, short, and unambiguous.
Keep the middle for evidence, not instructions. Inject retrieved documents, examples, and context in the middle. The model is less likely to follow instructions buried here, but it will still draw facts from this region. Separate the evidence from the directives structurally.
Reduce interference. Two documents that say subtly different things about the same topic will confuse the model the same way two people giving you contradictory directions confuse you. Deduplicate, filter, and rerank before you inject. The context window is not a dumping ground.
Same 20-document context, same question. Only the position of the document with the answer changes.
| Position | Content type | Principle |
|---|---|---|
| Top (primacy zone) | System instructions, role, safety constraints | First impression sets the behavioral frame; use primacy |
| Upper middle | Few-shot examples, output format schema | Examples work well here: they're structural, not directional |
| Middle | Retrieved documents, source material | Evidence zone: model reads but doesn't treat as instructions |
| Lower middle | Clarifications, conversation history | Context that provides nuance without overriding the frame |
| End (recency zone) | Final constraints, the actual question | Last thing read, highest recall: put what must not be ignored here |
Caching is the memory system
Once you accept that context is working memory, prompt caching becomes obvious: it's the equivalent of long-term memory. The parts of your context that don't change between requests (system instructions, tool definitions, static examples) can be cached server-side and reloaded at a fraction of the cost.
Anthropic's cache reduces costs on cached tokens by 90% with near-zero latency. The architectural implication: design your prompts so the stable elements sit above cache breakpoints and the volatile elements sit below. What you're really doing is distinguishing between long-term memory (what the system always knows) and working memory (what it needs to think about right now).
A prompt that wasn't designed with this separation in mind is paying full price every request for context that hasn't changed.
The simplest test
Here is a test worth running on any important prompt you've built. Take the most critical behavioral constraint in your system: the rule that absolutely must be followed. Now ask: where is it in the context? If the answer is "somewhere in the middle," you have a working memory problem, not a model problem.
This is precisely the engineering problem people are starting to name explicitly:
say you wanted to design an LLM interface that let you chat with an LLM ~indefinitely, with apparently persistent memory, despite context window limitations. one thing you might try is to periodically replace, say, the older half of the context window with a short summary of it Show more
References: Lost in the Middle (Liu et al., 2023), Anthropic Prompt Caching, Hugging Face