Attention is weighted averaging, and that's enough

Read this slowly: "The engineer fixed the server because it crashed."

When you reached it, you didn't pause. You already knew it meant the server, not the engineer, even though server was four words back. You reached across the sentence and grabbed the right word, instantly and without effort.

That reach is the whole story. For decades, the dominant way to process language with a neural network was to read left to right, one word at a time, cramming everything seen so far into a single running summary. That worked until the reach got long. The model would read it, and the memory of server had already faded into a blur of everything else.

Attention is the mechanism that lets a model make that reach directly: to look back at server and pull it forward, on demand.

This article is about how that works. By the end, the famous formula should read like a sentence you already know.

The core idea: a soft dictionary lookup

Here is the analogy I keep coming back to. Attention is a dictionary lookup that refuses to commit.

A normal dictionary lookup is hard. You have a query ("the word bank"), you scan keys (the entries), you find the one that matches exactly, and you return its value (the definition). One key matches; you take its value; done.

Attention does the same three roles (query, key, value) but softly. Instead of returning the single best-matching value, it returns a blend of all the values, weighted by how well each key matched the query. A key that matches the query a lot contributes most of its value; a key that barely matches still contributes a whisper.

Three things make this work as machinery, not just metaphor:

Query, key, and value are different views of the same token. Each token is projected (multiplied by a learned matrix) into a query vector, a key vector, and a value vector. So one word plays all three roles: it can ask a question, advertise what it offers, and carry content.
Matching is a dot product. To score how well a query matches a key, take their dot product. Two vectors pointing the same way score high; perpendicular vectors score near zero.
The blend is differentiable. Because the weighting is soft, gradients flow through it. The model can learn better queries and keys by gradient descent.

The mechanism, step by step

Now the actual procedure, for a single query token. Four steps, no heavy math.

Emit a query. The token produces its query vector q, its question.
Score against every key. Take the dot product of q with each token's key vector k. This gives one raw relevance score per token.
Softmax into weights. Push the scores through a softmax. This turns a list of arbitrary numbers into a set of positive weights that sum to 1.
Weighted sum of values. Multiply each token's value vector v by its weight and add them all up. The result is the output for this query token: a custom blend of the whole sequence, tilted toward whatever was relevant.

That is the entire operation. Run those four steps for every token, and you've computed self-attention for the whole sequence.

One attention step, in slow motion

Query token:

The output for cat is the weighted sum of the value vectors — each token's value, scaled by how much attention it got.

weights

The: 0.21

cat: 0.29

sat: 0.26

down: 0.23

output vector

[ 0.21, 0.42, 0.36 ]

Pick a query and step through it: dot product → scale → softmax → weighted sum. That four-step loop, run for every token at once, is attention. Vectors are toy 3-D values for legibility.

If you want it as one line, here is the scaled dot-product attention formula from the Transformer paper (Vaswani et al., 2017):

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V

Read it aloud and it says: score every query against every key, scale, soften into weights, then average the values. Same four steps.

Self, cross, and causal attention

Once you have the four-step move, the variations are just who supplies the queries, keys, and values.

Self-attention. Queries, keys, and values all come from the same sequence. This is what's running inside each layer of a language model.
Cross-attention. Queries come from one sequence, keys and values from another. E.g., in translation, the decoder's queries attend to the encoder's keys and values.
Causal (masked) attention. When a model generates text, it must not peek at the future. Before the softmax, every score from a token to any later token is set to -infinity, which softmax turns into a weight of exactly 0.

Self-attention heatmap — who attends to whom

query \ key	animal	it
The
animal
didn't
cross
the
street
because
it	46	22
was
tired

Each row is one token's query; brighter cells are the keys it weights most. Hover a row to see its attention pattern — note how it points back to animal, its antecedent. Weights are illustrative, not measured.

Many heads, many jobs

A single attention pattern has to commit: one set of weights, one notion of "relevant." But relevance has many flavors at once. "it" needs to track its antecedent (coreference). A verb needs to find its subject and object (syntax). One pattern can't do all of that.

The fix is almost embarrassingly direct: run several attention operations in parallel. These are heads. Multi-head attention lets each head specialize, develop its own idea of what to look for, and the layer combines their findings.

Three heads, three jobs — same sentence

query \ key	The	engineer	fixed	the	server	because	crashed
The	70
engineer		70
fixed			60
the				60
server					56
because						55
it					72
crashed							35

Coreference head (pronoun → antecedent). The pronoun “it” concentrates almost all its attention on “server,” the noun it refers to. Resolving references is exactly the kind of long-range link attention makes cheap.

Rows are query tokens, columns are key tokens; brighter cells mean more attention. The same sentence produces three very different maps — that division of labor is why we run many heads in parallel. Patterns are illustrative caricatures of documented head behaviors, not measured.

Engineering Implications

This is where the elegant idea meets your AWS bill.

That beautiful all-pairs comparison has a cost: QK^T compares every token to every token. For a sequence of length n, that's n^2 comparisons. Double the context, quadruple the attention compute and the memory to hold the score grid.

This O(n^2) scaling is the single most important number in long-context engineering. It's why a model that handles 8K tokens cheaply gets expensive at 128K and brutal at 1M.

The second practical artifact is the KV cache. When a model generates text one token at a time, each new token attends to all the previous tokens' keys and values. Recomputing those from scratch every step would be wasteful, so we cache them. That's the KV cache, and it's why generation is fast, but also why long conversations eat memory.

The chain of consequences runs straight from the math to your infrastructure: attention is all-pairs → all-pairs is O(n^2) → long context is expensive in compute and memory → the entire industry of efficient-attention variants exists to bend that curve. FlashAttention is the canonical example: by reordering the computation to avoid materializing the full attention matrix in GPU memory, Dao et al. (2022) reported a 3x wall-clock speedup on GPT-2 at 1K sequence length and a 15% end-to-end speedup training BERT-large, without approximating the attention weights at all (Dao et al., 2022). You can't reason about the cost of a long-context feature without reasoning about this one mechanism.

That sparsity isn't theoretical: it shows up directly when you look at real attention maps.

Aran Komatsuzaki on attention maps forming sparse, consistent vertical patterns rather than per-token magic.

References

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473
Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762
Vig, J. (2019). A Multiscale Visualization of Attention in the Transformer Model (BertViz). arXiv:1906.05714
Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Anthropic
Olsson, C., et al. (2022). In-context Learning and Induction Heads. arXiv:2209.11895
Dao, T., Fu, D., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135