Embeddings are coordinates, not encodings
Embeddings position text in a learned space where distance corresponds to meaning. Semantic search is nearest-neighbor navigation on that geometry, not fuzzy matching.
A few years ago I typed a clumsy query into a search box, something like "that movie where a guy relives the same day over and over", and it returned Groundhog Day without me ever naming it. No keyword in my query appears in the title or the plot summary I was thinking of. The old mental model of search (match the words, rank by how many match) simply can't do this.
Something else was happening. The system understood what I meant, not what I said.
That something is embeddings. The more I sat with how they work, the more I became convinced they're not a clever engineering hack bolted onto language. They're closer to a discovery: that meaning, the slippery thing philosophers have argued about for centuries, turns out to have a shape. Words and sentences can be placed as points in a high-dimensional space where distance and direction encode semantics. Get the map right, and "search" becomes "navigation."
From sparse symbols to dense vectors
The naive way to feed words to a machine is the one-hot vector. Pick a vocabulary of 50,000 words. Each word becomes a vector that's 50,000 numbers long, all zeros except for a single 1 at that word's position.
This works, technically, but it's semantically blind. "King" and "queen" are exactly as far apart as "king" and "kayak." The representation carries identity but zero meaning. It's also wildly wasteful: 50,000 dimensions to say one thing.
The fix sounds almost too simple. Instead of a long sparse vector that's mostly zeros, give each word a short, dense vector: a few hundred real numbers, all of them carrying signal. "King" might be [0.21, -0.43, 0.88, ...] across 300 dimensions. The trick isn't the shape of the vector; it's where the numbers come from.
You shall know a word by the company it keeps
The engine underneath all of this is an idea linguists call the distributional hypothesis: "You shall know a word by the company it keeps." The meaning of a word is reflected in the contexts it appears in.
Consider a word you've never seen, let's say "wug." If I show you enough sentences, "she poured the wug into her cup," "the wug was too hot to drink," "he ordered a wug with two sugars", you don't need a dictionary. You know a wug is a hot drink, because it keeps the same company those words keep.
That's exactly what embedding models like word2vec (Mikolov et al., 2013) do at scale. The model trains on a simple task: given a word, predict the words around it. Slide a window across billions of words of text and keep nudging each word's vector so it gets better at predicting its neighbors.
Nobody ever labels what any dimension means. There's no "royalty" dial. And yet, when the dust settles, words used in similar contexts have ended up near each other in the space. Meaning fell out of statistics.
Real embeddings live in hundreds of dimensions, so this 2D picture is a flattened cartoon. But the headline survives the squashing: animals huddle with animals, countries with countries, emotions with emotions. Nobody drew those boundaries. They emerged.
Why direction beats distance: cosine vs. Euclidean
Now we have points in space. To use them, we need to measure how similar two points are.
The obvious one is Euclidean distance: the straight-line gap. The less obvious one is cosine similarity: the angle between the two vectors, measured from the origin.
For meaning, direction usually wins. Imagine two documents about basketball. One is a three-sentence note; the other is a 50-page report. The long one mentions "basketball" many more times, so its raw vector is longer. By straight-line distance, the short note and the long report look far apart. Magnitude got conflated with topic.
Cosine similarity fixes this by throwing magnitude away. Both documents point in the "basketball direction," so their angle is small and their cosine is near 1. Direction encodes what a thing is about; magnitude often just encodes how much of it there is.
cos 90° = 0 (unrelated)
cos 180° = −1 (opposite)
The famous trick: king − man + woman ≈ queen
If clustering is the appetizer, analogy arithmetic is the dish everyone remembers. The word2vec authors noticed that relationships between words show up as consistent directions in the space.
Take the vector for "king," subtract "man," add "woman," and the resulting point lands closest to "queen."
vector("king") − vector("man") + vector("woman") ≈ vector("queen")
The arrow that points from "man" to "king" (call it the "royalty" direction) is roughly the same arrow that points from "woman" to "queen." To answer an analogy, you just start at a new word and walk the relevant direction.
The honest caveat
The crisp king − man + woman = queen demo is a best case. The standard way of solving these analogies explicitly excludes the three input words from the list of possible answers. Without it, the closest vector is very often just "king" itself (Nissim et al., 2020). It's a useful diagnostic, but don't over-index on it as a perfect mathematical proof of reasoning.
Engineering Implications
Strip away the theory and embeddings are one of the highest-leverage tools in the applied-AI toolbox.
- Semantic search. Embed your documents once into a vector database. At query time, embed the query and find its nearest neighbors by cosine similarity.
- RAG retrieval. Retrieval-augmented generation lives or dies on this step: embed a user's question, pull the most relevant chunks of your knowledge base, and hand them to the model as context.
- Clustering and deduplication. Near-duplicate support tickets sit close together in the space, so you can group or dedupe them without exact text matches.
Two engineering decisions quietly dominate whether any of this works:
- The chunking problem. You rarely embed whole documents. Chunk too small and you sever the context a passage needs to make sense; chunk too large and the vector becomes a vague average that matches everything weakly.
- Distance is not truth. Two sentences can be close in embedding space while one is true and the other false: "the Earth orbits the Sun" and "the Sun orbits the Earth" are near-neighbors. Embeddings measure aboutness, not correctness.
A third decision is dimensionality, and it trades off against both of the others: higher-dimensional vectors carry more signal but cost more in storage, latency, and noise.
The right dimensionality also depends on what you're actually using the embedding for: search, clustering, and classification each fail differently when the model is a poor fit.
| Need | Measure | Risk |
|---|---|---|
| Search | Recall@k | False positives |
| Clustering | Silhouette score | Topic blending |
| Classification | Linear probe accuracy | Shortcut features |
| Recommendations | CTR lift | Popularity bias |
Meaning has a shape, and once you can see the shape, you can compute with it. Just remember the map is not the territory. The original Sentence-BERT paper put a number on why this is worth doing operationally, not just philosophically: finding the most similar pair in a collection of 10,000 sentences took about 65 hours of pairwise comparison with BERT cross-encoding. With SBERT embeddings and cosine similarity, the same search took about 5 seconds, at comparable accuracy (Reimers & Gurevych, 2019).
Not everyone agrees the geometry is this clean, and that skepticism is worth sitting with:
There's an idea that LLMs encode high-level concepts linearly in representation space, and that these can be understood using geometric operations (e.g., cosine similarity) But: What does "linear" even mean? And, why would (Euclidean) geometry encode meaning? Show more
References
- Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781. arxiv.org/abs/1301.3781
- Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. NAACL-HLT. aclanthology.org/N13-1090
- Pennington, J., Socher, R., & Manning, C. (2014). GloVe: Global Vectors for Word Representation. EMNLP. aclanthology.org/D14-1162
- Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. EMNLP-IJCNLP. arxiv.org/abs/1908.10084
- Linzen, T. (2016). Issues in Evaluating Semantic Spaces Using Word Analogies. RepEval Workshop, ACL. aclanthology.org/W16-2503
- Nissim, M., van Noord, R., & van der Goot, R. (2020). Fair Is Better than Sensational... Computational Linguistics. aclanthology.org/2020.cl-2.7
- Ethayarajh, K. (2019). How Contextual Are Contextualized Word Representations? EMNLP-IJCNLP. aclanthology.org/D19-1006