Your model never reads text, it reads tokens

A while ago, I watched a strong engineer lose an afternoon to a one-character bug. Their prompt ended with a trailing space: "Answer:" versus "Answer: ". The model's output flipped between clean and garbled depending on which version shipped. No logic changed. No temperature changed. Just a space.

The culprit wasn't the model's reasoning. It was the layer underneath it: the step that turns your text into the units the model actually consumes. We tend to think a language model reads our words. It doesn't. By the time the first transformer block sees your prompt, your carefully chosen sentence has been chopped, merged, and renumbered into something you'd struggle to recognize.

That preprocessing step is tokenization, and it has consequences that bite engineers every day: in your bill, your accuracy, your non-English users, and your weirdest bugs.

What a token actually is

Imagine you had to design the alphabet a model reads. You have two obvious options, and both are bad.

Option one: words. Give every word its own symbol. Clean and intuitive, until you meet antidisestablishmentarianism, a typo like teh, a username, a URL, or any of the endless words your vocabulary never saw. A word-level model has no way to represent anything outside its fixed list. Language is open-ended; a word list is not. This is the "open-vocabulary problem," and it's exactly what motivated subword methods in the first place (Sennrich et al., 2016).

Option two: characters. Give every character its own symbol. Now nothing is out of vocabulary; you can spell anything. But your sequences explode in length. A short paragraph becomes hundreds of symbols, and the model has to do far more work to learn that c-a-t means the same thing every time it appears. You've traded an impossible vocabulary for impossibly long inputs.

Tokens are the compromise in between. A token is a chunk of text: sometimes a whole common word ( the), sometimes a word fragment (ization), sometimes a single character or even a single byte. Common things get short representations; rare things get spelled out from smaller pieces. The model gets a manageable vocabulary (typically tens of thousands of tokens) and can still represent any string by falling back to smaller units.

How Byte-Pair Encoding builds a vocabulary

Byte-Pair Encoding (BPE) started life as a data-compression algorithm and was repurposed for tokenization by Sennrich, Haddow, and Birch (2016). The intuition is almost suspiciously simple: start small, then greedily glue together whatever pair shows up most often.

Here's the recipe:

1. Start with the smallest possible vocabulary: every individual character (or byte) in your data.
2. Scan your training corpus and count every adjacent pair of symbols.
3. Find the most frequent pair and merge it into a single new symbol. Add that merge to your vocabulary.
4. Repeat: re-count, find the new most-frequent pair, merge.
5. Stop when you've reached your target vocabulary size.

Each merge records a rule like "whenever you see t followed by h, combine them into th." Apply those rules in order and you can re-tokenize any text deterministically.

Type into the demo below to watch this kind of splitting happen. (It's an illustrative splitter, not a real GPT tokenizer, but the behavior is faithful: common chunks survive whole, rare ones shatter.)

Note: Real production vocabularies are built once, offline, on enormous corpora: GPT-2's vocabulary is 50,257 tokens. At inference time the tokenizer just applies the learned merges; it doesn't re-learn anything from your prompt.

Where it bites: five surprising consequences

Here's the part that surprised me when I first internalized it. Tokenization isn't a neutral pipe. Its quirks become the model's quirks. Five of them show up constantly in production.

1. Numbers fragment, so arithmetic is fragile

A model doesn't see the number 9.11. It sees a sequence of tokens. Depending on the tokenizer, a long number gets chopped into chunks of digits, and the decimal point is its own token. The value's magnitude is never handed to the model directly; it has to reconstruct it from pieces.

This is part of why the now-infamous question "is 9.11 bigger than 9.9?" trips models up. Look at how the two numbers fragment:

9.11 and 9.9 share a leading 9 and a decimal point, then diverge into 11 versus 9. Nothing in that token sequence says "compare the fractional parts as decimals." The model is reasoning over fragments inside an ambiguous frame.

2. Non-English text costs more, sometimes far more

This one has real money and fairness implications. Tokenizers are trained mostly on English-heavy data, so English compresses efficiently. Other languages, especially those in non-Latin scripts, get spelled out from many more, smaller pieces.

Petrov et al. (2023) measured this directly across parallel text and found that the same content, translated into different languages, can differ in tokenized length by up to about 15x. The same meaning, fifteen times the tokens.

Why this matters in three concrete ways:

• Cost. Most APIs bill per token. If your Hindi users' text inflates into many more tokens, they pay multiples more for identical requests.
• Context window. A fixed context window holds far less meaning in an inflated language. Your non-English user fills the window sooner.
• Latency and quality. More tokens means more compute per request, and heavily fragmented representation can make the language harder for the model to handle well.

3. Whitespace and trailing spaces change outputs

Back to the opening bug. Because a leading space is usually baked into the token ( the ≠ the), the boundary you leave at the end of your prompt matters. If your prompt ends in "Answer:" the model is poised to emit a token that starts with a space ( Paris). If you end it in "Answer: ", with a trailing space, you've already committed that space, and now the model is choosing among tokens that don't start with a space, a slightly different and often worse distribution.

Rule of thumb: don't put a trailing space before where you expect the model to continue.

4. Code and rare strings shatter

Watch what a tokenizer does to source code: indentation, camelCase identifiers, operators, and unusual variable names rarely earned their own tokens, so they fragment heavily. A long, unique identifier like getUserAccountBalanceByRegionId might cost far more tokens than its meaning warrants.

Practically: code-heavy prompts burn through your token budget faster than equivalent prose, and pasting a giant log or a long ID into context is more expensive than it feels.

Engineering Implications

Strip away the curiosities and a single practical reality remains: tokens are the unit of everything. You don't pay per word or per character; you pay per token. Your context window isn't measured in sentences; it's measured in tokens.

That reframes a lot of day-to-day decisions:

• Budgeting and cost. Estimate prompts in tokens, not words. Measure with the real tokenizer for your model rather than guessing.
• Few-shot examples cost what they cost. Every example you stuff into the prompt is re-tokenized and re-sent on every call. Few-shot prompting isn't free context; it's a recurring per-request token bill.
• Numbers and IDs. Don't assume the model "sees" a number. For high-stakes arithmetic, lean on tools or step-by-step reasoning rather than trusting the raw token sequence.

Tokenization is the hidden layer beneath your prompts: invisible, consequential, and worth understanding before you debug anything above it.

That frustration with the tokenizer as the weak link is widely shared:

Andrew Carr 🤸

@andrew_n_carr

·Follow

There's a weird reality that we mostly ignore in language modeling. It's the fact that we don't _actually_ train these models end-to-end. That's because we have the tokenizer! It's actually a really frustrating piece to tune with sometimes small changes mattering a lot and Show more

3:23 AM · Dec 1, 2023

228

Reply

Copy link

Read 19 replies

References

1. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016. arXiv:1508.07909
2. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). PDF
3. Kudo, T., & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer... EMNLP 2018. arXiv:1808.06226
4. Petrov, A., La Malfa, E., Torr, P. H. S., & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023. arXiv:2305.15425