What are tokens?

A model cannot read. It can only do arithmetic.

Everything inside a language model is multiplication and addition on numbers. So the very first thing that has to happen to your sentence is conversion: text in, integers out. Nothing else can proceed until that is done.

The only question is where to cut.

That choice sounds like an implementation detail. It decides how much you pay, what the model can spell, and how well it works in your language.

Why not just use letters?

Twenty-six letters plus punctuation is a tiny, tidy vocabulary. Nothing is ever unknown — every possible word can be spelled. It is the obvious answer, and it was tried.

The problem is length. A page becomes thousands of units, and the model has to hold every relationship between them. Attention cost grows roughly with the square of the length, so this gets expensive very fast — and each individual letter carries almost no meaning to work with.

Character-level models do exist. They are excellent at spelling and slow at everything else.

Why not just use words?

Short sequences, each unit meaningful. Also the obvious answer, also tried, also broken — but in the opposite direction.

English has hundreds of thousands of words, and that is before names, typos, hashtags, product codes and every language other than English. Worse, anything outside the list becomes literally unrepresentable: the dreaded unknown token, where information simply disappears.

You cannot enumerate a living language. People invent words faster than you can ship a vocabulary.

So: chunks. Frequent things whole, rare things in pieces.

The answer everyone landed on sits between the two. Start with individual characters, then repeatedly find the most common adjacent pair in a huge pile of text and glue it together into a new unit. Do that fifty thousand times.

Common words end up as single units because they appeared constantly. Rare ones stay as fragments, assembled from pieces.

Nobody chose “low” as a unit. It won because it turned up in every word. The vocabulary is discovered, not designed.

Type something. Watch where it breaks.

Common words survive whole. Unusual ones shatter into pieces. Try a name, a long word, a number, some code — the places it fragments are exactly the places models get unreliable.

This is an approximation of a real byte-pair tokeniser, not the genuine vocabulary — close enough for English that the behaviour is honest, and worth saying out loud rather than hiding.

This is why it cannot count the r’s.

The question that made the internet laugh at these systems has a boring answer. By the time the model begins working, the letters are gone. It received a handful of opaque ids, not a string of characters to inspect.

Asking it to count letters is like asking you to count the brushstrokes in a photograph of a painting.

Newer models often get this right — because they have been trained on people asking, not because they can suddenly see letters.

It also explains the bad arithmetic.

Numbers are chopped like everything else, and the chopping does not respect place value. The same digits can split differently depending on what surrounds them, so the model never reliably sees units, tens and hundreds as a structure.

It is pattern-matching the shape of sums it has read, not calculating.

Which is why serious systems hand arithmetic to a calculator through a tool call rather than trusting the model with it.

You are charged per token, not per word.

Every provider prices by the token, in and out. So the ratio between words and tokens is not trivia — it is the exchange rate between your writing and your invoice.

Plain English runs around 1.3 tokens per word. Code, tables, unusual names and non-English text all run higher, sometimes much higher.

The same sentence costs more in other languages.

Tokenisers are trained mostly on English text, so English gets the efficient pieces. Everything else is assembled from smaller fragments — which means more tokens for the same meaning.

More tokens means a higher bill, slower responses, and less that fits in the context window. For the same sentence.

This is a fairness problem, not a footnote. The people who most need cheap access to these tools are frequently the ones paying several times more for it.

The context window is measured in tokens too.

When a model advertises a limit, that limit is tokens — not words, not pages. So the real capacity depends entirely on what you put in it.

A window that holds a long novel in English might hold a fraction of that in code, or in Hindi. Discovering this in production is a rite of passage.

Rough conversion for English prose: multiply words by about 1.3. For code, closer to 2. For non-Latin scripts, sometimes 3 or more.

Spaces count. Capitals count. Everything counts.

A leading space is usually part of the token, so “ cat” and “cat” are different ids entirely. A capitalised word is often a different token from its lowercase twin. Repeated whitespace in a pasted document becomes tokens you are paying for.

None of this is intuitive, and all of it is measurable.

This is also why prompt formatting sometimes changes an answer for no apparent reason: you changed the tokens without meaning to.

Each id becomes a position in space.

A token id is just a row number — 1642 is no more related to 1643 than to 90210. On its own it carries no meaning whatsoever.

So immediately after this step, every id is swapped for a long list of numbers: coordinates in a space where words used similarly end up near each other. That is the next lesson, and it is where meaning actually enters the machine.

Tokens are the alphabet. Embeddings are the meaning. Neither works without the other.

Spelling failures

The letters are discarded before the model starts. It is recalling what it has read about spelling, not looking.

Arithmetic failures

Digits split in ways that ignore place value, so the structure of a number is never reliably visible.

Your bill

Priced per token. The word-to-token ratio is the exchange rate between what you write and what you pay.

The language tax

English gets the efficient pieces. Other languages pay more for the same meaning, in money and in context.

Context limits

The window is tokens, not pages. Code and non-Latin scripts fill it several times faster than English prose.

Odd prompt sensitivity

A space or a capital changes the ids. Sometimes that is enough to change the answer.