- Before an AI reads anything, it chops the text into chunks called tokens, and it never sees the individual letters. That is why it has historically struggled to count or spell.
- The same system makes some languages far more expensive than others, since the chunking is tuned for English and treats everything else as costly.
- Tokenization quietly shapes what a model can do, what it costs, and how much fits in its memory, all before any thinking happens.
The Setup
Ask an older AI how many R's are in strawberry and it often gets it wrong. People treat that as proof the thing is dumb. It is actually a clue about how these models read, and the answer is a step almost nobody talks about: tokenization. It happens before the model thinks at all, and it shapes everything after.What Tokenization Actually Is
A language model does not read letters. The first thing it does with your text is chop it into chunks called tokens, then turn each chunk into a number it can do math on. A token is usually a word-piece, somewhere between a letter and a word. Common words like the become one token. A word like strawberry often splits into two, straw and berry. From that point on, the model works with those chunks, not the original letters.Why It Cannot Count the R's
Now the strawberry mystery makes sense. The model sees the tokens straw and berry, not s-t-r-a-w-b-e-r-r-y. It was never shown the individual letters or their order, so asking it to count the R's is like asking you to count letters in a word you only ever heard, never saw spelled. Newer reasoning models handle this better by working through it step by step, but the underlying blind spot is baked into how text gets fed in.Why Some Languages Cost Double
Here is the part that hits the wallet. Tokenizers are trained mostly on English, so English compresses efficiently, around four characters per token. Other languages do not. Chinese averages closer to two characters per token, so the same meaning takes about twice as many tokens, and some low-resource languages need ten to fifteen times more. Since AI is billed per token, a service in Hindi can cost hundreds of thousands of dollars a year more than the identical English one. Same product, very different bill.What It Means For Investors and Users
Tokenization is a quiet lever under a lot of AI economics. It sets the real cost of serving non-English markets, which shapes where products launch first and who they serve. It also defines context limits, since a model's memory is measured in tokens, not words, so a long document in a token-heavy language fills the window faster. Anyone building or buying AI at scale should look at token efficiency alongside the headline price per token.Why It Is Worth Understanding
The lesson is bigger than one word puzzle. A lot of AI's strange failures, spelling, simple arithmetic, odd behavior on rare words, trace back to this first invisible step. The model is not reasoning over reality. It is reasoning over chunks of text it was handed. Understanding the input explains a surprising amount of the output.FAQ
So is the strawberry thing fixed now?Mostly, for the big models. Reasoning models spell the word out step by step and get it right. But the blind spot is still there underneath, it is just being worked around, not removed, since the model still reads tokens, not letters.
Why don't they just tokenize by letter?
They could, and a few systems do, but it is far less efficient. Letter-by-letter means many more tokens per sentence, which is slower and more expensive. Word-piece tokens are a trade: cheaper and faster, at the cost of these blind spots.
How does this affect me practically?
If you use AI in a non-English language, you are likely paying more per message and filling the context window faster, often without realizing it. And if you need exact letter or number work, give the model room to reason step by step rather than trusting a one-shot answer.