Token

A token is the basic unit of text that a language model processes – usually a word, a piece of a word, a punctuation mark or a space. Before a model reads a text, a tokenizer splits it into tokens and maps each to a number; the model then predicts, one token at a time, what comes next.

Why fragments, not words

Models need a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Whole words would leave too many gaps – names, typos, new coinages – and single letters would make texts very long. Most tokenizers therefore use subword methods such as byte-pair encoding, adapted for neural machine translation in a 2015 paper, which keep common words whole and split rare ones into familiar pieces. In English, a token averages about four characters, or roughly three-quarters of a word, according to OpenAI’s rule of thumb.

Tokens as currency

Tokens are how the AI industry counts. Context windows are measured in tokens, training data in trillions of tokens, and commercial models are priced per million tokens of input and output. Output tokens usually cost several times as much as input tokens, largely because the model has to generate them one by one rather than read them in a single pass. The price of a given level of capability has fallen steeply – by 9 to 900 times a year depending on the task, by one estimate – while usage has reached enormous volumes: on one developer platform, a single low-cost model processed 59 trillion tokens in a week. The pattern is discussed under Jevons Paradox.

Side effects

Tokenization explains some strange failures. Models long struggled to count the letters in a word such as “strawberry,” largely because they see tokens rather than individual letters. Some tokens that were rare in training produced erratic behavior – the “glitch token” SolidGoldMagikarp became famous in 2023 and turns up in the jailbreak song DO ANYTHING NOW. And tokenizers are not neutral between languages: a 2023 study found that the same text can take up to 15 times as many tokens in some languages as in English, which makes using AI slower and more expensive for their speakers.

Why it matters

Tokens are the hidden unit behind much of what users notice about AI: how long a conversation can run, what it costs, why a model stumbles over spelling and why some languages are served worse than others. Knowing what a token is turns several mysteries of AI into ordinary engineering trade-offs – and points to a fairness question worth pressing, since better tokenizers and more training data for smaller languages can narrow the gap.

Written with Claude, made by Anthropic, which prices its Claude models per token.

Related

Large Language Model · Context Window · Embedding · Inference · Jevons Paradox · DO ANYTHING NOW

Sources: Rico Sennrich, Barry Haddow, Alexandra Birch: Neural Machine Translation of Rare Words with Subword Units (arXiv) (2015; ACL 2016) · OpenAI Help Center: What are tokens and how to count them? · Claude: Pricing · Epoch AI: LLM inference prices have fallen rapidly but unequally across tasks (March 12, 2025) · Tairan Fu et al.: Why Do Large Language Models (LLMs) Struggle to Count Letters? (arXiv) (December 2024) · Jessica Rumbelow, Matthew Watkins: SolidGoldMagikarp (plus, prompt generation) (LessWrong) (February 5, 2023) · Aleksandar Petrov et al.: Language Model Tokenizers Introduce Unfairness Between Languages (arXiv; NeurIPS 2023) (May 2023)

Leave a Reply

Your email address will not be published. Required fields are marked *