What is a token?
A token is the small unit of text that an AI language model reads and writes. It can be a whole word, part of a word, a single character or even a byte. For Claude, one token is about 3.5 English characters, though that varies with the language.
When a model is given text, the text is first encoded into a series of tokens for the model to process. Tokens are usually hidden when you talk to a model in plain text, but they matter when you look at exactly what goes into and comes out of it.
Larger tokens make training and running a model more efficient, while smaller tokens let it handle uncommon or never-seen words. The way text is split into tokens can affect a model's performance and vocabulary.
Background from ai notis, not part of the news. Checked against its sources on 4 Oct 2026.
Our sources
- Glossary · Anthropic