Token

A small piece of text, often part of a word, that a language model reads and writes.

concept

Language models do not see text as letters or whole words. They split it into tokens: chunks that may be a short word, part of a longer word, a space or a punctuation mark.

Everything a model reads is counted in tokens, and so is everything it writes. That is why limits and prices for language models are usually measured in tokens rather than words.

In English, a common rough rule is that one token is about four characters of text. The real split depends on the tokenizer a model uses and on the language.

Example

A long word like “unbelievable” may be split into several tokens, while a short word like “the” is usually one.

Related