Token · Text unit of a language model
A token is the unit of text a language model processes: a word, part of a word, a character or a punctuation mark. The number of tokens, not words, determines what fits in the context window and what a call costs.
Standalone definition · 41 words
How does text become tokens?
A tokenizer splits text according to an encoding specific to each model. For English, OpenAI gives a rough rule: one token is about four characters or three quarters of a word, so 100 tokens for 75 words. Spaces and capitalization change the split, and other languages split differently.
Why is it useful in GEO?
The context window and retrieval budgets are counted in tokens: long or verbose content takes more room for the same information. Metehan Yeşilyurt measured the same Arabic text going from 70 tokens with GPT-2 encoding to 21 with o200k, a reminder that splitting varies by model and language.
What should you take away for writing?
Aim for information density rather than length, avoid filler and put the essentials early on the page. To count precisely, use the tokenizer or tiktoken of the targeted model’s provider.
Frequently asked questions
Is a token the same as a word?
No. A common word can be one token, a rare word several, and punctuation counts too.
Does the token count vary by model?
Yes. The same text can produce different counts depending on the model, its encoding and the language.
Related terms
Further reading
- Profound vs Otterly vs Ahrefs Brand Radar: query fan-out comparison
- ← Back to the GEO glossary (114 terms)