Token (LLM)
A token (LLM) is the basic unit of text a language model reads and writes, often a word, part of a word, a punctuation mark or a byte. Prices, speed and context limits are all counted in tokens.
Also called: tokens
By Divya Sharma · Checked Nov 15, 2026 · How we review
Language models never see letters or words directly. A tokenizer splits text into tokens and maps each one to a number, and the model predicts the next token from the ones before it.
Why it matters
Tokens are the unit for almost every practical limit. The context window is a token count. API bills are per input and output token, as covered in token pricing. Speed is quoted in tokens per second. Compare prices in our AI model pricing research.
How many characters is a token?
It depends on the tokenizer and the language. Common English words are often a single token, while rare words, code and many non-English scripts split into several. Anthropic's documentation says a token is about 3.5 English characters on its earlier models, and that its newer tokenizer produces about 30 percent more tokens for the same text. The same text can therefore cost different amounts on different models. Use the provider's token counter for exact numbers.
Not the other tokens
This page is about text units in language models. It is unrelated to API or authentication tokens, such as an API key or OAuth access token, and to crypto tokens.
Related terms
- Tokenizer
- A tokenizer is the component that splits text into tokens and turns them into numeric IDs a language model can process, then turns the model's output IDs back into text.
- Token pricing
- Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
- Context window
- A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
- Output tokens
- Output tokens are the tokens a language model generates in its reply. They are counted and billed separately from input tokens, and a max_tokens setting caps how many one response may contain.
- Tokens per second
- Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.