Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Token (LLM)

A token (LLM) is the basic unit of text a language model reads and writes, often a word, part of a word, a punctuation mark or a byte. Prices, speed and context limits are all counted in tokens.

Also called: tokens

By Divya Sharma · Checked Nov 15, 2026 · How we review

Language models never see letters or words directly. A tokenizer splits text into tokens and maps each one to a number, and the model predicts the next token from the ones before it.

Why it matters

Tokens are the unit for almost every practical limit. The context window is a token count. API bills are per input and output token, as covered in token pricing. Speed is quoted in tokens per second. Compare prices in our AI model pricing research.

How many characters is a token?

It depends on the tokenizer and the language. Common English words are often a single token, while rare words, code and many non-English scripts split into several. Anthropic's documentation says a token is about 3.5 English characters on its earlier models, and that its newer tokenizer produces about 30 percent more tokens for the same text. The same text can therefore cost different amounts on different models. Use the provider's token counter for exact numbers.

Not the other tokens

This page is about text units in language models. It is unrelated to API or authentication tokens, such as an API key or OAuth access token, and to crypto tokens.

Related terms

Tokenizer
A tokenizer is the component that splits text into tokens and turns them into numeric IDs a language model can process, then turns the model's output IDs back into text.
Token pricing
Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
Context window
A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
Output tokens
Output tokens are the tokens a language model generates in its reply. They are counted and billed separately from input tokens, and a max_tokens setting caps how many one response may contain.
Tokens per second
Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.

Sources