Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Tokenizer

A tokenizer is the component that splits text into tokens and turns them into numeric IDs a language model can process, then turns the model's output IDs back into text.

Also called: tokenization, BPE, byte pair encoding

By Divya Sharma · Checked Nov 15, 2026 · How we review

Every language model ships with its own tokenizer, fixed at training time. You cannot swap it for another one without retraining the model.

How it works

Most modern tokenizers use subword units. A 2015 paper by Sennrich and colleagues showed that rare and unknown words can be encoded as sequences of subword units, using a segmentation based on byte pair encoding (BPE). BPE starts from characters or bytes and repeatedly merges the most frequent pairs into new vocabulary entries. Common words end up as one token, and rare words split into pieces, so the model can still handle a word it has never seen.

Why builders care

The tokenizer decides how many tokens your text becomes, which drives cost, speed and how much fits in the context window. Tokenizers trained mostly on English can split other languages into many more tokens, making them slower and dearer. Spaces, numbers and code formatting can tokenize in surprising ways, which helps explain why models sometimes struggle to count letters or do digit arithmetic.

In practice

Use the tokenizer that matches your model when counting tokens. The Hugging Face Transformers library loads the matching tokenizer for each model.

Related terms

Token (LLM)
A token (LLM) is the basic unit of text a language model reads and writes, often a word, part of a word, a punctuation mark or a byte. Prices, speed and context limits are all counted in tokens.
Large language model
A large language model (LLM) is a neural network with billions of parameters, trained on large amounts of text to predict the next token, which lets it write, answer questions, summarise and follow instructions.
Token pricing
Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
Context window
A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
Embedding
An embedding is a list of numbers (a vector) that represents a piece of text, an image or other data, produced so that items with similar meaning end up with vectors close together.

Sources