What is tiktoken?
tiktoken turns text into the token IDs OpenAI's models read and back again, which is how developers count tokens before sending a request. It ships OpenAI's encodings: o200k_base for GPT-4o, GPT-4.1, GPT-5 and the o-series reasoning models, cl100k_base for GPT-4, GPT-3.5 and the current embedding models, o200k_harmony for the open-weight gpt-oss models, and older encodings for retired models. encoding_for_model("gpt-4o") picks the right one by name.
The core is written in Rust with Python bindings, and the README says it is 3x to 6x faster than a comparable open-source tokeniser on 1 GB of text. It is MIT-licensed, has about 19,700 GitHub stars, and version 0.14.0 was released on 17 August 2026. PyPI recorded about 225 million downloads in the last month, mostly as a dependency of other libraries.
On first use each encoding file is downloaded from OpenAI's public blob storage, checked against a SHA-256 hash and cached in the system temp folder, or in TIKTOKEN_CACHE_DIR if set.
Is tiktoken worth it?
tiktoken is the reference tokeniser for OpenAI models, and for that job there is nothing to weigh against it: it is OpenAI's own code, it is fast, and it is already installed in most Python AI stacks because langchain-openai, llama-index-core and others depend on it. Counting a GPT-5 prompt with o200k_base gives the same number the API bills.
The trap is using it for everything else. Claude, Gemini, Llama, Mistral and Qwen each use their own tokenisers, so a tiktoken count for those models is an estimate that can be off by a noticeable margin, more so for code, non-English text and unusual formatting. Anthropic's docs warn that even its own tokeniser changed with Claude Opus 4.7, and that counts from older models should not be reused for newer ones. Anthropic offers a free token-counting endpoint and Google's Gemini API a countTokens call; local tokenisers for open-weight models come with their Hugging Face repositories.
Two operational details catch people out. The encoding files are fetched from openaipublic.blob.core.windows.net on first use, so air-gapped servers and locked-down containers fail until the files are pre-cached and TIKTOKEN_CACHE_DIR points at them. And the model-to-encoding table is a hard-coded dictionary in the library, with a comment that it may become an API; a new model name can raise an error until the package updates, which is why release 0.14.0 made GPT-5 variants fall back to o200k_base.
The catch: tiktoken counts text, not requests. Chat formatting, tool definitions, images and reasoning tokens add to the bill in ways a plain encode() call does not see, so an OpenAI cookbook-style helper or the API's own usage field is still needed for exact cost.
Our take: keep tiktoken for GPT models and budget checks, and give every other provider its own counter; a single "tokens" number across vendors is a comparison of different units.
Where it earns its place
A backend developer adding a usage cap to a GPT-powered feature
Reject or trim prompts that would exceed a per-user token budget
Call encoding_for_model for the deployed model and len(enc.encode(text)) before each request. It runs locally in microseconds for typical prompts and adds no API cost.
A data engineer chunking documents for an OpenAI embedding index
Split text into chunks that stay under the embedding model's limit
Use cl100k_base, the encoding for text-embedding-3-small and -large, to measure chunks exactly. For a Claude or Gemini pipeline, switch to that provider's counting API instead of reusing these numbers.
A finance analyst comparing LLM vendors on cost
Estimate monthly spend on the same workload across providers
Count a sample of real prompts with tiktoken for GPT models, Anthropic's free count_tokens endpoint for Claude and countTokens for Gemini, then apply each price list. Using one tokeniser for all three would skew the comparison.
Before you commit
- Setup
- pip install tiktoken (Python 3.9 or newer); the first get_encoding call downloads the encoding file, so pre-cache it with TIKTOKEN_CACHE_DIR for offline machines.
- Integrations
- Used by LangChain, LlamaIndex, LLMLingua and many others; ports exist for JavaScript and other languages, maintained outside this repository.
- Data and privacy
- Runs locally; the only network call is the one-time encoding download from OpenAI's public storage, verified by SHA-256 hash.
- Cost at scale
- Free. Counting is CPU-bound and fast enough to run on every request.
- Limits
- OpenAI encodings only; 61 open issues; model names newer than the installed release may not resolve; message, tool and image overhead not included.
What’s good
- Exact counts for OpenAI models, from OpenAI
- Rust core, 3x to 6x faster than a comparable tokeniser per its README
- Small dependency footprint (regex and requests)
- Plugin mechanism for custom encodings
What’s not
- Counts for Claude, Gemini or Llama are only approximations
- Downloads encodings from the internet on first use
- Hard-coded model table lags new model names
- Does not count chat, tool or image overhead
Best for
Developers building on OpenAI models who need fast, exact token counts for budgeting, chunking and context-window checks.
tiktoken alternatives
tiktoken: common questions
- Is tiktoken worth it?
- The correct way to count GPT tokens and a wrong way to count anyone else's. We rate it 4.5 out of 5.
- How much does tiktoken cost?
- Free and open source under MIT.
- What are the downsides of tiktoken?
- Counts for Claude, Gemini or Llama are only approximations; downloads encodings from the internet on first use; hard-coded model table lags new model names; does not count chat, tool or image overhead.
- Who is tiktoken best for?
- Developers building on OpenAI models who need fast, exact token counts for budgeting, chunking and context-window checks.
- How much does tiktoken cost for a team or in production?
- Free. Counting is CPU-bound and fast enough to run on every request.
- Is tiktoken safe for company data?
- Runs locally; the only network call is the one-time encoding download from OpenAI's public storage, verified by SHA-256 hash.
- What are the best tiktoken alternatives?
- The closest alternatives we have reviewed are Hugging Face Transformers, LiteLLM and Chonkie.
Answers checked Nov 20, 2026.