When a team fine-tunes an open model for one narrow job, it usually changes the weights and leaves the tokenizer alone. Dharma-AI argues that this is a costly habit. In a Hugging Face post previewing a paper still under review, it reports cutting the measured inference cost of a production model by 77% by rebuilding only its vocabulary, with the same base model, data and training recipe.
The model is a Qwen3-4B grader for Portuguese essays that returns a JSON assessment for each competency. Its tokenizer was the original one, built for many languages and kinds of text, with 151,669 entries. That causes two kinds of waste. The phrases the grader writes on every request, such as its JSON envelope and competency names, are split into many tokens. And at every generated token the model scores the whole vocabulary, including entries this task never uses; in a model this size, the output layer that does that holds almost 10% of the parameters.
The method, called VocabX, changes what the vocabulary optimises for. Instead of choosing new tokens by how often text appears, it weights each saving by its cost, and in this case a generated token cost 7.4 times as much as one read, so savings in the response count far more. It also works at two levels at once, reshaping how words are cut into pieces and turning long repeated spans, from phrases to whole JSON structures, into single macro tokens. A statistical estimate for candidates missing from a sample raised token reduction on a contracts corpus from 44.7% to 52.8%. In the grader, average responses fell from 350 to 101 tokens and a quality score rose from 0.703 to 0.810, measured on 500 requests with vLLM on an RTX 5090.
The result matters to anyone running a small fine-tuned model at volume, whether on Qwen or Gemma. The authors compare their approach with existing methods, including Liquid AI's in-place tokenizer expansion, and say those choose tokens by frequency and work on one level only.
Caution is needed. The figures come from a single production task with a highly repetitive output format, which is close to the best case for this technique. The paper is not yet published, and the authors say the numbers may change. The cost ratio between generated and read tokens depends on hardware, batch size and context length, so the savings will vary. The method looks worth testing, but not yet worth assuming.