Squeezing a model into 4-bit weights is the usual way to run an agent on a cheaper GPU, and the usual cost is that the agent gets worse at long, multi-step jobs. SteerDistill, a preprint from Tsinghua University's Institute for AI Industry Research, goes after that loss directly. It matters to anyone serving agents on modest hardware, because the method targets the steps where compressed models actually go wrong rather than general text quality.
The idea is to let the full-precision model act as a coach. When the 4-bit student fails a turn, the original model suggests a revision, the student tries it in the environment, and the coach adjusts its advice based on what happened. The fixes that work are then trained into the student, which learns to copy its own coached behaviour on those turns and the original model everywhere else. At run time the student works alone. Weights are stored in 4-bit integer groups of 64 while activations stay at 16 bits, the setting known as W4A16.
The authors tested three models, Qwen3-8B, Qwen3.5-4B and Gemma 4 12B-it, on five agent benchmarks: BFCL V4, τ²-Bench, WebShop, ALFWorld and a search question-answering set built from seven datasets. Averaged across them, Gemma 4 12B-it scored 64.31 against 65.13 at full precision, and Qwen3-8B scored 53.18 against 53.54. The strongest rival in each case was QeRL, a reinforcement learning method, which trailed by roughly two points. On τ²-Bench with Qwen3-8B, SteerDistill reached 31.68% after 32 GPU hours on A100s, while QeRL needed 96 hours to land 1.84 points lower. Pushed to 2-bit weights, Qwen3-8B fell to 42.97, still ahead of the other methods tested.
For readers of our Gemma 4 review, this is the kind of result that could make a 12B model practical on a single consumer card for tool-calling work. Most quantisation research still reports perplexity or short-answer accuracy, which says little about whether an agent will finish a twelve-step task. Measuring success on agent benchmarks is the more honest yardstick, and the authors have linked a GitHub repository and project page.
The limits are real. This is a preprint and has not been peer reviewed. Every model received task-specific post-training before quantisation, so the numbers describe agents tuned for these benchmarks, not general chat models compressed off the shelf. The coaching step only works on tasks where the full-precision model succeeds, so failures it cannot solve itself are left as they are. QeRL and one other baseline were built for NVIDIA's NVFP4 format and had to be adapted to integer quantisation, which may have handicapped them. Gains of two points are useful but modest, and independent replication on other model families would settle how far they carry.