An agent that says a job is done is not the same as an agent that did it. Microsoft's ThinkingBox benchmark was built around that gap: each of its 507 tasks gives an agent a real backend, a set of MCP tools and a simulated customer, then checks what actually changed in the database. In a joint post with Hugging Face, the team now reports turning those same checks into a training signal, and the results speak to anyone choosing a model for work that touches production records.
The idea is simple. If an executable check can score the final database state as pass or fail, it can also serve as a reward for reinforcement learning. The harder part was picking practice tasks. Tasks the model already solves, or cannot solve at all, teach it nothing, so the team estimated each candidate's success rate once with the untrained model and kept a fixed pool of 187 tasks in the useful middle. They then fine-tuned Qwen3.8-27B with GRPO for 50 steps on 24 H100s and tested it on the 507 benchmark tasks, whose domains and databases it had not seen in training.
Average success rose from 51.70% to 60.78%. The bigger change was consistency: tasks passed on all 20 attempts went from 38 to 107. In one hotel-booking case, the base model changed a booking without the required manager approval in 15 of 20 runs; the trained model left it alone every time. For comparison, the team's earlier figures put Claude Opus 5.5 at 241 tasks passed every time and GPT-6 Astra at 231, while GPT-5.6 Sol scored slightly higher on average than the fine-tuned Qwen, at 61.91%, but passed fewer tasks every time, 82. The authors estimate $8.00 per dependable task for the trained model, near GPT-6 Astra's $7.45 and Opus 5.5's $7.80.
The trade-off is reported honestly. Tasks solved at least once fell slightly, from 453 to 446, and dropped in three of five domains, with auto insurance losing 7.0 points. The model became more repeatable, not broader.
This is a blog result on the authors' own benchmark, with GPT-5 Chat playing the simulated user and occasionally judging, and the cost figures are a comparison index rather than real bills. The training code is on GitHub, which makes replication possible. Until someone does that on other agent benchmarks, treat it as a promising recipe rather than a settled one.