A new class of AI model skips writing out an answer and simply picks from a list of options in one quick pass. Researchers call these System One models, and they are increasingly used for routing requests, choosing a tool or checking whether a task is done. SO-Arena, a benchmark posted to arXiv, asks two questions about them: how capable are they, and can you trust their decisions to stay put. Its main finding is that the answers differ.
The benchmark covers 2,919 tasks across 11 domains grouped into five families: understanding text, knowledge and reasoning, safety and policy, judging evidence, and completing agent tasks. Reference labels were checked by hand against what the model can actually see, and most domains are split into difficulty tiers. Agent tasks run in sandboxes, testing both whether a model can pick the right next action on the web or with tools and whether it can judge a recorded run correctly. Reliability is measured separately with two tests: whether the answer survives the options being reordered, and whether the model notices when the correct option has been removed.
The 21 models tested include 16 open-weight System One models from 0.4B to 31B parameters, two closed decision services, Jev API 1.14 and OpenAI Decisions, and two conventional reasoning models, GPT-6 Sol and Luna, as references. The reasoning models still lead, with GPT-6 Sol scoring 82.3 and Luna 79.7, while the best fast models, GLiDE and KnowLine-31B, trail Luna by about ten points; Jev API 1.14 scored 60.4. On reliability the order changes. The abstract says the best-performing System One model ranks only seventh, and the most reliable, JevAny-27B, sits well down the capability table. Most models kept their answer when options were shuffled, with 14 of 18 scoring between 0.83 and 0.94, but spotting a missing correct answer varied widely.
For developers this is practical guidance. If you use a decision model such as Jev, which we review, or Google's new Decisions API to route traffic or enforce policy, the capability leaderboard is the wrong number to optimise alone. A model that confidently picks one of the offered options when none is right can be worse than a slightly weaker one that flags the gap. The paper also finds performance drops with task difficulty, especially in agent settings.
The limits are those of a fresh preprint. It has not been peer reviewed, several models are from small or little-known developers, and the reliability score combines two measures in a way readers may weigh differently. Results reflect each model's documented settings at one point in time; these services update often.