Ask a modern AI to find a fact buried in a long document and it usually can. Ask it to say what that fact is now, after the document has changed it several times, and it starts to slip. That is the gap Ai2's new benchmark, StateBench, is built to measure, and it matters for anyone who feeds an agent a long log, a contract or a codebase where values keep getting overwritten.
StateBench sorts the problem into four levels of difficulty. The easiest, which Ai2 calls r-trivial, is a value that is set once and never changes, like a birth date. Aperiodic problems reassign a value, so the model must find the most recent one. Soluble problems build on the current value, like adding to a running score. The hardest, general finite, let one variable take another's value or swap two around, which is the magician's cups-and-ball problem of tracking where things end up. The same underlying case is posed in several formats, because Ai2 found that framing a task as a spreadsheet edit rather than plain text could swing a small model from near perfect to near chance.
The frontier results are the headline. Gemini 3.8 Flash, Claude Opus 5.5 and OpenAI's GPT-6 models, all of which we review, handle simple overwrites well but fall to 13% or lower on the longest swap-style problems. Ai2's explanation is sensitivity: on the hardest level many more steps feed the final answer, so one missed update is enough to make the whole thing wrong. Retrieving a fact and tracking its current state are not the same skill.
The more interesting finding concerns why. Transformers have known theoretical limits on long chains of state changes, while recurrent designs such as Gated DeltaNet carry a running memory. When Ai2 trained small 275-million-parameter models directly on the task, the hybrid and recurrent models generalised to inputs 250 times longer than their training data, while transformers never learned the hardest level even with direct training. For the hybrids, the weakness traced to training data rather than architecture, which is the part model builders should watch.
The limits are worth keeping in view. StateBench assumes a fixed set of entities, while real conversations introduce new people and objects as they go, so it does not yet cover that case. The work is a tech report with public data and code rather than a peer-reviewed result, and the practical lesson is simple: if your agent must know the current value, not just recall that a value existed, test it on changing facts before you trust it.