📄 Tech Report | 🤗 Information | 💻 Code
We’re introducing StateBench, a benchmark for long-context state monitoring—how properly language fashions hold monitor of details about entities in lengthy paperwork as they evolve.
State monitoring underlies duties from studying code, the place variables are assigned and reassigned, to following a narrative as characters bodily transfer round and objects change arms. For instance, if a thriller novel states that “John gave Mary the knife” proper earlier than a homicide happens, it’s essential to know that the state of every character has modified. Nevertheless, not all state-tracking issues are created equal: some require solely discovering the most recent replace; more durable ones require following a sequence of modifications, the place one missed step can throw off the ultimate reply. For instance, a more durable variant of our thriller state monitoring can be if John gave Mary the merchandise in his backpack, with it beforehand recognized that Edward put a knife in John’s backpack.
Transformers have well-documented theoretical limits on exhausting variants of state monitoring with lengthy dependency chains, serving to inspire recurrent and hybrid architectures comparable to Gated DeltaNet that carry a working reminiscence ahead. Present state-tracking exams are inclined to deal with the toughest instances, however as these designs grow to be extra frequent, researchers have to know which sorts of state monitoring every structure can deal with. We want to perceive the variations between architectures not simply on the toughest sorts of state monitoring, but in addition on the easier, doubtlessly extra frequent varieties.
To this finish, we developed the StateBench benchmark to seize state-tracking issues of various complexity, which we used to judge the variations between language fashions with completely different architectures. We discover that recurrent and hybrid fashions can generalize a lot farther than transformers when educated instantly on these duties, whereas frontier fashions together with Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna nonetheless break down on more durable types of state monitoring as inputs develop longer.
4 ranges of state monitoring
StateBench consists of 4 problem ranges that allow more and more advanced state replace operations. For programmers, these ranges of state monitoring complexity may be understood as permitting several types of variable project operations:
- R-trivial issues use immutable project. A variable’s worth by no means modifications, so the mannequin solely must get better what it was first set to. In code, this appears to be like like let x = 4; in on a regular basis life, it’s like a beginning date—it stays mounted.
- Aperiodic issues enable reassignment by worth. The mannequin wants to seek out the newest worth and might ignore earlier ones. In code, this appears to be like like x = 4 adopted later by x = 2; in on a regular basis life, it’s like a thermostat, the place the most recent setting determines the present temperature.
- Soluble issues add self-referential updates. Every replace builds on the present worth, so the mannequin has to hold modifications ahead reasonably than merely discover the most recent project. In code, this appears to be like like x = x + 5; in on a regular basis life, it’s like a recreation rating, the place every level provides to the working whole.
- Common finite issues add project by reference. One variable can tackle one other variable’s present worth, creating dependencies throughout updates. In code, this appears to be like like x = y or swapping two variables. Think about a magician hiding a ball below one in all 5 cups, then repeatedly swapping their positions. Discovering the ball means protecting monitor of how these swaps change its location. This illustrates the S₅ state-tracking drawback: following how successive swaps rearrange 5 objects. It belongs to the toughest of our 4 ranges and is mentioned in earlier work on the bounds of state monitoring.
These ranges may be intuitively understood by way of variable assignments, however in addition they symbolize theoretically motivated distinctions in state monitoring rooted in Krohn–Rhodes idea, a department of algebra that describes how state-tracking issues – which may be considered as phrase issues over finite monoids – may be decomposed into atomic primitives. Every degree in our hierarchy corresponds to a category of state-tracking issues that may be constructed out of a distinct set of primitives.
To check the 4 ranges pretty, we begin with a normal finite sequence of variable assignments and derive three simpler variations from it, producing 4 comparable StateBench issues—one at every problem degree. All 4 use the identical variety of variables and distinct values, with assignments showing on the similar factors within the sequence; solely the kind of project modifications.
What we discovered
Pretrained fashions
We examined a variety of open pretrained fashions on the r-trivial, aperiodic, and normal finite ranges of StateBench to see the place their state monitoring breaks down, and whether or not completely different architectures fail the place idea predicts. Underneath commonplace theoretical assumptions, transformers can deal with r-trivial robustly (and soluble and aperiodic to a point) however are predicted to fail on normal issues. In distinction, Gated DeltaNet with a extra versatile reminiscence replace can, in precept, deal with all 4 ranges.
We began with open pretrained fashions, the place the coaching information and context lengths are totally documented. These included two 7-billion-parameter fashions, Olmo 3 (a transformer) and Olmo Hybrid (which provides Gated DeltaNet layers), together with smaller fashions with 275 million and a pair of.7 billion parameters. All of those fashions have been pretrained on the identical information; none had seen contexts longer than 64,000 tokens throughout coaching, and we examined most of them on inputs properly past that size.
After we scored every mannequin, nearly all carried out properly on r-trivial issues, misplaced accuracy on aperiodic ones as inputs grew, and stayed at probability on normal finite issues. Drawback format mattered right here—the 275-million-parameter hybrid and Gated DeltaNet fashions scored close to the ceiling on RULER-style r-trivial issues however have been close to probability on the identical issues framed as spreadsheet edits. That hole is one motive StateBench exams the identical underlying drawback in a number of codecs.
Frontier fashions
We then examined 4 frontier fashions – Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna – asking them to learn every sequence of variable assignments and generate the ultimate worth of each variable.
All 4 carried out properly on r-trivial and aperiodic issues. However on soluble and normal finite issues, accuracy fell sharply as inputs grew longer, dropping to 13% or decrease on the longest normal finite issues. One doubtless issue is sensitivity; at these more durable ranges, many extra steps can have an effect on the ultimate reply, leaving much less room for a mannequin to overlook an replace with out getting the end result unsuitable.
Fashions educated instantly on the duty
Subsequent, we investigated whether or not fashions’ problem with the more durable ranges displays limits of the architectures themselves, or whether or not pretraining merely doesn’t educate fashions to resolve them. Focused direct coaching is a extra favorable setting that reveals to what diploma state-tracking points come from the underlying structure and optimizer reasonably than the info used to pretrain a mannequin.
We educated a number of 275-million-parameter fashions from scratch on StateBench: two transformer variants, two hybrids, and two Gated DeltaNet variants with completely different memory-update constraints. The fashions educated on sequences of as much as 256 assignments and have been examined on sequences so long as 64,000 assignments, or 250 instances the utmost coaching size.
The architectures started to diverge as sequences grew longer. Gated DeltaNet and hybrid fashions generalized nearly completely on r-trivial, aperiodic, and soluble issues out to the longest sequences we examined, tens of 1000’s of assignments. Transformers realized the better ranges however degraded on longer sequences, they usually by no means realized normal finite issues. On the toughest degree, the Gated DeltaNet variant with a extra versatile reminiscence replace generalized the farthest, staying close to excellent out to roughly 10 instances the size of the longest sequences it noticed throughout coaching.
For Gated DeltaNet and hybrid fashions, the hole factors to coaching reasonably than structure; educated instantly, they realized to deal with state-tracking ranges their pretrained counterparts couldn’t. Transformers have been completely different—even educated instantly, they may by no means be taught the toughest degree.
Open questions and the discharge
Some questions stay open. Our new proof exhibits that Gated DeltaNet with a extra restricted reminiscence replace (non-negative eigenvalues) can resolve all aperiodic duties and sure periodic duties, comparable to monitoring whether or not a repeated enter has occurred an excellent or odd variety of instances, below the paper’s fixed-precision arithmetic with rounding. Its actual placement within the hierarchy stays open, together with whether or not it may resolve all soluble duties and the way numerical precision or combining it with consideration layers impacts its capabilities. StateBench additionally assumes a hard and fast variety of variables, whereas actual duties typically introduce new entities over time. In a protracted dialog, for instance, new individuals and objects can seem because it unfolds, and the mannequin has to start monitoring them too. The present framework doesn’t but cowl that setting.
As new flavors of recurrent and hybrid architectures emerge, understanding which sorts of state monitoring every can deal with may assist information how future techniques are developed and educated. We hope StateBench offers researchers a finer-grained approach to discover the place fashions break down—and to construct coaching and analysis information for the capabilities that also fall quick. Extra basically, we hope it serves as a North Star for researchers to develop new architectures with extra sturdy state-tracking talents in comparison with present fashions.
