acceptodds
Under review as a conference paper at ICLR 2027

StateTrackBench: Executable Trajectories for Locating Where Language Models Lose Track of Evolving State

Abstract

Long-running agents must maintain the current state of a world that changes as messages arrive. A model may observe every update yet lose its cumulative effect, leaving its state incorrect even when the final answer is right. Final-answer scores alone cannot locate failures in message interpretation, state representation, or transition computation. We introduce StateTrackBench, a benchmark of 6,055 executable trajectories that makes these failures observable. Each trajectory pairs messages with typed updates, deterministic transitions, and a query readout. Replay reconstructs the gold state after every message. Executing the same model-parsed updates through different state interfaces then separates computing and serializing state transitions from interpreting messages. The benchmark combines template-rendered platform records with generated task families. A four-model evaluation shows that three models approach ceiling on the platform strata, whereas generated histories remain challenging. GPT-4o diagnostics reveal corrupted state behind correct answers and transition errors on an auxiliary ledger evaluation. Using Parse–Then–Replay (PTR) as a parse-then-execute reference, we apply replay-time checks to locate messages for conflict-stated re-parsing. On GitHub label histories held out from repair development, the configured GPT4o route improves with repair from 0.918 to 0.990 accuracy; matched controls support the contribution of conflict-specific feedback. Intermediate-state checks therefore reveal errors hidden by answer scores and provide a basis for targeted repair. Code and data will be made public upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, State Tracking, Agent Memory, Evaluation Benchmarks, Error Diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_E8Q4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces StateTrackBench, a benchmark of 6,055 executable trajectories for testing whether language models can keep track of evolving state. Each message is paired with a typed update, and replaying the updates gives the gold state after every message. The data comes from GitHub, GitLab and npm event records plus 15 generated task families, and all text is template-rendered. The authors evaluate GPT-5, Gemini-2.5-Pro, Doubao-Seed-1.8 and GPT-4o, and run diagnostics with GPT-4o. They also propose Parse–Then–Replay (PTR), where the model parses each message and a program replays the updates, and a repair method that re-parses messages flagged by replay checks, which improves GPT-4o from 0.918 to 0.990 on GitHub label histories.

Critical strengths
  1. In this benchmark, replay gives a gold state after every message, which makes step-level error analysis possible.
Critical weaknesses
  1. The main claim is not supported by the main evaluation. The paper says final-answer scores hide errors in intermediate states, but the leaderboard only scores final answers. Intermediate states are only checked for GPT-4o on a few diagnostic subsets, so it is unclear whether this problem still exists for stronger models.
  2. The benchmark is mostly saturated and the data is not realistic. Strong models are near 100% on the platform data (about 74% of the test set), and a simple answer prior already gets 0.875 on GitLab. The generated part is similar to bAbI, with template-rendered text and only 10 distractor sentences, and none of it is checked by humans. Also I don't find some most recent models on this benchmark, like GPT-5.6/GPT-6, Claude Opus 5/5.5 etc.
  3. PTR and the repair method only work in this benchmark. They rely on a predefined update schema and the benchmark’s own transition function, which are usually not available in real agent settings. The repair is only tested on one task with GPT-4o, and on generated data its alarms have only 0.75% precision and 4.72% recall.
Questions for authors
  1. Can PTR or similar repair method be used when the update schema and transition rules are not given?
Reviewer_r8PZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces StateTrackBench which is a benchmark of 6,055 executable trajectories. This benchmarks aims to makes locate failures in message interpretation, state representation, or transition computation as opposed to focusing only on final-answer.

Critical strengths
  • The problem space is relevant, understanding and improving failures in reasonings and interpretations along the way is as important as final task, improvements can decrease token cost and improve accuracy.
Critical weaknesses

The paper is poorly written and very hard to understand, the text ends abruptly without explanation. Here are a few examples:

  • It is not clear what the paper adds to the listed and compared benchmarks? What is Disjoint split and Fixed Parse?
  • It is very hard to understand how the benchmark is created, what are the details and how it is evaluated, and what it is measuring and how?
  • What is Table 2 and what is is trying to convey?
  • The evaluation conditions in Table 3 and the learnings that follow that are not clear.
Questions for authors
  • The paper needs a major rewrite to be understandable.
Reviewer_VJz9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces StateTrackBench, a benchmark of 6,055 executable trajectories from GitHub, GitLab and npm records and 15 generated task families, for evaluating whether language models maintain evolving state. Each trajectory pairs template-rendered messages with typed intended updates and a deterministic transition function, so replay reconstructs the gold state after every message; source-object splits and shortcut readers control for leakage and shortcuts. The paper defines state interfaces, including the parse-then-execute reference PTR and fixed-parse folds, to separate message interpretation from transition computation, and proposes replay-grounded, conflict-stated re-parsing as a repair. Validation includes a 650-example test leaderboard for GPT-5, Gemini-2.5-Pro, Doubao-Seed-1.8 and GPT-4o, plus GPT-4o interface and step-level diagnostics on platform cohorts and a ledger grid. On held-out GitHub label histories, the repair raises PTR accuracy from 0.918 to 0.990.

Critical strengths
  1. Typed intended updates with deterministic replay give gold state after every message, so error types can be checked separately (Eq. 1, Sec. 3.1).
  2. Fixed-parse comparison isolates the executor. On the ledger grid, GPT-4o folds PTR's parses at 0.720, while program replay of the same parses scores 1.000 (Table 17).
  3. Validity controls are rigorous: replay of all 6,055 examples, source-object splits, shortcut readers (Table 8), a closed-book probe (Table 14) and a deviation log (App. E).
  4. Correct answers can hide corrupted states: 84/468 and 162/600 correct reopen answers end in an incorrect state (Table 20).
  5. The repair ablation is matched for call budget and attributes the gain to conflict feedback (22/0 vs. a generic re-parse), not targeting (7/2, n.s.) (Table 21).
Critical weaknesses
  1. Little headroom. Three of four models score 0.94–1.00 on the platform strata (Table 3), so difficulty rests on 172 generated test examples with 7–17 per family (Table 15).
  2. The diagnostics may not generalize beyond GPT-4o and the templates. Step-level analyses use GPT-4o, and its omissions cluster in a few paraphrase templates (C.4), so they may be rendering artifacts.
  3. The text is not human-validated. All text is template-rendered, and judge agreement on unique readability is κ = 0.169 (A.3).
  4. PTR's ceiling is partly by construction. It replays the same transition function that defines gold (Eq. 2), at about 198 calls per generated history (Sec. 5.2).
  5. The repair evidence is narrow. It covers one task and one model. On generated histories the precondition alarm has 0.75% precision and the omission alarm never fires (Table 22).
Questions for authors
  1. Do the step-level and fixed-parse diagnostics (Tables 17, 20) still show silent corruption and transition errors with GPT-5 or another strong model?
  2. How many omissions come from the three templates in C.4? Does accuracy change if those templates are reworded?
  3. Will you add a human fidelity audit and a harder platform stratum before release, and remove the 253 test examples that overlap diagnostic cohorts (Table 11)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.