- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces StateTrackBench, a benchmark of 6,055 executable trajectories for testing whether language models can keep track of evolving state. Each message is paired with a typed update, and replaying the updates gives the gold state after every message. The data comes from GitHub, GitLab and npm event records plus 15 generated task families, and all text is template-rendered. The authors evaluate GPT-5, Gemini-2.5-Pro, Doubao-Seed-1.8 and GPT-4o, and run diagnostics with GPT-4o. They also propose Parse–Then–Replay (PTR), where the model parses each message and a program replays the updates, and a repair method that re-parses messages flagged by replay checks, which improves GPT-4o from 0.918 to 0.990 on GitHub label histories.
Critical strengths
- In this benchmark, replay gives a gold state after every message, which makes step-level error analysis possible.
Critical weaknesses
- The main claim is not supported by the main evaluation. The paper says final-answer scores hide errors in intermediate states, but the leaderboard only scores final answers. Intermediate states are only checked for GPT-4o on a few diagnostic subsets, so it is unclear whether this problem still exists for stronger models.
- The benchmark is mostly saturated and the data is not realistic. Strong models are near 100% on the platform data (about 74% of the test set), and a simple answer prior already gets 0.875 on GitLab. The generated part is similar to bAbI, with template-rendered text and only 10 distractor sentences, and none of it is checked by humans. Also I don't find some most recent models on this benchmark, like GPT-5.6/GPT-6, Claude Opus 5/5.5 etc.
- PTR and the repair method only work in this benchmark. They rely on a predefined update schema and the benchmark’s own transition function, which are usually not available in real agent settings. The repair is only tested on one task with GPT-4o, and on generated data its alarms have only 0.75% precision and 4.72% recall.
Questions for authors
- Can PTR or similar repair method be used when the update schema and transition rules are not given?