- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
StateCross tests whether agents can choose the predefined program that meets current requirements when rules change across sessions. The main contributions are turning when a rule takes effect and what it applies to into executable checks, comparing agents with project files alone against agents that also use explicit memory, and analyzing failures stage by stage across storage, retrieval, and final input.
Critical strengths
- The paper focuses on a practical question: which requirements are still in effect.
- The project-file baseline helps measure what explicit memory adds, and the stage-wise checks offer a useful diagnostic framework.
Critical weaknesses
- The core evaluation conflates wording with state. An old procedure that is still valid is labeled “withdrawn” in the candidate descriptions, so the resulting errors cannot be reliably attributed to memory or state tracking. Early choices can also affect later states, so excluding this condition from the aggregate score does not fix trajectories that have already been affected. The authors acknowledge this in Section 7, but it directly weakens the main comparison.
- The conclusions have limited coverage and are hard to verify. The main comparison comes from only nine instances of a single template, with one run per system and task. The memory-stage diagnostics come from a separate five-task GPT calibration rather than the main runs, and rely on archived aggregates without the underlying records. This evidence does not support a stable system ranking across task types, and the reported points of information loss are hard to verify independently.
Questions for authors
- If the candidate descriptions are reworded neutrally and the experiments are rerun from the initial state, do the main performance gaps and system rankings still hold?
- Can you recollect complete diagnostic records to verify the reported conclusions about the storage, retrieval, and input stages, and separate the effect of the memory system itself from how it is integrated and filtered?