acceptodds
Withdrawn from ICLR 2027

StateCross: A Benchmark for Memory and State Tracking in Long-Horizon Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon tasks require agents to follow instructions that may change across sessions. An instruction may be withdrawn, take effect later, or apply only to one part of a task. Agents must therefore remember information and decide which requirements govern their next action. Existing memory evaluations often focus on recall or answer consistency. These measures do not by themselves show whether retained instructions lead to correct actions. Evaluations centered on memory can also overlook workspace files, which preserve instructions and earlier results after the conversation is cleared. We introduce StateCross, a benchmark for evaluating whether retained information supports actions that follow current requirements. Controlled coding tasks combine changing requirements with context resets while workspace files and explicit memory persist. Agents select from predefined programs, whose executed effects are checked against the requirements in force. We compare agents with and without explicit memory using decision accuracy, requirement-specific errors, and consistency across consecutive actions. The framework also distinguishes evidence gaps at storage, retrieval, and model input, supporting separate analysis of information availability and action correctness.

Primary area:
datasets and benchmarks
Keywords:
Agent Memory, Long-Horizon Agents, State Tracking, Memory Evaluation, Multi-Session Tasks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AmecOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

StateCross tests whether agents can choose the predefined program that meets current requirements when rules change across sessions. The main contributions are turning when a rule takes effect and what it applies to into executable checks, comparing agents with project files alone against agents that also use explicit memory, and analyzing failures stage by stage across storage, retrieval, and final input.

Critical strengths
  1. The paper focuses on a practical question: which requirements are still in effect.
  2. The project-file baseline helps measure what explicit memory adds, and the stage-wise checks offer a useful diagnostic framework.
Critical weaknesses
  1. The core evaluation conflates wording with state. An old procedure that is still valid is labeled “withdrawn” in the candidate descriptions, so the resulting errors cannot be reliably attributed to memory or state tracking. Early choices can also affect later states, so excluding this condition from the aggregate score does not fix trajectories that have already been affected. The authors acknowledge this in Section 7, but it directly weakens the main comparison.
  2. The conclusions have limited coverage and are hard to verify. The main comparison comes from only nine instances of a single template, with one run per system and task. The memory-stage diagnostics come from a separate five-task GPT calibration rather than the main runs, and rely on archived aggregates without the underlying records. This evidence does not support a stable system ranking across task types, and the reported points of information loss are hard to verify independently.
Questions for authors
  1. If the candidate descriptions are reworded neutrally and the experiments are rerun from the initial state, do the main performance gaps and system rankings still hold?
  2. Can you recollect complete diagnostic records to verify the reported conclusions about the storage, retrieval, and input stages, and separate the effect of the memory system itself from how it is integrated and filtered?
Reviewer_Y2wAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StateCross evaluates whether agents follow requirements that are currently valid and applicable to an operation across multiple sessions. Conversation history is reset between sessions, while workspace files and explicit memory persist. Agents select among four predefined programs, and executable checks evaluate their decisions. The benchmark additionally traces whether required information was stored, retrieved, and exposed to the model.

Critical strengths
  1. Appropriate statistical units and candid limitations.
  2. The diagnostic framework is promising.
Critical weaknesses
  1. The candidate wording directly confounds the intended measurement. Section 7 explicitly states that the old-valid candidate is described as withdrawn even when it remains valid. Across the seven systems, only 17 of 4,144 decisions in this condition are correct, approximately 0.41%. These results cannot cleanly distinguish failures of temporal state tracking from responses to a misleading candidate description. This is an acknowledged contradiction in the evaluation interface, rather than a hypothetical concern about wording. More importantly, earlier choices modify subsequent workspace states and requests. Excluding old-valid decisions from the aggregate score therefore does not recover an unaffected trajectory.

  2. Structural coverage is limited. The main evaluation comprises nine instances of one structural template. Five narrative settings do not necessarily provide five distinct decision structures, and the large number of sequential decisions does not increase the number of independent task instances. After resolving the wording issue, I would encourage evaluation on at least one materially different structure, such as multiple concurrent entities, overlapping local exceptions, or less predictable update schedules.

  3. A substantial discrepancy between experimental batches remains unexplained. For the same six tasks used in the file-visibility experiment, the workspace-only results in Table 10 average approximately 37.70%, whereas the fresh files-visible results in Table 17 average 50.91%, a difference of approximately 13.22 percentage points. This exceeds the largest reported main-panel memory improvement of 8.98 percentage points.

Questions for authors
  1. Can you provide a complete rerun using neutral and accurate candidate descriptions? Do the condition-level results and system rankings persist?
  2. What explains the approximately 13.22-point workspace-only difference across the two experimental batches?
  3. Can you release or reconstruct a fully auditable diagnostic subset, including the provenance-matching procedure?
Reviewer_bNHPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

StateCross evaluates whether agents retain and correctly apply changing requirements across context resets. Tasks span 16 sessions and 256 decisions, with agents choosing among four predefined programs whose execution is checked against current requirements. The benchmark compares workspace-only agents with six memory systems, measuring action accuracy, consistency, and information availability across storage, retrieval, and model input. On nine task instances, MemOS and GraphRAG improve accuracy over the 38.4% workspace-only baseline by 9.0 and 7.3 percentage points. Separate fixed-decision experiments show larger gains when agents receive full history or explicit current requirements.

Critical strengths

The benchmark is motivated well. Evaluating whether retained information produces correct actions under timed and scoped rule changes, rather than whether it can be recalled, is an important and under-tested direction. The "pause" condition, where the old plan is withdrawn and the new one is not yet effective, is a thoughtful addition.

Critical weaknesses
  1. A known wording defect undermines benchmark validity. The old-valid candidate is described as withdrawn when it remains valid (Sec. 7). This affects 25.7% of decisions and yields only 0.41% accuracy across systems. Because errors can alter later workspaces, excluding this condition through B3B_3 does not repair the trajectories. Neutral wording and complete reruns are needed.

  2. Repeatability is untested, with a substantial unexplained baseline gap. Each system–task pair has one run, so confidence intervals capture task variation rather than repeated-run stability. On the same six tasks, workspace-only accuracy rises from 37.70% in Table 10 to 50.91% in Table 17—exceeding the largest reported memory gain. This could reflect randomness or configuration differences and requires clarification and repeated runs under matched settings.

  3. Results depend strongly on memory integration choices. The adapters modify native outputs, and a shared reranker exposes at most five records. In the separate GPT calibration, common selection changes accuracy by +13.33 points for MemOS, −8.89 for Mem0, and −6.67 for A-MEM (Table 24). Native-versus-common comparisons and token/compute costs are needed to distinguish system capabilities from the effects of these particular integrations.

Questions for authors
  1. Can you rerun the benchmark with neutral old-valid wording and report the complete trajectories? How do the system rankings change?
  2. How do the fresh workspace-only runs in App. D.5 differ from the main-panel workspace-only runs (prompt, API route, model version), and what explains the 12–13-point gap on the same six tasks? Please report repeated runs per cell and the sampling temperature actually used.
  3. Several diagnostics (Tables 6, 22 and 23) are reconstructed from archived summaries. Can they be recomputed on the main DeepSeek trajectories, for which App. B says candidate and exposed record IDs are saved for all 13,824 memory decisions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.