- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether performance plateaus in memory-augmented agents indicate stable capability improvements. The authors propose a longitudinal audit combining item-level gain/loss analysis, repeated evaluations of frozen memory states, and benchmark coverage statistics. Experiments on ALFWorld and ScienceWorld show that similar aggregate success curves can conceal different patterns, including gain-loss cancellation, execution variability, and benchmark saturation.
Critical strengths
S1. Relevant evaluation problem. The paper highlights an important limitation of aggregate success rates in evaluating continuously evolving agents. Distinguishing stable improvements from hidden regressions and execution variability is practically meaningful.
S2. Systematic empirical analysis. The authors evaluate multiple backbones and memory mechanisms, and the matched evaluation of frozen memory states provides useful observations about performance changes across successive updates.
Critical weaknesses
W1. Limited novelty and methodological contribution. The central observation that aggregate performance can mask instance-level changes is well established. The proposed audit primarily combines straightforward measurements, including prediction flips, repeated-run disagreement, and benchmark coverage. It remains unclear what fundamentally new methodological insight the paper provides beyond applying these existing analyses to memory-augmented agents.
W2. Insufficient experimental rigor. The analysis relies on only three seeds and one repeated evaluation per memory state, which provides limited evidence for distinguishing genuine changes from stochastic variability. In particular, the proposed excess-turnover measure is based on a simple subtraction of repeat disagreement, without sufficient statistical justification or uncertainty estimation. This weakens the reliability of the resulting diagnoses.
W3. Limited generalizability and analytical depth. The evaluation primarily uses 140 ALFWorld tasks and a small 30-task ScienceWorld panel. Moreover, differences in backbones and evaluation protocols prevent controlled cross-benchmark comparisons. More importantly, the analysis identifies which tasks change but provides limited insight into why memory updates cause improvements or regressions. Consequently, the findings remain largely descriptive rather than offering deeper understanding of memory evolution.
Questions for authors
Q1. What is the main methodological novelty of the proposed audit compared with existing instance-level performance analysis and stochastic evaluation methods?
Q2. How reliable is the excess-turnover measure given that each frozen memory state is evaluated with only one matched repeat? Would additional repetitions or statistical uncertainty estimates change the conclusions?
Q3. Can the authors provide deeper analysis linking specific memory updates or retrieval behaviors to the observed task-level gains and regressions, rather than reporting outcome changes alone?