acceptodds
Under review as a conference paper at ICLR 2027

Same Curve, Different Regimes: A Longitudinal Audit of Memory-Augmented Agents

Abstract

Memory-augmented agents adapt after deployment by accumulating interaction experience, and their aggregate success often rises before flattening. Does a plateau indicate a stable capability gain? Similar curves can conceal a stable solved-task set, gains offset by losses, changes at the scale of repeated execution, or a task bank with little remaining discriminative range. We propose a longitudinal audit of frozen memory states that jointly examines across-seed coverage, matched reruns of the same state, and the item-level composition of change. On a fixed ALFWorld bank, three backbone operating points and two memory mechanisms reveal gain–loss cancellation, repeat-scale change, and ceiling-limited evidence. On a smaller ScienceWorld panel, two further backbones show different early trajectories and later changes near the repeat scale. A plateau should therefore prompt this audit before it supports a claim of convergence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory-Augmented Agents, Longitudinal Evaluation, Agent Reliability, Agent Evaluation, Benchmark Saturation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RmVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether performance plateaus in memory-augmented agents indicate stable capability improvements. The authors propose a longitudinal audit combining item-level gain/loss analysis, repeated evaluations of frozen memory states, and benchmark coverage statistics. Experiments on ALFWorld and ScienceWorld show that similar aggregate success curves can conceal different patterns, including gain-loss cancellation, execution variability, and benchmark saturation.

Critical strengths

S1. Relevant evaluation problem. The paper highlights an important limitation of aggregate success rates in evaluating continuously evolving agents. Distinguishing stable improvements from hidden regressions and execution variability is practically meaningful.

S2. Systematic empirical analysis. The authors evaluate multiple backbones and memory mechanisms, and the matched evaluation of frozen memory states provides useful observations about performance changes across successive updates.

Critical weaknesses

W1. Limited novelty and methodological contribution. The central observation that aggregate performance can mask instance-level changes is well established. The proposed audit primarily combines straightforward measurements, including prediction flips, repeated-run disagreement, and benchmark coverage. It remains unclear what fundamentally new methodological insight the paper provides beyond applying these existing analyses to memory-augmented agents.

W2. Insufficient experimental rigor. The analysis relies on only three seeds and one repeated evaluation per memory state, which provides limited evidence for distinguishing genuine changes from stochastic variability. In particular, the proposed excess-turnover measure is based on a simple subtraction of repeat disagreement, without sufficient statistical justification or uncertainty estimation. This weakens the reliability of the resulting diagnoses.

W3. Limited generalizability and analytical depth. The evaluation primarily uses 140 ALFWorld tasks and a small 30-task ScienceWorld panel. Moreover, differences in backbones and evaluation protocols prevent controlled cross-benchmark comparisons. More importantly, the analysis identifies which tasks change but provides limited insight into why memory updates cause improvements or regressions. Consequently, the findings remain largely descriptive rather than offering deeper understanding of memory evolution.

Questions for authors

Q1. What is the main methodological novelty of the proposed audit compared with existing instance-level performance analysis and stochastic evaluation methods?

Q2. How reliable is the excess-turnover measure given that each frozen memory state is evaluated with only one matched repeat? Would additional repetitions or statistical uncertainty estimates change the conclusions?

Q3. Can the authors provide deeper analysis linking specific memory updates or retrieval behaviors to the observed task-level gains and regressions, rather than reporting outcome changes alone?

Reviewer_MFvtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to interpret the aggregate success curve of memory-augmented agents whose memory accumulates after deployment while the model weights stay fixed. It argues that a rising and then flattening curve can reflect different situations, such as a stable set of solved tasks, gains offset by losses on different tasks, change at the scale of simply rerunning the same state, or a task bank with little remaining range, and these call for different evaluation decisions. The paper proposes a longitudinal audit that evaluates frozen memory states on a fixed task bank and reads three diagnostics together without combining them into one score. The first measures how many items still separate across seeds, the second compares the turnover of outcomes between adjacent states with the disagreement obtained by rerunning the same state, and the third reports the gains and losses at the item level. The audit is applied to several backbones and two memory mechanisms on an embodied household benchmark, and to a smaller panel on a science benchmark, where it is reported to yield different diagnoses for similar aggregate curves. The paper emphasizes that the diagnostics are descriptive, that cross-panel comparisons are not rankings, and that the contribution is the reporting procedure rather than a numerical threshold.

Critical strengths

S1. The paper frames a clear and practically relevant evaluation question, namely what a flat success curve of a memory-augmented agent can and cannot support. It makes this concrete by reporting item-level gains, losses, and rerun disagreement for every adjacent transition, so readers can see cases where similar aggregate scores correspond to different underlying changes, and it links these readings to evaluation decisions.

S2. The paper is careful about the limits of its own claims and reports them openly. It states that excess turnover is descriptive rather than an effect estimate, that the shaded ranges are not confidence intervals, and that cross-panel contrasts do not support rankings, and it retains negative values and acknowledges what its exports cannot reconstruct. This makes the evidence easier to interpret and to build on.

Critical weaknesses

W1. The proposed audit is a combination of established ideas, and its validity is not demonstrated. Item-level change between model versions, disagreement between reruns, and floor and ceiling fractions are all known tools, and the paper's own related work describes the new element as adding an identical-state rerun. The coverage statistic is a count of items whose outcome is constant across three seeds, and the paper notes that a related entropy measure adds nothing independent. The excess turnover quantity is explicitly descriptive rather than an estimator, and the diagnoses are read by judgment instead of explicit rules, so it is unclear that two analysts would reach the same conclusion on the same data. The main paper does not test whether the audit recovers known situations, for example by injecting known gains, losses, or noise, and it does not compare against simpler alternatives such as paired tests or variance decomposition on the same data. It also does not show a case where the audit changes a conclusion that standard reporting would have supported, and the recommended follow-up actions are not carried out to check that they resolve the ambiguity. The contribution therefore reduces to a recommended reporting practice, and its benefit over existing practice remains an assertion.

W2. The empirical evidence is thin for the diagnoses it is used to support. Each condition uses three seeds and a single matched rerun per state, so the repeat baseline used to compute excess turnover is itself a noisy single-sample estimate, as the reported negative excess values suggest. No uncertainty is quantified, and the paper states that the ranges are not confidence intervals and that overlap between bands is not a significance test. The task banks are small, and on the second benchmark a single item moves a per-seed rate by several points while only a handful of items are informative, so reported excess values can be driven by very few items. The assumption that turnover between two different states carries noise comparable to the disagreement between reruns of one state is not examined, even though item difficulty distributions differ across states. The cases are also not matched. The two memory mechanisms differ in training exposure, harness, and embedder, one is evaluated under a protocol that departs from its original one, no backbone appears on both benchmarks, and the saturation case reflects the interaction of a strong model with an easy bank. Together these points make it difficult to judge how robust the three diagnoses are or whether they would hold beyond the observed panels.

Questions for authors

Q1. I wonder whether the audit's diagnoses can be validated in a setting where the ground truth is known. For example, injecting a known number of gains and losses, or adding controlled execution noise to frozen states, would show whether the audit separates cancellation, repeat-scale change, and saturation correctly. It would also help to see how the same data would be read using simpler alternatives, such as paired tests on item outcomes or a variance decomposition, and where the audit gives a different answer.

Q2. The repeat baseline relies on one matched rerun per state and three seeds. I am curious how stable the excess turnover values and the resulting diagnoses are when more reruns and seeds are used, and whether some form of uncertainty estimate, such as a bootstrap over items, can accompany the reported values. This would help me judge whether the negative and near-zero excess values reflect real repeat-scale change or estimation noise, especially on the small benchmark where a few items can drive the result.

Q3. The diagnoses are read by judgment rather than by explicit rules. I wonder whether the authors can give decision criteria, for example thresholds or conditions that map the three diagnostics to a recommended action. It would also be informative to see the recommended follow-up carried out in at least one case, for instance collecting more reruns for the controls with repeat-scale change, and whether this resolves the ambiguity as intended.

Q4. The repeat baseline for a transition is the average of the rerun disagreement at its two endpoint states. I wonder whether this is a reasonable estimate of the noise that two different states would show when run once each, given that the distribution of item difficulty differs across states. It would help to know how sensitive the excess turnover values are to this choice, including the alternative baselines mentioned in the supplementary material.

Reviewer_YQ2NOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper audits frozen memory snapshots using task-level gains/losses, matched reruns, and across-seed coverage. Experiments on ALFWorld and a smaller ScienceWorld panel illustrate how similar success-rate plateaus can conceal different patterns of change.

Critical strengths

Concrete evidence of hidden regressions. Qwen2.5-14B MemRL exhibits approximately 20–31% task turnover despite small changes in aggregate success (Table 2). This is the strongest empirical observation and supports reporting task-level regressions alongside average performance where a lot of memory agent works focuses now.

Critical weaknesses

Limited novelty and incremental insight.

  • MemRL already measures success-to-failure forgetting (§5.4.2; Appendix B.1) and discusses capability–difficulty matching and ceiling effects.
  • The paper does not sufficiently establish what substantive conclusions become possible specifically through its combination of reruns and coverage.
  • The 27B model’s approximately 97% success largely demonstrates an expected benchmark ceiling. Testing a suitably harder bank would provide stronger evidence that the audit identifies otherwise hidden learning dynamics.

Narrow validation and insufficiently established diagnostic interpretation.

  • Three seeds and one matched rerun per snapshot provide limited precision for interpreting excess turnover around 1–3%. Across-seed coverage also lacks clear validation as a measure of remaining benchmark headroom beyond success rate.
  • Evidence is concentrated in MemRL. AdaMEM uses only one backbone and offline index prefixes; ScienceWorld contributes only 30 easy-setting variations with different backbones. These extensions provide limited support for general applicability. And why not consider other methods?
Questions for authors
  • What conclusions change when coverage is added to success rates, gains/losses, and reruns? How does this extend MemRL’s existing forgetting analysis?

  • How stable are the small excess estimates under original/repeat swapping and the alternative baselines mentioned in the supplementary material?

  • I suggest you to draw your own figure of figure 1, it's pretty AI and low quality IMO

  • Why select MemRL and AdaMEM? What aspect of continual memory updating does the AdaMEM offline-prefix experiment validate?

  • Would evaluating the same system on a harder task bank reveal changes hidden by the reported ceiling? This would materially strengthen the ceiling interpretation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.