acceptodds
Under review as a conference paper at ICLR 2027

On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification

Abstract

Memory-based self-improving agents—those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank—have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple self-improving runs to quantify variance, and (2) randomly shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Empirically, when these methods are applied, we observe the variance across runs increase in 71% of cases, and the gap between the best and worst runs of the same experiment can reach up to 10 percentage points. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. In contrast, when evaluated under a shuffled task order, agent performance degrades (-4.5%) instead of exhibiting the expected improvement (+1.5%). To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. Without clear specifications, agents generate plausible yet inapplicable memories (e.g., recommending API usage in a browser-only environment) that may distract the agent from feasible strategies. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under realistic, challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
self-improving agents, robustness, reliability, evaluation, continual learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P5p1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies instability issue in memory-bsaed self-evolving agent. It probes two aspects: 1) cross-run variance; 2) sample ordering - both can lead to significantly degradation of agent performance, raising questions about previously reported results from Agent Workflow Memory and Reasoning Bank. It identified two specific issues that cause the instablity: task and environment underspecification. By specifying the tasks and getting environmental feedback, the baseline results can be further improved.

However, I think the overall contribution is slightly below the bar of a ICLR paper. To improve the paper, the authors can consider 1) propose a more comprehensive solutions for the observed issues; 2) expand the study to more benchmarks or agent harness.

Critical strengths
  • The paper is clearly motivated, rooted in the stability of self-evolving agents. This is serious issue in my work that the reported numbers might be exaggerated.
  • The two identified issue: cross-run variance and sample order are both intuitive and well investigated with clear evidence.
  • The additional analysis that helped proposed effective solutions (task and environment specifications) are good findings and effective backed by the experimental results.
Critical weaknesses
  • The first weakness is the novelty - since self-evolving a process of optimization, we can expect the two identified issues (not saying these are trivial), but solutions based on these findings are relatively weak based on some manual observations.
  • I also have concerns that these solutions themselve (despite useful in the tested data and agents), may be overfitting to these small range of benchmarks and agents. Hence, a more comprehensive needed to improve the soundness of the work.
  • Memory-based agent now is a relatively narrow scope of harness based agent. The authors should consider extending the work to the entire harness evolution.
Questions for authors
  • Have you tried the proposed solution on other benchmarks?
  • What is the reason to focus only on Agent Workflow Memory and Reasoning Bank?
3 comments

Thank you for the detailed and constructive review! We are glad that you found the paper written clearly, and the two identified issues well supported by evidence. We respond to your concerns and questions below.

To improve the paper, the authors can consider: … expand the study to more benchmarks or agent harness.

  • Thank you for the suggestion. Beyond the three benchmarks and two methods in the main text, appendix E (page 20-21) contains experiments that we deferred due to space.
  • For example, on EnterpriseOps-Gym (new benchmark), using a filesystem-based memory method built on Letta Code (new memory method), simply enabling memory does not improve the overall success rate (24.5 -> 22.3 with GPT-OSS-120B; 30.5 -> 29.7 with GPT-5.4-mini) and increases variance in 10 of 16 cases.
  • These results support the same conclusions as the main paper, and we will reference them more clearly in the main text.

Weakness 1: The first weakness is the novelty - since self-evolving a process of optimization, we can expect the two identified issues (not saying these are trivial), but solutions based on these findings are relatively weak based on some manual observations.

  • We agree that variance and order sensitivity can be expected in kind. What was not known is their magnitude and the implications for evaluation: under the single-run, default-order protocol of prior work, these issues are hidden behind a single performance number. We quantify them: ReasoningBank's +1.5% gain is not statistically significant (p=0.23), and under shuffled task orders it becomes a 4.5% degradation.
  • The interventions in section 4.2 serve two purposes, and we see the first as their main value. (1) As a diagnostic, they provide direct evidence for the underspecification hypothesis, which until then rested on manual observations: each type of added information helps, and together they recover 31% of the degradation. (2) As a solution, they are simple and non-disruptive, so we recommend using them whenever such information is available.

Question 1: Have you tried the proposed solution on other benchmarks?

  • Yes, for the rubric component (+Rub). In a follow-up project, we tested it on EnterpriseOps-Gym (a 76-task subset) with the filesystem-based memory built on Letta Code. The agent attempts each task 8 times, updating its memory after each attempt, under different levels of feedback. The table reports pass rate (%) at trial k. p0 is the trial 0, before any learning; p7 is the trial 7.
Feedbackp0p1p3p5p7Lift (p≥1 minus p0)
rubrics + targets + score (+Rub)1680929697+75
rubrics + score1863697376+52
score only1737434345+26
LLM judge (tuned)1728293131+13
LLM judge (naive)1727262928+10
  • We see a clear ladder from the result table: richer feedback yields better learning. Our +Rub setting corresponds to the first row, which raises the lift from +26 (score only) to +75. When the agent has to rely on an LLM-judge (bottom two rows) instead of ground-truth feedback, it learns much less (+10 to +13). Note that this setup differs from the main paper in several ways, and we will add the full details and results to the revised paper.

Weakness 2: I also have concerns that these solutions themselve (despite useful in the tested data and agents), may be overfitting to these small range of benchmarks and agents. Hence, a more comprehensive needed to improve the soundness of the work.

  • +Rub and +Env are benchmark- and method-agnostic by design: they pass information that the evaluator or environment already produces to the memory construction step, without benchmark-specific tuning. Our response to Q1 above also shows that +Rub helps on another benchmark and another memory method. We therefore do not believe these two interventions are overfit to the settings in this paper.
  • +PMod is different: its content is specific to our experiments (e.g., no API access, no user confirmation). However, we believe the procedure behind it is transferable: when unintended strategies are observed in memory, the corresponding environment constraints can be stated explicitly in the prompt to steer self-improvement back on track.

Weakness 3: Memory-based agent now is a relatively narrow scope of harness based agent. The authors should consider extending the work to the entire harness evolution.

  • Thank you for the suggestion. We agree this is an important direction, and Appendix E already takes a small step toward it. The letta code based method is not limited to a memory bank: its reflection agent can edit the system prompt and create or edit skill files. The same reliability issue appears: enabling it does not improve overall success and increases variance in the majority of cases.
  • More broadly, we expect the causes we identify to extend beyond memory. Any agent that persistently modifies itself based on task outcomes is stateful, so early randomness can compound, and results can depend on task order and on how well the feedback is specified. Harness evolution is a recent and fast-moving direction that gives the agent a larger action space, so these issues may matter at least as much there. Our recommendations still apply: stress-test the system and build mechanisms for human intervention. We will make this connection explicit in the revised paper.

Question 2: What is the reason to focus only on Agent Workflow Memory and Reasoning Bank?

  • We chose these two because both are established methods with public implementations, and both were validated on web browsing tasks, the focus of this paper. This lets us use their reported results as a reference point (Table 2).
  • Our conclusions do not rest on these two alone. In Appendix E we evaluated a third, more recent filesystem-based method, where enabling memory still fails to give consistent gains.

Thank you again and please let us know if you have follow-up questions!

AuthorsOpenReview ↗

Follow up on our response

Dear Reviewer P5p1,

Thank you again for your review and suggestions! We hope our earlier response has helped address your concerns.

Since then, we have posted two additional sets of results in our responses to other reviewers, which improved the strength of our work, and may also be relevant to your concerns: (1) experiments with controlled easy-to-hard and hard-to-easy task orders, which show that difficulty direction affects the learning outcome; and (2) a leakage-controlled analysis showing that the gain from rubrics is essentially unchanged after excluding tasks with shared answers, and thus not explained by answer leakage.

Please let us know if you have any remaining concerns. Thank you!

Reviewer_hpkoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper re-evaluates two memory-based self-improving agents that learn from a stream of tasks by maintaining a textual memory bank. The evaluation is conducted on web-browsing and enterprise benchmarks with a stronger base agent than in the original work, and it is broadened along two axes, namely repeated identical self-improving runs to measure variance and shuffled task orders to test dependence on order. The paper reports that run-to-run variance is substantial even without memory and increases in most cases when memory is added, that the gains of the methods over the stronger baseline are small and not clearly distinguishable from noise, and that the default task order follows an implicit easy-to-hard progression, with performance degrading under shuffled orders. A manual inspection of the memories suggests that environment and task underspecification contribute, through plausible but inapplicable memories, spurious memories from ambiguous tasks or evaluator issues, and unintended strategies that are reinforced over time. Adding evaluation rubrics, environment feedback, and revised prompts during memory construction partially closes the degradation. Supplementary experiments with other models, another benchmark, and another memory method are provided, and the paper concludes with recommendations for evaluation and for memory validation and human oversight.

Critical strengths

S1. The paper addresses a timely and practically relevant question, namely whether the reported benefits of memory-based self-improving agents are reliable. It provides suggestive evidence that single-run, fixed-order evaluation can give a misleading picture, and it translates this into concrete evaluation recommendations, such as reporting several runs and testing under shuffled task orders. The per-domain tables and the supplementary replications with other models, another benchmark, and another memory method broadly point in the same direction, which makes the evaluation message more actionable for the community.

S2. The failure analysis is concrete and inspectable. The paper shows memories that sound plausible but cannot be executed in the environment, spurious memories created by evaluator problems, and a strategy that appears to spread through repeated reinforcement, and it releases the prompts and example trajectories. It is also candid that the inspection is qualitative and the mitigation is only partial. These observations give useful hypotheses for work on memory validation and human oversight, which makes the study more informative than a pure benchmark re-run.

Critical weaknesses

W1. The contribution is not sharply defined, the novelty is limited, and the scope is narrower than the claims. The paper combines a re-evaluation, a diagnosis of failure causes, a partial mitigation, and a set of recommendations without stating which is the primary claim. Evaluating over multiple runs and under altered task orders are established evaluation practices, and the paper itself acknowledges recent benchmarks that adopt them, so the new element is mainly their application to memory-based agents together with the underspecification diagnosis, which is supported mostly by qualitative evidence. The analysis centers on two methods that both accumulate textual memory items, with only one additional method examined in the appendix. Whole categories of self-improving agents are not covered, including methods that update parameters, learn skills or executable tools, curate or validate memory, or combine memory with test-time scaling, and the paper itself recommends memory validation without testing it. Despite this, the title and conclusions speak about self-improving agents in general. For a re-evaluation paper, the breadth of the methods studied is a large part of its value, so the narrow method coverage limits how far the findings can be trusted to hold beyond these two methods.

W2. Several central claims rest on limited statistical evidence and on an analysis with technical weaknesses. Each setting uses three runs, which gives very imprecise estimates of run-to-run variability, and the claim that self-improvement amplifies variance is supported by counting domain-level cells without a test, where the cells share runs and a common baseline. The headline gap of up to ten points is a maximum over many cells and comes from a domain with 48 tasks, where it corresponds to about five tasks, while variability of the overall score is not reported in the main tables. For task order, only two shuffled orders are used, each with three runs, so order effects and run noise are not separated, the notion of significance in the figure is not defined, and the abstract's contrast between a gain and a degradation reflects one method on one benchmark while that method stays above the baseline under shuffled orders on the other benchmark. The implicit curriculum explanation is not tested with controlled easy-to-hard or hard-to-easy orders. The underspecification account relies on manual inspection of a subset of memories, and the mitigation supplies evaluation rubrics that can contain answer information, which confounds better specification with label leakage. The fraction of the degradation that is closed depends on a reference that is not defined. Together these points make it difficult to judge how robust the reported fragility and its explanation are.

Questions for authors

Q1. The claim that self-improvement amplifies run-to-run variance is based on three runs per setting. I wonder how this was tested, and what the mean, standard deviation, and confidence interval look like for the overall pass rate rather than for individual domains. Since some domains contain few tasks, it would also help to know how domain size was taken into account when interpreting the gaps, and whether the conclusions hold with more runs in the other settings beyond the single supplementary experiment.

Q2. The task order analysis uses only two shuffled orders. I am curious whether the order effect can be separated from run-to-run noise, for example by using more shuffled orders or by explicitly ordering tasks from easy to hard and from hard to easy according to the baseline pass rate. It would also help to clarify how significance is defined in the corresponding figure, and why one method is hurt by shuffling on one benchmark but not on the other.

Q3. The underspecification account is supported mainly by manual inspection and examples. It would be helpful to see whether removing or correcting the problematic memories restores performance, and how the authors ruled out that the evaluation rubrics given to the memory constructor carry answer information that could benefit later tasks. I am also curious which reference was used to compute the fraction of degradation that is closed, for instance the no-memory baseline or the default task order.

Q4. The re-evaluation uses its own implementations, a different agent harness, and ground-truth rewards during memory construction, whereas the original methods rely on self-judged outcomes. I wonder whether the implementations reproduce the originally reported gains under the original settings, and how to interpret the smaller gains observed even when the same base model as in the original report is used. It would help to separate the contribution of the stronger base model from that of the implementation, harness, and reward differences.

7 comments

Thank you for the exceptionally thorough and careful review. We are glad that you found the question timely and practically relevant. Your comments on the statistical analysis (e.g., reporting the variability of the overall pass rate) were very helpful, and we will incorporate them in the revised version! Please see our responses below.

W1-Part 1. The contribution is not sharply defined, … The paper combines a re-evaluation, a diagnosis of failure causes, a partial mitigation, and a set of recommendations without stating which is the primary claim.

  • Thank you for raising this. As an analysis paper, our work has several components, but they are not parallel. The primary claim is the re-evaluation result: under multiple runs and shuffled task orders, the gains of memory-based self-improving methods over a strong baseline are not reliable. They are within run-to-run noise, often come with higher variance, and can turn into degradation when the task order changes.
  • The other components support this claim. The underspecification diagnosis offers a partial explanation. The interventions test that explanation and serve as a practical remedy when the information is available. The recommendations follow from these findings. We will revise our work to state this structure explicitly.

W1-Part 2. …the novelty is limited, …Evaluating over multiple runs and under altered task orders are established evaluation practices, and the paper itself acknowledges recent benchmarks that adopt them, so the new element is mainly their application to memory-based agents together with the underspecification diagnosis, which is supported mostly by qualitative evidence.

  • We agree that multiple runs and altered task orders are established practices, and we do not claim them as contributions of this work. Our contribution is what they reveal when applied to published methods, whose original evaluations report a single run under the default order: the gains turn out to be less reliable, and this had not been quantified before.
  • The recent benchmarks we cite adopt these practices for new evaluations. Our work is complementary: it re-examines existing methods and their evaluations, and investigates why they fail.
  • On the diagnosis being mostly qualitative: manual inspection is the starting point, but we also track the identified patterns quantitatively across runs (Figure 4) and test the hypothesis with interventions (Figure 5). Together, these go beyond qualitative observation.

W1-Part 3. … the scope is narrower than the claims. … The analysis centers on two methods that both accumulate textual memory items, with only one additional method examined in the appendix. Whole categories of self-improving agents are not covered, including methods that update parameters, learn skills or executable tools, curate or validate memory, or combine memory with test-time scaling, and the paper itself recommends memory validation without testing it. Despite this, the title and conclusions speak about self-improving agents in general.

  • Thank you for the suggestion. We agree and will narrow the stated scope. The abstract already specifies memory-based self-improving agents; we will make the title and conclusion consistent with it (e.g., “On the Fragility of Memory-Based Self-Improving Agents”).
  • Regarding coverage, we would like to mention that the additional method in Appendix E goes beyond a memory bank, as its reflection agent can edit the system prompt and create skill files. The other categories you list (parameter updates, executable tools, memory curation, test-time scaling) are important directions. We chose to study one family in depth, across multiple runs, task orders, benchmarks and models, so they fall outside the scope of this work, and we do not claim our results for them.
  • That said, we expect similar issues to arise there: any method that persistently updates itself from task outcomes is stateful, so early randomness can compound and results can depend on task order. We will include this in the discussion of the revised paper.

W1-Part 4. For a re-evaluation paper, the breadth of the methods studied is a large part of its value, so the narrow method coverage limits how far the findings can be trusted to hold beyond these two methods.

  • We agree that method breadth matters for a re-evaluation. Breadth and depth trade off, and we prioritized depth because our findings depend on it: every method is evaluated over repeated runs, and the study covers four benchmarks, three task orders and four backbone models in total. This depth is what makes the variance and order effects visible; a broader comparison with a single run per method would have missed them.
  • Within the memory-based family, we chose representative methods that span different designs: workflow memory (AWM), retrieved reasoning items (ReasoningBank), and a filesystem-based memory (Appendix E).
AuthorsOpenReview ↗

Response to reviewer hpko (Part 2)

W2-Part 1: Each setting uses three runs, which gives very imprecise estimates of run-to-run variability, and the claim that self-improvement amplifies variance is supported by counting domain-level cells without a test, where the cells share runs and a common baseline.

  • We would like to first clarify our experiment setting, and then provide additional evidence.
  • In our experiments, each domain is a separate task stream with its own memory, so cells from different domains come from different, independent runs and do not share runs. We should have stated this explicitly and will do so in the revised paper. We agree that three runs may give imprecise estimates of variability. For ReasoningBank on WebArena, we therefore conducted three additional runs (Appendix E, Table 11), making six in total. With this more precise estimate, ReasoningBank still shows a larger standard deviation than the no-memory baseline in 5 of 6 domains.
  • We agree that counting cells is coarse, and that the two methods share a common baseline within each domain. We therefore apply a two-sided sign test to each method separately over the 12 domains in Table 1. For ReasoningBank, variance increases in 10 of 12 domains (p=0.039). For AWM, the direction is the same but the effect is weaker (7 increases, 3 decreases, 2 ties; p=0.34). This is consistent with our claim that self-improvement can amplify variance, and we will report the two methods separately in the revised paper.

W2-Part 2: The headline gap of up to ten points is a maximum over many cells and comes from a domain with 48 tasks, where it corresponds to about five tasks, while variability of the overall score is not reported in the main tables.

  • We agree that the “up to 10 points” number is a maximum over cells and comes from a small domain (multisite, 48 tasks). We will replace it with the GitLab example already used in the introduction (180 tasks; the best-worst gap grows from 4.4 to 7.8 points with ReasoningBank). We will also report gaps in the number of tasks, in addition to percentage points, in a new row in Table 1.
  • On overall variability: since each domain is an independent stream, the variance of the overall pass rate is the size-weighted sum of per-domain variances (the standard formula for a stratified mean). This also accounts for domain size. Computed from the per-domain values in Tables 7, 8 and 10, the overall sample standard deviations (baseline / AWM / ReasoningBank) are 0.84 / 1.08 / 1.39 on WebArena, 0.68 / 1.08 / 1.14 on VisualWebArena, and 0.84 / 1.66 / 1.45 on SCUBA. Both methods increase overall variability on all three benchmarks. The full table with means and confidence intervals is in our response to Q1 below, and we will add it to the paper.

W2-Part 3: For task order, only two shuffled orders are used, each with three runs, so order effects and run noise are not separated, the notion of significance in the figure is not defined, and the abstract's contrast between a gain and a degradation reflects one method on one benchmark while that method stays above the baseline under shuffled orders on the other benchmark.

  • On the number of orders: task orders are shuffled independently within each domain, and each domain is a separate stream. Therefore, on WebArena, each method is evaluated on 12 distinct shuffled streams (6 domains x 2 shuffles), each with three runs.
  • On separating order effects from run noise: on WebArena, the overall pass rate drops by 4.3 to 6.5 points from the default order to the shuffled orders for both methods, while its run-to-run standard deviation within a given order is about 1 point (0.8 to 1.4). Welch's t-tests on the overall pass rate (using the overall variance described in W2-Part 2) give p<0.01 in all four WebArena cases.
  • On significance in Figure 3: the word was used informally, and we will replace it with the tests above. Of the 8 cases, 4 show a significant drop (both methods on WebArena), 2 show a non-significant drop (AWM/VisualWebArena, 1.6 to 2.1 points), and 2 show no drop (ReasoningBank/VisualWebArena). Even so, a drop of 4 to 6 points below the no-memory baseline on one benchmark is a concern for deployment.
  • On the abstract: we agree that the contrast between +1.5% and -4.5% reflects ReasoningBank on WebArena only. We will state this more clearly in the abstract.
AuthorsOpenReview ↗

Response to reviewer hpko (Part 3)

W2-Part 4: The implicit curriculum explanation is not tested with controlled easy-to-hard or hard-to-easy orders.

  • We agree that this explanation is currently observational: Figure 2 shows that the default order starts with easier tasks, but we have not tested it with controlled task orders.
  • We will run the controlled experiment you suggest on WebArena with ReasoningBank. Tasks will be ordered from easy to hard and from hard to easy, with difficulty defined by the no-memory baseline pass rate at the task-template level. We will share the results during the discussion period if they complete in time, and include them in the revised paper in either case.

W2-Part 5: The underspecification account relies on manual inspection of a subset of memories, and the mitigation supplies evaluation rubrics that can contain answer information, which confounds better specification with label leakage.

  • To clarify the setup: rubrics are provided only during memory construction, after the task is finished, and each task appears once per stream. Since different tasks ask different questions, we expect answer leakage to be limited. Still, this is a fair concern, so we checked how often it can happen in WebArena.
  • Of the 812 tasks, 76 (9.4%) have an earlier task in the same domain whose reference answer is identical and is shown to the memory constructor. For 52 of them the answer is uninformative (e.g., “N/A”, “0”, “yes”), leaving 24 tasks (3.0%) with an informative shared answer.
  • This is an upper bound. The memory prompt asks for generalizable insights and not specific strings (Appendix F), and the rubric message additionally instructs the model not to mention rubrics or scores in the memory items, but to turn them into actionable hints for similar tasks.

W2-Part 6: The fraction of the degradation that is closed depends on a reference that is not defined.

  • The reference is ReasoningBank under the default task order (56.3 on WebArena): the degradation is its drop under a shuffled order, and the closed fraction is the part recovered by +All (Figure 5b). This gives (52.7-49.8)/(56.3-49.8)=45% on Shuffle-1 and (51.8-50.7)/(56.3-50.7)=20% on Shuffle-2, about 32% on average.
  • We will define the reference explicitly in the revised paper. The paper states 31%; the small difference is likely due to rounding, and we will verify this against the unrounded results.

Q1. The claim that self-improvement amplifies run-to-run variance is based on three runs per setting. I wonder how this was tested, and what the mean, standard deviation, and confidence interval look like for the overall pass rate rather than for individual domains. Since some domains contain few tasks, it would also help to know how domain size was taken into account when interpreting the gaps, and whether the conclusions hold with more runs in the other settings beyond the single supplementary experiment.

  • On how the claim was tested: please see our response to W2-Part 1 (per-method sign tests over domains). The mean, standard deviation and confidence interval of the overall pass rate are reported below.

Overall pass rate (%) across 3 runs

BenchmarkMethodMeanStd (population)Std (sample)95% CI of the mean
WebArenaBaseline54.80.690.84[53.6, 55.9]
AWM54.10.881.08[52.5, 55.7]
ReasoningBank56.31.141.39[54.5, 58.2]
VisualWebArenaBaseline54.90.550.68[53.9, 56.0]
AWM54.50.881.08[53.0, 56.0]
ReasoningBank55.60.931.14[53.7, 57.6]
SCUBABaseline29.70.680.84[28.5, 31.0]
AWM28.01.361.66[24.8, 31.2]
ReasoningBank31.11.181.45[28.8, 33.4]

WebArena, 6 runs (Table 11)

MethodMeanStd (population)Std (sample)95% CI of the mean
Baseline54.20.800.88[53.4, 54.9]
ReasoningBank55.61.101.20[54.6, 56.7]
  • Note that, confidence intervals use the sample variance and Welch-Satterthwaite degrees of freedom. Table 1 in the paper reports population standard deviations (dividing by n); we will switch to sample standard deviations (dividing by n-1) in the revised paper. This scales all values by 1.22 and does not change any comparison.
  • On domain size and the gaps: in addition to the size-weighted overall numbers above, we will add a row to Table 1 that reports the best-worst gap in number of tasks.
  • On more runs: we have 6 runs only for ReasoningBank on WebArena under the default order. In that setting, the overall standard deviation remains higher than the baseline (1.20 vs. 0.88), and the gain remains not significant (p=0.07). We have not extended the other settings beyond three runs due to budget constraints.
Reviewer_EbDCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper re-evaluates memory-based agents across repeated runs and task orderings. Memory can amplify variability and reduce performance under shuffled orders. Qualitative analysis identifies misleading memories, while additional specifications and feedback partially mitigate degradation.

Critical strengths
  • It addresses an important evaluation weakness: conclusions drawn from a single run and fixed task order.
  • Strong baselines, multiple benchmarks, and supplementary models support the practical relevance.
  • Concrete failure examples and intervention ablations make the analysis informative.
Critical weaknesses
  • Limited statistical precision: Most settings use three runs and two shuffled orders. The six-run supplement helps, but variance estimates remain uncertain.
  • I think curriculum attribution is incomplete: Shuffling changes both difficulty order and task similarity; the experiments do not isolate an easy-to-hard curriculum effect.
  • Mechanistic evidence remains preliminary: Selected examples and keyword counts do not establish how much degradation comes from harmful memories. Gold rubrics also provide additional supervision, complicating attribution to reduced underspecification.
  • Some claims need qualification: ReasoningBank improves under shuffled orders on VisualWebArena, so the headline degradation is not universal.
Questions for authors

Can controlled memory removal and difficulty-controlled ordering experiments establish the proposed causes?

3 comments

Thank you for your feedback and suggestions! We are glad that you found our work to address an important evaluation weakness, and that you appreciated the strong baselines and the range of benchmarks and models! Please see our responses below.

Weakness 1: Limited statistical precision: Most settings use three runs and two shuffled orders. The six-run supplement helps, but variance estimates remain uncertain.

  • This is a reasonable concern, and we agree that three runs could still give imprecise estimates. We see three runs and two shuffled orders as a step forward from the single-run, fixed-order evaluations common in prior work, but not as the final word.
  • To make better use of the existing runs, we now report variability at the benchmark level by pooling across domains, which is more stable than single-domain estimates. Both memory methods show a higher standard deviation than the no-memory baseline on all three benchmarks (e.g., 1.39 for ReasoningBank vs. 0.84 for the baseline on WebArena). We see this as an improved (though still imperfect) measure, and will add these results to the revised paper.

Weakness 2: I think curriculum attribution is incomplete: Shuffling changes both difficulty order and task similarity; the experiments do not isolate an easy-to-hard curriculum effect.

  • Thank you for the suggestion! We are running a controlled experiment that orders tasks from easy to hard and from hard to easy (based on no-memory baseline performance), while keeping similar (same template) tasks adjacent in both orders. This holds task similarity fixed and varies only the direction of difficulty. We will share the results if they finish in time.

Weakness 3.1: Mechanistic evidence remains preliminary: Selected examples and keyword counts do not establish how much degradation comes from harmful memories.

  • We would like to note that the examples and keyword counts are findings in their own right: they show that misleading memories exist and are increasingly retrieved, which is a reliability concern regardless of their exact share of the degradation.
  • These observations directly motivated our interventions, which recovered about a third of the drop under shuffled orders. This indicates that such memories indeed contribute to the degradation.
  • However, we agree that the current experiments do not cleanly attribute how much. The degradation likely arises from several interacting factors, which we are unable to investigate fully in this work. We will state this limitation more clearly in the revised paper.

Weakness 3.2: Gold rubrics also provide additional supervision, complicating attribution to reduced underspecification.

  • Thank you for raising this. Our writing may have caused some confusion here, and we would like to clarify by separating two concepts: underspecification at task execution time, and underspecification at memory construction time. This experiment targets the latter.
  • You are right that gold rubrics are additional supervision. To clarify, in our main experiments, ReasoningBank already receives the ground-truth pass/fail outcome (Section 3.1). What the rubrics add beyond that is a description of what the task required. We therefore view this as an oracle setting: it measures how much can be recovered when the memory construction step is well specified, similar to a human giving detailed feedback after a mistake.
  • Addressing underspecification at execution time would require rewriting and revalidating the task queries, which we leave to future work.

Weakness 4: Some claims need qualification: ReasoningBank improves under shuffled orders on VisualWebArena, so the headline degradation is not universal.

  • Thank you for pointing this out. We agree that our current wording is not precise enough. Performance drops under shuffled orders in 3 of 4 method-benchmark combinations, and ReasoningBank on VisualWebArena is the exception. Our intended claim is that a reliable method should not fall below the no-memory baseline under any task order, and the concern is that this happens in most of our settings. We will revise the claim in the abstract and main paper to state it more clearly.

Question 1: Can controlled memory removal and difficulty-controlled ordering experiments establish the proposed causes?

  • Thank you for these suggestions. We think both are good ways to test the proposed causes.
  • For difficulty-controlled ordering, we are running this experiment (see our response to Weakness 2) and will share the results if they finish in time.
  • For controlled memory removal, unfortunately this is non-trivial: it requires identifying which memories are problematic among the large volume the agent produces, and we do not yet have a reliable way to do so at scale. The closest evidence we have is +PMod, which discourages some of these memories from being written and gives a modest gain (see Figure 5a). We believe much more work is needed in this direction and leave it to future work.

Thank you again for your feedback!! It has helped us make our claims more precise, and we hope the experiments you suggested will strengthen our findings. We will post an update once the ongoing experiments finish. Please let us know if you have any follow-up questions in the meantime.

AuthorsOpenReview ↗

Follow up on isolating the easy-to-hard curriculum effect

Weakness 2: I think curriculum attribution is incomplete: Shuffling changes both difficulty order and task similarity; the experiments do not isolate an easy-to-hard curriculum effect.

To separate task difficulty from task similarity, we ran three additional orders with ReasoningBank on WebArena (3 runs each): template-level easy-to-hard, shuffled, and hard-to-easy. In all three, tasks from the same template stay adjacent, and template difficulty is determined by the no-memory baseline pass rate.

Task orderOverall pass rate (%)
No-memory54.76 ± 0.84
Default56.32 ± 1.39
Instance-level shuffle
Shuffle-149.84 ± 0.77
Shuffle-250.74 ± 1.14
Template-level control
Easy-to-hard53.16 ± 1.23
Shuffle50.74 ± 0.82
Hard-to-easy48.69 ± 1.46
  • From the last three rows, we find that difficulty direction does matter: easy-to-hard outperforms hard-to-easy by 4.5 points, and the template-level shuffle falls in between.
  • Task similarity appears to matter less: the template-level shuffle, which keeps similar tasks adjacent, performs about the same as the instance-level shuffles, which scatter them.
  • Difficulty alone does not explain the advantage of the default order: easy-to-hard is still 3.2 points below the default order, and does not exceed the no-memory baseline. The default order likely has additional structure (e.g., subtask or skill dependencies) that helps the agent learn.

We will revise the paper to include this set of results, and make it clear that difficulty is one contributing factor rather than the full explanation. Thank you again for providing feedback to our work!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear reviewers,

Thank you all for your thoughtful comments! We are grateful that reviewers found our work clearly motivated (P5p1), timely and practically relevant (hpko), and addressing an important evaluation weakness (EbDC), with a concrete failure analysis (hpko, EbDC).

We have uploaded a revised version, with changes marked in blue. Below is a summary.


Edits:

  • Title, abstract and introduction: Narrowed the scope to memory-based methods in the title, stated the primary claim explicitly in the abstract, and made the claims on variance and task order more precise. (Reviewers hpko, EbDC)
  • Section 3: Clarified that each domain is a separate task stream, added statistical tests to support the significance claims, noted that ReasoningBank on VisualWebArena is an exception to the degradation under shuffled orders, and added a row to Table 1 reporting gaps in number of tasks. (Reviewers hpko, EbDC)
  • Section 4.2: Explained how the recovered fraction of degradation (32%) is computed (Reviewers hpko)
  • Appendix A (Limitations): Reviewers suggested extending the study to other self-improving methods (e.g., harness evolution) and attributing the degradation more cleanly. These are valuable directions beyond the scope of this work, and we now discuss them as future work. (Reviewers P5p1, hpko, EbDC)

New results:

  • Section 3.3: Controlled task orders (easy-to-hard, shuffled, hard-to-easy). We verified that difficulty direction matters, but it explains only part of the advantage of the default order. (Reviewers hpko, EbDC)
  • Appendix E: Benchmark-level standard deviation with confidence intervals. (Reviewers hpko)
  • Appendix E: Leakage-controlled results, showing that the gains from rubrics are not explained by prior tasks leaking answers to subsequent tasks. (Reviewer hpko)
  • Appendix E: Experiments on using five different levels of feedback, from a noisy LLM judge to the most detailed rubrics. The quality of feedback greatly influences the agent’s learning outcome. (Reviewer P5p1)

Thank you again for reviewing our work. Your feedback has helped us strengthen the paper, and we are grateful for the time you have invested.

Authors of Submission 50649

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.