acceptodds
Under review as a conference paper at ICLR 2027

AMBER: Training Long-Horizon Web Agents through Append-Only Memory

Abstract

Modern language-model agents increasingly interact with external environments over long-horizon, multi-step trajectories, where the accumulated interaction history can quickly exceed practical context budgets. To ensure reliability, agents must maintain factual information over long horizons, remember execution errors and corrective feedback, and track progress across actions. Several approaches have been proposed to achieve this without the need for maintaining the entire execution history in context, such as using the reasoning and action history, learning to maintain a fixed-size memory through an overwrite mechanism, and periodic summarization. Although overwrite memory can in principle retain anything an append-only memory can, it must learn to carry each fact through every subsequent rewrite, which is difficult to learn from sparse outcome rewards; for interactive applications like web agents, we find that trained overwrite memories delete key information required by the trajectory, as well as corrective feedback received from the environment. We introduce AMBER (Append-only Memory Bank for Evidence Retention)—a simple and scalable framework where an agent jointly learns to reason, act, and write free-form memory, while an append-only rule guarantees retention by construction. This allows AMBER to be trained end-to-end with reinforcement learning from outcome rewards without the need for extensive curated SFT data. On WebArena Lite, AMBER improves average success over overwrite-based memory by 4.09 percentage points, increases the fraction of tasks solved in five repeated runs by 4.8 percentage points, and matches an overwrite baseline trained on substantially more expensive curated supervision. AMBER achieves these improvements while maintaining a practical token budget, providing a strong balance between context efficiency, task performance, and reliable long-horizon execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-horizon tasks, web agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K2W5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AMBER for long-horizon web agents. At each step, the agent generates reasoning, an action, and a free-form memory entry. The entry is appended to a persistent memory bank. Retention is fixed by an append-only rule; only "what to write" is learned. The policy starts from light SFT (memory labels annotated by GPT-5.5 over Go-Browse trajectories), then trains end-to-end with GRPO from a terminal outcome reward. On WebArena Lite (Qwen3.5-9B/27B), AMBER beats an overwrite-memory baseline (MemAgent) by 4.09/2.05 points in average success. It improves repeated-run reliability (5/5: 15.7%→20.5% at 9B) and matches an overwrite baseline retrained with refined supervision. Trajectory analysis shows overwrite memory loses environment feedback within ~4 steps (retention ~4.5%) and drops recorded evidence. The authors conclude the gap is learnability, not memory capacity.

Critical strengths

S1. A clean single-axis controlled comparison. Prior memory papers vary model, observation, training recipe, and environment together. The retention rule's effect cannot be isolated. This paper fixes the other axes and varies only the retention rule, under an identical outcome-reward RL recipe, at two model scales. Such controlled studies are rare in the agent-memory literature. This is the core evidence for the central claim.

S2. Mechanism-level evidence. Table 5 shows environment feedback survives in overwrite memory for only ~4 steps (100→25.6→7.5→4.5→0.8%). The invalid-action analysis (1.29 vs 1.95 re-issues per event; ≥6 re-issues in 3.3% vs 15.4% of events) supports the causal chain: overwrite erases feedback, so the agent repeats mistakes. Appendices E/F give audited trajectories with full memory dumps, so the qualitative claims are checkable.

Critical weaknesses

W1. The statistical evidence does not support the precision of the paper's comparative claims. The paper makes many fine-grained comparisons. The abstract claims AMBER "matches an overwrite baseline trained on substantially more expensive curated supervision." Section 5.2 claims AMBER's "main advantage appears in solving tasks consistently across repeated runs." But none of these claims is backed by a significance test. No table reports variance except Table 1's seed spread. The evaluation sample is small: 165 tasks × 5 seeds. At this size, the reported gaps are of the same order as seed noise. As the paper asserts directional conclusions ("matches", "improves reliability", and "AMBER leads at k of five thresholds") from numbers that cannot, at this sample size, distinguish signal from noise. A paired analysis over tasks (e.g., McNemar's test or bootstrap CIs) is required before any of these claims can carry weight.

W2. The learnability claim is never tested against the strongest competing paradigm. The central claim: the append-only prior is easier to learn from sparse outcome rewards than learned rewriting. Memory-R1 and Memory-as-Action learn explicit memory operations under the same outcome-reward RL paradigm. They are the direct opponents of this claim. Yet they get zero experimental comparison, and one passing sentence in related work. AMBER itself also has no ablation: no SFT-free variant, no ablation of the two-turn memory timing, no entry-length penalty control, no training curves. The gain's attribution — to the retention prior's learnability, or to the data pipeline, or to the RL budget — remains unestablished.

Questions for authors

Q1. The abstract states that AMBER "matches an overwrite baseline trained on substantially more expensive curated supervision." However, in Table 4 the refined MemAgent wins at ≥1/5 and — more importantly — at the 5/5 threshold. The 5/5 threshold is the consistency metric that Section 5.2 itself argues is AMBER's main advantage. So on the paper's own reliability metric, the refined baseline numerically wins. Please clarify paired per-task statistics (e.g., McNemar's test or a bootstrap CI over tasks) for Tables 1–4. With n=165 tasks and 5 seeds, the observed gaps of 0.8–1.5 points are within one to two tasks' worth of difference. Without variance estimates, neither "matches" nor "outperforms" can be distinguished from noise.

Q2. Why is there no comparison with Memory-R1 or Memory-as-Action? The paper's central claim is that the append-only prior is easier to learn from sparse outcome rewards than learned memory management. Memory-R1 learns ADD/UPDATE/DELETE/NOOP operations with an RL reward; Memory-as-Action learns in-place context editing (deletion/insertion) end-to-end with RL. Both use the same training paradigm as this paper: outcome-driven RL, no dense supervision. They are therefore the direct experimental opponents of the learnability claim. Currently they receive no experiment and only one sentence in related work ("they enlarge what the policy must learn"). If your position is that these methods are harder to train and underperform, it needs the head-to-head experiment, or at least a discussion of why the comparison would not be informative.

Q3. What about the symmetric failure mode on the append side? In the two-turn design, the memory entry m_t is written after action a_t but before the outcome observation o_{t+1}. So a fact that appears only in the action outcome is captured only if a later step re-records it, it may be omitted. Table 5 does not tell us how often the append agent fails to record outcome information that later steps need ("omission" loss). The claim "durability by construction" covers only what was written down. Please report the append-side omission rate — e.g., the fraction of steps whose outcome facts are never present in any memory entry before they are needed.

Q4. RL training uses only 100 optimizer steps with group size G=6 (Appendix I.2). The two methods differ in how quickly useful memory behavior emerges under a fixed budget. If the overwrite baseline is trained longer (or with a larger G), does the gap to AMBER persist? If the gap closes with more RL steps, the correct conclusion may be a sample-efficiency advantage, not a learnability gap. Please report training curves for both methods under matched budgets.

5 comments

Dear Reviewer K2W5, thank you for your feedback. We address your concerns as follows. We have incorporated them into the revised version, with the changes marked in red.

Scope of our claim:

  • Under one fixed protocol (same model, observations, SFT trajectories, RL recipe and budget), replacing learned overwrite retention with an append-only rule improves a web agent trained from outcome rewards on WebArena.
  • The improvement is consistent with overwrite memory dropping information it had recorded.
  • Where a request falls outside this scope, we say so below, and the revised Limitations (Appendix A) now state it.

W1/Q1: Can the claims that AMBER outperforms overwrite memory and matches refined overwrite be distinguished from noise?

  • As per the reviewer's suggestion, we run the paired per-task analysis. The evaluation set has 127 tasks (WebArena Lite without GitLab), each evaluated in five runs, or 635 episodes per method. For each comparison, we report:

    • a 95% CI for the difference, written as Δ\Delta ± margin. This is the standard paired t-interval (margin = 1.98 × standard error of the per-task differences);
    • a p-value from a paired sign-flip permutation test.
  • AMBER outperforms overwrite memory: AMBER's success rate is 4.09 points higher than MemAgent's (36.85% vs. 32.76%, Table R1). This gain is statistically significant, as seen by its 95% CI, +4.09 ± 3.66 (+0.44 to +7.75), not containing zero (p = 0.034).

AMBER's advantage holds when a task must be solved in several runs:

  • To better demonstrate reliability, we additionally report pass^k [1]: the probability that k runs of a task all succeed.
    • For a task solved in c of the five runs, pass^k = C(c, k) / C(5, k).
    • We average it over tasks. pass^1 is the success rate.
    • Unlike the thresholds, it uses every run of every task.
  • AMBER's pass^k is significantly higher than MemAgent's for k = 1 to 4, as seen by each margin being smaller than the gap, so that the CIs do not contain zero (Table R2).
  • The gap is of similar size at every k (+4.1 to +5.2 points). Hence, it does not shrink when a task must be solved in several runs.

Most of AMBER's gain comes from solving tasks more consistently:

  • To separate consistency from task difficulty, we group the tasks by which method solves them at least once. We then compare the methods on the same tasks (Table R3).
  • On the 55 tasks that both methods solve, AMBER succeeds in 78.2% of runs and MemAgent in 71.6%. These tasks contribute 2.83 of the 4.09 points.
  • Solving more tasks contributes the remaining 1.26 points (2.99 − 1.73). AMBER alone solves 13 tasks and MemAgent alone solves 7. This difference is not statistically significant (McNemar p = 0.26).
  • This supports the claim in Section 5.2 that most of AMBER's advantage is in solving tasks consistently.

Refined overwrite is comparable to AMBER, but needs a costlier annotation pipeline:

  • Tables R1 (paired success rate differences) and R2 (pass^k) show that with a refined pipeline for SFT data (Refined MemAgent), the difference from AMBER is not statistically significant, both in success rate and in reliability.
  • The CI also bounds the difference. At the 95% level, Refined MemAgent is at most 2.69 points better than AMBER in success rate (CI for AMBER minus Refined MemAgent: −2.69 to +5.21).
  • Refined MemAgent's lead at 5/5 (21.3% vs. 20.5%) is a single task, and is not significant (McNemar p = 1.0).
  • Refined MemAgent also has a larger generation budget for its memory turn, in both RL training and evaluation: 3,072 tokens, versus 1,024 tokens for AMBER and MemAgent. It uses 283.7 tokens per step on average (Appendix D.6, Table 13).
  • As claimed in the manuscript, AMBER reaches comparable performance without the additional refinement pipeline used for creating the SFT data for the Refined MemAgent baseline (Appendix I).
  • This result is valuable because refining the SFT data costs ~2 times as much. Thus, the benefit of AMBER is that it can take advantage of less refined SFT data.

(Continued) W1/Q1: Can the claims that AMBER outperforms overwrite memory and matches refined overwrite be distinguished from noise?

Table R1. Paired success-rate differences (Qwen 3.5 9B). Δ\Delta is AMBER minus the baseline, with a 95% CI margin (paired t-interval over tasks) and a sign-flip p-value. Bold: statistically significant.

ComparisonSuccess rate (%)Δ\Delta (points)95% CIp
AMBER vs. MemAgent36.85 vs. 32.76+4.09± 3.660.034
AMBER vs. Refined MemAgent36.85 vs. 35.59+1.26± 3.950.58

Table R2. Repeated-run reliability (Qwen 3.5 9B). ≥1/5 is the percentage of tasks solved in at least one run. pass^k is the probability (%) that k of the five runs all succeed, averaged over tasks. Δ\Delta rows give AMBER minus the baseline: 95% CIs for pass^k as Δ\Delta ± margin (paired t-interval over tasks), and an exact McNemar p-value for ≥1/5. Bold: statistically significant.

≥1/5pass^1pass^2pass^3pass^4pass^5
AMBER53.536.929.325.622.820.5
MemAgent48.832.824.720.517.615.7
Refined MemAgent54.335.627.924.722.721.3
Δ\Delta vs. MemAgent+4.7 (p = 0.26)+4.1 ± 3.7+4.6 ± 4.0+5.1 ± 4.4+5.2 ± 5.1+4.8 ± 5.8
Δ\Delta vs. Refined MemAgent−0.8 (p = 1.0)+1.3 ± 4.0+1.4 ± 4.5+0.9 ± 5.0+0.1 ± 5.7−0.8 ± 6.4

Table R3. Where AMBER's success-rate gain over MemAgent comes from (Qwen 3.5 9B, 127 tasks). Tasks are grouped by which method solves them in at least one of the five runs. The contribution is the group's share of the 4.09-point gap in success rate over all tasks.

Task groupTasksSuccess rate (%), AMBER vs. MemAgentContribution (points)
Solved by both5578.2 vs. 71.6+2.83
Solved only by AMBER1329.2 vs. 0+2.99
Solved only by MemAgent70 vs. 31.4−1.73
Solved by neither520 vs. 00
Total12736.85 vs. 32.76+4.09

We have added this analysis to the appendix (Appendices D.1 and D.2), with pointers in Sections 5.1, 5.2 and 5.4. The abstract and the Introduction now report the pass^4 gain (+5.2 points), which is statistically significant, instead of the 5/5 gain (+4.8 points), which is not.


References

[1] Yao et al., "τ\tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", ICLR25

W2/Q2: Why is there no comparison with Memory-R1 or Memory-as-Action, which learn memory operations from outcome rewards?

The comparison with MemAgent tests the learnability claim for learned retention:

  • By learned retention, we mean that the policy decides which information stays in memory.
  • Overwrite memory (MemAgent), context deletion (Memory-as-Action) and UPDATE/DELETE operations (Memory-R1) all use learned retention.
  • AMBER instead keeps every entry by an append-only rule. MemAgent is one representative of learned retention. It shares AMBER's observations, model, SFT trajectories and RL recipe, so the retained context is the only axis that changes (the single-axis comparison the reviewer notes in S1).

Choice of evaluation benchmark:

  • Memory-R1 and Memory-as-Action are evaluated primarily on QA benchmarks where the actions do not change the environment: Memory-R1 manages memory over multi-session dialogues. It is evaluated on LoCoMo, MSC and LongMemEval.
  • Memory-as-Action edits the working context of a retrieval agent. It is evaluated on search QA (HotpotQA, 2WikiMultihopQA, Bamboogle, Frames and BrowseComp-Plus).
  • We choose WebArena to evaluate memory mechanisms under demanding settings. Its websites require diverse interaction patterns. It is also a state-changing environment, where actions modify the website state. The agent must therefore remember what it has already done and the feedback it received.
  • Hence, each method must be re-implemented and re-trained on WebArena.

Practical limitations:

  • We re-implement and re-train MemAgent and WebAgent-R1 for fair comparisons with AMBER.
  • Memory-as-Action uses its own RL algorithm (DCPO). It also needs a cold-start SFT dataset of memory-editing trajectories.
  • This SFT dataset is built by running a large model (DeepSeek-V3.1 in the original work) as the agent, and keeping only successful episodes. On WebArena, this means paid API calls for a frontier model acting on the live websites.
  • Memory-R1 trains a memory manager and an answer agent separately. Both need to be adapted to web trajectories. Moreover, Memory-R1 released the implementation on November 7th, 2026, prior to which the repository was a placeholder and hence difficult to reproduce.
  • Hence, under a limited compute, cloud VM and API budget, we prioritize experiments with reasoning-and-action history (WebAgent-R1), overwrite memory (MemAgent), and a baseline with improved SFT data for overwrite memory (Refined MemAgent).

Training curves:

  • As per the reviewer's suggestion, we add the RL training reward curves of AMBER, MemAgent and WebAgent-R1 (Appendix D.3 of the revised manuscript).

The gain arises during RL, not from the SFT data:

  • AMBER and MemAgent use the same SFT trajectories, and the same RL recipe and budget (100 RL steps, group size G = 6).
  • After SFT alone, the two methods have similar success rates (30.71% for AMBER vs. 31.50% for MemAgent, Table R4).
  • RL then improves AMBER by 6.14 points (30.71 → 36.85), but MemAgent by only 1.26 points (31.50 → 32.76).
  • Hence, the gap is not explained by the SFT data. It opens during RL, under the same budget for both methods.

Table R4. Success rate (%) after SFT and after SFT + RL (Qwen 3.5 9B).

MethodSFTSFT + RLGain from RL (points)
MemAgent31.5032.76+1.26
AMBER30.7136.85+6.14

Ablations of AMBER:

  • SFT-free variant: Table 1 in the manuscript reports AMBER's interface without any training (zero-shot Qwen 3.5 9B). It reaches 21.82%, compared with 34.22% for the zero-shot full-context agent. Hence, the base model cannot write step-wise memory reliably without the supervised warm-up. We did not run RL without SFT.
  • Two-turn memory timing and entry-length penalty: each ablation needs a new RL run, of about 384 GPU-hours (8 GPUs × 48 hours). This is beyond our compute budget, and we leave them to future work. AMBER uses no length penalty: each entry is only capped at 1024 tokens, and AMBER's entries are shorter than MemAgent's memories (122.6 vs. 212.9 tokens per step).

We have clarified in the paper that our conclusion is shown for overwrite memory, as one form of learned retention (Section 2). We have also defined learned retention in the related work, citing Memory-R1 and Memory-as-Action (Section 2), and added both to the design-space table (Appendix C, Table 6). We have also stated why we chose WebArena (Section 5). The Limitations (Appendix A) now state that we test learned retention only as overwrite memory, and list the AMBER ablations we did not run.

Reviewer_ShbeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how a web agent should retain information during a long, multi-step task. AMBER trains one policy to reason, act, and write a free-form memory entry at each step, while a deterministic append-only rule keeps all earlier entries available. The authors compare this design with reasoning-and-action history and an adapted overwrite-memory agent under a shared WebArena Lite setup. They report higher average success and repeated-run consistency, along with examples in which an overwrite memory loses factual evidence or corrective feedback. The main proposed insight is that fixing the retention rule makes useful memory easier to learn from sparse outcome rewards.

Critical strengths
  1. A simple, well-motivated retention rule tested with recent open-weight models. The append-only update cleanly separates learning what to record from deciding whether an earlier entry survives. Experiments with Qwen3.5 9B and 27B test this idea at two model sizes. These are capable and relatively recent open-weight models, although I would not describe these particular sizes as frontier models.
Critical weaknesses
  1. The evaluation does not establish broad reliability or the claimed efficiency. All principal results come from WebArena Lite, with GitLab excluded and evaluation capped at 30 steps. Section 5 does define success rate, repeated-run reliability, and action-prompt length, so the issue is not an absence of metrics; it is the absence of cross-benchmark evidence, paired uncertainty estimates for the method differences, and end-to-end cost or latency measurements. The repeated-run thresholds mix task difficulty with consistency, and the operational meaning of repeated attempts is unclear for state-changing web tasks. At step 30, AMBER's reported action prompt is about 7.4k tokens versus 4k for overwrite memory, while AMBER also generates a separate memory turn at each step. Training efficiency matters as well as inference efficiency, but the paper reports neither comparative GPU-hours or annotation costs nor episode-level inference tokens and latency. The assertion that AMBER is "cheaper and easier to train" rests on the absence of an additional refinement pipeline, rather than a quantified comparison. The rollout architecture reuses established asynchronous-training ingredients without a throughput or cost ablation establishing a distinct systems contribution.

  2. The claimed contributions are largely a recombination of prior ideas, and the results do not isolate a unique append-only benefit. The selected baselines represent sensible context-retention categories, but MemAgent was developed for segmented long-text processing rather than interactive web navigation; the exact adaptation and parity of prompts, memory-generation turns, training data, and compute budgets require more detail. The assertion that reasoning-and-action history is insufficient and explicit memory is required is not new as a motivation: Self-MAP and HiAgent already advocate explicit working memory, while WebGym shows a memory-prompt benefit in outcome-trained web agents. Joint outcome-based training of web actions was established by WebAgent-R1; joint learning of reasoning and memory appears in MEM1 and MemSearcher. The retained-log rule is simple and may be useful, but the paper does not show that it, rather than a larger prompt budget or a different memory-writing protocol, uniquely causes the reported gains. Higher success, repeated-run consistency, and fewer repeated errors can also arise from other memory or error-control designs; for example, WebGym separately penalizes repeated ineffective actions. AMBER's reasoning-history baseline still completes many tasks, and the full-context reference is not training-matched. The defensible contribution is a controlled comparison of immutable versus overwrite retention in this setting, not the individual ingredients or a general necessity result. The "minimal supervision" framing also needs qualification: 2,328 SFT trajectories received retrospective GPT-5.5 memory labels.

  3. Likely generated by Auto-research agents.

Questions for authors
  1. Generality and reliability. Can you evaluate on another interactive benchmark or a substantially longer task horizon, including the omitted GitLab site if feasible? Please report per-task outcomes across the five seeds, paired confidence intervals for AMBER-versus-baseline differences, and a consistency measure conditioned on tasks that at least one method can solve. What practical use case motivates the at-least-one-of-five and five-of-five thresholds for tasks with side effects?

  2. Baseline adaptation and incremental contribution. How exactly were MemAgent and WebAgent-R1 adapted to the common observation, action, memory-generation, and training protocol? Were memory turns, initial supervision, training compute, and token limits matched? Can you include a budget-matched overwrite baseline and a trained full-history or simpler append-memory control? Which observed benefit specifically requires immutable append-only retention, as opposed to another memory design or explicit repeated-action control?

  3. Inference and training cost. For each method, can you report total prefill and generated tokens per completed episode, number of model calls, wall-clock latency, throughput, training GPU-hours, and annotation cost? In particular, how much of AMBER's success gain remains when comparison methods receive the same inference-token or latency budget, and how does cost grow beyond the reported 30-step horizon? What measured saving supports "cheaper and easier to train," and what throughput gain, if any, is attributable to the proposed rollout architecture?

  4. Scope of the claims. Given earlier explicit-memory agents and the success of your reasoning-history baseline on some tasks, what evidence supports the statement that explicit memory is required, rather than beneficial under this particular protocol? What evidence shows that retaining every generated entry improves usable evidence rather than merely preserving text, especially when entries are erroneous, stale, or difficult to retrieve? How many of the 2,328 retrospectively labeled SFT trajectories and how much GPT-5.5 annotation cost were necessary for the reported gains? Please align the abstract's claims about minimal supervision, scalability, and reliability with these measurements.

7 comments

Dear Reviewer Shbe, thank you for your feedback. We address your concerns as follows. We have incorporated the relevant changes into the revised version, marked in red.

Scope of our claim:

  • Under one fixed protocol (same model, observations, SFT trajectories, RL recipe and budget), replacing learned overwrite retention with an append-only rule improves a web agent trained from outcome rewards on WebArena.
  • The improvement is consistent with overwrite memory dropping information it had recorded.
  • Where a request falls outside this scope, we say so below, and the revised Limitations (Appendix A) now state it.

W1/Q1: Are the AMBER-versus-baseline differences significant under paired tests, and what are the per-task outcomes across the five seeds?

  • We run the paired per-task analysis the reviewer asks for. All methods are evaluated on the same 127 tasks (WebArena Lite without GitLab), each in five runs, or 635 episodes per method. We therefore compare them task by task. For each comparison, we report:

    • a 95% CI for the difference, written as Δ\Delta ± margin. This is the standard paired t-interval (margin = 1.98 × standard error of the per-task differences);
    • a p-value from a paired sign-flip permutation test.
  • AMBER outperforms overwrite memory: AMBER's success rate is 4.09 points higher than MemAgent's (36.85% vs. 32.76%, Table R1). This gain is statistically significant, as seen by its 95% CI, +4.09 ± 3.66 (+0.44 to +7.75), not containing zero (p = 0.034).

  • AMBER and refined overwrite memory are comparable: AMBER's success rate is 1.26 points higher than refined overwrite (Refined MemAgent) - 36.85% for AMBER vs. 35.59% for Refined MemAgent. This difference is not statistically significant, as seen by its 95% CI, +1.26 ± 3.95 (−2.69 to +5.21), containing zero (p = 0.58).

  • The CI also bounds the difference: at the 95% level, Refined MemAgent is at most 2.69 points better than AMBER. Hence, as claimed in the manuscript, the two methods are comparable, not that AMBER outperforms Refined MemAgent.

  • However, to reach AMBER's performance, Refined MemAgent needs a more expensive SFT data curation pipeline, which costs ~2 times as much in API credits (please see our response to W2/Q4 on the annotation of the 2,328 SFT trajectories for the exact costs).

Table R1. Paired success-rate differences (Qwen 3.5 9B). Δ\Delta is AMBER minus the baseline, with a 95% CI margin (paired t-interval over tasks) and a sign-flip p-value. Bold: statistically significant.

ComparisonSuccess rate (%)Δ\Delta (points)95% CIp
AMBER vs. MemAgent36.85 vs. 32.76+4.09± 3.660.034
AMBER vs. Refined MemAgent36.85 vs. 35.59+1.26± 3.950.58

Per-task outcomes across the five runs:

  • Table R2 gives the number of tasks that each method solves in exactly c of the five runs.
  • AMBER has fewer tasks with no success than MemAgent (59 vs. 65), and more tasks solved in all five runs (26 vs. 20).

Table R2. Number of tasks solved in exactly c of the five runs (Qwen 3.5 9B, 127 tasks).

Methodc = 012345
AMBER5915751526
MemAgent65126121220
Refined MemAgent581968927

W1/Q1: The k-of-5 thresholds mix task difficulty with consistency. How consistent is each method on tasks that at least one method can solve?

AMBER is more consistent on the tasks that at least one method solves:

  • As per the reviewer's suggestion, we restrict the comparison to the 75 tasks that AMBER or MemAgent solves in at least one run.
  • On these tasks, AMBER's success rate is 6.93 points higher than MemAgent's (62.4% vs. 55.5%). This gain is statistically significant, as seen by its 95% CI, +6.93 ± 6.13 (+0.80 to +13.06), not containing zero (p = 0.034).
  • On the 78 tasks that AMBER or Refined MemAgent solves, the difference is not statistically significant (+2.05 ± 6.45, p = 0.58).

Most of AMBER's gain comes from solving tasks more consistently:

  • To separate consistency from task difficulty, we group the tasks by which method solves them at least once (Table R4).
  • On the 55 tasks that both methods solve, AMBER succeeds in 78.2% of runs and MemAgent in 71.6%. These tasks contribute 2.83 of the 4.09 points.
  • Solving more tasks contributes the remaining 1.26 points (2.99 − 1.73). AMBER alone solves 13 tasks and MemAgent alone solves 7. This difference is not statistically significant (McNemar p = 0.26).
  • This supports the claim in Section 5.2 that most of AMBER's advantage is in solving tasks consistently.

AMBER's advantage holds when a task must be solved in several runs:

  • To better demonstrate reliability, we additionally report pass^k [1]: the probability that k runs of a task all succeed.
    • For a task solved in c of the five runs, pass^k = C(c, k) / C(5, k).
    • We average it over tasks. pass^1 is the success rate.
    • Unlike the thresholds, it uses every run of every task.
  • AMBER's pass^k is significantly higher than MemAgent's for k = 1 to 4, as seen by each margin being smaller than the gap, so that the CIs do not contain zero (Table R3).
  • The gap is of similar size at every k (+4.1 to +5.2 points). Hence, it does not shrink when a task must be solved in several runs.

Refined overwrite is comparable to AMBER, but needs a costlier annotation pipeline:

  • Paired success rate (Table R1) and pass^k (Table R3) show that with a refined pipeline for SFT data (Refined MemAgent), the difference from AMBER is not statistically significant, both in success rate and in reliability.
  • Refined MemAgent's lead at 5/5 (21.3% vs. 20.5%) is a single task, and is not significant (McNemar p = 1.0).
  • As claimed in the manuscript, AMBER reaches comparable performance without the additional refinement pipeline used for creating the SFT data for the Refined MemAgent baseline (Appendix I).
  • This result is valuable because refining the SFT data costs ~2 times as much (please see our response to W2/Q4 on the annotation of the 2,328 SFT trajectories for the exact costs). Thus, the benefit of AMBER is that it can take advantage of less refined SFT data.

Table R3. Repeated-run reliability (Qwen 3.5 9B). ≥1/5 is the percentage of tasks solved in at least one run. pass^k is the probability (%) that k of the five runs all succeed, averaged over tasks. Δ\Delta rows give AMBER minus the baseline: 95% CIs for pass^k as Δ\Delta ± margin (paired t-interval over tasks), and an exact McNemar p-value for ≥1/5. Bold: statistically significant.

≥1/5pass^1pass^2pass^3pass^4pass^5
AMBER53.536.929.325.622.820.5
MemAgent48.832.824.720.517.615.7
Refined MemAgent54.335.627.924.722.721.3
Δ\Delta vs. MemAgent+4.7 (p = 0.26)+4.1 ± 3.7+4.6 ± 4.0+5.1 ± 4.4+5.2 ± 5.1+4.8 ± 5.8
Δ\Delta vs. Refined MemAgent−0.8 (p = 1.0)+1.3 ± 4.0+1.4 ± 4.5+0.9 ± 5.0+0.1 ± 5.7−0.8 ± 6.4

Table R4. Where AMBER's success-rate gain over MemAgent comes from (Qwen 3.5 9B, 127 tasks). Tasks are grouped by which method solves them in at least one of the five runs. The contribution is the group's share of the 4.09-point gap in success rate over all tasks.

Task groupTasksSuccess rate (%), AMBER vs. MemAgentContribution (points)
Solved by both5578.2 vs. 71.6+2.83
Solved only by AMBER1329.2 vs. 0+2.99
Solved only by MemAgent70 vs. 31.4−1.73
Solved by neither520 vs. 00
Total12736.85 vs. 32.76+4.09

We have added this analysis to the appendix (Appendices D.1 and D.2), with pointers in Sections 5.1, 5.2 and 5.4.


References

[1] Yao et al., "τ\tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", ICLR25

W1/Q1: What do repeated attempts mean for state-changing tasks, and what practical use case motivates the ≥1/5 and 5/5 thresholds?

  • Repeated attempts are independent trials. The website containers are reset after every episode, so one run does not change the state that another run sees. We have clarified this in the manuscript (Section 5, Evaluation).

The thresholds separate capability from reliability:

  • ≥1/5 measures whether a method can solve a task at all. It is a capability diagnostic, not a deployment measure.
  • 5/5 and pass^k measure whether it solves the task every time. These are the deployment-relevant measures.
  • Reliability matters when an agent runs unattended, for example when it places an order or edits a record in a store's admin panel. A failed run there leaves side effects, so the task cannot simply be retried.

W1/Q1: Can you evaluate on another interactive benchmark, a longer horizon, or the omitted GitLab site?

  • We exclude GitLab because parallel GitLab containers started on the same VM are unstable. Larger VMs or sandbox services could address this, but they are beyond our budget.
  • We chose WebArena because it is demanding for memory. It is a state-changing environment: actions modify the website state. The agent must therefore remember what it has already done and the feedback it received.
  • WebArena evaluates the memory generation from multiple perspectives: tracking task progress and environment state, retaining factual information, and retaining and utilizing environment feedback. Hence, we leave evaluating on other benchmarks to future work, as WebArena covers diverse memory skills that any long-horizon agent must possess.

We have stated this scope in the Limitations (Appendix A).

W1/Q3: For each method, what are the total prefill and generated tokens, and wall-clock latency per episode, and how does cost grow beyond 30 steps?

Tokens per step:

  • Table R5 gives the tokens per step. Its input tokens are those of the action turn. They are dominated by the observation (accessibility tree) tokens.
  • Append (AMBER) uses an average of 4,568.0 input tokens per step, and overwrite (MemAgent) uses 3,846.6. These are means over all steps. The 7.4k vs. 4k tokens in Section 5.3 are the action prompt at step 30, where AMBER's memory bank is longest.
  • Both methods generate roughly the same number of reasoning and action tokens (240.8 for AMBER and 228.3 for MemAgent). AMBER generates fewer memory tokens (122.6 for AMBER vs. 212.9 for MemAgent).

Table R5. Input and output tokens per step (Qwen 3.5 9B, 5 runs). Mean ± standard deviation over all steps. Input tokens are those of the action turn. Both methods have a memory budget of 1024 tokens per step.

MethodInput tokens (accessibility tree)Input tokens (total)Output tokens (action)Output tokens (memory)Output tokens (total)
MemAgent2,253.6 ± 1,304.63,846.6 ± 1,365.0228.3 ± 128.3212.9 ± 127.7441.2 ± 208.8
AMBER2,296.0 ± 1,387.64,568.0 ± 1,964.1240.8 ± 145.9122.6 ± 74.9363.4 ± 191.6

AMBER uses fewer model calls and generated tokens per episode, and 4% more prompt tokens:

  • Table R6 sums the tokens of both turns (action and memory) over each episode.
  • AMBER and MemAgent make two model calls per step, one per turn. WebAgent-R1 makes one, since it has no memory turn.
  • AMBER's episodes are shorter (10.06 vs. 11.38 steps), so it makes fewer model calls per episode (20.1 vs. 22.8).
  • AMBER's prompt tokens per episode are 4% higher (96,628 vs. 92,726). It generates 27% fewer tokens (3,656 vs. 5,021).
  • The memory turn's prompt extends the action turn's prompt, so most of it is served from the prefix cache. The uncached prompt tokens are 43,031 for AMBER vs. 39,768 for MemAgent.

Table R6. Cost per episode, summed over both turns of every step (Qwen 3.5 9B, 635 episodes per method). Mean ± standard deviation over episodes. Uncached prompt tokens are the prompt tokens not served from the prefix cache.

MethodStepsModel callsPrompt tokensUncached prompt tokensGenerated tokens
MemAgent11.38 ± 8.5422.892,726 ± 78,61839,768 ± 31,7115,021 ± 4,639
AMBER10.06 ± 7.4520.196,628 ± 97,55643,031 ± 41,6193,656 ± 3,419

Cost beyond 30 steps:

  • We believe that 30 steps is a reasonable trajectory length to analyze the shortcomings of overwrite memory, and how an append memory can improve the results.
  • While analyzing the behavior under longer trajectories is valuable, we leave this for future work. Longer episodes increase the execution time and, in effect, the computational resources required, which is beyond our budget (Limitations, Appendix A).

We have added the per-episode costs to the appendix (Appendix D.6).

Reviewer_L88JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMBER, an append-only memory framework for web agents performing multi-step tasks. At each step, the agent generates reasoning and an action, then writes a memory entry that is retained for subsequent interactions. The approach separates learning what to record from deciding what to preserve, avoiding the need to repeatedly rewrite earlier information. The same model is trained jointly for reasoning, action selection, and memory generation through supervised fine-tuning followed by GRPO with task-outcome rewards.

The authors evaluate Qwen3.5-9B and 27B on WebArena-Lite with GitLab excluded, comparing against reasoning-and-action history and overwrite memory under a shared setup. AMBER improves average success over overwrite memory by 4.09 and 2.05 percentage points, respectively. Additional analyses examine reliability across repeated runs, context growth, stronger supervision for overwrite memory, and retention of facts and error feedback. The main contribution is empirical evidence that append-only retention can improve task performance and preserve useful information with less elaborate memory supervision, while keeping context manageable over the evaluated horizons.

Critical strengths
  1. A simple, practical approach with consistent improvements across two model sizes* AMBER separates the learned choice of what to record from a deterministic rule that preserves existing entries. Under the shared experimental setup, it improves average success over overwrite memory by 4.09 percentage points for the 9B model and 2.05 points for the 27B model. These results support the practical value of a readily implementable memory design.

  2. The paper provides useful evidence about why memory rewriting can fail. The trajectory analyses illustrate the loss of previously recorded facts and corrective feedback, while the refined-supervision experiment shows that better memory annotations narrow the overwrite baseline’s performance gap. Together, these findings support the paper’s argument that learning to preserve information is a meaningful challenge, and that the retention rule can reduce the supervision needed to obtain useful memory behavior.

Critical weaknesses
  1. The experiments do not sufficiently establish the claimed learnability advantage. AMBER retains an expanding collection of memories, whereas the overwrite baseline maintains a bounded memory; their action contexts reach approximately 7.4K and 4K tokens, respectively, at step 30. Without matched memory budgets, SFT-only results, or learning curves, it is difficult to determine whether the gains arise from avoiding repeated rewriting, retaining more information, or differences established during supervised training.

  2. The evidence for the efficiency–performance tradeoff is incomplete. The cost analysis measures action-prompt length, but AMBER also generates a memory response at every step. It does not report total input/output tokens, execution latency, or the annotation costs underlying the comparison with refined overwrite supervision.

Questions for authors
  1. What are the exact memory-generation and retained-context limits for AMBER and the overwrite baseline? Do existing SFT checkpoints or RL learning curves show when their performance diverges? A limited comparison with a larger overwrite memory would help determine whether the advantage persists at comparable memory budgets.

  2. What are the total input/output tokens and execution times when both action and memory generation are included? Can you quantify the additional annotation cost of refined overwrite supervision and show how AMBER compares in performance at a similar total cost?

3 comments

Dear Reviewer L88J, thank you for your feedback. We address your concerns as follows. We have incorporated them into the revised version, with the changes marked in red.

Scope of our claim:

  • Under one fixed protocol (same model, observations, SFT trajectories, RL recipe and budget), replacing learned overwrite retention with an append-only rule improves a web agent trained from outcome rewards on WebArena.
  • The improvement is consistent with overwrite memory dropping information it had recorded.
  • Where a request falls outside this scope, we say so below, and the revised Limitations (Appendix A) now state it.

W1/Q1: Do AMBER's gains come from avoiding repeated rewriting, from retaining more information, or from differences established during supervised training? What are the memory limits, and would a larger overwrite memory close the gap?

AMBER's gains appear during RL training:

  • AMBER's advantage appears during RL training, not during supervised fine-tuning. After SFT alone, both methods have similar success rates (30.71% for AMBER vs. 31.50% for MemAgent, Table R1). RL then improves AMBER by 6.14 points (30.71 → 36.85), but MemAgent by only 1.26 points (31.50 → 32.76). This supports our claim that append-only memory is easier to learn from outcome rewards.
  • We also add the RL training reward curves of AMBER and MemAgent (Appendix D.3 of the revised manuscript). Their training rewards are close over all 100 RL steps.
  • Hence, the two methods improve similarly on the training tasks, but only AMBER's improvement carries over to the evaluation tasks.
  • AMBER and MemAgent use the same SFT trajectories, so the gap does not come from differences established during supervised training.

Table R1. Success rate (%) after SFT and after SFT + RL (Qwen 3.5 9B).

MethodSFTSFT + RLGain from RL (points)
MemAgent31.5032.76+1.26
AMBER30.7136.85+6.14

Memory limits:

  • At each step, both AMBER and MemAgent are given a budget of 1024 tokens to generate memory.
  • The retained context differs. MemAgent retains one memory of at most 1024 tokens. AMBER retains every entry: up to 30 × 1024 tokens in principle, and about 3.7k tokens by step 30 in practice (30 × 122.6 tokens per entry).

Can a larger token budget close the gap?

  • We observe that MemAgent does not utilize the full budget. MemAgent writes 212.9 tokens per step on average, about a fifth of its 1024-token budget (Table R2).
  • During RL training, we observe that MemAgent rarely reaches the budget. The trajectories that do reach it are runaway generations, such as repeated newlines or repeated sentences. MemAgent therefore has room to keep more information, but does not use it.
  • Hence, the 1024-token cap is not what limits MemAgent.
  • We have not trained an overwrite variant that is supervised to write longer memories (Limitations, Appendix A). The closest existing variant is refined overwrite (Refined MemAgent), which is supervised to keep a cumulative memory. We use a budget of 3,072 tokens for its memory turn in both RL training and evaluation, versus 1,024 for AMBER and MemAgent, to provide enough room for the cumulative memory. It writes 283.7 tokens per step, about 1.3 times as many as MemAgent (Table R2).
  • Refined MemAgent drops 16.9% of the facts that need memory after recording them, about half of MemAgent's 30.3% (string-matching tasks, Appendix D.5). AMBER drops none, since it never deletes an entry.

Table R2. Memory tokens generated per step (Qwen 3.5 9B). Mean ± standard deviation over steps, and median. AMBER and MemAgent have a memory budget of 1024 tokens per step, and Refined MemAgent is given a budget of 3,072 tokens.

MethodMean ± stdMedian
MemAgent212.9 ± 127.7187
AMBER122.6 ± 74.9103
Refined MemAgent283.7 ± 152.8249

We have added the SFT and SFT + RL results to the appendix (Appendix D.4), and the memory budget analysis to Appendix D.6, with pointers in Sections 5.1 and 5.3. We have also stated in the Limitations (Appendix A) that we have not trained an overwrite variant supervised to write longer memories.

W2/Q2: What are the total input/output tokens and execution times when both action and memory generation are included?

Tokens per step:

  • Append (AMBER) uses an average of 4,568.0 input tokens per step in the action turn, and overwrite (MemAgent) uses 3,846.6. The input tokens are dominated by the observation (accessibility tree) tokens.
  • AMBER uses more input tokens since the memory generated at each step is appended and provided as input to the subsequent steps.
  • Both methods generate roughly the same number of reasoning and action tokens (240.8 for AMBER and 228.3 for MemAgent), and AMBER generates fewer output memory tokens (122.6 for AMBER vs. 212.9 for MemAgent).
  • Table R3 shows the input and output tokens per step, averaged over all the steps with the Qwen 3.5 9B model.

Table R3. Input and output tokens per step (Qwen 3.5 9B, 5 runs). Mean ± standard deviation over all steps. Input tokens are those of the action turn. Both methods have a memory budget of 1024 tokens per step.

MethodInput tokens (accessibility tree)Input tokens (total)Output tokens (action)Output tokens (memory)Output tokens (total)
MemAgent2,253.6 ± 1,304.63,846.6 ± 1,365.0228.3 ± 128.3212.9 ± 127.7441.2 ± 208.8
AMBER2,296.0 ± 1,387.64,568.0 ± 1,964.1240.8 ± 145.9122.6 ± 74.9363.4 ± 191.6

AMBER uses fewer model calls and generated tokens per episode, and 4% more prompt tokens:

  • Table R4 sums the tokens of both turns (action and memory) over each episode. Each step makes two model calls, one per turn.
  • AMBER's episodes are shorter (10.06 vs. 11.38 steps), so it makes fewer model calls per episode (20.1 vs. 22.8).
  • AMBER's prompt tokens per episode are 4% higher (96,628 vs. 92,726). It generates 27% fewer tokens (3,656 vs. 5,021).
  • The memory turn's prompt extends the action turn's prompt, so most of it is served from the prefix cache. The uncached prompt tokens are 43,031 for AMBER vs. 39,768 for MemAgent.

Table R4. Cost per episode, summed over both turns of every step (Qwen 3.5 9B, 635 episodes per method). Mean ± standard deviation over episodes. Uncached prompt tokens are the prompt tokens not served from the prefix cache.

MethodStepsModel callsPrompt tokensUncached prompt tokensGenerated tokens
MemAgent11.38 ± 8.5422.892,726 ± 78,61839,768 ± 31,7115,021 ± 4,639
AMBER10.06 ± 7.4520.196,628 ± 97,55643,031 ± 41,6193,656 ± 3,419

We have added these token counts to the appendix (Appendix D.6), with a pointer in Section 5.3.

Q2: What is the additional annotation cost of refined overwrite supervision, and how does AMBER compare at a similar total cost?

  • For the 2,328 SFT trajectories used from GoBrowse (which contain only the reasoning and action traces), generating the memory annotations costs about $250 ($0.107 per trajectory). This cost is the same for append (AMBER) and overwrite (MemAgent).
  • The refinement pipeline of refined overwrite supervision costs another $240 ($0.103 per trajectory). Refined supervision therefore costs about $490 in total (250 + 240).
  • Hence, using refined SFT data brings the performance of overwrite memory to a level comparable to AMBER (36.85% success rate for AMBER vs. 35.59% for refined overwrite), but at double the monetary cost of data annotation.
  • AMBER thus reaches comparable performance at about half the annotation cost.

We have added this cost breakdown to the appendix (Appendix I), and stated the cost difference in Section 5.4.

Thank you for the additional analyses. My main concern remains only partially resolved: matching per-step generation limits does not match retained-memory capacity. Without a matched-capacity comparison, it remains unclear whether AMBER’s gains reflect easier learning or simply retaining more information. I will therefore maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.