- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AMBER for long-horizon web agents. At each step, the agent generates reasoning, an action, and a free-form memory entry. The entry is appended to a persistent memory bank. Retention is fixed by an append-only rule; only "what to write" is learned. The policy starts from light SFT (memory labels annotated by GPT-5.5 over Go-Browse trajectories), then trains end-to-end with GRPO from a terminal outcome reward. On WebArena Lite (Qwen3.5-9B/27B), AMBER beats an overwrite-memory baseline (MemAgent) by 4.09/2.05 points in average success. It improves repeated-run reliability (5/5: 15.7%→20.5% at 9B) and matches an overwrite baseline retrained with refined supervision. Trajectory analysis shows overwrite memory loses environment feedback within ~4 steps (retention ~4.5%) and drops recorded evidence. The authors conclude the gap is learnability, not memory capacity.
Critical strengths
S1. A clean single-axis controlled comparison. Prior memory papers vary model, observation, training recipe, and environment together. The retention rule's effect cannot be isolated. This paper fixes the other axes and varies only the retention rule, under an identical outcome-reward RL recipe, at two model scales. Such controlled studies are rare in the agent-memory literature. This is the core evidence for the central claim.
S2. Mechanism-level evidence. Table 5 shows environment feedback survives in overwrite memory for only ~4 steps (100→25.6→7.5→4.5→0.8%). The invalid-action analysis (1.29 vs 1.95 re-issues per event; ≥6 re-issues in 3.3% vs 15.4% of events) supports the causal chain: overwrite erases feedback, so the agent repeats mistakes. Appendices E/F give audited trajectories with full memory dumps, so the qualitative claims are checkable.
Critical weaknesses
W1. The statistical evidence does not support the precision of the paper's comparative claims. The paper makes many fine-grained comparisons. The abstract claims AMBER "matches an overwrite baseline trained on substantially more expensive curated supervision." Section 5.2 claims AMBER's "main advantage appears in solving tasks consistently across repeated runs." But none of these claims is backed by a significance test. No table reports variance except Table 1's seed spread. The evaluation sample is small: 165 tasks × 5 seeds. At this size, the reported gaps are of the same order as seed noise. As the paper asserts directional conclusions ("matches", "improves reliability", and "AMBER leads at k of five thresholds") from numbers that cannot, at this sample size, distinguish signal from noise. A paired analysis over tasks (e.g., McNemar's test or bootstrap CIs) is required before any of these claims can carry weight.
W2. The learnability claim is never tested against the strongest competing paradigm. The central claim: the append-only prior is easier to learn from sparse outcome rewards than learned rewriting. Memory-R1 and Memory-as-Action learn explicit memory operations under the same outcome-reward RL paradigm. They are the direct opponents of this claim. Yet they get zero experimental comparison, and one passing sentence in related work. AMBER itself also has no ablation: no SFT-free variant, no ablation of the two-turn memory timing, no entry-length penalty control, no training curves. The gain's attribution — to the retention prior's learnability, or to the data pipeline, or to the RL budget — remains unestablished.
Questions for authors
Q1. The abstract states that AMBER "matches an overwrite baseline trained on substantially more expensive curated supervision." However, in Table 4 the refined MemAgent wins at ≥1/5 and — more importantly — at the 5/5 threshold. The 5/5 threshold is the consistency metric that Section 5.2 itself argues is AMBER's main advantage. So on the paper's own reliability metric, the refined baseline numerically wins. Please clarify paired per-task statistics (e.g., McNemar's test or a bootstrap CI over tasks) for Tables 1–4. With n=165 tasks and 5 seeds, the observed gaps of 0.8–1.5 points are within one to two tasks' worth of difference. Without variance estimates, neither "matches" nor "outperforms" can be distinguished from noise.
Q2. Why is there no comparison with Memory-R1 or Memory-as-Action? The paper's central claim is that the append-only prior is easier to learn from sparse outcome rewards than learned memory management. Memory-R1 learns ADD/UPDATE/DELETE/NOOP operations with an RL reward; Memory-as-Action learns in-place context editing (deletion/insertion) end-to-end with RL. Both use the same training paradigm as this paper: outcome-driven RL, no dense supervision. They are therefore the direct experimental opponents of the learnability claim. Currently they receive no experiment and only one sentence in related work ("they enlarge what the policy must learn"). If your position is that these methods are harder to train and underperform, it needs the head-to-head experiment, or at least a discussion of why the comparison would not be informative.
Q3. What about the symmetric failure mode on the append side? In the two-turn design, the memory entry m_t is written after action a_t but before the outcome observation o_{t+1}. So a fact that appears only in the action outcome is captured only if a later step re-records it, it may be omitted. Table 5 does not tell us how often the append agent fails to record outcome information that later steps need ("omission" loss). The claim "durability by construction" covers only what was written down. Please report the append-side omission rate — e.g., the fraction of steps whose outcome facts are never present in any memory entry before they are needed.
Q4. RL training uses only 100 optimizer steps with group size G=6 (Appendix I.2). The two methods differ in how quickly useful memory behavior emerges under a fixed budget. If the overwrite baseline is trained longer (or with a larger G), does the gap to AMBER persist? If the gap closes with more RL steps, the correct conclusion may be a sample-efficiency advantage, not a learnability gap. Please report training curves for both methods under matched budgets.
5 comments
(Continued) W1/Q1: Can the claims that AMBER outperforms overwrite memory and matches refined overwrite be distinguished from noise?
Table R1. Paired success-rate differences (Qwen 3.5 9B). is AMBER minus the baseline, with a 95% CI margin (paired t-interval over tasks) and a sign-flip p-value. Bold: statistically significant.
| Comparison | Success rate (%) | (points) | 95% CI | p |
|---|---|---|---|---|
| AMBER vs. MemAgent | 36.85 vs. 32.76 | +4.09 | ± 3.66 | 0.034 |
| AMBER vs. Refined MemAgent | 36.85 vs. 35.59 | +1.26 | ± 3.95 | 0.58 |
Table R2. Repeated-run reliability (Qwen 3.5 9B). ≥1/5 is the percentage of tasks solved in at least one run. pass^k is the probability (%) that k of the five runs all succeed, averaged over tasks. rows give AMBER minus the baseline: 95% CIs for pass^k as ± margin (paired t-interval over tasks), and an exact McNemar p-value for ≥1/5. Bold: statistically significant.
| ≥1/5 | pass^1 | pass^2 | pass^3 | pass^4 | pass^5 | |
|---|---|---|---|---|---|---|
| AMBER | 53.5 | 36.9 | 29.3 | 25.6 | 22.8 | 20.5 |
| MemAgent | 48.8 | 32.8 | 24.7 | 20.5 | 17.6 | 15.7 |
| Refined MemAgent | 54.3 | 35.6 | 27.9 | 24.7 | 22.7 | 21.3 |
| vs. MemAgent | +4.7 (p = 0.26) | +4.1 ± 3.7 | +4.6 ± 4.0 | +5.1 ± 4.4 | +5.2 ± 5.1 | +4.8 ± 5.8 |
| vs. Refined MemAgent | −0.8 (p = 1.0) | +1.3 ± 4.0 | +1.4 ± 4.5 | +0.9 ± 5.0 | +0.1 ± 5.7 | −0.8 ± 6.4 |
Table R3. Where AMBER's success-rate gain over MemAgent comes from (Qwen 3.5 9B, 127 tasks). Tasks are grouped by which method solves them in at least one of the five runs. The contribution is the group's share of the 4.09-point gap in success rate over all tasks.
| Task group | Tasks | Success rate (%), AMBER vs. MemAgent | Contribution (points) |
|---|---|---|---|
| Solved by both | 55 | 78.2 vs. 71.6 | +2.83 |
| Solved only by AMBER | 13 | 29.2 vs. 0 | +2.99 |
| Solved only by MemAgent | 7 | 0 vs. 31.4 | −1.73 |
| Solved by neither | 52 | 0 vs. 0 | 0 |
| Total | 127 | 36.85 vs. 32.76 | +4.09 |
We have added this analysis to the appendix (Appendices D.1 and D.2), with pointers in Sections 5.1, 5.2 and 5.4. The abstract and the Introduction now report the pass^4 gain (+5.2 points), which is statistically significant, instead of the 5/5 gain (+4.8 points), which is not.
References
[1] Yao et al., "-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", ICLR25
W2/Q2: Why is there no comparison with Memory-R1 or Memory-as-Action, which learn memory operations from outcome rewards?
The comparison with MemAgent tests the learnability claim for learned retention:
- By learned retention, we mean that the policy decides which information stays in memory.
- Overwrite memory (MemAgent), context deletion (Memory-as-Action) and UPDATE/DELETE operations (Memory-R1) all use learned retention.
- AMBER instead keeps every entry by an append-only rule. MemAgent is one representative of learned retention. It shares AMBER's observations, model, SFT trajectories and RL recipe, so the retained context is the only axis that changes (the single-axis comparison the reviewer notes in S1).
Choice of evaluation benchmark:
- Memory-R1 and Memory-as-Action are evaluated primarily on QA benchmarks where the actions do not change the environment: Memory-R1 manages memory over multi-session dialogues. It is evaluated on LoCoMo, MSC and LongMemEval.
- Memory-as-Action edits the working context of a retrieval agent. It is evaluated on search QA (HotpotQA, 2WikiMultihopQA, Bamboogle, Frames and BrowseComp-Plus).
- We choose WebArena to evaluate memory mechanisms under demanding settings. Its websites require diverse interaction patterns. It is also a state-changing environment, where actions modify the website state. The agent must therefore remember what it has already done and the feedback it received.
- Hence, each method must be re-implemented and re-trained on WebArena.
Practical limitations:
- We re-implement and re-train MemAgent and WebAgent-R1 for fair comparisons with AMBER.
- Memory-as-Action uses its own RL algorithm (DCPO). It also needs a cold-start SFT dataset of memory-editing trajectories.
- This SFT dataset is built by running a large model (DeepSeek-V3.1 in the original work) as the agent, and keeping only successful episodes. On WebArena, this means paid API calls for a frontier model acting on the live websites.
- Memory-R1 trains a memory manager and an answer agent separately. Both need to be adapted to web trajectories. Moreover, Memory-R1 released the implementation on November 7th, 2026, prior to which the repository was a placeholder and hence difficult to reproduce.
- Hence, under a limited compute, cloud VM and API budget, we prioritize experiments with reasoning-and-action history (WebAgent-R1), overwrite memory (MemAgent), and a baseline with improved SFT data for overwrite memory (Refined MemAgent).
Training curves:
- As per the reviewer's suggestion, we add the RL training reward curves of AMBER, MemAgent and WebAgent-R1 (Appendix D.3 of the revised manuscript).
The gain arises during RL, not from the SFT data:
- AMBER and MemAgent use the same SFT trajectories, and the same RL recipe and budget (100 RL steps, group size G = 6).
- After SFT alone, the two methods have similar success rates (30.71% for AMBER vs. 31.50% for MemAgent, Table R4).
- RL then improves AMBER by 6.14 points (30.71 → 36.85), but MemAgent by only 1.26 points (31.50 → 32.76).
- Hence, the gap is not explained by the SFT data. It opens during RL, under the same budget for both methods.
Table R4. Success rate (%) after SFT and after SFT + RL (Qwen 3.5 9B).
| Method | SFT | SFT + RL | Gain from RL (points) |
|---|---|---|---|
| MemAgent | 31.50 | 32.76 | +1.26 |
| AMBER | 30.71 | 36.85 | +6.14 |
Ablations of AMBER:
- SFT-free variant: Table 1 in the manuscript reports AMBER's interface without any training (zero-shot Qwen 3.5 9B). It reaches 21.82%, compared with 34.22% for the zero-shot full-context agent. Hence, the base model cannot write step-wise memory reliably without the supervised warm-up. We did not run RL without SFT.
- Two-turn memory timing and entry-length penalty: each ablation needs a new RL run, of about 384 GPU-hours (8 GPUs × 48 hours). This is beyond our compute budget, and we leave them to future work. AMBER uses no length penalty: each entry is only capped at 1024 tokens, and AMBER's entries are shorter than MemAgent's memories (122.6 vs. 212.9 tokens per step).
We have clarified in the paper that our conclusion is shown for overwrite memory, as one form of learned retention (Section 2). We have also defined learned retention in the related work, citing Memory-R1 and Memory-as-Action (Section 2), and added both to the design-space table (Appendix C, Table 6). We have also stated why we chose WebArena (Section 5). The Limitations (Appendix A) now state that we test learned retention only as overwrite memory, and list the AMBER ablations we did not run.
Dear Reviewer K2W5, thank you for your feedback. We address your concerns as follows. We have incorporated them into the revised version, with the changes marked in red.
Scope of our claim:
As per the reviewer's suggestion, we run the paired per-task analysis. The evaluation set has 127 tasks (WebArena Lite without GitLab), each evaluated in five runs, or 635 episodes per method. For each comparison, we report:
AMBER outperforms overwrite memory: AMBER's success rate is 4.09 points higher than MemAgent's (36.85% vs. 32.76%, Table R1). This gain is statistically significant, as seen by its 95% CI, +4.09 ± 3.66 (+0.44 to +7.75), not containing zero (p = 0.034).
AMBER's advantage holds when a task must be solved in several runs:
Most of AMBER's gain comes from solving tasks more consistently:
Refined overwrite is comparable to AMBER, but needs a costlier annotation pipeline: