- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes AERCA, a method for training LLM agents on long, multi-step tasks. AERCA combines Adaptive Experience Regulation (AER) and Memory-Aware Credit Assignment (CA). AER stores past interactions, and groups them into i. how to complete a given task, and ii. how to find missing information, helping the agent to decide what to retrieve, use, keep or drop. CA then assigns extra reward to trajectories that make more progress than others in their rollout group, and assigns that reward to the periods of memory use during which the task score increased
The method is trained with GRPO on Qwen2.5-7B-Instruct and evaluated on ScienceWorld (45.08% full success, task score 30.70) and WebShop (78.13% success, score 87.83). The paper includes ablations and a comparison in which all RL baselines share the same memory system.
Critical strengths
- The authors ask a genuinely relevant and timely question, and their proposed method is intuitive.
- The authors share their code, and provide genuinely insightful ablation studies.
Critical weaknesses
-
AERCA is positioned as a method for long-horizons, yet it does not outperform the baselines for long horizon tasks, and on WebShop, it does not beat RetroAgent.
-
Several aspects of the experimental design are not entirely clear to me, and it looks like the authors did not provide the code for their baselines (see Questions).
Questions for authors
Q1: Why did the authors copy the results of MemRL, RetroAgent and EMPO for Webshop, but re-ran the experiments for ScienceWorld?
Q2: Could the authors please share their code for the baseline methods?
Q3: Could the authors please help me understand why the memory-based methods in ScienceWorld show such a weak performance (almost on par with the zero shot baseline), while they appear to perform substantially better on WebShop?
Q4: Could the authors re-run the baselines on WebShop so the two benchmarks are comparable?
Q5: Could the authors please clarify which precise scoring functions are used for the WebShop results in Table 1? In the code, it looks like a four-stage milestone score is used (webshop_milestones.py); was the same metric used for the results that were reported from other papers?
9 comments
Response to Reviewer DVrA (Part 2/2): ScienceWorld Baselines and Long-Horizon Performance
Q3: Regarding the Performance of Memory Baselines on ScienceWorld
Q3. Could the authors please help me understand why the memory-based methods in ScienceWorld show such a weak performance (almost on par with the zero shot baseline), while they appear to perform substantially better on WebShop?
Thank you for raising this question! Unlike WebShop’s search and purchase tasks, ScienceWorld tasks may require gathering materials, setting up experiments, and taking measurements. Partial progress earns credit, but Full Success requires completing every step required by the task. In Table 1, RetroAgent and EMPO² substantially improve Task Score over zero-shot, with smaller gains in Full Success.
| Method | Full Success (%) | Task Score |
|---|---|---|
| Zero-shot | 11.29 | −30.42 |
| RetroAgent | 13.60 | 24.85 |
| EMPO² | 15.49 | 26.52 |
| AERCA | 45.33 ± 1.27 | 30.27 ± 3.01 |
Two factors may help explain why these gains in Task Score do not translate into comparable gains in Full Success:
- Memory can help with individual steps without resolving the entire task. Retrieved experience may help the agent make progress or avoid costly errors, raising Task Score even if some requirements remain unmet.
- Retrieved guidance may not fit the current state. In Section 2.1, RetroAgent [1] follows retrieved advice to search the bedroom even though the thermometer is visible in the kitchen. This illustrates how guidance from an earlier task variation can hinder completion when it conflicts with the current observation.
Our evaluation protocol also matters when comparing with prior work. We require a final score of 100 for Full Success, whereas the released BEACON implementation [2] counts termination with a positive score as success. This stricter criterion can yield lower success rates. We also test on unseen variations across all task types and cap the number per type to keep coverage balanced (Appendices A.1–A.2).
After checking the five AERCA runs and recomputing the summary statistics, we updated the reported mean Full Success from 45.08% to 45.33% (sample SD: 1.27 percentage points). The evaluation protocol did not change.
W1: Regarding Whether AERCA Improves Performance on Long Tasks
W1. AERCA is positioned as a method for long-horizons, yet it does not outperform the baselines for long horizon tasks, ...
Thank you for raising this concern! To more robustly assess performance on long tasks, we expanded the single run comparison to five runs per method under the same conditions (see Figure 4(a) in our revised paper). The main evaluation allows up to 30 interaction steps per task.
Full Success (%) at 30 steps, mean ± standard deviation
| Method | Short | Medium | Long |
|---|---|---|---|
| GRPO | 50.18 ± 10.01 | 26.51 ± 4.22 | 37.76 ± 3.44 |
| GiGPO | 56.73 ± 5.95 | 28.37 ± 6.86 | 39.88 ± 2.54 |
| BEACON (MiGPO) | 58.55 ± 7.97 | 35.81 ± 6.06 | 34.91 ± 3.76 |
| AERCA | 63.27 ± 5.05 | 39.07 ± 7.61 | 40.87 ± 1.35 |
AERCA has the highest mean Full Success in all three groups and the highest mean Task Score on Long tasks (22.41 ± 2.84).
Individual Long results at 30 steps: Full Success (%) / Task Score. Runs are not paired across methods.
| Method | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 |
|---|---|---|---|---|---|
| GRPO | 36.02 / 22.85 | 41.61 / 19.60 | 40.99 / 27.11 | 33.54 / 0.53 | 36.65 / 17.61 |
| GiGPO | 43.48 / 30.83 | 38.51 / 16.37 | 40.37 / 13.98 | 40.37 / 15.15 | 36.65 / 7.71 |
| BEACON (MiGPO) | 39.13 / 22.13 | 36.02 / 22.06 | 33.54 / 5.11 | 36.65 / 22.01 | 29.19 / 5.69 |
| AERCA | 39.75 / 24.22 | 39.75 / 19.96 | 40.37 / 26.02 | 41.61 / 22.58 | 42.86 / 19.27 |
We also evaluated the same groups at 40 steps, with two runs per method.
Full Success (%) at 40 steps, mean ± standard deviation
| Method | Short | Medium | Long |
|---|---|---|---|
| GRPO | 50.91 ± 10.29 | 23.26 ± 13.16 | 42.86 ± 3.51 |
| GiGPO | 59.09 ± 3.86 | 34.88 ± 3.29 | 40.99 ± 0.88 |
| BEACON (MiGPO) | 70.91 ± 7.71 | 29.07 ± 1.64 | 37.89 ± 7.03 |
| AERCA | 56.36 ± 5.14 | 34.88 ± 6.58 | 45.96 ± 1.76 |
At 40 steps, MiGPO leads on both Short metrics. On Medium tasks, AERCA ties GiGPO in mean Full Success and leads in mean Task Score. On Long tasks, AERCA leads in both mean Full Success (45.96%) and Task Score (26.16). See Section 4.3 and Figure 4(a) for the main analysis and Appendix D.3, Table 19 for the 40-step results.
Thank you again for prompting us to revisit this comparison! AERCA leads on both Long task metrics at both budgets, with higher means in the 40-step evaluation. These additional runs strengthen the evidence beyond the original single run.
References
[1] RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback. 2026.
[2] Milestone-Guided Policy Learning for Long-Horizon Language Agents. ICML 2026.
Thank you for the detailed responses, new runs, controls, and baseline code. The controlled ScienceWorld comparison against GiGPO with the same Adaptive Experience Regulation provides evidence for the credit-assignment component, and the WebShop scoring clarification answers my Q5. Some key concerns remain. I may have misread parts of the responses or code and would welcome corrections.
-
Long-horizon claim. As far as I can tell, the revised horizon breakdown does not yet show a specific benefit on long horizons. At 30 steps, AERCA's mean lead over the strongest baseline is +4.72 pp on Short, +3.26 pp on Medium and +0.99 pp on Long tasks. The Long-task lead over GiGPO corresponds to about 1.6 of 161 instances per run and is smaller than the run-to-run standard deviations (1.35 and 2.54). Long tasks make up 62% of the test set but account for only about 16% of the overall gain over GiGPO. The 40-step results rest on two runs per method. Could you provide significance tests or confidence intervals for the horizon-specific claims? If they are not supported, I would suggest narrowing them.
-
Statistical support. Many of the reported differences seem comparable to the run-to-run variation at 3–5 runs: for example, +0.78 pp over BEACON with the same regulation on WebShop, and most Table 3 ablations (e.g., w/o Dynamic Regulation, 41.03 ± 4.51 vs. 45.33 ± 1.27). The Qwen3-4B results appear to be single runs, and the attribution controls use two runs each. Your response notes that distinct seeds were not selected, and the baseline launcher fixes env.seed=0, so the spread may understate seed-to-seed variance. (The overall ScienceWorld gain over GiGPO does pass a simple Welch test, p ≈ 0.02.) Could you report significance tests or confidence intervals for the main comparisons (Tables 2–3 and the horizon breakdown), ideally with independently seeded runs and paired per-instance comparisons against GiGPO and BEACON under the same regulation? Where differences are not significant, I would suggest tempering the claims.
-
Reported runs. The original 45.08% / 30.70 appears to correspond exactly to the mean of runs 1–4 in your per-run table, while the responses describe the change both as an added run and as a recomputation over five runs. I also could not obtain the original GiGPO result (40.41%) from any three of the five runs now listed, and the GiGPO run in Table 17 (39.77%) does not seem to be among them. Separately, several Table 1 means do not seem compatible with five runs over 259 instances. If I understand correctly, a five-run mean must be a multiple of 1/1295 (about 0.077 pp): for example, BEACON's 21.67 falls between the closest attainable values, 21.62 and 21.70. The same holds for RetroAgent (13.60), EMPO² (15.49) and GiGPO (32.24). There may be a simple explanation, such as a different number of runs or missing episodes. Could you clarify how many runs each number is based on and whether any runs were replaced, and provide per-run counts for Table 1?
-
Baseline reproductions. If I read the RetroAgent ScienceWorld code correctly (ScienceWorldEnvironmentManager.step), the strict 10 * won reward is applied only during evaluation, and during training the per-step score changes are summed. RetroAgent would then be trained on task score rather than strict completion. This seems to differ from Appendix A.2 and might explain the pattern behind my Q3 (13.60% full success, versus 28.19% for plain GRPO). For EMPO² on WebShop, the public EMPO² recipe does not appear to include WebShop, so the pipeline seems to be your own port. As your response notes, each training goal is drawn once; since the port also keys tips by goal ID, tips would essentially never be reused, so the matched 65.23% may not reflect EMPO²'s memory mechanism. Could you comment on both, and document any deviations in the paper?
-
Paper vs. code. The evaluation README in the archive appears to state that WebShop evaluation goals differ per checkpoint (steps 100 and 150 share 36 of 128 goals), whereas Appendix A.1 describes one fixed set. If so, the move from 100 to 150 iterations also changed the test set. Appendix A.3 mentions the "easy" ScienceWorld simplification for evaluation, while protocol.json and the launcher appear to set none. Could you clarify which settings were used?
-
Code. The archive does not appear to include GRPO, GiGPO, the shared-AER baselines in Table 2, EMPO² on ScienceWorld, or the new controls, and its READMEs say end-to-end training is unverified. Could you add these, or indicate which results cannot be reproduced from the released code? Please also add the matched reruns and controls to the paper; I could not find them in Tables 1–2 or Appendix A.4.
For now I am keeping my score. Clear answers on points 1–4 would help me reassess.
Response to Reviewer DVrA (Part 1/2): Baseline Comparisons and WebShop Evaluation
Thank you for your careful review and thoughtful feedback! We appreciate your comments on the research question, the design of AERCA, and the ablation studies. Your questions prompted us to add baseline comparisons, clarify our evaluation protocols, and revisit the analysis of longer tasks. We address each point in the two comments below and would be happy to discuss any remaining questions.
Q1 and Q4: Regarding Baseline Selection and WebShop Reproduction
Thank you for asking about the different sources. On ScienceWorld, MemRL and RetroAgent did not report directly comparable results, so we reran them under our evaluation protocol. We also reran EMPO² [1] because its main results report Average Return rather than Full Success, which requires a final score of 100 in our evaluation.
On WebShop, the published results for SkillRL, MemRL, and Mem0+GRPO come from SkillRL [2]. Our revised setup uses the same base model as SkillRL and increases the training budget from 100 to 150 iterations, in line with the budgets reported by SkillRL and RetroAgent [3]. We did not rerun SkillRL's full pipeline because regenerating its skills and supervised training data would require access to OpenAI o3.
Following your suggestion, we prioritized rerunning RetroAgent and EMPO². Both use the same setup as AERCA: Qwen2.5-7B-Instruct, 16 prompts per iteration, 8 rollouts per prompt, 150 training iterations, a limit of 15 interaction steps, and the same 128 evaluation tasks. See Tables 1–2 and Appendix A.4.
AERCA's mean Success is below RetroAgent's published value, but it leads both baselines on both metrics under the matched protocol. We discuss the gaps between the published results and our reruns below:
Q2: Regarding the Availability of Baseline Code
Following your suggestion, we added
AERCA-baselines.zipto the supplementary materials. It includes the adapted implementations, configurations, and instructions for training and evaluation.Q5: Regarding How WebShop Results Are Scored
Thank you for pointing out this ambiguity. The milestones in
webshop_milestones.py, inspired by BEACON [4], are used only to assign memory credit during training. For evaluation, Success is the percentage of episodes with a final WebShop reward of 1, and Task Score is the mean final reward multiplied by 100.We recomputed the WebShop Task Scores for our own runs in Tables 1–3 and kept the results from prior work as published, without assuming identical evaluation implementations. We clarified the distinction in Appendix A.1 and will update the WebShop evaluation code to clearly separate the reported Task Score from training progress.
Thank you again for these suggestions. They helped us add direct baseline comparisons and make the WebShop scoring easier to interpret.
References
[1] Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization. ICLR 2026.
[2] SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. 2026.
[3] RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback. 2026.
[4] Milestone-Guided Policy Learning for Long-Horizon Language Agents. ICML 2026.