acceptodds
Under review as a conference paper at ICLR 2027

AERCA: Adaptive Experience Regulation and Credit Assignment for Long-Horizon Agents

Abstract

Reliable end-to-end completion remains challenging for LLM agents in long-horizon tasks. We identify two recurring obstacles. First, retrieved experience can remain influential after becoming poorly matched to the agent's evolving state or information need, causing previously useful guidance to bias subsequent decisions. Second, long-horizon RL exhibits a credit assignment dilemma: some credit schemes improve strict completion while leaving many unresolved trajectories with limited progress, whereas milestone-aware credit can substantially improve intermediate progress without yielding correspondingly stronger completion. To address these challenges, we introduce AERCA (Adaptive Experience Regulation and Memory-Aware Credit Assignment), which couples adaptive experience use during execution with selective credit assignment during policy learning. AERCA structures past interactions into Task-level Completion Experience for how to complete and Gap-level Cognition Experience for how to know, while dynamically regulating whether retrieved experience should be used, retained, or discarded as execution evolves. For policy learning, relative trajectory progress determines how much memory-aware credit a trajectory receives, while breakthroughs during memory use events determine where that credit is assigned. On ScienceWorld, AERCA achieves full success with a task score of , while reaching success and a task score of on WebShop. More importantly, even when all compared methods use our Adaptive Experience Regulation, AERCA improves full success over GiGPO by percentage points on ScienceWorld, isolating the contribution of Memory-Aware Credit Assignment. Ablation studies further support the complementary contributions of the two components. Our code and datasets are available at https://anonymous.4open.science/r/AERCA-6FF2/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, long-horizon task completion, reinforcement learning, credit assignment, agent memory, experience regulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DVrAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes AERCA, a method for training LLM agents on long, multi-step tasks. AERCA combines Adaptive Experience Regulation (AER) and Memory-Aware Credit Assignment (CA). AER stores past interactions, and groups them into i. how to complete a given task, and ii. how to find missing information, helping the agent to decide what to retrieve, use, keep or drop. CA then assigns extra reward to trajectories that make more progress than others in their rollout group, and assigns that reward to the periods of memory use during which the task score increased

The method is trained with GRPO on Qwen2.5-7B-Instruct and evaluated on ScienceWorld (45.08% full success, task score 30.70) and WebShop (78.13% success, score 87.83). The paper includes ablations and a comparison in which all RL baselines share the same memory system.

Critical strengths
  1. The authors ask a genuinely relevant and timely question, and their proposed method is intuitive.
  2. The authors share their code, and provide genuinely insightful ablation studies.
Critical weaknesses
  1. AERCA is positioned as a method for long-horizons, yet it does not outperform the baselines for long horizon tasks, and on WebShop, it does not beat RetroAgent.

  2. Several aspects of the experimental design are not entirely clear to me, and it looks like the authors did not provide the code for their baselines (see Questions).

Questions for authors

Q1: Why did the authors copy the results of MemRL, RetroAgent and EMPO for Webshop, but re-ran the experiments for ScienceWorld?

Q2: Could the authors please share their code for the baseline methods?

Q3: Could the authors please help me understand why the memory-based methods in ScienceWorld show such a weak performance (almost on par with the zero shot baseline), while they appear to perform substantially better on WebShop?

Q4: Could the authors re-run the baselines on WebShop so the two benchmarks are comparable?

Q5: Could the authors please clarify which precise scoring functions are used for the WebShop results in Table 1? In the code, it looks like a four-stage milestone score is used (webshop_milestones.py); was the same metric used for the results that were reported from other papers?

9 comments
AuthorsOpenReview ↗

Response to Reviewer DVrA (Part 1/2): Baseline Comparisons and WebShop Evaluation

Thank you for your careful review and thoughtful feedback! We appreciate your comments on the research question, the design of AERCA, and the ablation studies. Your questions prompted us to add baseline comparisons, clarify our evaluation protocols, and revisit the analysis of longer tasks. We address each point in the two comments below and would be happy to discuss any remaining questions.

Q1 and Q4: Regarding Baseline Selection and WebShop Reproduction

Q1. Why did the authors copy the results of MemRL, RetroAgent and EMPO for Webshop, but re-ran the experiments for ScienceWorld?

Q4. Could the authors re-run the baselines on WebShop so the two benchmarks are comparable?

W1. ... and on WebShop, it does not beat RetroAgent.

Thank you for asking about the different sources. On ScienceWorld, MemRL and RetroAgent did not report directly comparable results, so we reran them under our evaluation protocol. We also reran EMPO² [1] because its main results report Average Return rather than Full Success, which requires a final score of 100 in our evaluation.

On WebShop, the published results for SkillRL, MemRL, and Mem0+GRPO come from SkillRL [2]. Our revised setup uses the same base model as SkillRL and increases the training budget from 100 to 150 iterations, in line with the budgets reported by SkillRL and RetroAgent [3]. We did not rerun SkillRL's full pipeline because regenerating its skills and supervised training data would require access to OpenAI o3.

Following your suggestion, we prioritized rerunning RetroAgent and EMPO². Both use the same setup as AERCA: Qwen2.5-7B-Instruct, 16 prompts per iteration, 8 rollouts per prompt, 150 training iterations, a limit of 15 interaction steps, and the same 128 evaluation tasks. See Tables 1–2 and Appendix A.4.

MethodPublished Success (%)Published Task ScoreMatched Success (%)Matched Task ScoreRuns
RetroAgent82.3088.9072.66 ± 1.1081.59 ± 3.542
EMPO²76.9088.3065.23 ± 6.0877.84 ± 2.622
AERCA——79.22 ± 1.0588.91 ± 0.765

AERCA's mean Success is below RetroAgent's published value, but it leads both baselines on both metrics under the matched protocol. We discuss the gaps between the published results and our reruns below:

  • RetroAgent. We used the official RetroAgent implementation. A similar WebShop reproduction gap was reported in GitHub Issue #2, but its cause remains unclear.
  • EMPO². During training, EMPO² can reuse tips from earlier attempts when revisiting a task and learn a policy that acts without external memory at evaluation [1]. In our protocol, each training task is sampled only once, and evaluation uses unseen tasks. This may limit opportunities to reuse tips on the same task, making our setting potentially less favorable to EMPO². We therefore interpret the result as performance under our protocol.

Q2: Regarding the Availability of Baseline Code

W2. ... it looks like the authors did not provide the code for their baselines (see Questions).

Q2. Could the authors please share their code for the baseline methods?

Following your suggestion, we added AERCA-baselines.zip to the supplementary materials. It includes the adapted implementations, configurations, and instructions for training and evaluation.

Q5: Regarding How WebShop Results Are Scored

Q5. Could the authors please clarify which precise scoring functions are used for the WebShop results in Table 1? In the code, it looks like a four-stage milestone score is used (webshop_milestones.py); was the same metric used for the results that were reported from other papers?

Thank you for pointing out this ambiguity. The milestones in webshop_milestones.py, inspired by BEACON [4], are used only to assign memory credit during training. For evaluation, Success is the percentage of episodes with a final WebShop reward of 1, and Task Score is the mean final reward multiplied by 100.

We recomputed the WebShop Task Scores for our own runs in Tables 1–3 and kept the results from prior work as published, without assuming identical evaluation implementations. We clarified the distinction in Appendix A.1 and will update the WebShop evaluation code to clearly separate the reported Task Score from training progress.

Thank you again for these suggestions. They helped us add direct baseline comparisons and make the WebShop scoring easier to interpret.

References

[1] Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization. ICLR 2026.

[2] SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. 2026.

[3] RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback. 2026.

[4] Milestone-Guided Policy Learning for Long-Horizon Language Agents. ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer DVrA (Part 2/2): ScienceWorld Baselines and Long-Horizon Performance

Q3: Regarding the Performance of Memory Baselines on ScienceWorld

Q3. Could the authors please help me understand why the memory-based methods in ScienceWorld show such a weak performance (almost on par with the zero shot baseline), while they appear to perform substantially better on WebShop?

Thank you for raising this question! Unlike WebShop’s search and purchase tasks, ScienceWorld tasks may require gathering materials, setting up experiments, and taking measurements. Partial progress earns credit, but Full Success requires completing every step required by the task. In Table 1, RetroAgent and EMPO² substantially improve Task Score over zero-shot, with smaller gains in Full Success.

MethodFull Success (%)Task Score
Zero-shot11.29−30.42
RetroAgent13.6024.85
EMPO²15.4926.52
AERCA45.33 ± 1.2730.27 ± 3.01

Two factors may help explain why these gains in Task Score do not translate into comparable gains in Full Success:

  1. Memory can help with individual steps without resolving the entire task. Retrieved experience may help the agent make progress or avoid costly errors, raising Task Score even if some requirements remain unmet.
  2. Retrieved guidance may not fit the current state. In Section 2.1, RetroAgent [1] follows retrieved advice to search the bedroom even though the thermometer is visible in the kitchen. This illustrates how guidance from an earlier task variation can hinder completion when it conflicts with the current observation.

Our evaluation protocol also matters when comparing with prior work. We require a final score of 100 for Full Success, whereas the released BEACON implementation [2] counts termination with a positive score as success. This stricter criterion can yield lower success rates. We also test on unseen variations across all task types and cap the number per type to keep coverage balanced (Appendices A.1–A.2).

After checking the five AERCA runs and recomputing the summary statistics, we updated the reported mean Full Success from 45.08% to 45.33% (sample SD: 1.27 percentage points). The evaluation protocol did not change.

W1: Regarding Whether AERCA Improves Performance on Long Tasks

W1. AERCA is positioned as a method for long-horizons, yet it does not outperform the baselines for long horizon tasks, ...

Thank you for raising this concern! To more robustly assess performance on long tasks, we expanded the single run comparison to five runs per method under the same conditions (see Figure 4(a) in our revised paper). The main evaluation allows up to 30 interaction steps per task.

Full Success (%) at 30 steps, mean ± standard deviation

MethodShortMediumLong
GRPO50.18 ± 10.0126.51 ± 4.2237.76 ± 3.44
GiGPO56.73 ± 5.9528.37 ± 6.8639.88 ± 2.54
BEACON (MiGPO)58.55 ± 7.9735.81 ± 6.0634.91 ± 3.76
AERCA63.27 ± 5.0539.07 ± 7.6140.87 ± 1.35

AERCA has the highest mean Full Success in all three groups and the highest mean Task Score on Long tasks (22.41 ± 2.84).

Individual Long results at 30 steps: Full Success (%) / Task Score. Runs are not paired across methods.

MethodRun 1Run 2Run 3Run 4Run 5
GRPO36.02 / 22.8541.61 / 19.6040.99 / 27.1133.54 / 0.5336.65 / 17.61
GiGPO43.48 / 30.8338.51 / 16.3740.37 / 13.9840.37 / 15.1536.65 / 7.71
BEACON (MiGPO)39.13 / 22.1336.02 / 22.0633.54 / 5.1136.65 / 22.0129.19 / 5.69
AERCA39.75 / 24.2239.75 / 19.9640.37 / 26.0241.61 / 22.5842.86 / 19.27

We also evaluated the same groups at 40 steps, with two runs per method.

Full Success (%) at 40 steps, mean ± standard deviation

MethodShortMediumLong
GRPO50.91 ± 10.2923.26 ± 13.1642.86 ± 3.51
GiGPO59.09 ± 3.8634.88 ± 3.2940.99 ± 0.88
BEACON (MiGPO)70.91 ± 7.7129.07 ± 1.6437.89 ± 7.03
AERCA56.36 ± 5.1434.88 ± 6.5845.96 ± 1.76

At 40 steps, MiGPO leads on both Short metrics. On Medium tasks, AERCA ties GiGPO in mean Full Success and leads in mean Task Score. On Long tasks, AERCA leads in both mean Full Success (45.96%) and Task Score (26.16). See Section 4.3 and Figure 4(a) for the main analysis and Appendix D.3, Table 19 for the 40-step results.

Thank you again for prompting us to revisit this comparison! AERCA leads on both Long task metrics at both budgets, with higher means in the 40-step evaluation. These additional runs strengthen the evidence beyond the original single run.

References

[1] RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback. 2026.

[2] Milestone-Guided Policy Learning for Long-Horizon Language Agents. ICML 2026.

Thank you for the detailed responses, new runs, controls, and baseline code. The controlled ScienceWorld comparison against GiGPO with the same Adaptive Experience Regulation provides evidence for the credit-assignment component, and the WebShop scoring clarification answers my Q5. Some key concerns remain. I may have misread parts of the responses or code and would welcome corrections.

  1. Long-horizon claim. As far as I can tell, the revised horizon breakdown does not yet show a specific benefit on long horizons. At 30 steps, AERCA's mean lead over the strongest baseline is +4.72 pp on Short, +3.26 pp on Medium and +0.99 pp on Long tasks. The Long-task lead over GiGPO corresponds to about 1.6 of 161 instances per run and is smaller than the run-to-run standard deviations (1.35 and 2.54). Long tasks make up 62% of the test set but account for only about 16% of the overall gain over GiGPO. The 40-step results rest on two runs per method. Could you provide significance tests or confidence intervals for the horizon-specific claims? If they are not supported, I would suggest narrowing them.

  2. Statistical support. Many of the reported differences seem comparable to the run-to-run variation at 3–5 runs: for example, +0.78 pp over BEACON with the same regulation on WebShop, and most Table 3 ablations (e.g., w/o Dynamic Regulation, 41.03 ± 4.51 vs. 45.33 ± 1.27). The Qwen3-4B results appear to be single runs, and the attribution controls use two runs each. Your response notes that distinct seeds were not selected, and the baseline launcher fixes env.seed=0, so the spread may understate seed-to-seed variance. (The overall ScienceWorld gain over GiGPO does pass a simple Welch test, p ≈ 0.02.) Could you report significance tests or confidence intervals for the main comparisons (Tables 2–3 and the horizon breakdown), ideally with independently seeded runs and paired per-instance comparisons against GiGPO and BEACON under the same regulation? Where differences are not significant, I would suggest tempering the claims.

  3. Reported runs. The original 45.08% / 30.70 appears to correspond exactly to the mean of runs 1–4 in your per-run table, while the responses describe the change both as an added run and as a recomputation over five runs. I also could not obtain the original GiGPO result (40.41%) from any three of the five runs now listed, and the GiGPO run in Table 17 (39.77%) does not seem to be among them. Separately, several Table 1 means do not seem compatible with five runs over 259 instances. If I understand correctly, a five-run mean must be a multiple of 1/1295 (about 0.077 pp): for example, BEACON's 21.67 falls between the closest attainable values, 21.62 and 21.70. The same holds for RetroAgent (13.60), EMPO² (15.49) and GiGPO (32.24). There may be a simple explanation, such as a different number of runs or missing episodes. Could you clarify how many runs each number is based on and whether any runs were replaced, and provide per-run counts for Table 1?

  4. Baseline reproductions. If I read the RetroAgent ScienceWorld code correctly (ScienceWorldEnvironmentManager.step), the strict 10 * won reward is applied only during evaluation, and during training the per-step score changes are summed. RetroAgent would then be trained on task score rather than strict completion. This seems to differ from Appendix A.2 and might explain the pattern behind my Q3 (13.60% full success, versus 28.19% for plain GRPO). For EMPO² on WebShop, the public EMPO² recipe does not appear to include WebShop, so the pipeline seems to be your own port. As your response notes, each training goal is drawn once; since the port also keys tips by goal ID, tips would essentially never be reused, so the matched 65.23% may not reflect EMPO²'s memory mechanism. Could you comment on both, and document any deviations in the paper?

  5. Paper vs. code. The evaluation README in the archive appears to state that WebShop evaluation goals differ per checkpoint (steps 100 and 150 share 36 of 128 goals), whereas Appendix A.1 describes one fixed set. If so, the move from 100 to 150 iterations also changed the test set. Appendix A.3 mentions the "easy" ScienceWorld simplification for evaluation, while protocol.json and the launcher appear to set none. Could you clarify which settings were used?

  6. Code. The archive does not appear to include GRPO, GiGPO, the shared-AER baselines in Table 2, EMPO² on ScienceWorld, or the new controls, and its READMEs say end-to-end training is unverified. Could you add these, or indicate which results cannot be reproduced from the released code? Please also add the matched reruns and controls to the paper; I could not find them in Tables 1–2 or Appendix A.4.

For now I am keeping my score. Clear answers on points 1–4 would help me reassess.

Reviewer_NN4dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies two difficulties in long-horizon agent learning: retrieved experience can become inappropriate as execution changes, and improved intermediate progress does not always translate into complete task success. AERCA combines Adaptive Experience Regulation with Memory-Aware Credit Assignment.

The regulation component separates successful task-completion experience from local procedures extracted from informative segments of failed trajectories. The agent explicitly retrieves, adopts, retains, or drops experience during execution. The learning component constructs an additional reward from above-group-average peak progress, then distributes this credit across memory-use events according to environment-verified breakthroughs during their active intervals.

The paper reports 45.08% full success on ScienceWorld and 78.13% success on WebShop. Under shared experience regulation, AERCA improves ScienceWorld full success over GiGPO by 4.67 percentage points. Ablations examine the experience types, dynamic regulation, and event-level credit attribution, with an additional backbone experiment in Appendix D.2.

Critical strengths
  • The motivation is practical. Retrieved guidance can remain relevant to the overall task while becoming inappropriate for the current observation. Explicitly controlling its continued use is a useful addition to retrieval and memory construction.

  • The controlled comparison in Table 2 is an important strength. GRPO, GiGPO, BEACON, and AERCA share the experience-construction, retrieval, and regulation mechanism, making the additional benefit of the learning rule more informative than a comparison between completely different agent systems.

  • The paper carefully distinguishes intermediate progress from full completion. Appendix A.2 explains the difference between positive-score termination and a final ScienceWorld score of 100, and aligns the terminal success criterion across the main comparisons. This is necessary for interpreting the proposed completion gains.

  • The evaluation includes five-run aggregate results, component ablations, and another policy backbone. On Qwen3-4B, AERCA improves full success from 37.84% to 41.70% over GRPO with the same experience regulation. The detailed lifecycle and credit-distribution statistics also make the method easier to inspect, although they do not independently establish the proposed mechanism.

Critical weaknesses
  1. The connection between temporal reward attribution and GRPO is underspecified. Equation (5) preserves the total memory credit of each trajectory, while Section 3.3 states that the augmented rewards are optimized using standard GRPO. If the implementation sums these rewards into one trajectory return and broadcasts its normalized advantage across the response, moving the same credit between steps would not change that advantage. A step-dependent return, advantage, or loss mask could make attribution effective, but this rule is not specified. Table 3 already includes a useful no-attribution control; my concern is how its optimization differs mathematically from the full method.

  2. The evidence does not fully isolate what is specific to memory-event attribution. Appendix C.2 provides detailed association statistics, but these do not establish whether the selected memory intervals are more useful than other progress-related intervals. Among positive-breakthrough events with recoverable latency, 94.81% have their first breakthrough at the adoption step. This is consistent with useful memory, but can also reflect adopting memory when the agent is already prepared to make progress. The no-attribution ablation supports localization, while a matched randomized-interval or generic breakthrough-localization control would help understand better why memory-event boundaries matter.

  3. The strongest evidence for improved completion comes from the shorter horizon groups. In Figure 4, AERCA reaches 39.75% full success on Long tasks, compared with 41.61% for GRPO and 40.37% for GiGPO. The authors appropriately describe this as competitive, so I would not interpret it as a contradictory claim. However, it limits the evidence that the method particularly improves the longest tasks. Table 3 and the horizon breakdown also omit uncertainty estimates, even though the main results include five-run standard deviations. Repeated ablations and experiments with larger interaction budgets would make this part more convincing.

Questions for authors
  1. Could you specify exactly how the augmented per-step rewards are converted into advantages and token-level updates? Equation (5) preserves total memory credit. If GRPO aggregates rewards into one trajectory return, redistributing that credit would leave the rollout-level advantage unchanged. I am not fully clear how the implementation makes temporal attribution affect the update, particularly relative to the no-attribution control in Table 3.

  2. How is memory credit handled in the ablation without dynamic regulation? Appendix A.5 removes the active experience state and USE/KEEP/DROP decisions. Does this variant still use same event-attribution rule, and how are event boundaries defined? This would clarify whether the comparison changes only experience regulation or also the credit mechanism.

  3. Could you include a control that assigns credit to randomized intervals with matched duration and total credit, or localizes it around breakthroughs without using memory boundaries? The existing no-attribution ablation is useful. This additional comparison would help understand better whether memory-event boundaries provide a specific benefit beyond temporal localization.

  4. Could you clarify the number of evaluation rollouts per instance and how full success is aggregated across instances and runs? The paper specifies 259 fixed ScienceWorld instances and five-run main results, but I may be missing some detail of the averaging procedure. Per-run completion counts and denominators would help reconcile the reported 45.08% with these settings.

  5. Could you report variation across runs for Table 3 and the horizon breakdown, as already provided for the main results? For Long group, AERCA is competitive rather than better than GRPO or GiGPO. Does this comparison change when the interaction budget increases beyond 30 steps, and is the trend consistent across runs?

4 comments
AuthorsOpenReview ↗

Response to Reviewer NN4d (Part 1/3)

Thank you for your careful reading and constructive feedback. We have revised the description of memory credit to remove ambiguity in the original manuscript and added experiments to address your questions. We respond to each point below.

Q1: Regarding How Memory Credit Affects the GRPO Update

W1. Table 3 already includes a useful no-attribution control; my concern is how its optimization differs mathematically from the full method.

Q1. Could you specify exactly how the augmented per-step rewards are converted into advantages and token-level updates?

Following prior work on agent RL [1–3], we treat each interaction step as a separate response for optimization. The response at step tt in rollout ii receives the episode reward plus its memory credit: r~i,t=Rienv+ri,tmem\tilde r_{i,t}=R_i^{\mathrm{env}}+r_{i,t}^{\mathrm{mem}}.

We compute the GRPO advantage [4] as Ai,t=(r~i,t−μg)/(σg+ϵ)A_{i,t}=(\tilde r_{i,t}-\mu_g)/(\sigma_g+\epsilon), where the mean and standard deviation are computed across all valid step responses in the same rollout group. We apply each response’s advantage to its valid generated tokens. We do not sum the rewards into one trajectory return, so moving credit between steps can change the update even though Equation (5) preserves the total credit.

  • Credit from Relative Progress determines how much credit a rollout receives. We compare its peak progress with the average peak progress of the rollouts with memory in the same group. Only rollouts that exceed this average receive positive credit.
  • Credit Attribution to Memory Use determines where that credit goes. An event earns a share if the agent reaches a new progress peak while the memory is active. Larger gains earn a larger share. We spread each event’s share evenly from retrieval until the event ends.

In Table 3, w/o Credit Attribution to Memory Use removes only the second component. It keeps the credit from relative progress but spreads it evenly over the initial responses, using the first peak score to set the cutoff. w/o Memory-Aware Credit Assignment removes both components. This gives plain GRPO trained only on the environment reward, with the same experience retrieval and regulation as AERCA. We clarified the update in Section 3.3.

Q2: Regarding How We Assign Credit Without Dynamic Regulation

How is memory credit handled in the ablation without dynamic regulation? Appendix A.5 removes the active experience state and USE/KEEP/DROP decisions. Does this variant still use same event-attribution rule, and how are event boundaries defined?

After the first retrieval, the retrieved experience stays fixed as guidance. If the agent requests another retrieval, we do not carry it out.

We still record USE, KEEP, and DROP to assign credit. A valid USE starts a new event and closes any previous one. KEEP leaves the event open. DROP or the end of the episode closes it. Thus, DROP can close an event without removing the fixed guidance. We apply the same credit assignment rule to the recorded events and clarify this distinction in Appendix A.5.

You may also be interested in whether memory credit helps when the agent uses only one type of experience. We added Table 8 in Appendix A.5 to compare GRPO with and without memory credit for TCE alone and for GCE alone.

ExperienceTrainingScienceWorld Full Success (%)ScienceWorld Task ScoreWebShop Success (%)WebShop Task Score
TCE onlyGRPO39.38 ± 1.0912.34 ± 3.6871.35 ± 1.2084.31 ± 2.26
TCE onlyGRPO + memory credit39.77 ± 5.0221.10 ± 7.5377.60 ± 1.2087.43 ± 0.49
GCE onlyGRPO29.15 ± 2.46−1.59 ± 1.0763.80 ± 1.8078.78 ± 3.61
GCE onlyGRPO + memory credit36.29 ± 4.1817.59 ± 10.3068.75 ± 6.7784.05 ± 2.48

Results are mean ± SD. The GRPO controls use two runs on ScienceWorld and three on WebShop. The variants with memory credit use three runs on each benchmark. Mean success and Task Score are higher with memory credit in all four comparisons.

References
[1] Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025.
[2] Milestone-Guided Policy Learning for Long-Horizon Language Agents. ICML 2026.
[3] RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents. ICLR 2026.
[4] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv 2024.

AuthorsOpenReview ↗

Response to Reviewer NN4d (Part 2/3)

Q3: Regarding Whether Memory Event Boundaries Matter

W2. This is consistent with useful memory, but can also reflect adopting memory when the agent is already prepared to make progress.

Q3. Could you include a control that assigns credit to randomized intervals with matched duration and total credit, or localizes it around breakthroughs without using memory boundaries?

We agree that the agent may adopt memory when it is already in a position to make progress. Our motivation is that finding and using relevant experience can help explain why some rollouts make more progress than others on the same task. We first check whether a rollout’s peak progress exceeds the group average. We then look for new peaks while each memory is active. In Appendix C.2, 42.13% of training rollouts receive positive memory credit. Within those rollouts, 46.18% of valid memory events contain no new breakthrough and receive no breakthrough credit.

Following your suggestion, we tested two ways to change where credit goes while keeping the total credit fixed within each rollout. Both controls use the same event detection and GRPO procedure.

  • Shuffled Event: We keep the original event intervals and shuffle the credit amounts among the events in the same rollout.
  • Random Interval: For each event, we keep the credit amount and the number of steps the same, but choose a random valid starting step in the same rollout.

We trained each control twice for 100 iterations and evaluated it on the same 259 ScienceWorld instances.

MethodRunFull Success (%)CompletionsTask Score
AERCA5 runs, mean ± SD45.33 ± 1.27—30.27 ± 3.01
Shuffled Event140.15104/25919.80
Shuffled Event240.15104/25926.92
Shuffled EventMean ± SD40.15 ± 0.00—23.36 ± 5.03
Random Interval135.5292/25922.11
Random Interval238.61100/25926.67
Random IntervalMean ± SD37.07 ± 2.18—24.39 ± 3.22

AERCA’s mean Full Success is 5.18 percentage points higher than Shuffled Event and 8.26 points higher than Random Interval. Its mean Task Score is also higher than both. These results support using breakthroughs to determine each event’s share and keeping that credit in the original event intervals, rather than shuffling the shares or moving the intervals at random.

Thank you for suggesting the matched control. It helped us test whether the placement of credit matters when its total stays the same.

AuthorsOpenReview ↗

Response to Reviewer NN4d (Part 3/3)

Q4: Regarding How We Calculate Full Success Across Runs

Could you clarify the number of evaluation rollouts per instance and how full success is aggregated across instances and runs?

Per-run completion counts and denominators would help reconcile the reported 45.08% with these settings.

For each of the five AERCA training runs, we evaluate the resulting policy with one rollout per instance on the same 259 ScienceWorld instances. Each rollout allows up to 30 interactions with the environment and counts as a full success if its final environment score is 100. For example, Run 2 completed 120 of the 259 instances, yielding a Full Success rate of 46.33% when rounded to two decimal places. Task Score is the average final environment score across all 259 instances in that run.

AERCA runCompletionsFull Success (%)Task Score
1112/25943.2430.52
2120/25946.3333.49
3117/25945.1732.65
4118/25945.5626.13
5120/25946.3328.55
Mean ± sample SD—45.33 ± 1.2730.27 ± 3.01

We compute both metrics for each run, then report the mean and sample standard deviation across the five runs. After checking the individual results and recomputing the aggregate, we updated the reported Full Success from 45.08% to 45.33 ± 1.27%. The evaluation protocol is unchanged.

Q5: Regarding Performance on Long Tasks and Variation Across Runs

W3. Table 3 and the horizon breakdown also omit uncertainty estimates, even though the main results include five-run standard deviations.

Q5. For Long group, AERCA is competitive rather than better than GRPO or GiGPO. Does this comparison change when the interaction budget increases beyond 30 steps, and is the trend consistent across runs?

Thank you for pointing out that the original manuscript did not show how these results vary across runs. To estimate average performance more reliably, we increased the number of runs for each baseline in Table 2 from three to five. Figure 4 now shows the mean and standard deviation across five runs for each horizon group. Table 3 reports mean ± standard deviation over five runs for full AERCA and three runs for each ablation.

To see what happens with a larger budget, we trained and evaluated each method twice with up to 40 interactions per episode. We kept the same 161 Long instances at both budgets. MiGPO is the optimizer used in BEACON [1].

Method30 steps: mean ± SD (%)40 steps: Run 1 / Run 2 (%)40 steps: mean ± SD (%)
GRPO37.76 ± 3.4445.34 / 40.3742.86 ± 3.51
GiGPO39.88 ± 2.5440.37 / 41.6140.99 ± 0.88
MiGPO34.91 ± 3.7632.92 / 42.8637.89 ± 7.03
AERCA40.87 ± 1.3544.72 / 47.2045.96 ± 1.76

At 30 steps, AERCA and GiGPO have similar Full Success rates on Long tasks. At 40 steps, the results for GiGPO and MiGPO suggest that finer credit assignment can improve overall completion without necessarily helping on Long tasks. AERCA improves on GRPO in both the Short and Medium groups and achieves the highest mean Full Success on Long tasks (45.96%, 3.10 percentage points above GRPO). Both AERCA runs outperform both runs of GiGPO and both runs of MiGPO on Long tasks.

Overall results at 40 steps on all 259 instances. Run numbers do not indicate paired seeds. The full horizon breakdown is in Appendix D.3, Table 19.

MethodRun 1: Full Success (%) / ScoreRun 2: Full Success (%) / ScoreFull Success (%), mean ± SDScore, mean ± SD
GRPO39.77 / 16.1242.86 / 17.5641.31 ± 2.1816.84 ± 1.02
GiGPO42.47 / 23.3045.17 / 24.8943.82 ± 1.9124.10 ± 1.12
MiGPO39.00 / 28.6847.88 / 32.0543.44 ± 6.2830.36 ± 2.39
AERCA45.56 / 32.2347.10 / 27.3646.33 ± 1.0929.80 ± 3.44

WebShop update. We also increased the training budget from 100 to 150 iterations to better align it with the training settings reported in SkillRL [2] and RetroAgent [3]. The updated results are in Tables 1 and 2, with training details in Appendix A.4.

Thank you again for asking for the individual results and suggesting the larger budget. We would be happy to clarify any remaining points.

References
[1] Milestone-Guided Policy Learning for Long-Horizon Language Agents. ICML 2026.
[2] SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. arXiv 2026.
[3] RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback. arXiv 2026.

Dear Reviewer NN4d,

We sincerely thank you again for your careful review, insightful feedback, and constructive suggestions! We particularly appreciate your recognition of the value of adaptive experience regulation and the importance of our controlled comparison under a shared experience pipeline.

In response to your concerns, we clarified how memory credit affects GRPO optimization, conducted additional randomized attribution controls and experiments with a larger interaction budget, and expanded our statistical analyses. We have also updated the revised manuscript accordingly.

We hope these clarifications and additional results help address your concerns. If you have any further questions or feedback, we would be happy to discuss them.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_8U85OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets reliable end-to-end completion for LLM agents on long-horizon tasks and starts from two observations: retrieved experience can keep influencing the policy after it no longer matches the current state, and existing credit schemes trade intermediate progress against strict completion (GRPO/GiGPO complete more tasks but leave failed trajectories with little progress, while BEACON's milestone credit does the opposite).

Critical strengths
  1. The paper gives the clear evidence for the credit-assignment component. GRPO, GiGPO, and BEACON all use the same TCE/GCE retrieval and regulation pipeline, yet AERCA improves ScienceWorld full success from GiGPO’s 40.41% to 45.08%. Holding the memory pipeline fixed makes this comparison more convincing than the overall results alone and supports the claim that the learning signal provides an additional benefit.
Critical weaknesses
  1. It is unclear what the proposed attribution mechanism can distinguish when progress is only observed at the end. Equations (2)–(4) use changes in task score to select trajectories and identify useful memory events. In standard WebShop, however, reward arrives when the agent purchases a product. Unlesss additional intermediate scores are constructed, attribution would credit the memory active at purchase, or fall back to trajectory-level credit, without distinguishing earlier memory events that helped find the product. The paper might need to explain how this setting produces the attribution gains in Table 3. This matters because identifying useful memory use is the main motivation for the credit scheme.

  2. The WebShop comparison leaves an important gap against existing memory-based agents. Table 1 takes the RetroAgent and EMPO² results from prior work without establishing that they use equivalent training and evaluation conditions. AERCA’s 78.13% success therefore cannot be compared confidently with RetroAgent’s 82.30% or EMPO²’s 76.9%. Table 2 provides a controlled comparison between learning signals, but does not settle whether the proposed memory system improves on existing memory-based approaches. Reproducing these methods under the same protocol would make the empirical contribution substantially clearer.

Questions for authors
  1. How is the step-level score defined on WebShop? If scores change only at purchase, what information allows the attribution mechanism to distinguish useful memory events earlier in the trajectory? Please explain how attribution works in this setting and accounts for the improvement from 75.78% to 78.13% in Table 3.

  2. Do the five runs represent independent training seeds, and is the improvement over GiGPO in Table 2 consistent across them? Per-run results would help assess its reliability. Please also clarify how many runs underlie Table 3 and report uncertainty for the key ablations.

  3. Because credit follows score increases during active memory intervals, how do you distinguish helpful memory use from memory that merely happens to be active when progress occurs? Does breakthrough-based attribution outperform shuffled event attribution when the total trajectory credit is held fixed?

  4. Are the published WebShop results for RetroAgent and EMPO2 comparable in backbone, training budget, and evaluation set? If these conditions differ, could you provide a matched comparison under your protocol? This would help establish AERCA’s benefit over existing memory-based agents.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8U85 (Part 1/2): Memory Credit Attribution

We sincerely thank for your careful review, positive feedback, and constructive suggestions! We particularly appreciate your recognition of our controlled comparison, which helps isolate the contribution of Memory-Aware Credit Assignment. Your comments have helped us clarify the attribution mechanism and strengthen our experimental evaluation.

To make our response clear and easy to follow, we address your concerns point by point below. We would be happy to discuss any further questions or suggestions.

Comment 1 (W1, Q1): Intermediate Progress in WebShop

It is unclear what the proposed attribution mechanism can distinguish when progress is only observed at the end.

How is the step-level score defined on WebShop? If scores change only at purchase, what information allows the attribution mechanism to distinguish useful memory events earlier in the trajectory?

Following BEACON’s approach to tracking progress through page stages in WebShop [1] (see its official implementation), we define four milestones: Search (25), Detail (50), Options (75), and Purchase (100), starting from 0. At each stage, we check the relevant task requirements, such as product attributes, price, and options, rather than relying on page transitions alone.

For example, opening a matching product can raise the progress score from 25 to 50. A memory use event active during this transition can therefore receive credit before purchase. These milestone scores are used only to assign credit during training and do not determine evaluation success. Thank you for pointing out that this distinction was unclear. We have clarified how the milestones are defined and used in Section 3.3 and Appendix A.1.

In the revised Table 3, removing event attribution lowers WebShop Success from 79.22 ± 1.05% to 75.52 ± 1.96%, a difference of 3.70 percentage points.

Comment 2 (W1, Q3): Memory Attribution and Randomized Controls

... how do you distinguish helpful memory use from memory that merely happens to be active when progress occurs? Does breakthrough-based attribution outperform shuffled event attribution when the total trajectory credit is held fixed?

We agree that the agent may make progress simply by choosing a better action, even while memory is active. AERCA does not assume that memory caused the progress. Instead, it uses observed progress to decide which rollouts receive memory credit and which memory use events receive a share of that credit:

  1. Trajectory credit.

    We compare rollouts on the same task with access to memory. A rollout receives positive credit only if its peak progress is above the group average.

  2. Memory use lifecycle credit.

    Within a rollout that earns credit, we look for new progress peaks during each memory use event. An event with no new peak receives no breakthrough credit.

As reported in Appendix C.2, 42.13% of trajectories receive positive memory credit, and 46.18% of valid memory use events contain no new breakthrough. Simply activating memory is not enough to earn breakthrough credit.

Following your suggestion, we tested whether assigning credit based on breakthroughs works better than shuffling it across events. We ran two controls on ScienceWorld, each preserving the total credit assigned to each trajectory at the attribution stage:

1.Shuffled Event. We keep the start and end steps of every memory event unchanged. Within each rollout, we randomly reassign the original credit amounts across these events, keeping the total credit unchanged.

2.Random Interval. We keep each event’s credit and duration but move its credit interval to a random location in the same trajectory.

We trained each control twice for 100 iterations and evaluated all methods on the same ScienceWorld test set.

MethodRunFull Success (%)Success CountTask Score
AERCA5 runs, mean ± SD45.33 ± 1.27—30.27 ± 3.01
Shuffled Event140.15104/25919.80
Shuffled Event240.15104/25926.92
Shuffled EventMean ± SD40.15 ± 0.00—23.36 ± 5.03
Random Interval135.5292/25922.11
Random Interval238.61100/25926.67
Random IntervalMean ± SD37.07 ± 2.18—24.39 ± 3.22

AERCA’s mean Full Success is 5.18 percentage points higher than Shuffled Event and 8.26 points higher than Random Interval. With the same total credit, these results suggest that both how much credit each event receives and where that credit is assigned matter for learning.

Thank you again for raising these points. We have clarified the attribution mechanism in the revised paper.

References

[1] Milestone-Guided Policy Learning for Long-Horizon Language Agents. ICML 2026. Paper

AuthorsOpenReview ↗

Response to Reviewer 8U85 (Part 2/2): Experimental Fairness and Reliability

Comment 3 (W2, Q4): Fair Comparison with Existing Memory Agents

The WebShop comparison leaves an important gap against existing memory-based agents.

Are the published WebShop results for RetroAgent and EMPO2 comparable in backbone, training budget, and evaluation set? If these conditions differ, could you provide a matched comparison under your protocol?

Thank you for highlighting the importance of fair baseline comparisons. We agree that the published results alone do not establish a comparison under equivalent conditions. We made two updates.

  • First, we increased the WebShop training budget from 100 to 150 iterations, matching the budget reported by SkillRL [1] and RetroAgent [2]. We reran the WebShop experiments, including the ablations, with this budget and updated the WebShop results in Tables 1–3. Training details are in Appendix A.4.

  • Second, we reproduced RetroAgent and EMPO² using the same backbone (Qwen2.5-7B-Instruct), training budget (150 iterations), and WebShop test set as AERCA.

MethodPublished Success (%)Success under our protocol (%)Task Score under our protocolRuns
RetroAgent82.3072.66 ± 1.1081.59 ± 3.542
EMPO²76.9065.23 ± 6.0877.84 ± 2.622
AERCA—79.22 ± 1.0588.91 ± 0.765

Under this protocol, AERCA’s mean Success is 6.56 percentage points higher than RetroAgent and 13.99 points higher than EMPO².We discuss the differences between our reproduced results and the published results below:

  • RetroAgent. We used the official RetroAgent implementation. A similar WebShop reproduction gap was reported in GitHub Issue #2, which remains open.

  • EMPO². During training, EMPO² can reuse tips from earlier attempts when revisiting a task and incorporate what it learns into a policy that can act without external memory [3]. In our protocol, each training task is sampled only once, and evaluation is conducted on unseen tasks. Sampling each training task once may limit opportunities to reuse tips on the same task, which could make our setting less favorable to EMPO².

To avoid judging EMPO² solely under a protocol that may be less favorable to it, we retain its published WebShop result in Table 1. We also retain RetroAgent’s published result and report both reproductions separately as a comparison under our shared protocol.

Comment 4 (Q2): Repeated Runs and Uncertainty

Do the five runs represent independent training seeds, and is the improvement over GiGPO in Table 2 consistent across them? Per-run results would help assess its reliability. Please also clarify how many runs underlie Table 3 and report uncertainty for the key ablations.

Thank you for asking us to clarify how the runs were conducted. For the controlled comparison in Table 2, we increased the number of baseline runs from three to five, added an AERCA run, and updated the means and standard deviations. These are separate training runs under the same configuration. We did not deliberately select five distinct seeds.

With the additional run, AERCA’s mean ScienceWorld Full Success changed from 45.08% to 45.33%, without changing the method or evaluation protocol. The table below shows the individual results for AERCA and GiGPO under the same Adaptive Experience Regulation.

RunAERCA Success (%)AERCA ScoreGiGPO Success (%)GiGPO Score
143.2430.5244.0236.89
246.3333.4938.2220.34
345.1732.6541.3115.70
445.5626.1340.9314.88
546.3328.5543.2414.19
Mean ± SD45.33 ± 1.2730.27 ± 3.0141.54 ± 2.2620.40 ± 9.53

Under the same Adaptive Experience Regulation, AERCA’s mean Full Success across five runs is 3.79 percentage points higher than GiGPO’s. This result supports the benefit of AERCA’s credit assignment method. For the ablations, the revised Table 3 reports mean ± standard deviation over three runs for each ablation and five runs for full AERCA.

Thank you again for your suggestions. They helped us make the experimental comparison clearer.

References

[1] SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. 2026. Paper

[2] RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback. 2026. Paper

[3] Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization. ICLR 2026. Paper

Dear Reviewer 8U85,

We sincerely thank you again for your thoughtful review, encouraging feedback, and constructive suggestions!

We particularly appreciate your recognition of our controlled comparison, which provides clear evidence for the contribution of Memory-Aware Credit Assignment. Your positive assessment of this key contribution is especially encouraging to us.

Following your valuable suggestions, we have clarified the memory attribution mechanism, conducted additional randomized attribution experiments, reproduced existing memory agents under a shared protocol, and expanded our experimental analysis. We have incorporated these improvements into the revised manuscript and updated the corresponding experimental results and implementation materials. We believe these additions further strengthen the evidence for the effectiveness and reliability of AERCA.

We hope our detailed responses, additional experiments, and revised manuscript have addressed your concerns and further clarified the contributions of our work. If you have any remaining questions or suggestions, we would be happy to discuss them.

Thank you once again for your time and valuable feedback!

Best regards,
The Authors

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their insightful feedback, thoughtful discussions, and constructive suggestions! We are delighted that all reviewers recognized the value of our work, particularly its practical motivation (Reviewer DVrA and Reviewer NN4d) and the controlled experimental evidence supporting Memory-Aware Credit Assignment (Reviewer 8U85, Reviewer NN4d, and Reviewer DVrA). We also greatly appreciate the recognition of Adaptive Experience Regulation as a useful mechanism for managing retrieved guidance (Reviewer NN4d).

The reviewers raised important questions regarding memory credit attribution, experimental fairness, statistical reliability, and long-horizon performance. We greatly appreciate these valuable suggestions, which have helped us strengthen our work. In response, we have conducted additional experiments, provided detailed clarifications, and incorporated the corresponding improvements into our revised manuscript. Our main responses are summarized below:

  1. Methodological Clarifications (Reviewer 8U85 and Reviewer NN4d): Clarified memory credit attribution, how per-step memory rewards affect GRPO policy updates, and evaluation protocols.
  2. Additional Experiments (Reviewer 8U85, Reviewer NN4d, and Reviewer DVrA): Introduced randomized attribution controls, reproduced memory-based baselines, and expanded long-horizon evaluations.
  3. Statistical Reliability and Reproducibility (Reviewer NN4d and Reviewer DVrA): Reported confidence intervals and additional runs, corrected numerical inconsistencies, refined claims, and expanded the anonymized code materials.

We are pleased that our detailed responses and revisions have helped clarify the reviewers' concerns, and that all three reviewers now hold positive assessments of our work, with both Reviewer NN4d and Reviewer DVrA raising their scores during the discussion. We are sincerely grateful for the reviewers' encouraging feedback and recognition of our efforts!

Once again, we deeply appreciate the time and expertise all reviewers have devoted to evaluating our work. Your thoughtful suggestions have substantially improved the clarity, rigor, and reproducibility of our paper, and your encouragement motivates us to continue advancing this research. We sincerely hope that our revised manuscript and detailed responses have adequately addressed the remaining concerns.

Best regards,
Authors of Paper 42910

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.