- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the "run-in forgetting" phenomenon in proxy reinforcement learning, where previously successful trajectories may become less likely under subsequent strategies, even if the task distribution remains unchanged. The authors measure this phenomenon by tracking the probability of 128 historical successful trajectories at the training checkpoints. To address this issue, this paper proposes a method called "success-constrained policy gradient projection", which maintains a backpropagation buffer of the maximum return trajectory and applies the A-GEM gradient projection when the current reinforcement learning gradient conflicts with the replay gradient. This method aims to avoid high-order growth of replay loss without modifying the original reinforcement learning advantage estimator.
Critical strengths
- An interesting observation about behavioral forgetting during the operation process.
- Performs well on WebShop and has a simple formula.
Critical weaknesses
- The methodological innovation is limited, and there is insufficient comparison with existing replay methods.
- The verification of the proposed forgetting mechanism is insufficient, and the experimental evidence is limited.
Questions for authors
Q1. Comparison with Existing Success Replay Methods.
This method applies the A-GEM gradient projection [1] to historical successful trajectories. However, existing methods such as SIL (Self-Implicit Learning) [2] and ARPO [3] have also studied how to utilize successful experiences for strategy optimization.
Could the authors compare these representative successful replay methods under matching training conditions? In particular, by constructing a simple baseline using auxiliary behavior cloning or negative log-likelihood loss on the same successful buffer, it helps to isolate the role of the gradient projection. If applicable, the comparison with SIL and ARPO will further demonstrate the superiority of this method over existing replay-based methods in agent reinforcement learning.
Q2. Direct Evaluation of Forgetting Mitigation.
Figure 2 shows that during the GRPO training process, the historical successful trajectories gradually reduce their likelihood values. Could the authors provide the corresponding likelihood curves for GRPO/GiGPO in both with and without projection? Since reducing the likelihood value of a specific successful trajectory does not necessarily mean forgetting at the task level, evaluating the success rate of a group of previously solved tasks at different checkpoints can also help verify the core argument.
Q3. Gradient Projection Behavior and Computational Cost.
During the training process, how frequently is the projection mechanism activated? Can the author report the cosine similarity of the gradients between the current reinforcement learning objective and the replay loss, as well as the magnitude of the gradient correction resulting from this? These statistical information helps to link theoretical analysis with actual optimization behavior.
Additionally, since the method requires computing an extra replay gradient, what is the additional wall-clock training and GPU memory overhead compared with the original GRPO and GiGPO implementations?
Q4. Experimental Robustness.
The ALFWorld experiment shows significant variability in the training process, and the main comparison is limited to a single non-crashing training seed. Can the author provide complete results for multiple seeds, including the frequency of policy crashes? The results of WebShop also indicate that the proposed method does not consistently improve the performance of GiGPO on all training seeds. Further analysis will help clarify under what circumstances successful gradient projection has an advantage.
References
[1] Chaudhry et al., "Efficient Lifelong Learning with A-GEM." ICLR 2019. https://openreview.net/forum?id=Hkf2_sC5FX
[2] Oh et al., "Self-Imitation Learning." ICML 2018. https://proceedings.mlr.press/v80/oh18b.html
[3] Lu et al., "ARPO: End-to-End Policy Optimization for GUI Agents with Experience Replay." 2025. https://arxiv.org/abs/2505.16282
5 comments
The rebuttal addresses part of my concerns by adding the AddBC comparison, likelihood analysis, and multi-seed results. However, three concerns remain.
-
Baseline comparison remains incomplete. No direct comparison with SIL/ARPO is provided, and AddBC outperforms projection on GiGPO (74.00% vs. 72.38%).
-
The forgetting mitigation is not consistent. Projection preserves historical likelihood better for GRPO, but not consistently for GiGPO, and this does not consistently translate to better task-level retention.
-
The method is highly sensitive across seeds. On ALFWorld GiGPO, performance drops from 76.43% to 5.24% on one seed, while the overall mean decreases from 70.87% to 50.16%.
Overall, these additional experiments do not support the effectiveness of the proposed method. Instead, they reveal limited advantage over simple replay and substantial instability across training settings.
1. Baseline comparison. Our study asks two research questions:
- Which historical trajectories should we protect as the policy changes?
- How can we adaptively protect them during an update?
On a given task, any maximum-return trajectory has nonnegative trajectory-level advantage relative to the current policy, regardless of which policy collected it. This provides theoretical support for selecting historical maximum-return trajectories as protection targets.
SIL uses past successes to encourage exploration, and ARPO uses replay to restore a learning signal in all-failure rollout groups. Answering our two research questions does not require outperforming SIL or ARPO on a particular benchmark. Our research objective is to identify the role of target selection and the role of adaptive protection. We therefore vary each choice separately.
To test target selection on WebShop, we held projection fixed and compared two replay buffers. One admits only maximum-return trajectories. The other admits completed trajectories of any return. In additional experiments on CartPole, we compared the same two selection rules.
| Setting and final metric | Maximum-return protection | Random-history protection |
|---|---|---|
| WebShop (GRPO), task success (%) | 71.40 ± 1.67 | 67.87 ± 5.93 |
| CartPole (PPO, width 16), episode return | 500.00 ± 0.00 | 481.43 ± 50.05 |
Maximum-return protection scored 3.53 percentage points higher on WebShop and 18.57 return points higher on CartPole on average. Entries are means ± SD across training seeds (three on WebShop, ten on CartPole).
To test the protection rule, we held maximum-return selection fixed and compared A-GEM projection with additive behavior cloning (AddBC). The cited 74.00% on GiGPO was the best final AddBC score among three imitation weights, identifiable only after evaluating the sweep. Besides, projection scored 1.39 percentage points higher on average across GRPO and GiGPO, even when compared with the best observed AddBC weight for each algorithm (Table B7). Projection uses gradient alignment to adaptively determine when and how much to correct, without tuning an imitation weight.
2. Forgetting mitigation. Figure 2 follows a fixed cohort of 128 archived successes per algorithm in a single-seed WebShop comparison. From checkpoint 50 to the final checkpoint, projection reduced the median decline in length-normalized trajectory likelihood for both GRPO (13.03% to 5.73%) and GiGPO (31.45% to 25.66%). The GiGPO curves cross during training.
With shared parameters, protecting a recorded action can change behavior at other states. The policy can therefore benefit even if success on the goal that supplied that action stays the same. We assess policy-level benefit through end-to-end performance across training seeds. Projection increased held-out success by 6.16 percentage points on WebShop with GRPO (p = 0.0023) and increased PPO’s summed final return across four MuJoCo tasks by 14.8% (p = 0.023). Nine of ten paired MuJoCo seed totals improved. On ALFWorld, the estimated effects were +30.32 percentage points for GRPO (p = 0.17) and −20.72 points for GiGPO (p = 0.51). The direction of the mean effect remains uncertain for each algorithm on ALFWorld. All four p-values are from two-sided t-tests paired by training seed.
3. Sensitivity across seeds. Both ALFWorld baselines already showed substantial variability. Final success ranged from 0.00% to 32.62% for GRPO and from 54.29% to 81.90% for GiGPO. The cited 76.43% vs. 5.24% shows a large performance difference for a single training seed. Attributing greater seed sensitivity to projection requires more statistical evidence.
The average performance effect is a separate question. If the observed standard deviation of the paired GiGPO differences is representative, a two-sided paired t-test with three training seeds has only about 8% power to detect a 20-percentage-point mean difference at α = 0.05. The average effect on ALFWorld remains uncertain, while WebShop GRPO and the four-task MuJoCo aggregate both show statistically significant improvements.
Thank you for your careful review.
Q1. Protection target and cloning control
We select historical behavior to protect within one run. In a tabular setting, a feasible shift from lower- to maximum-return trajectories raises expected return when other tasks’ policies stay fixed. With shared parameters, updates can affect other tasks. Our anchor is each task’s historical trajectory distribution under a mixture of past policies, conditioned on maximum return (Sec. 3.1, Appendix A.3).
SIL uses past good decisions to encourage exploration. ARPO restores a learning signal when a GRPO group contains only failures. We address forgetting: even an informative update can make earlier successes less likely.
We compared A-GEM projection with additive behavior cloning (AddBC) at matched admission, warmup, capacity and replay budget. Bold marks the best AddBC mean over three tested weights (three training seeds, Table B7).
Projection scored 4.40 points above the best AddBC on GRPO and 1.62 below on GiGPO, using gradient alignment to set its correction without an imitation-weight sweep.
Q2. Does projection reduce forgetting?
Both policies rescored the same 128 baseline successes per algorithm (one seed). Entries give median change (%) in length-normalized likelihood from each policy’s checkpoint 50 (Figure 2, Table B6).
Projection reduced the final decline for both algorithms. Because the policy shares parameters across states, protecting a recorded action can change behavior elsewhere. The policy can therefore benefit even if its probability of success from that particular state does not increase. All four policies solved more original goals at 149 than at 50, with no consistent gain from projection across those goals. Held-out success measures overall performance (Sec. 5.4).
Q3. Gradient behavior and cost
Projection activated more often late in training (mean within-round rates, one run per setting, Table B8).
In a separate WebShop GiGPO run, 154/323 updates triggered projection. Their median gradient cosine was −0.034, giving a median correction of 3.39% of the main gradient norm (IQR 1.24–5.72%).
Wall-clock. Replay uses four stored trajectories without new rollouts. Times cover 150 rounds on 16 GPUs (three seeds per arm).
GPU memory. Peak allocated memory rose by 0.36 GiB on both tasks. Mean peak reserved memory rose from 98.39 to 153.31 GiB on WebShop and from 62.45 to 91.74 GiB on ALFWorld (Appendix B.9).
Q4. Robustness across seeds and tasks
Task interactivity contributes to high variance because early actions change the states and successes encountered. WebShop GRPO improved in all three paired seeds (+6.16 points, paired 95% t-interval [4.89, 7.42]). GiGPO gained 2.14 points on average, improving in two of three seeds (Table 1).
All 12 ALFWorld runs completed (140 ValidSeen games, three decoding seeds, Table B5). The original Table 2 used training seed 1.
Two final policies regularly failed at the 50-turn cap: baseline GRPO seed 2 (420/420 episodes) and protected GiGPO seed 0 (398/420).
We further tested high-return protection with PPO on MuJoCo (final return, mean ± SD across ten paired seeds, Table 4).
Summed final return increased 14.8%, with gains in nine of ten paired seed totals (two-sided paired t-test, p=0.023, paired bootstrap 95% CI for the mean gain [525, 2,735]).