acceptodds
Under review as a conference paper at ICLR 2027

Continual Learning in Agentic RL via Success-Constrained Policy Gradient Projection

Abstract

Agentic reinforcement learning can forget previously successful behaviors even within a single training run on a fixed task distribution. We quantify this trajectory-level forgetting, revealing a continual learning problem arising without an explicit sequence of tasks. At the trajectory level, maximum-return oracle trajectories have nonnegative advantage relative to the current policy's expected return, regardless of which policy generated them. This provides a policy-improvement motivation for selecting historical successes as protection targets. We introduce Success-Constrained Policy Gradient Projection, which makes the minimum Euclidean correction to the main loss gradient needed to satisfy an aggregate success-replay constraint. Our analysis characterizes the tradeoff between protecting historical successes and optimizing the current objective: for a gradient-descent step, the correction removes a first-order increase in aggregate replay loss, with a cost to the main objective determined by gradient alignment. Averaged over three training seeds on WebShop, the method improves success by 6.2 and 2.1 percentage points for GRPO and GiGPO, respectively, without changing their advantage estimators. GRPO with the correction reaches 71.4% mean success, surpassing the GiGPO baseline at 70.2%. In a supplementary ALFWorld comparison using one training seed without policy collapse, the method improves final success and reduces the decline from validation-selected saved checkpoints with both algorithms. These results demonstrate that retaining historical successes can yield gains comparable to finer-grained advantage estimation, and establish the practical value of success-based protection for continual learning in agentic reinforcement learning.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, continual learning, catastrophic forgetting, experience replay, policy gradient

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s1NaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the "run-in forgetting" phenomenon in proxy reinforcement learning, where previously successful trajectories may become less likely under subsequent strategies, even if the task distribution remains unchanged. The authors measure this phenomenon by tracking the probability of 128 historical successful trajectories at the training checkpoints. To address this issue, this paper proposes a method called "success-constrained policy gradient projection", which maintains a backpropagation buffer of the maximum return trajectory and applies the A-GEM gradient projection when the current reinforcement learning gradient conflicts with the replay gradient. This method aims to avoid high-order growth of replay loss without modifying the original reinforcement learning advantage estimator.

Critical strengths
  1. An interesting observation about behavioral forgetting during the operation process.
  2. Performs well on WebShop and has a simple formula.
Critical weaknesses
  1. The methodological innovation is limited, and there is insufficient comparison with existing replay methods.
  2. The verification of the proposed forgetting mechanism is insufficient, and the experimental evidence is limited.
Questions for authors

Q1. Comparison with Existing Success Replay Methods.

This method applies the A-GEM gradient projection [1] to historical successful trajectories. However, existing methods such as SIL (Self-Implicit Learning) [2] and ARPO [3] have also studied how to utilize successful experiences for strategy optimization.

Could the authors compare these representative successful replay methods under matching training conditions? In particular, by constructing a simple baseline using auxiliary behavior cloning or negative log-likelihood loss on the same successful buffer, it helps to isolate the role of the gradient projection. If applicable, the comparison with SIL and ARPO will further demonstrate the superiority of this method over existing replay-based methods in agent reinforcement learning.

Q2. Direct Evaluation of Forgetting Mitigation.

Figure 2 shows that during the GRPO training process, the historical successful trajectories gradually reduce their likelihood values. Could the authors provide the corresponding likelihood curves for GRPO/GiGPO in both with and without projection? Since reducing the likelihood value of a specific successful trajectory does not necessarily mean forgetting at the task level, evaluating the success rate of a group of previously solved tasks at different checkpoints can also help verify the core argument.

Q3. Gradient Projection Behavior and Computational Cost.

During the training process, how frequently is the projection mechanism activated? Can the author report the cosine similarity of the gradients between the current reinforcement learning objective and the replay loss, as well as the magnitude of the gradient correction resulting from this? These statistical information helps to link theoretical analysis with actual optimization behavior.

Additionally, since the method requires computing an extra replay gradient, what is the additional wall-clock training and GPU memory overhead compared with the original GRPO and GiGPO implementations?

Q4. Experimental Robustness.

The ALFWorld experiment shows significant variability in the training process, and the main comparison is limited to a single non-crashing training seed. Can the author provide complete results for multiple seeds, including the frequency of policy crashes? The results of WebShop also indicate that the proposed method does not consistently improve the performance of GiGPO on all training seeds. Further analysis will help clarify under what circumstances successful gradient projection has an advantage.

References

[1] Chaudhry et al., "Efficient Lifelong Learning with A-GEM." ICLR 2019. https://openreview.net/forum?id=Hkf2_sC5FX

[2] Oh et al., "Self-Imitation Learning." ICML 2018. https://proceedings.mlr.press/v80/oh18b.html

[3] Lu et al., "ARPO: End-to-End Policy Optimization for GUI Agents with Experience Replay." 2025. https://arxiv.org/abs/2505.16282

5 comments

Thank you for your careful review.

Q1. Protection target and cloning control

We select historical behavior to protect within one run. In a tabular setting, a feasible shift from lower- to maximum-return trajectories raises expected return when other tasks’ policies stay fixed. With shared parameters, updates can affect other tasks. Our anchor is each task’s historical trajectory distribution under a mixture of past policies, conditioned on maximum return (Sec. 3.1, Appendix A.3).

SIL uses past good decisions to encourage exploration. ARPO restores a learning signal when a GRPO group contains only failures. We address forgetting: even an informative update can make earlier successes less likely.

We compared A-GEM projection with additive behavior cloning (AddBC) at matched admission, warmup, capacity and replay budget. Bold marks the best AddBC mean over three tested weights (three training seeds, Table B7).

Final WebShop First-500 success (%)GRPOGiGPO
Base65.24 ± 2.1670.24 ± 2.17
Success projection71.40 ± 1.6772.38 ± 3.31
AddBC, λ = 0.0166.47 ± 7.6873.33 ± 0.92
AddBC, λ = 0.0367.00 ± 4.4774.00 ± 2.27
AddBC, λ = 0.1065.33 ± 2.4270.07 ± 6.03

Projection scored 4.40 points above the best AddBC on GRPO and 1.62 below on GiGPO, using gradient alignment to set its correction without an imitation-weight sweep.

Q2. Does projection reduce forgetting?

Both policies rescored the same 128 baseline successes per algorithm (one seed). Entries give median change (%) in length-normalized likelihood from each policy’s checkpoint 50 (Figure 2, Table B6).

PolicyCkpt 75Ckpt 100Ckpt 125Ckpt 149
GRPO−6.88−9.83−11.15−13.03
GRPO + projection−1.92−4.47−4.63−5.73
GiGPO−5.29−9.02−15.74−31.45
GiGPO + projection−3.44−11.21−20.65−25.66

Projection reduced the final decline for both algorithms. Because the policy shares parameters across states, protecting a recorded action can change behavior elsewhere. The policy can therefore benefit even if its probability of success from that particular state does not increase. All four policies solved more original goals at 149 than at 50, with no consistent gain from projection across those goals. Held-out success measures overall performance (Sec. 5.4).

Q3. Gradient behavior and cost

Projection activated more often late in training (mean within-round rates, one run per setting, Table B8).

SettingUpdates corrected, rounds 25–49Updates corrected, rounds 125–149
WebShop GRPO38.57%40.66%
WebShop GiGPO36.96%47.93%
ALFWorld GRPO28.80%46.12%
ALFWorld GiGPO30.75%58.45%

In a separate WebShop GiGPO run, 154/323 updates triggered projection. Their median gradient cosine was −0.034, giving a median correction of 3.39% of the main gradient norm (IQR 1.24–5.72%).

Wall-clock. Replay uses four stored trajectories without new rollouts. Times cover 150 rounds on 16 GPUs (three seeds per arm).

SettingBase (h)Projection (h)Change
WebShop GRPO7.2067.205−0.01%
WebShop GiGPO6.0506.195+2.40%
ALFWorld GRPO13.75314.285+3.87%
ALFWorld GiGPO11.35713.832+21.79%

GPU memory. Peak allocated memory rose by 0.36 GiB on both tasks. Mean peak reserved memory rose from 98.39 to 153.31 GiB on WebShop and from 62.45 to 91.74 GiB on ALFWorld (Appendix B.9).

Q4. Robustness across seeds and tasks

Task interactivity contributes to high variance because early actions change the states and successes encountered. WebShop GRPO improved in all three paired seeds (+6.16 points, paired 95% tt-interval [4.89, 7.42]). GiGPO gained 2.14 points on average, improving in two of three seeds (Table 1).

All 12 ALFWorld runs completed (140 ValidSeen games, three decoding seeds, Table B5). The original Table 2 used training seed 1.

ALFWorld final success (%)Seed 0Seed 1Seed 2Mean ± SD
GRPO13.5732.620.0015.40 ± 16.39
GRPO + projection46.9036.6753.5745.72 ± 8.51
GiGPO76.4354.2981.9070.87 ± 14.62
GiGPO + projection5.2470.0075.2450.16 ± 38.99

Two final policies regularly failed at the 50-turn cap: baseline GRPO seed 2 (420/420 episodes) and protected GiGPO seed 0 (398/420).

We further tested high-return protection with PPO on MuJoCo (final return, mean ± SD across ten paired seeds, Table 4).

EnvironmentPPOPPO + projection
HalfCheetah-v52,280 ± 1,2542,272 ± 1,176
Hopper-v52,029 ± 8852,911 ± 902
Walker2d-v53,009 ± 1,1893,504 ± 462
Ant-v53,701 ± 6323,961 ± 720
Sum across tasks11,019 ± 1,70312,649 ± 1,428

Summed final return increased 14.8%, with gains in nine of ten paired seed totals (two-sided paired tt-test, p=0.023p=0.023, paired bootstrap 95% CI for the mean gain [525, 2,735]).

The rebuttal addresses part of my concerns by adding the AddBC comparison, likelihood analysis, and multi-seed results. However, three concerns remain.

  1. Baseline comparison remains incomplete. No direct comparison with SIL/ARPO is provided, and AddBC outperforms projection on GiGPO (74.00% vs. 72.38%).

  2. The forgetting mitigation is not consistent. Projection preserves historical likelihood better for GRPO, but not consistently for GiGPO, and this does not consistently translate to better task-level retention.

  3. The method is highly sensitive across seeds. On ALFWorld GiGPO, performance drops from 76.43% to 5.24% on one seed, while the overall mean decreases from 70.87% to 50.16%.

Overall, these additional experiments do not support the effectiveness of the proposed method. Instead, they reveal limited advantage over simple replay and substantial instability across training settings.

1. Baseline comparison. Our study asks two research questions:

  • Which historical trajectories should we protect as the policy changes?
  • How can we adaptively protect them during an update?

On a given task, any maximum-return trajectory has nonnegative trajectory-level advantage relative to the current policy, regardless of which policy collected it. This provides theoretical support for selecting historical maximum-return trajectories as protection targets.

SIL uses past successes to encourage exploration, and ARPO uses replay to restore a learning signal in all-failure rollout groups. Answering our two research questions does not require outperforming SIL or ARPO on a particular benchmark. Our research objective is to identify the role of target selection and the role of adaptive protection. We therefore vary each choice separately.

To test target selection on WebShop, we held projection fixed and compared two replay buffers. One admits only maximum-return trajectories. The other admits completed trajectories of any return. In additional experiments on CartPole, we compared the same two selection rules.

Setting and final metricMaximum-return protectionRandom-history protection
WebShop (GRPO), task success (%)71.40 ± 1.6767.87 ± 5.93
CartPole (PPO, width 16), episode return500.00 ± 0.00481.43 ± 50.05

Maximum-return protection scored 3.53 percentage points higher on WebShop and 18.57 return points higher on CartPole on average. Entries are means ± SD across training seeds (three on WebShop, ten on CartPole).

To test the protection rule, we held maximum-return selection fixed and compared A-GEM projection with additive behavior cloning (AddBC). The cited 74.00% on GiGPO was the best final AddBC score among three imitation weights, identifiable only after evaluating the sweep. Besides, projection scored 1.39 percentage points higher on average across GRPO and GiGPO, even when compared with the best observed AddBC weight for each algorithm (Table B7). Projection uses gradient alignment to adaptively determine when and how much to correct, without tuning an imitation weight.

2. Forgetting mitigation. Figure 2 follows a fixed cohort of 128 archived successes per algorithm in a single-seed WebShop comparison. From checkpoint 50 to the final checkpoint, projection reduced the median decline in length-normalized trajectory likelihood for both GRPO (13.03% to 5.73%) and GiGPO (31.45% to 25.66%). The GiGPO curves cross during training.

With shared parameters, protecting a recorded action can change behavior at other states. The policy can therefore benefit even if success on the goal that supplied that action stays the same. We assess policy-level benefit through end-to-end performance across training seeds. Projection increased held-out success by 6.16 percentage points on WebShop with GRPO (p = 0.0023) and increased PPO’s summed final return across four MuJoCo tasks by 14.8% (p = 0.023). Nine of ten paired MuJoCo seed totals improved. On ALFWorld, the estimated effects were +30.32 percentage points for GRPO (p = 0.17) and −20.72 points for GiGPO (p = 0.51). The direction of the mean effect remains uncertain for each algorithm on ALFWorld. All four p-values are from two-sided t-tests paired by training seed.

3. Sensitivity across seeds. Both ALFWorld baselines already showed substantial variability. Final success ranged from 0.00% to 32.62% for GRPO and from 54.29% to 81.90% for GiGPO. The cited 76.43% vs. 5.24% shows a large performance difference for a single training seed. Attributing greater seed sensitivity to projection requires more statistical evidence.

The average performance effect is a separate question. If the observed standard deviation of the paired GiGPO differences is representative, a two-sided paired t-test with three training seeds has only about 8% power to detect a 20-percentage-point mean difference at α = 0.05. The average effect on ALFWorld remains uncertain, while WebShop GRPO and the four-task MuJoCo aggregate both show statistically significant improvements.

Reviewer_qud7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies forgetting in agentic reinforcement learning, where previously successful trajectories can become less likely under the policy as training continues, even when the task distribution remains fixed. The authors propose Success-Constrained Policy Gradient Projection, which stores maximum-return trajectories and applies A-GEM’s gradient projection when the current RL gradient conflicts with the negative log-likelihood gradient of sampled historical successes. The correction aims to preserve their aggregate likelihood to first order while minimally modifying the base gradient. Experiments with GRPO and GiGPO on WebShop show improved mean task success across three training seeds.

Critical strengths
  1. The paper investigates interference with historical successful behavior during agentic RL training. Its application of A-GEM provides a clear correction that can be incorporated into GRPO and GiGPO without changing their advantage estimators.
  2. The method improves final success in all three training seeds, raising average success from 65.24% to 71.40%. Evaluation includes multiple training and decoding seeds, making this the strongest empirical result.
  3. The experiments examine memory warmup, FIFO versus reservoir retention, and uniform versus prefix importance weighting, providing practical insights into how the protected replay distribution affects performance.
Critical weaknesses
  1. The connection between the forgetting diagnostic and the proposed mechanism remains unclear. Figure 2 demonstrates declining likelihood of historical successful trajectories, but this does not necessarily indicate a loss of task-solving ability: the policy could instead learn alternative successful behaviors. Moreover, the paper does not repeat this diagnostic with the proposed correction, leaving it unclear whether the method mitigates the phenomenon used to motivate it. A matched comparison tracking both historical-trajectory likelihood and success on the associated tasks would help establish the relevance of the diagnostic and the proposed retention mechanism.

  2. Additional controls would help isolate the contributions of success selection and gradient projection. The reported replay variants all retain maximum-return trajectories. A control that protects randomly collected historical trajectories would help test whether selecting successes is important beyond historical replay more generally. Comparisons with alternative ways of incorporating the same successful data, such as an auxiliary replay NLL objective, would further clarify the benefit of the projection rule.

  3. The empirical scope limits conclusions about robustness and generality. Evaluation covers two environments and one model scale, with the strongest evidence coming from GRPO on WebShop. GiGPO’s WebShop gains are inconsistent across training seeds, while the main ALFWorld comparison reports one seed without collapse. The replay ablations also use only one training seed.

  4. The presentation explains the intervention more thoroughly than it investigates why and when it works. Considerable space is devoted to the replay formulation and existing A-GEM projection, whereas the empirical analysis relies mainly on final-performance tables. Comparative learning curves and diagnostics of projection frequency, gradient conflict, and realized replay-loss changes would provide a clearer account of the method’s behavior and better establish its contribution beyond applying an existing projection technique.

Questions for authors
  1. Can you repeat the fixed-cohort analysis in Figure 2 with the proposed correction enabled, and evaluate task success on the associated prompts? This would help distinguish reduced likelihood of particular successful trajectories from loss of task-solving ability, and test whether the method mitigates the diagnosed phenomenon.

  2. How important is restricting the protected buffer to maximum-return trajectories? A matched control using randomly collected historical trajectories would help isolate the contribution of success-based selection.

  3. How does gradient projection compare with alternative ways of incorporating the same successful trajectories, such as an auxiliary replay NLL objective under a matched replay budget?

  4. Could you provide further analysis of how the correction behaves during training? For example, learning curves, projection frequency, or gradient-alignment measurements could help explain when the method is most useful and how it affects training dynamics.

3 comments

Thank you for your careful reading of our paper and your constructive feedback.

Q1. Historical paths and their original goals

For each algorithm, we rescored a fixed cohort of 128 baseline WebShop successes with and without projection (one training seed). From checkpoint 50 to 149, the median decline in length-normalized likelihood fell from 13.03% to 5.73% for GRPO and from 31.45% to 25.66% for GiGPO (Figure 2, Table B6).

Because the policy shares parameters across states, protecting a recorded action can change behavior elsewhere. The policy can therefore benefit even if its probability of success from that particular state does not increase. In fresh rollouts on the original goals, we observed no consistent gain over the baselines. We evaluate the overall policy by end-to-end success on held-out goals (Appendix B.7).

Q2. Success selection versus random history

In a tabular setting, treat each task as a state and each feasible trajectory as an action. Moving probability from lower-return to maximum-return trajectories improves expected return on that task. With other tasks’ policies held fixed, overall expected return cannot decrease. With shared parameters, we use the historical policy mixture’s trajectory distribution conditioned on maximum return as a regularization anchor (Sec. 3.1, Appendix A.3).

On WebShop, the random-history control matched buffer capacity (2,048), uniform draws (four trajectories per update) and replay onset (128 distinct goals with stored successes). We also ran this control on CartPole with PPO.

Task and final-checkpoint metricSuccess onlyRandom history
WebShop (GRPO), First-500 success (%)71.40 ± 1.6767.87 ± 5.93
CartPole (PPO, width 16), return500.00 ± 0.00481.43 ± 50.05

Success-only protection had higher mean final performance in both reported comparisons (Appendix B.8 and Table B9).

Q3. Additive cloning on successful trajectories

Additive behavior cloning (AddBC) adds a weighted imitation loss at each replay update. Projection intervenes when the RL gradient conflicts with the replay gradient. Both used the same success-admission rule, warmup, buffer capacity and replay budget (Appendix B.8, Table B7).

Final WebShop First-500 success (%)GRPOGiGPO
Base65.24 ± 2.1670.24 ± 2.17
Success projection71.40 ± 1.6772.38 ± 3.31
AddBC, λ = 0.0166.47 ± 7.6873.33 ± 0.92
AddBC, λ = 0.0367.00 ± 4.4774.00 ± 2.27
AddBC, λ = 0.1065.33 ± 2.4270.07 ± 6.03

Entries are means ± SD across three training seeds. Bold marks the highest mean First-500 success among the three AddBC weights. This best observed AddBC result was 4.40 points below projection on GRPO and 1.62 points above projection on GiGPO. Projection was competitive and set its correction size from gradient alignment without sweeping an imitation weight.

Q4. Frequency and magnitude of correction

In each setting, a larger fraction of updates triggered projection late in training than early. Entries average the fraction corrected within each logged round, using one run per setting (Table B8).

SettingUpdates corrected, rounds 25–49Updates corrected, rounds 125–149
WebShop GRPO38.57%40.66%
WebShop GiGPO36.96%47.93%
ALFWorld GRPO28.80%46.12%
ALFWorld GiGPO30.75%58.45%

In a separate WebShop GiGPO run, 154 of 323 updates triggered projection. Among those updates, the median gradient cosine was −0.034. The median correction was 3.39% of the main gradient norm (interquartile range 1.24–5.72%).

Q5. Robustness and experimental scope

The interactive nature of WebShop and ALFWorld is itself a source of high variance: early actions affect which states and successes the policy learns from. In the original WebShop runs, GRPO improved in all three paired training seeds, from 65.24% to 71.40% (+6.16 points, paired 95% tt-interval [4.89, 7.42]). GiGPO gained 2.14 points on average and improved in two of three seeds (Table 1). All twelve final ALFWorld results are in Table B5.

To further evaluate the algorithm across more training seeds, we compared PPO with and without high-return protection on four MuJoCo tasks. Entries are final returns, mean ± SD across ten paired training seeds (Table 4).

EnvironmentPPOPPO + projection
HalfCheetah-v52,280 ± 1,2542,272 ± 1,176
Hopper-v52,029 ± 8852,911 ± 902
Walker2d-v53,009 ± 1,1893,504 ± 462
Ant-v53,701 ± 6323,961 ± 720
Sum across tasks11,019 ± 1,70312,649 ± 1,428

The summed final return rose 14.8%. Nine of ten paired seed totals improved (two-sided paired tt-test, p=0.023p=0.023, paired bootstrap 95% CI for the mean gain [525, 2,735]). Hopper showed the largest gain and HalfCheetah changed little.

Thank you for the extensive rebuttal and experiments on the new likelihood analysis, random-history and AddBC controls, gradient statistics, and paired tests, which address several of my questions. But my central concern remains. Projection slows the likelihood decline of archived successes, but the baselines also solved more of those goals over training, and projection gave no consistent gain on them. And together with AddBC being competitive on GiGPO and the near orthogonal gradients (median cosine −0.034), it remains unclear whether the gains come from forgetting mitigation, from projection itself, or simply from replaying successes. I encourage the authors to either isolate this mechanism more directly or reframe the contribution accordingly.

Thank you for clarifying your central concern. We hope the revisions make our contribution and its empirical basis clearer.

1. Policy forgetting and task success (§5.4, Figure 2, Appendix B.7). We study policy forgetting: as the policy improves overall, some previously effective behaviors can become less likely. We regularize the shared policy using actions from historical successes, so the benefit need not appear as higher success on the goals that supplied those actions. Indeed, both WebShop baselines solved more of the original goals later in training even as the archived paths became less likely. These results are compatible because a goal can be solved along several paths. In the single-seed comparison in Figure 2, projection reduced the final likelihood decline for both GRPO and GiGPO. We assess the overall policy benefit through end-to-end success on held-out goals.

2. Projection and AddBC (§5.2, Table B7). Both methods replay maximum-return trajectories. AddBC adds an imitation loss with a fixed weight. The reported 74.00% GiGPO score was the best result obtained by sweeping three imitation weights and evaluating each one. Using the best observed weight for each algorithm, AddBC scored 1.62 percentage points above projection on GiGPO and 4.40 points below it on GRPO. In Table B7, projection averaged 1.39 points higher across the two algorithms, with gradient alignment determining when and how much to intervene, without an imitation-weight sweep.

3. Near-orthogonal gradients (§3.3, Appendix B.9). In the separate WebShop GiGPO diagnostic, the median cosine was −0.034 among the 154 updates that triggered projection. The corresponding median correction was 3.39% of the full RL gradient norm. When the gradients are nearly orthogonal, only a small component of the RL update would increase replay loss to first order. A small correction can therefore remove the entire conflicting component. If successive updates erode the same behavior, repeatedly removing that component can slow its cumulative decline.

4. Scope of the contribution (Abstract, §1, §6). Our goal is to mitigate policy forgetting within a single training run. Maximum-return replay selects which behavior to protect, and projection determines how to protect it during an update. Because the policy shares parameters across states, protection can influence behavior beyond the goals that supplied the replay data. We now state explicitly in §6 that how much retaining particular behaviors contributes to end-to-end gains remains open. This question concerns how regularization on recorded actions generalizes to held-out goals through shared policy parameters.

Reviewer_3ZtKOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Success-Constrained Policy Gradient Projection, an constrained RL training algorithm that combats catastrophic forgetting in a stationary RL problem. Essentially, the authors posit that even in a stationary task distribution, GRPO and GiGPO can forget prior successful trajectories, leading to policy collapse. Their proposed methodology combats this by identifying historical successful trajectories, and replaying them under the latest snapshot. Using the gradients from the replay, the policy gradient can be adjusted to not incur forgetfulness on successful trajectories up to first order effects in the loss. They evaluate this approach in two agentic RL task setups, WebShop and ALFWorld. They show that on Webshop, this approach improves final success rate by 2-6%, and on ALFWorld improves the final checkpoint success rate, while having unclear effects on the success rate from the best validation loss checkpoint.

Critical strengths

First, I'd like to thank the authors for their submission. I found this paper quite intriguing:

  • The paper is well-written. The organization of the content, narrative, and level of detail is expertly crafted. Well done!
  • The idea is interesting as a general approach for training stabilization in high-variance RL tasks. I do believe this approach is usable beyond just agentic RL setup in particular.
  • The experimental results are promising.
Critical weaknesses

Despite my enthusiasm, there are several flaws that undermine the potential of this work:

  • I wish the authors would include a theoretical basis for why this approach improves stability beyond intuition. I understand the logic behind advantages being positive and that this implies gradient updates should not reduce the likelihood of successful trajectories. But there is a logical leap between "successful tasks should not become less likely under training" and "this constraint improves long-term success". Even with a policy based on an ideal function approximator with infinite expressiveness (e.g., assume an infinitely wide multi-layer FCN), it is not clear why this would improve overall success rate after multiple training iterations. To be clear, I do not mean you need a theoretical analysis with an infinitely expressive function approximator! This is just to say that even in that scenario where plasticity is not a concern, I am not sure why this would improve overall success rate.
  • While the experimental results are promising, I wish authors had more tasks in their evaluations. I also wish this extra constraint would be evaluated in tasks that are not so unstable to learn, to understand what the downside to this constrained optimization approach is. There will be a downside (unless there is theory showing not), and if we are to use this as a general RL approach, it is important to understand broadly where it helps and where it doesn't. To be clear, I don't think these tasks have to necessarily be agentic RL; nothing in this work, to me, ties it to agentic RL scenarios. It seems broader than that and applicable even to more classical RL tasks. If agentic RL tasks are too expensive to add, you should experiment with more classical environments.
  • I think the abstract and intro should be more clear about what the goal of this constrained optimization is. Is it maximizing average return by stabilizing training? Is it maximizing the best attained return all over during the training run? Is it about avoiding overfitting in low-data RL training regimes? I find the objective to be somewhat vague, and I had to interpret it after reading the whole paper.
  • I think there is a discussion about plasticity here. This constraint, conceptually, forces the function approximator to look for a gradient direction that maximizes the policy gradient objective while not changing the policy in certain locations. This always inhibits training in setups where the function approximator is constrained in its expressiveness. This may not be the case in agentic RL, since LLMs are high-dimensional and it is not difficult to find a gradient direction that achieves both. But in more parameterc-constrained scenarios, I think this approach will hinder eventual learning. This can potentially be evaluated with an ablation on model size, or simply noted in the discussion section.

I think this paper has great potential. The above weaknesses are meant to be guides for authors to achieve that potential.

Questions for authors

Please see weaknesses.

1 comment

Thank you for your careful reading of our paper and your constructive feedback.

Q1. Why protect maximum-return trajectories?

Consider a tabular setting where each task is a state and each complete trajectory is an action. A feasible shift of probability from a lower-return trajectory to a maximum-return trajectory improves expected return on that task. Holding the other tasks’ policies fixed, successive such shifts preserve or improve average return over the fixed task distribution. An idealized network inherits this result if it realizes the same sequence of policies.

With shared parameters, an update on one task can change behavior on others. We take a regularization perspective and choose an anchor specifying which historical behavior to preserve. For a task, let pHp_H be the trajectory distribution induced by a mixture of past policies and OO the maximum-return event. If pH(O)>0p_H(O)>0, the anchor qH=pH(⋅∣O)q_H=p_H(\cdot\mid O) minimizes DKL(u∥pH)D_{\mathrm{KL}}(u\Vert p_H) over distributions uu supported on OO. It preserves the relative probabilities of oracle trajectories. RL’s Razor, Lemma A.1.

Reference KLSuccess-constrained projection
Behavior to preserve at a given historyAction distribution under the reference policyAction distribution under qHq_H
How it enters trainingPenalize divergence from the reference policyUse sampled successes to estimate replay loss. Project the RL gradient when a descent step would increase that loss to first order.

The tabular argument explains the choice of anchor. Projection limits first-order increases in the sampled replay loss, and we assess the resulting policy by its final return (Sec. 3.1 and Appendix A.3).

Q2. Additional tasks

We compared PPO with and without protection on four MuJoCo locomotion tasks. Each run used 3M interactions and two 64-unit hidden layers. We sampled four of the 64 most recent trajectories meeting both a rolling 75th-percentile return threshold and a fixed pilot threshold. Entries are final returns (mean ± SD across ten paired training seeds, 100 evaluation episodes per policy).

EnvironmentPPOPPO + projection
HalfCheetah-v52,280 ± 1,2542,272 ± 1,176
Hopper-v52,029 ± 8852,911 ± 902
Walker2d-v53,009 ± 1,1893,504 ± 462
Ant-v53,701 ± 6323,961 ± 720
Sum across tasks11,019 ± 1,70312,649 ± 1,428

Summing the four raw returns within each seed, protection raised aggregate return by 14.8%. Nine of ten paired sums improved (two-sided paired tt-test, p=0.023p=0.023, paired bootstrap 95% CI for the mean gain [525, 2,735]). Hopper gained the most and HalfCheetah changed little (Table 4).

On CartPole, all ten narrow PPO policies reached mean return ≥475 during training, but four finished below 300. All ten protected policies finished at the maximum of 500. With wider networks both methods finished at 500, and protection improved average return over training (Table B9).

CartPole hidden units per layerEvaluation return (mean ± SD, 10 seeds)PPOPPO + success projection
16Final377.18 ± 163.21500.00 ± 0.00
16Average over training339.14 ± 57.72397.02 ± 35.37
64Final500.00 ± 0.00500.00 ± 0.00
64Average over training437.51 ± 21.92450.08 ± 20.36

Q3. Which performance objective?

The revised Introduction (pp. 1–2) now states our two research questions explicitly: which historical trajectories to protect and how to protect them adaptively during policy updates. For both comparisons, we evaluate the final checkpoint by measuring end-to-end task success on held-out goals. To address the first question, Section 5.2 (p. 8) now reports a WebShop experiment comparing maximum-return replay with random-history replay while holding the projection rule fixed. We hope these revisions address your concerns about clarity.

Q4. Plasticity with limited capacity

Let cc be the cosine between the RL and replay loss gradients. When c<0c<0, projection gives up a fraction c2c^2 of the first-order decrease in the RL loss at the same learning rate. When c≥0c\geq0, it leaves the update unchanged (Appendix A.3).

MethodWidth 16 final returnWidth 64 final return
PPO377.18 ± 163.21500.00 ± 0.00
Success-history projection500.00 ± 0.00500.00 ± 0.00

With 16 units per hidden layer, the paired mean final gain was 122.82 (bootstrap 95% CI [30.49, 220.19]), and average return during training rose from 339.14 to 397.02. Thus protection can improve performance even in a narrow policy network (Appendix B.10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.