acceptodds
Under review as a conference paper at ICLR 2027

Rollout Aggregation Stabilizes On-Policy Reinforcement Learning

Abstract

Large-scale parallel PPO can struggle on complex, long-horizon tasks with high-dimensional action spaces. Prior work has emphasized exploration bottlenecks and diminishing returns from increasing environment parallelism. We instead present evidence that learning instability offers a compelling alternative explanation for PPO's poor performance. To improve stability, we introduce rollout aggregation (RAgger), a minimal modification that retains the previous rollout and trains on it alongside newly collected samples. We show that overlapping rollout batches regularize the policy's optimization trajectory by preserving pressure to retain recently learned behavior while incorporating new experience. Across several challenging manipulation tasks, RAgger prevents the collapse observed with standard PPO and outperforms strong baselines, including SAPG and CPO, with several-fold improvements in sample efficiency. Our findings motivate studying recent-data reuse as a mechanism for stabilizing policy learning and achieving strong, sample-efficient performance at scale.

Primary area:
reinforcement learning
Keywords:
on-policy reinforcement learning, proximal policy optimization, large-scale reinforcement learning, dexterous manipulation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_VajhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary. The paper attributes PPO's failure on long-horizon dexterous manipulation at scale to instability rather than insufficient exploration, and proposes RAGGER: train each update on the previous rollout together with the new one. This prevents PPO's collapse, outperforms SAPG and CPO on four AllegroKuka tasks, and improves PPO on hand-object tracking.

Critical strengths

S1. The presentation is clear and easy to follow; the method itself is a minimal, well-motivated modification of PPO.

S2. The ablations are well designed and give a good interpretation of where the performance gains come from.

S3. The experimental settings are clear and rigorous: challenging tasks, strong baselines, multiple seeds, and full reproducibility details.

Critical weaknesses

W1. Novelty. AMBER (Han & Sung, 2018) and off-policy PPO (Meng et al., 2023) already train PPO on the current batch together with past batches, with importance ratios computed against the collecting policy; parameter variants of these methods (e.g., reusing a single past batch) correspond to the core of this work. The main difference is the setting: massively parallel training. In this regime, however, data is abundant by construction, so the sample-efficiency motivation behind this line of work, and the paper's sample-efficiency claims are of limited importance.

W2. Work sanity. Fig. 3 shows no seed aggregation and no shaded regions: it plots raw per-seed curves, and the caption's claim ("Lines show mean over 3 seeds, and shaded regions show ±1 standard deviation") does not match what is shown. The figure quality (axis fonts, plotting) is also careless. These issues substantially reduce the credibility of the results and fall below the quality expected at ICLR.

Separately: the paper lists "how many rollouts to carry" as an open limitation, but this is answerable at the paper's own ablation scale (2B frames, 3 seeds), where the failure mode already manifests (Fig. 4c). Carrying K=2–3 rollouts would also directly test the paper's own claim in Sec. 4.4 that older rollouts are too stale to help. I ask the authors to add this ablation on Regrasping and Reorientation.

W3. Baseline fairness. PPO, SAPG and CPO all run in fp16 (the upstream default), while RAGGER runs in fp32 (mixed_precision=False) on the AllegroKuka tasks; this difference is not reported in the paper and appears only in the code repository. In addition, each RAGGER update trains on 2× data and performs 2× gradient compute per collected frame: frames are matched, but FLOPs and wall-clock are not, and no wall-clock numbers are reported anywhere.

W4. Benchmark coverage. The number of tasks and baselines is limited and not convincing: all results come from dexterous manipulation in a single simulator ecosystem.

W5. Missing baselines. The closest prior methods — AMBER, off-policy PPO — are cited but never run as baselines.

Questions for authors

Q1. How does RAGGER differ from AMBER (Han & Sung, 2018) restricted to a single past batch, and from off-policy PPO (Meng et al., 2023)? What is the methodological delta beyond this prior work?

Q2. Fig. 3's caption states "Lines show mean over 3 seeds, and shaded regions show ±1 standard deviation", but the figure appears to plot raw per-seed curves with no aggregation and no shading. Could you clarify? Relatedly, why do curves within the same panel terminate at different frame counts?

Q3. Could you add the K=2–3 carried-rollout ablation on Regrasping and Reorientation at your ablation scale (2B frames, 3 seeds)?

Q4. Why does RAGGER run in fp32 while PPO, SAPG and CPO all run in fp16, and why is this difference not reported in the paper? Could you provide a PPO+fp32 control, and report wall-clock / update-time comparisons?

Q5. Could you extend the evaluation beyond dexterous manipulation tasks in a single simulator ecosystem, and run the closest prior methods (AMBER, off-policy PPO) as baselines?

2 comments

We thank reviewer Vajh for their time and constructive feedback. We address the main concerns below:

Novelty. AMBER (Han & Sung, 2018) and off-policy PPO (Meng et al., 2023) already train PPO on the current batch together with past batches, with importance ratios computed against the collecting policy … the main difference is the setting: massively parallel training

As the reviewer points out, the setting is very different: AMBER's entire training run (~10^6 steps) is shorter than three of our rollouts, while we train for 20B frames with 24,576 parallel environments. Beyond this, the two papers differ in their goals, tasks, analysis and findings. AMBER evaluates on Gym tasks with at most 17 action dimensions, and even finds that aggregating rollouts degrades performance on high-action-dimensional tasks like Humanoid. We study long-horizon dexterous manipulation with 23 and 46 action dimensions, ask why large-scale PPO fails there, and find the opposite.

In this regime, however, data is abundant by construction…

We agree (our ablations in Sec 4.2 show this), and this is what makes our finding non-trivial. PPO collapses inspite of data being abundant, showing reuse plays a different role in this setting, which we analyze in App. E and Sec. 4.4.

…so the sample-efficiency motivation behind this line of work, and the paper's sample-efficiency claims are of limited importance.

We should clarify: our paper does not attribute RAGGER's gains to sample efficiency. RAGGER is introduced to improve stability, and on the AllegroKuka tasks it improves on every axis we measure, not just sample efficiency: it achieves the best final success of all methods (Table 1), and reaches SAPG's and CPO's final performance in less wall-clock time. Nor is sample efficiency the mechanism in this paper: PPO does not merely learn more slowly but gets worse as it trains, and giving PPO more data per update or more passes per sample does not close the gap (Sec. 4.3). The sample-efficiency gains are a consequence of stable learning, not the cause.

Fig. 3 doesn’t match caption.

We thank the reviewer for pointing out this error in the caption. The lines show the traces of the 3 individual seeds. We have updated the diagram.

why do curves within the same panel terminate at different frame counts?

DexMachina involves a 70-step curriculum (L304), for which graduation depends on high average reward. After this point the task is considered a success, and training ends. We have updated the diagram.

Carrying K=2–3 rollouts would also directly test the paper's own claim in Sec. 4.4 that older rollouts are too stale to help. I ask the authors to add this ablation on Regrasping and Reorientation.

We have added this experiment in the paper, on Regrasping and Reorientation at the ablation scale (2B frames, 3 seeds). We empirically find that K=2 is comparable with our original setting, while K=3 degrades performance.

RegraspingReorientation
PPO0.41 ± 0.211.08 ± 0.68
K = 1 (paper)37.22 ± 0.5427.57 ± 3.53
K = 236.17 ± 0.2421.76 ± 1.79
K = 332.80 ± 1.135.08 ± 2.71

Baseline fairness. PPO, SAPG and CPO all run in fp16, while RAGGER runs in fp32

We have updated the hyperparameter tables to report the precision of every method. We ablated the importance of this setting, and find no difference in RAGGER's performance between fp16 and fp32. PPO continues to collapse with fp32. Since RAGGER performs the same in fp16, the comparison with the baselines is not affected by precision.

TaskPPO, mixed FP16PPO, FP32PPO+RAGGER, FP32PPO+RAGGER, mixed FP16
Regrasping0.41 ± 0.210.23 ± 0.0837.22 ± 0.5437.17 ± 0.82
Reorientation1.08 ± 0.680.25 ± 0.0927.57 ± 3.5329.39 ± 1.11
Two-Arms Reorientation0.03 ± 0.020.81 ± 0.949.82 ± 3.7713.95 ± 6.20

each RAGGER update trains on 2× data and performs 2× gradient compute per collected frame: frames are matched, but FLOPs and wall-clock are not

We have ablated both differences in Sec. 4.3 of the main paper, and find that neither explains RAGGER's gains.

no wall-clock numbers are reported anywhere

We have added the wall-clock times (in hours) for all methods to the paper. Because RAGGER learns much faster, it still reaches SAPG's and CPO's final performance in a fraction of their wall-clock time.

(* indicates that other GPUs were used; L40S was used for the rest)

TaskPPOSAPGCPOPPO+RAGGER
Shadow Hand61.6*48.455.655.7
Allegro Hand81.893.9*80.686.4
Regrasping54.858.882.9*71.0
Reorientation50.261.272.472.9
Throw54.464.072.576.2
Two-Arms Reorientation97.2115.2*138.1*127.9

Benchmark coverage. The number of tasks and baselines is limited and not convincing: all results come from dexterous manipulation in a single simulator ecosystem.

In the main paper, we restricted ourselves to the DexPBT tasks studied in prior work (SAPG, CPO), and in addition included object tracking. We note that the object-tracking tasks run in Genesis [2], so the results already span two simulators. We have also added results on 4 standard Isaac Gym tasks in App.D. We find that RAGGER matches or outperforms PPO on most tasks and settings. We also find that recomputing advantages (instead of freezing them) helps in this setting.

Missing baselines. The closest prior methods — AMBER, off-policy PPO — are cited but never run as baselines.

We chose baselines that address the same problem as our paper: improving PPO in the massively parallel setting for long-horizon high-dimensional tasks. SAPG and CPO were designed specifically for this setting and evaluated on these tasks, and both report outperforming DexPBT. AMBER and off-policy PPO instead target sample efficiency in small-batch settings. We also note that the closest instantiation of PPO-MBER in our setting, a single carried rollout with frozen advantages, is essentially our frozen-advantage variant.

[1] AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control, Han & Sung, 2018

[2] DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation, Mandi et al., 2025

We hope we have addressed your questions. Please feel free to ask if you have more questions and we will be happy to answer.

Reviewer_nmHzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RAGGER, a simple variant of PPO where the last rollout batch trained on in addition to the current rollout batch. They find that this change enables significantly more stable and sample-efficient performance on a variety of long-horizon dexterous manipulation tasks. The authors attribute this improvement to a “local forgetting phenomenon” and introduce a metric and experiments to demonstrate this claim, along with various ablations to rule out other explanations for RAGGERs improved performance.

Critical strengths
  • The method is very simple to implement and yields improvements over baselines like SAPG or CPO on a wide range of long-horizon manipulation tasks (Figure 2). On tasks where PPO typically collapses or fails, RAGGER is able to successfully converge to a solution.

  • The paper ablates several plausible explanations for RAGGER’s improved performance (e.g. more sample reuse) and finds that they are not sufficient.

  • The paper formulates a metric to measure “local forgetting” and demonstrates that RAGGER forgets significantly less than PPO as training continues.

Critical weaknesses
  • The method is not novel and seems to be a special case of existing methods like AMBER, which also train on existing batches. Therefore, it seems most of the value of the work is more in the application and analysis of this particular set of tasks, along with the local forgetting experiments.

  • The experiments do not demonstrate strong causal relation between the local forgetting metric and PPO’s collapse on such tasks.

Questions for authors

In general, because the method is not novel, I would feel positive if I could get more insight into why the method works.

  • Can you report experimental results for repeating more than 2 batches? Does performance peak or plateau? It would be interesting to see this both for success/reward on the tasks and the proposed forgetting metric.
  • Is there a “minimal” amount of replay that needs to occur? I.E. would it be sufficient to say, replay the last batch once every N epochs. Does it need to be the last batch, or can we sample uniformly at random?
  • Does local forgetting still occur on easier tasks that PPO can solve? I.E. what does the local forgetting graph/chart look like when PPO converges well.
  • Can you discuss in further detail the key differences between this setting and the setting considered in AMBER? The claim being made is that although the methods are similar, the regimes and thus claims being considered are different.
3 comments

We thank reviewer nmHz for their time and constructive feedback. We address the main concerns below:

Can you report experimental results for repeating more than 2 batches? Does performance peak or plateau?

We try this experiment in App.B.5. We empirically find that K=2 is comparable with our original setting (K=1), while K=3 degrades performance. We hypothesize that this is due to older rollouts being optimized repeatedly.

RegraspingReorientation
PPO0.41 ± 0.211.08 ± 0.68
K = 1 (paper)37.22 ± 0.5427.57 ± 3.53
K = 236.17 ± 0.2421.76 ± 1.79
K = 332.80 ± 1.135.08 ± 2.71

Is there a “minimal” amount of replay that needs to occur? I.E. would it be sufficient to say, replay the last batch once every N epochs.

We test this in App.B.5, by carrying the last rollout either 3 of every 4 updates, or once every two updates. We find that how often the rollout should be carried is task dependent: carrying less frequently is more harmful for Reorientation.

Carried rolloutRegraspingReorientation
PPO (none)0.41 ± 0.211.08 ± 0.68
Every epoch (paper)37.22 ± 0.5427.57 ± 3.53
3 of every 4 epochs33.91 ± 0.9816.92 ± 4.99
1 of every 2 epochs14.27 ± 10.602.15 ± 0.78

Does local forgetting still occur on easier tasks that PPO can solve? I.E. what does the local forgetting graph/chart look like when PPO converges well.

To examine how local forgetting relates to task reward in general, including on easier tasks, we take a PPO/RAGGER run and, midway through training, branch each into several variants of PPO and RAGGER, all starting from an identical checkpoint, so that differences in subsequent reward come only from the intervention. For each branch, we measure local forgetting with ρ, a normalized version of F that is comparable across tasks (App. E.1), together with the reward reached after the branch point. We study both the AllegroKuka and IsaacGym tasks, and observe the following:

  • On the AllegroKuka tasks, the number of successes after the intervention is inversely correlated with ρ: branches that forget more reach lower reward.
  • We also measure ρ against task reward for different frequencies of carrying the previous rollout, and again observe a strong correlation.
  • On the easier IsaacGym tasks, we observe that several traces clump together, for both PPO and RAGGER.

Together, these results show that local forgetting can predict the reward reached after the intervention. We refer the reviewer to App. E for the complete analysis and figures.

Key differences in setting between PPO and AMBER

Unlike AMBER [1], we tackle long-horizon, high-dimensional dexterous manipulation tasks in a massively parallel setting. AMBER's entire training run (~10^6 steps, on Gym tasks with at most 17 action dimensions) is shorter than three of our rollouts; we train for 20B frames with 24,576 parallel environments, on tasks with 23 and 46 action dimensions.

The goals of the two papers also differ. AMBER reuses past batches to improve sample efficiency, which matters when data is scarce. At our scale, data is abundant, and sample efficiency cannot explain RAGGER's gains.

As a consequence, the findings of the two papers are very different. For example, AMBER finds that aggregating rollouts degrades performance on high-action-dimensional tasks like Humanoid. On our tasks, which have even more action dimensions, we find the opposite. The main contribution of this paper is therefore the empirical finding: a simple change prevents PPO's collapse at scale and beats far more complicated methods, together with an analysis of why this works.

We hope we have addressed your questions. Please feel free to ask if you have more questions and we will be happy to answer.

[1] AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control, Han & Sung, 2018

The new experiments in the appendix address my questions about whether the forgetting metric correlates well with increases and decreases in performance, and provide stronger evidence for the validity of the forgetting explanation. Specifically, they show that in cases where you switch from PPO to RAGGER in the middle of training, you an increase in performance and decrease in the forgetting metric. Likewise, the reverse happens when switching from RAGGER to PPO. Although I agree with other reviewers that the method itself is not novel relative to AMBER, I think the depth of analysis for why the method works is sufficiently interesting and novel to warrant a new paper.

To the AC, I would be happy to champion the paper here. While other reviewers raise fair concerns about e.g. generalizability and algorithmic novelty, I think the key contribution here is significantly better analysis and insights over AMBER. While I maintain my score at a 3 as an "objective" measure of the paper's contributions, you may consider it as strongly/confidently positive.

Dear Reviewer nmHz,

Thank you again for your thoughtful feedback, which has helped improve our work, and for engaging with us during the discussion period.

Reviewer_Xt45OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies instability in large-scale PPO and proposes rollout aggregation (RAGGER), which trains on the immediately preceding rollout alongside newly collected data. The method retains each sample’s collecting policy as the reference for PPO probability ratios and recomputes advantages over the combined rollout window. Experiments on dexterous manipulation and hand–object tracking show improved stability and sample efficiency. Ablations suggest that larger batches or additional optimization passes alone do not explain the gains, while a surrogate-objective diagnostic shows reduced forgetting on recent rollouts.

Critical strengths
  1. The paper is clearly written and easy to follow. The motivation, algorithm, and experimental analysis are presented coherently, with useful illustrations and explicit descriptions of how RAGGER modifies PPO.

  2. The experiments demonstrate improvements on challenging dexterous manipulation tasks, with additional evidence from hand–object tracking. Comparisons against PPO, SAPG, and CPO show that a simple rollout-reuse scheme can be competitive with more elaborate approaches.

  3. The ablations examine plausible alternative explanations, including larger training batches, additional optimization passes, and recomputed advantages. The local-forgetting analysis provides further evidence consistent with the proposed stabilization mechanism.

Critical weaknesses
  1. The algorithm closely overlaps with PPO-MBER and its adaptive variant, AMBER [1], which already combine current and previous rollout batches using each sample’s collecting policy in the probability ratio. RAGGER’s default advantage recomputation is a distinction, but its ablations suggest that this is not essential on the tested tasks. Without a comparison against these prior methods, the algorithmic benefit beyond applying recent-rollout reuse at scale remains unclear.

  2. The local-forgetting metric evaluates the old-batch surrogate objective that RAGGER explicitly continues to optimize, so reduced deterioration is partly expected. Although the analysis supports better retention of recent training objectives, it does not establish that this retention explains the improvement in actual task performance. The proposed explanation for preventing collapse therefore remains suggestive.

[1] Seungyul Han and Youngchul Sung. “AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control.” arXiv:1710.04423v2, 2018.

Questions for authors
  1. Beyond advantage recomputation, what methodological differences distinguish RAGGER from PPO-MBER with a two-batch replay window? Given that the frozen-advantage variant performs comparably, clarifying this relationship would help assess the algorithmic contribution.

  2. Tables 2 and 3 indicate that value-function clipping is enabled for PPO but disabled for RAGGER. Was this difference also present in the ablations, and is there existing evidence that it does not account for the reported gains?

3 comments

We thank reviewer Xt45 for their time and constructive feedback. We address the main concerns below:

The algorithm closely overlaps with PPO-MBER and its adaptive variant, AMBER

Our frozen-advantage variant (Fig. 4a) corresponds to PPO-MBER [1] with L=2. Recomputing advantages performs comparably on Regrasping and Reorientation, but on the IsaacGym tasks it clearly outperforms the frozen variant (App. D). Beyond this, the two papers differ in almost every other respect: the setting, the tasks, the goals, the analysis and the findings. We tackle long-horizon, high-dimensional dexterous manipulation in a massively parallel setting. AMBER's entire training run (~10^6 steps, on Gym tasks with at most 17 action dimensions) is shorter than three of our rollouts; we train for 20B frames with 24,576 parallel environments, on tasks with 23 and 46 action dimensions.

The goals of the two papers also differ. AMBER reuses past batches to improve sample efficiency, which matters when data is scarce. At our scale, data is abundant, and sample efficiency cannot explain RAGGER's gains. We study why reuse helps at scale: not through more data per update or more passes per sample (Sec. 4.3), but by limiting how much each update undoes recently learned behavior (Sec. 4.4, App. E).

As a consequence, the findings of the two papers are very different. For example, AMBER finds that aggregating rollouts degrades performance on high-action-dimensional tasks like Humanoid. On our tasks, which have even more action dimensions, we find the opposite. The main contribution of this paper is therefore the empirical finding: a simple change prevents PPO's collapse at scale and beats far more complicated methods, together with an analysis of why this works.

The local-forgetting metric evaluates the old-batch surrogate objective that RAGGER explicitly continues to optimize, so reduced deterioration is partly expected.

This concern applies to the first two epochs of the forgetting experiment, but not to the later ones. In each superepoch (Algorithm 2), the first rollout D0D_0 is in RAGGER's training batch for only one epoch. From the third epoch on, RAGGER trains on its two most recent rollouts, which no longer include D0D_0.

Although the analysis supports better retention of recent training objectives, it does not establish that this retention explains the improvement in actual task performance.

We have added experiments that test the link directly. We take a PPO/RAGGER run and, midway through training, branch each into several variants of PPO and RAGGER, all starting from an identical checkpoint, so that differences in subsequent reward come only from the intervention. For each branch, we measure local forgetting with ρ, a normalized version of F that is more comparable across tasks (App. E.1), together with the reward reached after the branch point.

We observe that forgetting predicts subsequent reward. On the AllegroKuka tasks, the number of successes after the intervention is inversely correlated with ρ (App. E.2): branches that forget more reach lower reward.

Together, these results show that local forgetting consistently predicts the reward reached after the intervention. We refer the reviewer to App. E for the complete analysis and figures.

value-function clipping is enabled for PPO but disabled for RAGGER … existing evidence that it does not account for the reported gains?

Thank you to the reviewer for pointing this out. We preserve the settings listed in the hyperparameter tables for the ablations. We have added the ablation for value clipping for RAGGER, and find that it has little effect on performance on all six tasks.

(Table is for 5 seeds, 20B frames)

TaskPPO+RAGGER, no value clippingPPO+RAGGER, with value clipping
Shadow Hand12321 ± 42212533 ± 163
Allegro Hand12891 ± 106312707 ± 821
Regrasping40.49 ± 0.4240.65 ± 1.04
Reorientation44.58 ± 0.6343.39 ± 0.82
Throw23.47 ± 0.7521.78 ± 1.49
Two-Arms Reorientation44.13 ± 0.6140.95 ± 5.65

We hope we have addressed your questions. Please feel free to ask if you have more questions and we will be happy to answer.

[1] AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control, Han & Sung, 2018

Thanks to the authors for the explanations and additional experiments. My concerns are largely addressed. Although I still feel the new experiments cannot fully establish forgetting as the causal mechanism, the PPO/PPO+RAGGER switching and interpolation results in Appendix E strengthen the link between forgetting and task performance. While the methodological novelty remains limited, I find this empirical contribution valuable to the community. I am therefore raising my score.

Dear Reviewer Xt45,

Thank you again for your constructive review, which has helped strengthen the paper, and for following up during the discussion period.

Based on the latest feedback, we will revise Sec 4.4 and App. E, and present ρ\rho as a diagnostic that correlates with collapse with local forgetting as the explanation these experiments support.

Reviewer_bCrjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that PPO's poor performance in large-scale, long-horizon dexterous manipulation stems primarily from learning instability rather than insufficient exploration. It proposes RAGGER, a simple rollout aggregation method that trains PPO on both current and previous rollout batches without changing its architecture or hyperparameters. Across AllegroKuka and DexMachina tasks, RAGGER reduces policy collapse, improves sample efficiency by up to ~4×, and generally matches or outperforms strong baselines (SAPG, CPO). Ablations suggest the gains are not explained by additional data, critic recomputation, or extra optimization passes. A proposed local forgetting metric further supports RAGGER's ability to preserve recently learned behaviors.

Critical strengths
  1. A remarkably simple modification outperforms substantially more complex leader-follower methods (SAPG, CPO). RAGGER achieves superior performance on all four AllegroKuka tasks with minimal changes to PPO. This is practically significant and challenges the assumption that policy diversity or population-based optimization is necessary for effective large-scale parallel RL.

  2. Well-designed ablations rule out key alternative explanations for the performance gains. Figure 4 controls for both data volume per update and optimization passes per sample, showing that neither alone reproduces RAGGER's improvements. These results provide convincing evidence that re-optimizing the previous rollout's surrogate objective is the key contributing factor, although the underlying stabilization mechanism remains theoretically unexplained.

Critical weaknesses
  1. The proposed stability mechanism is not causally established. The local forgetting metric is defined using the same surrogate objective that RAGGER explicitly re-optimizes, making reduced forgetting partly an expected consequence of the method's design. While this demonstrates objective preservation, the paper does not establish that reduced surrogate forgetting causally explains improved task performance. This weakens the central "stability rather than exploration" interpretation, leaving a strong empirical result without a fully validated mechanism.

  2. Limited generalizability beyond dexterous manipulation. All experiments focus on closely related simulated dexterous manipulation and tracking tasks. Although the results are consistent across multiple benchmarks, it remains unclear whether rollout aggregation provides similar stability benefits in other RL domains or under different levels of environment parallelism. This limits the evidence supporting the paper's broader claims about stabilizing on-policy reinforcement learning.

  3. If AMBER already demonstrated that reusing previous PPO rollouts improves convergence speed and stability, what is the distinct algorithmic contribution of RAGGER? Why is AMBER not included as a baseline?

Questions for authors

Can the authors establish that reduced local forgetting explains the reward gains, rather than merely reflecting RAGGER's optimization objective? The forgetting metric F is defined on the clipped surrogate that RAGGER explicitly re-optimizes, making reduced forgetting partly expected by construction. A per-seed/per-task analysis relating F to subsequent reward improvements would provide supporting evidence, while an independent intervention that reduces forgetting without RAGGER would offer stronger causal evidence. A convincing answer would improve my assessment of Soundness and Informativeness.

3 comments

We thank reviewer bCrj for their time and constructive feedback. We address the main concerns below:

The local forgetting metric is defined using the same surrogate objective that RAGGER explicitly re-optimizes, making reduced forgetting partly an expected consequence of the method's design.

This concern applies to the first two epochs of the forgetting experiment, but not to the later ones. In each superepoch (Algorithm 2), the first rollout D0D_0 is in RAGGER's training batch for only one epoch. From the third epoch on, RAGGER trains on its two most recent rollouts, which no longer include D0D_0.

Can the authors establish that reduced local forgetting explains the reward gains…while an independent intervention that reduces forgetting … would offer stronger causal evidence.

We thank the reviewer for this suggestion and have run such interventions (App. E). We take a PPO/RAGGER run and, midway through training, branch it into several variants of PPO and RAGGER, all starting from the identical checkpoint, so that differences in subsequent reward only come from the intervention. For each branch, we measure local forgetting with ρ, a normalized version of F that is more comparable across tasks (App. E.1), together with the reward reached after the branch point.

We observe that forgetting predicts subsequent reward. On the AllegroKuka tasks, we perform various interventions and find that the number of successes after the intervention is inversely correlated with ρ (App.E.2): branches that forget more reach lower reward and often collapse. App. E.2 also analyzes the effect of mini-epoch count in PPO, where more mini-epochs are able to lower ρ independently and indeed, usually end up with higher success. So reducing forgetting even within PPO improves performance.

We have since performed various other independent interventions that we would be happy to discuss if the reviewer wishes.

Together, these results show that local forgetting consistently predicts the reward reached after the intervention. We refer the reviewer to App. E for the complete analysis and figures.

Limited generalizability beyond dexterous manipulation.

We restricted ourselves to the DexPBT tasks studied in prior work (SAPG, CPO), and in addition included object tracking. We have extended our results to 4 standard Isaac Gym tasks in App.D, under various settings. We find that RAGGER matches or outperforms PPO on most tasks and settings. We also find that recomputing advantages (instead of freezing them) helps in this setting.

Distinction between AMBER and RAGGER

Our frozen-advantage variant (Fig. 4a) corresponds to PPO-MBER [1] with L=2. Recomputing advantages performs comparably on Regrasping and Reorientation, but on the IsaacGym tasks it clearly outperforms the frozen variant (App. D). Beyond this, the two papers differ in almost every other respect: the setting, the tasks, the goals, the analysis and the findings. We tackle long-horizon, high-dimensional dexterous manipulation in a massively parallel setting. AMBER's entire training run (~10^6 steps, on Gym tasks with at most 17 action dimensions) is shorter than three of our rollouts; we train for 20B frames with 24,576 parallel environments, on tasks with 23 and 46 action dimensions.

The goals of the two papers also differ. AMBER reuses past batches to improve sample efficiency, which matters when data is scarce. At our scale, data is abundant, and sample efficiency cannot explain RAGGER's gains. We study why reuse helps at scale: not through more data per update or more passes per sample (Sec. 4.3), but by limiting how much each update undoes recently learned behavior (Sec. 4.4, App. E).

As a consequence, the findings of the two papers are very different. For example, AMBER finds that aggregating rollouts degrades performance on high-action-dimensional tasks like Humanoid. On our tasks, which have even more action dimensions, we find the opposite. The main contribution of this paper is therefore the empirical finding: a simple change prevents PPO's collapse at scale and beats far more complicated methods, together with an analysis of why this works.

We hope we have addressed your questions. Please feel free to ask if you have more questions and we will be happy to answer.

[1] AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control, Han & Sung, 2018

Dear Reviewer bCrj,

Thank you again for your review. As the discussion period nears its end, we wanted to check whether our response addresses your concerns.

Thank you for the Appendix E interventions and for clarifying the diagnostic role of ρ. The additional experiments strengthen the empirical evidence, including the extension to standard Isaac Gym tasks. However, the proposed causal explanation remains insufficiently isolated, as interventions that reduce forgetting also alter optimization dynamics. While the results support ρ as a useful diagnostic, they do not fully establish reduced forgetting as the mechanism driving reward improvements. I will maintain my rating of 2 and appreciate the more cautious framing of the claims.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank all reviewers for their time and constructive feedback. We are encouraged that they broadly agree on the empirical value of the paper, the strength of its results, and the quality and rigor of its experiments and ablations:

  • Reviewer bCrj highlights that a "remarkably simple modification outperforms substantially more complex leader-follower methods", which is "practically significant and challenges the assumption that policy diversity … is necessary", and finds the ablations "well-designed".
  • Reviewer Xt45 concurs that "a simple rollout-reuse scheme can be competitive with more elaborate approaches", and finds "this empirical contribution valuable to the community".
  • Reviewer nmHz also agrees that "the method is very simple to implement and yields improvements over baselines", and highlights the ablations, analysis and insights.
  • Reviewer Vajh finds the ablations "well designed" and the experimental settings "clear and rigorous".

Summary of Discussion Period

During the rebuttal, reviewers brought up two main concerns. We summarize the main points below:

  1. Novelty of the algorithmic contribution. The overlap of RAGGER with AMBER [1] was discussed; specifically, the version of RAGGER with frozen advantages can be viewed as MBER with one carried batch. We highlighted two main points:

    1. The major contributions of this paper are empirical, and are substantially different from AMBER's. We evaluate the method in a completely different regime: AMBER's entire training run (~10^6 steps, on Gym tasks with at most 17 action dimensions) is shorter than three of our 50,000+ rollouts on higher-dimensional tasks. The study itself is also different: AMBER reuses past batches to improve sample efficiency in a setting where samples are not abundant, while this paper analyzes the stability of PPO in a massively parallel setting for large-scale, long-horizon tasks, where data is abundant. Aggregation helps here for a different reason: not through additional data or optimization passes, but by limiting how much each update undoes recently learned behavior (Sec. 4.4, App. E). Consequently, the analysis, experiments and ablations differ entirely, as do the conclusions (for instance, we find that aggregating rollouts helps on high-dimensional tasks, while AMBER finds the opposite).
    2. Recomputing advantages, the main algorithmic difference from MBER, performs comparably to freezing them on the AllegroKuka tasks, but clearly improves performance on the Isaac Gym tasks (App. D).

    We have revised the paper accordingly and toned down claims of algorithmic novelty.

  2. Analysis of local forgetting. Following the discussion with the reviewers, we added several experiments (App. E) that intervene on the update from an identical mid-training checkpoint and track both success and local forgetting. We measured local forgetting by ρ\rho, the metric of Sec. 4.4, normalized by each batch's own gain so that it is comparable across tasks. Specifically, we showed that:

    • Switching the update rule between PPO and RAGGER makes the metric jump, after which a corresponding collapse or recovery follows.
    • Interpolating between the two update rules interpolates the metric, which is paralleled by the success rate.
    • Other interventions that do not change the update rule (e.g., the number of mini-epochs within PPO, an intervention without RAGGER as Reviewer bCrj suggested) also show that lower ρ\rho corresponds to higher success, and vice versa.

    We have revised the writing to present ρ\rho as a diagnostic that tracks collapse, with local forgetting as the explanation these experiments support.

We thank the reviewers again for engaging with us during the discussion period and for helping to strengthen the paper.

[1] AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control, Han & Sung, 2018

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.