- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary. The paper attributes PPO's failure on long-horizon dexterous manipulation at scale to instability rather than insufficient exploration, and proposes RAGGER: train each update on the previous rollout together with the new one. This prevents PPO's collapse, outperforms SAPG and CPO on four AllegroKuka tasks, and improves PPO on hand-object tracking.
Critical strengths
S1. The presentation is clear and easy to follow; the method itself is a minimal, well-motivated modification of PPO.
S2. The ablations are well designed and give a good interpretation of where the performance gains come from.
S3. The experimental settings are clear and rigorous: challenging tasks, strong baselines, multiple seeds, and full reproducibility details.
Critical weaknesses
W1. Novelty. AMBER (Han & Sung, 2018) and off-policy PPO (Meng et al., 2023) already train PPO on the current batch together with past batches, with importance ratios computed against the collecting policy; parameter variants of these methods (e.g., reusing a single past batch) correspond to the core of this work. The main difference is the setting: massively parallel training. In this regime, however, data is abundant by construction, so the sample-efficiency motivation behind this line of work, and the paper's sample-efficiency claims are of limited importance.
W2. Work sanity. Fig. 3 shows no seed aggregation and no shaded regions: it plots raw per-seed curves, and the caption's claim ("Lines show mean over 3 seeds, and shaded regions show ±1 standard deviation") does not match what is shown. The figure quality (axis fonts, plotting) is also careless. These issues substantially reduce the credibility of the results and fall below the quality expected at ICLR.
Separately: the paper lists "how many rollouts to carry" as an open limitation, but this is answerable at the paper's own ablation scale (2B frames, 3 seeds), where the failure mode already manifests (Fig. 4c). Carrying K=2–3 rollouts would also directly test the paper's own claim in Sec. 4.4 that older rollouts are too stale to help. I ask the authors to add this ablation on Regrasping and Reorientation.
W3. Baseline fairness. PPO, SAPG and CPO all run in fp16 (the upstream default), while RAGGER runs in fp32 (mixed_precision=False) on the AllegroKuka tasks; this difference is not reported in the paper and appears only in the code repository. In addition, each RAGGER update trains on 2× data and performs 2× gradient compute per collected frame: frames are matched, but FLOPs and wall-clock are not, and no wall-clock numbers are reported anywhere.
W4. Benchmark coverage. The number of tasks and baselines is limited and not convincing: all results come from dexterous manipulation in a single simulator ecosystem.
W5. Missing baselines. The closest prior methods — AMBER, off-policy PPO — are cited but never run as baselines.
Questions for authors
Q1. How does RAGGER differ from AMBER (Han & Sung, 2018) restricted to a single past batch, and from off-policy PPO (Meng et al., 2023)? What is the methodological delta beyond this prior work?
Q2. Fig. 3's caption states "Lines show mean over 3 seeds, and shaded regions show ±1 standard deviation", but the figure appears to plot raw per-seed curves with no aggregation and no shading. Could you clarify? Relatedly, why do curves within the same panel terminate at different frame counts?
Q3. Could you add the K=2–3 carried-rollout ablation on Regrasping and Reorientation at your ablation scale (2B frames, 3 seeds)?
Q4. Why does RAGGER run in fp32 while PPO, SAPG and CPO all run in fp16, and why is this difference not reported in the paper? Could you provide a PPO+fp32 control, and report wall-clock / update-time comparisons?
Q5. Could you extend the evaluation beyond dexterous manipulation tasks in a single simulator ecosystem, and run the closest prior methods (AMBER, off-policy PPO) as baselines?
2 comments
Benchmark coverage. The number of tasks and baselines is limited and not convincing: all results come from dexterous manipulation in a single simulator ecosystem.
In the main paper, we restricted ourselves to the DexPBT tasks studied in prior work (SAPG, CPO), and in addition included object tracking. We note that the object-tracking tasks run in Genesis [2], so the results already span two simulators. We have also added results on 4 standard Isaac Gym tasks in App.D. We find that RAGGER matches or outperforms PPO on most tasks and settings. We also find that recomputing advantages (instead of freezing them) helps in this setting.
Missing baselines. The closest prior methods — AMBER, off-policy PPO — are cited but never run as baselines.
We chose baselines that address the same problem as our paper: improving PPO in the massively parallel setting for long-horizon high-dimensional tasks. SAPG and CPO were designed specifically for this setting and evaluated on these tasks, and both report outperforming DexPBT. AMBER and off-policy PPO instead target sample efficiency in small-batch settings. We also note that the closest instantiation of PPO-MBER in our setting, a single carried rollout with frozen advantages, is essentially our frozen-advantage variant.
[1] AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control, Han & Sung, 2018
[2] DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation, Mandi et al., 2025
We hope we have addressed your questions. Please feel free to ask if you have more questions and we will be happy to answer.
We thank reviewer Vajh for their time and constructive feedback. We address the main concerns below:
As the reviewer points out, the setting is very different: AMBER's entire training run (~10^6 steps) is shorter than three of our rollouts, while we train for 20B frames with 24,576 parallel environments. Beyond this, the two papers differ in their goals, tasks, analysis and findings. AMBER evaluates on Gym tasks with at most 17 action dimensions, and even finds that aggregating rollouts degrades performance on high-action-dimensional tasks like Humanoid. We study long-horizon dexterous manipulation with 23 and 46 action dimensions, ask why large-scale PPO fails there, and find the opposite.
We agree (our ablations in Sec 4.2 show this), and this is what makes our finding non-trivial. PPO collapses inspite of data being abundant, showing reuse plays a different role in this setting, which we analyze in App. E and Sec. 4.4.
We should clarify: our paper does not attribute RAGGER's gains to sample efficiency. RAGGER is introduced to improve stability, and on the AllegroKuka tasks it improves on every axis we measure, not just sample efficiency: it achieves the best final success of all methods (Table 1), and reaches SAPG's and CPO's final performance in less wall-clock time. Nor is sample efficiency the mechanism in this paper: PPO does not merely learn more slowly but gets worse as it trains, and giving PPO more data per update or more passes per sample does not close the gap (Sec. 4.3). The sample-efficiency gains are a consequence of stable learning, not the cause.
We thank the reviewer for pointing out this error in the caption. The lines show the traces of the 3 individual seeds. We have updated the diagram.
DexMachina involves a 70-step curriculum (L304), for which graduation depends on high average reward. After this point the task is considered a success, and training ends. We have updated the diagram.
We have added this experiment in the paper, on Regrasping and Reorientation at the ablation scale (2B frames, 3 seeds). We empirically find that K=2 is comparable with our original setting, while K=3 degrades performance.
We have updated the hyperparameter tables to report the precision of every method. We ablated the importance of this setting, and find no difference in RAGGER's performance between fp16 and fp32. PPO continues to collapse with fp32. Since RAGGER performs the same in fp16, the comparison with the baselines is not affected by precision.
We have ablated both differences in Sec. 4.3 of the main paper, and find that neither explains RAGGER's gains.
We have added the wall-clock times (in hours) for all methods to the paper. Because RAGGER learns much faster, it still reaches SAPG's and CPO's final performance in a fraction of their wall-clock time.
(* indicates that other GPUs were used; L40S was used for the rest)