- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines whether standard off-policy RL design choices remain beneficial under massively parallel simulation. Building on FlashSAC, it studies three components: the previously proposed Sample Weight Decay (SWD), parameter-projection normalization, and clipped double-Q learning. It introduces two recommended configurations: WarpSAC-L, which retains normalization and double-Q for CPU-scale training, and WarpSAC-A, which removes parameter projection and uses a single critic for GPU-parallel training. The central contribution is an empirical design prescription: stabilization should depend on the data regime, while recency-weighted replay is broadly useful. The paper reports improved aggregate sample efficiency over FlashSAC and faster training, although some wall-clock comparisons use proxy timing traces.
Critical strengths
The paper asks whether stabilization mechanisms inherited from conventional off-policy RL remain appropriate when simulation becomes highly parallel. This is consequential because unnecessary constraints can reduce both learning performance and computational efficiency. The resulting configurations are straightforward to implement and do not require auxiliary networks or a substantially more complicated training pipeline.
The evaluation spans locomotion, dexterous manipulation, and humanoid control across eight benchmark families, generally using five seeds. The network-capacity experiments add useful evidence about when replay weighting and parameter projection matter, while the Unitree G1 deployment provides a practical demonstration beyond simulator-only learning curves.
Critical weaknesses
WarpSAC primarily combines the previously proposed SWD mechanism with selective removal of parameter-projection normalization and a second critic from FlashSAC. It does not introduce a new learning objective, update rule, or network architecture; its algorithmic contribution is therefore largely a configuration prescription for existing components. A systematic empirical reassessment can constitute a meaningful contribution, but the paper needs to establish what substantially new understanding emerges beyond existing work on adapting off-policy algorithms to parallel simulation. The current experiments demonstrate useful configuration-dependent improvements over FlashSAC, yet do not sufficiently isolate the proposed data-coverage explanation or establish when the prescription generalizes.
the comparisons primarily establish improvements over FlashSAC, while closely related work is missing from the positioning: Raffin’s parallel-SAC studies (2025), Parallel Q-Learning (Li et al., 2023), and LIFT (Huang et al., 2026). These works address parallel off-policy training, configuration choices, or scalable SAC pretraining. Given that WarpSAC mainly recombines existing mechanisms, clearer differentiation and representative comparisons are important to establish the extent of its contribution.
[1] Antonin Raffin. Getting SAC to Work on a Massive Parallel Simulator: Tuning for Speed (Part II). Technical blog, July 2025.
[2] Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel Q-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulation. Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202:19440–19459, 2023.
[3] Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, and Jingwen Zhang. Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control. International Conference on Learning Representations (ICLR), 2026.
Questions for authors
-
Can the data-regime hypothesis be tested within the same simulator? Why do the data-limited experiments use separate CPU-scale benchmarks rather than the same GPU-based simulator with one or a small number of environments? Could the authors vary the number of parallel environments within the same task and simulator, while controlling the network architecture, update-to-data ratio, and training budget? Such a comparison would more directly test the proposed data-regime hypothesis by reducing confounding factors arising from differences in tasks, simulators, and implementations.
-
Do the conclusions hold under deterministic data collection? SSRL[4] use deterministic actions for real-world data collection while retaining stochastic exploration within learned world models. Motivated by this setting, could the authors compare the effects of SWD, parameter-projection normalization, and clipped double-Q when training data are collected using deterministic policy actions?
[4] Jacob Levy, Tyler Westenbroek, and David Fridovich-Keil. Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models. Proceedings of the 8th Conference on Robot Learning, PMLR 270:2061–2079, 2025.
I thank the authors for their response, which addresses my questions and concerns. Incorporating these clarifications into the final version will certainly improve the paper's overall quality. Overall, I will increase the score from 2 to 3.