acceptodds
Under review as a conference paper at ICLR 2027

WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation

Abstract

Scaling off‑policy reinforcement learning (RL) through massively parallel simulation changes the data regime assumptions, under which RL algorithms are designed. Canonical stabilizers are motivated by data‑limited training, where replay buffers provide narrow state–action coverage. By contrast, massively parallel simulation offers diverse experience at high throughput, naturally challenging the canonical roles of the stabilizers in this new data regime. Through comprehensive and controlled empirical study across eight benchmark families spanning CPU‑scale locomotion, GPU‑parallel robotic simulation, dexterous manipulation, humanoid whole‑body control and we find that these stabilizers are strongly data‑regime‑dependent: parameter normalization helps under narrow replay coverage but restricts value fitting when data are abundant, clipped double‑Q can be safely relaxed in high‑throughput manipulation, and age‑biased replay weighting is broadly useful for improving learning efficiency, especially under limited network capacity. Turning this analysis into a prescription, we build WarpSAC\xspace, a regime‑aware family of off‑policy RL algorithms. WarpSAC\xspace uses Sample Weight Decay as a regime‑agnostic component for efficient exploitation and matches each regime with a prescribed variant: WarpSAC\xspace‑L (Norm ON, clipped double‑Q) for data‑limited CPU‑scale training, and WarpSAC\xspace‑A (Norm OFF, single‑Q) for data‑abundant GPU‑parallel training. appendixpurpleWarpSAC\xspace improves normalized score–step AUC over FlashSAC by across nine CPU‑scale environments and across fourteen GPU‑parallel environments; lifts UnitreeG1TransportBox‑v1 success rate from to , and gains in mean normalized wall‑time AUC on MuJoCo Playground; achieves faster sim‑to‑real deployment on Unitree G1 than FlashSAC by 36.4% in terms of wall time. These results argue that scalable off‑policy RL should adapt its stabilizers to the available data regime. Under this principle, WarpSAC\xspace advances the state of the art of scalable off‑policy RL, delivering consistent gains over FlashSAC across different data regimes.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, off‑policy RL, parallel simulation, robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YLVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines whether standard off-policy RL design choices remain beneficial under massively parallel simulation. Building on FlashSAC, it studies three components: the previously proposed Sample Weight Decay (SWD), parameter-projection normalization, and clipped double-Q learning. It introduces two recommended configurations: WarpSAC-L, which retains normalization and double-Q for CPU-scale training, and WarpSAC-A, which removes parameter projection and uses a single critic for GPU-parallel training. The central contribution is an empirical design prescription: stabilization should depend on the data regime, while recency-weighted replay is broadly useful. The paper reports improved aggregate sample efficiency over FlashSAC and faster training, although some wall-clock comparisons use proxy timing traces.

Critical strengths

The paper asks whether stabilization mechanisms inherited from conventional off-policy RL remain appropriate when simulation becomes highly parallel. This is consequential because unnecessary constraints can reduce both learning performance and computational efficiency. The resulting configurations are straightforward to implement and do not require auxiliary networks or a substantially more complicated training pipeline.

The evaluation spans locomotion, dexterous manipulation, and humanoid control across eight benchmark families, generally using five seeds. The network-capacity experiments add useful evidence about when replay weighting and parameter projection matter, while the Unitree G1 deployment provides a practical demonstration beyond simulator-only learning curves.

Critical weaknesses

WarpSAC primarily combines the previously proposed SWD mechanism with selective removal of parameter-projection normalization and a second critic from FlashSAC. It does not introduce a new learning objective, update rule, or network architecture; its algorithmic contribution is therefore largely a configuration prescription for existing components. A systematic empirical reassessment can constitute a meaningful contribution, but the paper needs to establish what substantially new understanding emerges beyond existing work on adapting off-policy algorithms to parallel simulation. The current experiments demonstrate useful configuration-dependent improvements over FlashSAC, yet do not sufficiently isolate the proposed data-coverage explanation or establish when the prescription generalizes.

the comparisons primarily establish improvements over FlashSAC, while closely related work is missing from the positioning: Raffin’s parallel-SAC studies (2025), Parallel Q-Learning (Li et al., 2023), and LIFT (Huang et al., 2026). These works address parallel off-policy training, configuration choices, or scalable SAC pretraining. Given that WarpSAC mainly recombines existing mechanisms, clearer differentiation and representative comparisons are important to establish the extent of its contribution.

[1] Antonin Raffin. Getting SAC to Work on a Massive Parallel Simulator: Tuning for Speed (Part II). Technical blog, July 2025.

[2] Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel Q-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulation. Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202:19440–19459, 2023.

[3] Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, and Jingwen Zhang. Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control. International Conference on Learning Representations (ICLR), 2026.

Questions for authors
  1. Can the data-regime hypothesis be tested within the same simulator? Why do the data-limited experiments use separate CPU-scale benchmarks rather than the same GPU-based simulator with one or a small number of environments? Could the authors vary the number of parallel environments within the same task and simulator, while controlling the network architecture, update-to-data ratio, and training budget? Such a comparison would more directly test the proposed data-regime hypothesis by reducing confounding factors arising from differences in tasks, simulators, and implementations.

  2. Do the conclusions hold under deterministic data collection? SSRL[4] use deterministic actions for real-world data collection while retaining stochastic exploration within learned world models. Motivated by this setting, could the authors compare the effects of SWD, parameter-projection normalization, and clipped double-Q when training data are collected using deterministic policy actions?

[4] Jacob Levy, Tyler Westenbroek, and David Fridovich-Keil. Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models. Proceedings of the 8th Conference on Robot Learning, PMLR 270:2061–2079, 2025.

Reviewer_xUZEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WarpSAC, a family of FlashSAC-based training recipes for different data regimes. It combines existing Sample Weight Decay with different choices of parameter projection and critic count, recommending separate configurations for CPU-scale and GPU-parallel training. Experiments across eight benchmark families and a Unitree G1 sim-to-real case study report improvements in performance and training time.

Critical strengths
  • The evaluation is broad, with five-seed results and per-task curves across eight benchmark families. The capacity ablations provide useful detail about when replay weighting and parameter projection matter.

  • The changes are straightforward to apply. In the G1 case study, training reaches the target performance in about 35 minutes versus 55 minutes for FlashSAC under the same setup, demonstrating a practical benefit.

Critical weaknesses
  • The main limitation is the amount of new insight. SWD is an existing method, and the other changes enable or disable FlashSAC components. The broad evaluation is useful, but the additional understanding beyond these configuration choices is limited.

  • The experiments do not isolate the proposed coverage-based explanation. The CPU and GPU profiles differ in tasks and training settings (Table 4), so their comparison does not establish coverage as the cause. The acknowledged single-Q inconsistency on rough-terrain G1 also leaves the scope of the recommended GPU recipe unclear.

Questions for authors
  • Is there a controlled comparison on the same task showing that changing the data-collection scale or replay coverage changes the preferred configuration, with other training choices held fixed?

  • Beyond the CPU/GPU distinction, what evidence can guide the choice between single-Q and double-Q on a new task, particularly given the rough-terrain G1 results?

1 comment

I thank the authors for their response, which addresses my questions and concerns. Incorporating these clarifications into the final version will certainly improve the paper's overall quality. Overall, I will increase the score from 2 to 3.

Reviewer_c6w1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether classical off-policy RL stabilizers remain appropriate when training shifts from data-limited CPU environments to massively parallel GPU simulation. Building on FlashSAC, it varies replay weighting (SWD), parameter-projection normalization, and clipped double-Q versus single-Q, yielding WarpSAC-L for CPU-scale training and WarpSAC-A for GPU-parallel training. It reports AUC gains over FlashSAC across CPU and GPU benchmark suites, a ManiSkill success-rate case, and a Unitree G1 sim-to-real demonstration.

Critical strengths
  1. Relevant, practical question. Classical stabilizers were designed for narrow replay coverage and extrapolation risk; GPU-parallel robot learning changes this regime, so asking when they help versus restrict learning is important.
  2. Broad, internally controlled study. Backbone, optimizer, interface, and architecture are held fixed while replay, normalization, and critic multiplicity vary; evaluation spans eight families and includes a real G1 case study.
Critical weaknesses
  1. The claimed mechanism is confounded and the ablation is incomplete.
    The paper attributes effects to replay coverage, but coverage is never measured or independently manipulated. CPU and GPU groups also differ in task family, simulator, action/observation dimensions, environment count, replay implementation and size, precision, horizon, discount, n-step return, and update schedule (Appendix Table 4). Moreover, only four of the eight 2×2×2 configurations are evaluated, and every non-baseline configuration includes SWD. Missing cells—Norm OFF without SWD, single-Q without SWD, SWD+Norm ON+single-Q—prevent separation of component effects and interactions; WarpSAC-A changes all three components simultaneously.

  2. Headline gains lack statistical support and strong external baselines.
    The 4.5%/23.1% aggregate AUC gains and the 19.8%→96.4% single-task success change are reported without confidence intervals, task-level uncertainty, significance tests, or win/tie/loss counts; with five seeds and heterogeneous normalization conventions, a few tasks can dominate means. The comparison is also largely internal to the FlashSAC family: no tuned PPO on GPU benchmarks, no PER/recent-replay/small-buffer controls, and no other scalable off-policy methods. The state-of-the-art claim therefore exceeds the evidence.

Questions for authors
  1. List the exact nine CPU and fourteen GPU tasks in the headline aggregates, explain their selection rule, and provide per-task AUC with uncertainty.
  2. Report bootstrap CIs or task-level uncertainty for aggregate gains, plus win/tie/loss counts against FlashSAC.
  3. Since the causal claim is replay coverage, report a buffer-turnover or state–action coverage metric, or vary the number of parallel actors in a fixed environment while holding other settings constant.
  4. Why are the missing factorial cells absent? In particular, how is SWD separated from removing normalization or the second critic?
  5. Provide an SWD horizon sweep and recent-only/small-buffer controls; the horizon changes from 80,000 (CPU) to 2,000 (GPU) with a 10M buffer, so it is unclear whether gains come from age weighting or a changed effective replay distribution.
  6. How does WarpSAC-A compare with tuned PPO under equal wall-clock budgets on the GPU-parallel benchmarks, not only in the G1 appendix?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.