acceptodds
Under review as a conference paper at ICLR 2027

ScaleAC: Scale Actor-Critic by Replay Ratio

Abstract

Employing a high replay ratio, the number of agent updates per environment interaction, has emerged as a promising strategy to improve sample efficiency in reinforcement learning (RL). However, most existing efforts to scale the replay ratio stagnate at small values, and what happens inside the agent's network under a high update frequency remains largely unexamined. In this paper, we bridge this gap by scaling the replay ratio from the perspective of network internals to achieve sample-efficient RL. We identify a critical pathology whereby the dormant neuron phenomenon intensifies with increasing replay ratios, which undermines network capacity and destroys learning. To address this problem, we propose ScaleAC, a synergy of three elements based on Actor-Critic (AC) algorithms, for high-replay-ratio settings with theoretical insights. First, to tackle dormant neurons under a high update frequency, ScaleAC introduces a periodic soft network reset in RL agents. Second, to further stabilize high-replay-ratio training, ScaleAC integrates an in-target random minimization into the Q target computation. Third, to prevent overfitting at high replay ratios, ScaleAC diversifies the replay experience through data augmentations. Extensive experiments in MuJoCo and DMC demonstrate that ScaleAC scales replay ratios in both vector-based RL (up to 256) and pixel-based RL, and extends to a modern RL network architecture (i.e., SimbaV2), yielding substantial learning acceleration and performance improvement.

Primary area:
reinforcement learning
Keywords:
Dormant nueron, high replay ratio, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_k4hUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ScaleAC to address neuron dormancy and training instability at high replay ratios by combining soft network resets, in-target random minimization, and data augmentation. Experiments on continuous control benchmarks demonstrate improved sample efficiency at replay ratios up to 256.

Critical strengths
  1. The experiments cover a broad range of tasks and standard baselines, with ablations supporting the contribution of each component.
  2. On some challenging tasks, increasing the replay ratio to 128 or 256 improves sample efficiency, which is an interesting empirical result.
  3. The proposed algorithmic recipe is simple and easy to understand.
Critical weaknesses
  1. Limited conceptual novelty. The connection between replay ratio and neuron dormancy has already been investigated in prior work (e.g., Sokar et al., 2023), and all three components have established precedents. The main contribution lies in their effective combination for continuous control at higher replay ratios. While this contribution is somewhat nontrivial, I still find that this paper offers little new insight to the community.

  2. Insufficient controls in visual RL experiments. ScaleAC uses RR=8, while DrQ-v2 and DrM use RR=0.5 and Adaptive RR increases only to 2. Matched-RR comparisons or baseline RR sweeps are needed to isolate the contribution of the proposed recipe from additional gradient updates.

A minor issue with Theorems 3.1 and 3.2: both theoretical analyses define dormancy as z≤0z\leq0, where z=w⊤xz=w^\top x. However, the actual criterion in Definition 2.1 is

\frac{\mathbb E_{x\sim D}[|\rho_j^l(x)|]} {\frac{1}{N_l}\sum_{k=1}^{N_l}\mathbb E_{x\sim D}[|\rho_k^l(x)|]}.$$ It is easy to see that these two criteria are related: $$\mathbb E_{x\sim D}[\rho(x)]=P(z>0)\cdot\mathbb E[z\mid z>0].$$ If a ReLU neuron consistently outputs zero, its contribution will also tend to be low. However, there is still a gap in the theoretical analysis. Moreover, there is an error in Theorem 3.2: if $z_0$ always comes from the original initialization, then using $z_0\sim\mathcal N(\mu,\sigma^2)$ in the proof is not justified, since the relevant distribution is actually the conditional distribution $$z_0\mid(z_t\leq0).$$ In fact, we can construct a complete counterexample. Consider a scalar ReLU neuron: $$x=1,\qquad w_0\sim\mathcal N(0,1).$$ Let the loss be $$L(w)=\frac12\operatorname{ReLU}(w)^2.$$ Performing one ordinary SGD step with learning rate $\eta=2$ gives $$w_t= \begin{cases} -w_0,&w_0>0,\\ w_0,&w_0\leq0, \end{cases} =-|w_0|\leq0.$$ The neuron is now certainly dormant. Applying the reset with the paper’s default $\alpha=0.8$ gives $$w_t'= \begin{cases} -0.6w_0<0,&w_0>0,\\ w_0\leq0,&w_0\leq0. \end{cases}$$ Therefore, $$\boxed{P(z_t'>0\mid z_t\leq0)=0.}$$ This directly contradicts the paper’s claim of a strictly positive probability. The learning rate of 2 is only chosen to construct a simple counterexample. Since the theorem does not exclude this training history, the counterexample is sufficient to show that the current statement does not hold. The solution is quite simple: to draw a fresh initialization $z_0'$ each time. Since it is independent of $z_t$, this would allow Theorem 3.2 to hold. If the authors provide the missing experiments and correct the theoretical errors, I would consider raising my score, although I would still have concerns about the paper’s limited novelty.
Questions for authors
  1. Can you provide matched-RR comparisons for the visual RL experiments?
  2. Why is the actor updated every 20 critic updates in MuJoCo but after every critic update in DMC? What motivates this difference, and how sensitive are the results to the actor update frequency?
7 comments
AuthorsOpenReview ↗

Contribution Clarification, Theoretical Corrections, and Matched Comparisons

We thank the reviewer for the detailed review and the explicit counterexample. We have completed RR8 visual controls and actor-update sensitivity experiments, and we correct the stored-initialization probability argument below.

[W1] "Limited conceptual novelty."

Our work investigates neuron dormancy and its mitigation in continuous control at high replay ratios, with experiments extending to RR256. Sokar et al. [1] studied the relationship between replay ratio and neuron dormancy in DQN at small replay ratios of 0.25, 0.5, 1, and 2. Building on this finding, we further examine whether reducing dormancy is sufficient to sustain effective learning under substantially more intensive reuse of experience.

A key observation in this regime is that reducing the dormant neuron rate alone is insufficient. In our matched-RR32 experiments, ReDo-SAC reduces the dormant neuron rate to nearly zero but learns poorly on all four MuJoCo tasks. This motivates a training approach that addresses dormancy while also retaining learned knowledge and stabilizing repeated updates. ScaleAC therefore uses soft resets to mitigate dormancy while retaining learned parameters. It also incorporates randomized target minimization and data augmentation for further enhancement.

Our additional RR32 comparisons further examine the choice of reset mechanism within this combination. We combine ReDo with randomized target minimization (RM) and random amplitude scaling (RAS), using the same ten-critic ensemble, M=2 target minimum, and RAS range as ScaleAC. The comparison covers four MuJoCo tasks, using six seeds per method and matched interaction budgets. The table reports AUC (the normalized integral of the return curve) as a metric for sample efficiency, with means ± 95% confidence intervals:

TaskBudgetReDo-SACReDo + RM + RASScaleAC (S&P + RM + RAS)
Hopper-v5100k147 ± 191065 ± 7782350 ± 398
Ant-v5300k989 ± 42257 ± 13343657 ± 199
Walker2d-v5300k105 ± 122951 ± 7193571 ± 538
Humanoid-v5300k198 ± 1197 ± 22874 ± 608

Adding RM and RAS improves ReDo's AUC means on Hopper, Ant, and Walker2d. ScaleAC achieves higher AUC means than ReDo + RM + RAS on all four tasks, supporting the use of soft resets as part of the combined approach to high-replay-ratio training.

Together, these results support ScaleAC’s integration of these components to sustain effective learning at high replay ratios.

[W2 + Q1] "Can you provide matched-RR comparisons for the visual RL experiments?"

Following the reviewer's suggestion, we added visual RL comparisons with DrQ-v2 and DrM at the same RR8 as ScaleAC. We also increased the upper bound of Adaptive RR to 8. Adaptive RR retains its adaptive schedule, so this comparison matches its maximum replay ratio.

The comparison covers three tasks, using six seeds per method and matched interaction budgets. The table reports AUC (the normalized integral of the return curve) as a metric for sample efficiency, with means ± 95% confidence intervals:

TaskDrQ-v2 (RR8)DrM (RR8)Adaptive RR (max 8)ScaleAC (RR8)
hopper-hop39 ± 3845 ± 3537 ± 2898 ± 27
quadruped-run212 ± 94247 ± 45171 ± 63236 ± 48
reacher-hard156 ± 35179 ± 79153 ± 48357 ± 29

ScaleAC has higher AUC means than all three baselines on hopper-hop and reacher-hard. On quadruped-run, ScaleAC has a higher AUC mean than DrQ-v2 and Adaptive RR, while DrM has the highest mean. These results show that ScaleAC's sample-efficiency benefits, especially on hopper-hop and reacher-hard, remain when DrQ-v2 and DrM use the same replay ratio.

We will clarify the replay-ratio settings of the visual RL baselines.

[W3a] "The gap between the theoretical and empirical definitions of dormancy."

Complete inactivity (z≤0z\le0) is a stricter case of the dormancy criterion in Definition 2.1. If a ReLU neuron is inactive almost surely under the input distribution, its expected activation is zero, corresponding to d=0≤τd=0\le\tau. However, inactivity on a fixed input does not imply inactivity across the distribution. In the newly uploaded manuscript, we have clarified the scope of Theorems 3.1 and 3.2 in Appendix A, with the added explanation marked in red.

[W3b] "The conditional-probability error in Theorem 3.2."

We thank the reviewer for the counterexample. The stored initialization w0w_0 and the trained parameters wtw_t are dependent, so the original Gaussian probability argument is not valid. We have retained the implemented reset rule and replaced the original positive-probability claim with a one-reset reactivation condition for a sufficiently small interpolation coefficient.

Revised Theorem 3.2. Under an arbitrary but fixed input xx to the neuron, consider a ReLU neuron with z0=w0⊤x>0z_0=w_0^\top x>0 and current pre-activation zt=wt⊤x≤0z_t=w_t^\top x\le0. Apply Shrink & Perturb: wt′=αwt+(1−α)w0,0<α<1.w_t'=\alpha w_t+(1-\alpha)w_0,\qquad 0<\alpha<1. Then the neuron is reactivated if and only if 0<α<z0z0+∣zt∣.0<\alpha<\frac{z_0}{z_0+|z_t|}. For any finite zt≤0z_t\le0 and z0>0z_0>0, this interval is nonempty. Therefore, there always exists a sufficiently small α∈(0,1)\alpha\in(0,1) such that one Shrink & Perturb update reactivates the neuron.

Proof. From the update rule,

=\alpha z_t+(1-\alpha)z_0.$$ Assume the neuron is dormant at step $t$, i.e., $z_t\le0$. Then the condition $z_t'>0$ is equivalent to $$z_0>-\frac{\alpha}{1-\alpha}z_t.$$ Since $z_t\le0$, we have $-z_t=|z_t|$. Using $1-\alpha>0$ and $z_0+|z_t|>0$, we obtain $$\begin{aligned} z_t'>0 &\quad\Longleftrightarrow\quad z_0>\frac{\alpha}{1-\alpha}|z_t| \\ &\quad\Longleftrightarrow\quad (1-\alpha)z_0>\alpha|z_t| \\ &\quad\Longleftrightarrow\quad z_0>\alpha(z_0+|z_t|) \\ &\quad\Longleftrightarrow\quad \alpha<\frac{z_0}{z_0+|z_t|}. \end{aligned}$$ The boundary is strict: equality gives $z_t'=0$, which remains inactive under ReLU. In the positive-initialization branch of the reviewer's example, $z_t=-z_0$, so reactivation requires $\alpha<1/2$. The default $\alpha=0.8$ does not satisfy this condition, consistent with the counterexample. Choosing any $0<\alpha<1/2$ instead reactivates the neuron in one reset. For example, choosing $\alpha=0.4$ gives $z_t'=0.4(-z_0)+0.6z_0=0.2z_0>0$. We also agree that a fresh independent Gaussian initialization with nonzero variance restores the one-reset positive-probability argument. In the revised manuscript, we have updated Theorems 3.2 and A.2 to state the explicit bound on $\alpha$, updated the proof, and added the fresh independent Gaussian initialization result as a separate proposition in Appendix A. These changes are marked in red in the revised manuscript. **[Q2] "Why is the actor updated every 20 critic updates in MuJoCo but after every critic update in DMC? What motivates this difference, and how sensitive are the results to the actor update frequency?"** We retained the default actor-update intervals of the original codebases: 20 in REDQ (PyTorch) for MuJoCo and 1 in SR-SAC (JAX) for DMC. These settings were inherited from the original implementations rather than retuned for ScaleAC. The actor-update interval is measured in critic gradient updates. Following the reviewer's suggestion, we additionally evaluated the alternative interval while keeping the other components unchanged: | RR64 task | Codebase | Original actor interval | Actor interval 1 | Actor interval 20 | | --- | --- | --- | --- | --- | | Hopper-v5 | REDQ (PyTorch) | 20 | 3398.5 ± 145.2 | 3265.5 ± 97.8 | | dog-run | SR-SAC (JAX) | 1 | 164.3 ± 93.4 | 112.8 ± 119.9 | | humanoid-run | SR-SAC (JAX) | 1 | 34.8 ± 85.4 | 1.5 ± 0.1 | The table reports final returns as means ± 95% confidence intervals across six seeds per method. On Hopper-v5, the final-return means are close. On dog-run and humanoid-run, interval 20 has lower final-return means than the original interval 1. The humanoid-run results remain weak under both settings, although the default actor interval of 1 yields a higher final-return mean. These results indicate sensitivity to the actor-update schedule, particularly on the DMC tasks. We will clarify the source of the actor-update settings in the revision. We hope the clarified empirical contribution (W1), the matched-RR visual controls (W2/Q1), the corrected theoretical statements (W3a/W3b), and the actor-schedule comparison (Q2) help assess the recipe on the evidence presented. Thank you again for the constructive discussion. **Reference** [1] Sokar, Agarwal, Castro, and Evci. *The Dormant Neuron Phenomenon in Deep Reinforcement Learning.* ICML 2023.

Thank you for your detailed response and the substantial effort devoted to the additional experiments.


I. Experimental controls and theoretical correction.
The additional visual controls and actor-update sensitivity study strengthen the empirical evaluation and address my corresponding concerns. I also appreciate your acknowledgment and correction of the theoretical error.


II. Novelty and mechanistic insight.
The observation that ReDo substantially reduces dormancy yet learns poorly is potentially valuable. I would welcome a clearer explanation of why reducing dormancy does not translate into improved performance, and how the proposed combination addresses the remaining difficulties. While the recipe is empirically effective, its components are established techniques, and the additional mechanistic understanding remains somewhat limited. Explaining this observation more deeply could strengthen the contribution.


I recognize the thorough empirical validation and appreciate your careful responses(experiments are indeed solid). My remaining reservation concerns the extent of the contribution: I found the new insights into what fundamentally matters for learning at high replay ratios relatively limited. As an academic work, I consider this paper borderline.

Reviewer_RhMCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ScaleAC, an actor-critic framework for improving sample efficiency by increasing the replay ratio. The authors identify dormant neurons as a potential bottleneck at high replay ratios and combine three existing techniques: Shrink & Perturb for periodic soft resets, randomized minimization of Q-targets, and data augmentation. The paper provides theoretical analysis of neuron dormancy and evaluates ScaleAC on MuJoCo, DMC, and visual control tasks. The results show improvements over SAC, REDQ, and SR-SAC, with replay ratios up to 256. The authors also extend ScaleAC to SimBaV2 to evaluate its compatibility with modern RL architectures.

Critical strengths
  • Relevant research question: Understanding whether high replay ratios remain beneficial with modern architectures is an important question. Since recent architectural improvements have already led to substantial gains in sample efficiency, revisiting existing high replay ratio approaches in this setting is worthwhile.
  • Broad evaluation across different settings: The paper evaluates ScaleAC on both state-based and visual control tasks, including challenging DMC-Hard environments. The ablation studies and comparisons with different network reset techniques also help explain the contribution of individual components.
Critical weaknesses
  • Lack of a fair comparison with baselines: My main concern is the lack of a fair comparison with modern baselines. In Section 4.7 and Appendix I, the paper compares vanilla SimBaV2 (RR=1) with SimBaV2 + ScaleAC (RR=4). However, this does not establish whether the improvement comes from ScaleAC or simply from increasing the replay ratio. The paper should provide replay ratio curves for both methods under standard 500K and 1M environment-step settings. Without this comparison, it is difficult to determine whether ScaleAC provides meaningful additional benefits over existing modern architectures.
  • Limited justification over existing approaches: ScaleAC combines existing techniques, including Shrink & Perturb, randomized Q-target minimization, and data augmentation. While combining these components can be valuable, the paper does not sufficiently establish why this combination is necessary with modern architectures. In particular, the SimBaV2 experiments only consider RR=4, despite the paper's main motivation of enabling much higher replay ratios.
  • Missing recent baselines: Recent methods such as XQC [1] and FlashSAC [2] have investigated sample-efficient and stable off-policy RL, but are not compared or discussed. I do not consider this omission critical by itself, but these comparisons would help establish the practical relevance of ScaleAC. Appendix J also reports a 5.9x wall-clock overhead over vanilla SimBaV2, making comparisons with stronger baselines particularly important.

[1] Palenicek et al. XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning. ICLR 2026. https://arxiv.org/abs/2509.25174

[2] Kim et al. FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control. 2026. https://arxiv.org/abs/2604.04539

Questions for authors

Please see weaknesses

3 comments
AuthorsOpenReview ↗

Matched-RR Comparisons, Higher Replay Ratios, and Recent Baselines

We thank the reviewer for the careful review. Following the suggestion, we added vanilla SimbaV2 at RR4 as the direct control for ScaleAC + SimbaV2 at RR4. We report this comparison first, followed by a discussion of the components, an evaluation at higher replay ratios, and comparisons with recent baselines.

[W1] "Lack of a fair comparison with baselines"

We agree that comparing vanilla SimbaV2 at RR1 with ScaleAC + SimbaV2 at RR4 does not isolate the effect of ScaleAC. Following the reviewer's suggestion, we added vanilla SimbaV2 at RR4 to compare it with ScaleAC + SimbaV2 under the same replay ratio and interaction budget. The table reports AUC (the normalized integral of the return curve) as a metric for sample efficiency, with means ± 95% confidence intervals across six seeds:

TaskComparison horizonSimbaV2 (RR=4)ScaleAC + SimbaV2 (RR=4)AUC change
Hopper-v5100k2203 ± 2542276 ± 62+3.3%
Ant-v5300k2147 ± 7993505 ± 677+63.2%
Walker2d-v5300k3949 ± 4024171 ± 462+5.6%
Humanoid-v5300k4475 ± 1924948 ± 429+10.6%

At the original interaction budgets, ScaleAC + SimbaV2 has higher AUC means on all four tasks, including a 63.2% increase on Ant-v5.

We additionally completed separate 1M-step comparisons on Ant-v5 and dog-run, with both methods at RR4 and six seeds per method. Each run uses the learning-rate schedule configured for the 1M budget. The AUC results and relative changes are:

Task at 1M stepsSimbaV2 (RR=4)ScaleAC + SimbaV2 (RR=4)AUC change
Ant-v54653 ± 7835436 ± 314+16.8%
dog-run374 ± 30454 ± 63+21.2%

At the 1M budget, ScaleAC + SimbaV2 achieves 16.8% and 21.2% higher AUC means than vanilla SimbaV2 on Ant-v5 and dog-run, respectively. In addition, we observe that ScaleAC enables faster learning, especially in the early stage, supporting its sample efficiency when environment interactions are limited.

[W2] Limited justification over existing approaches and higher replay ratios

The motivation for combining ScaleAC with modern architectures is to make additional gradient updates on limited experience more effective. Increasing the replay ratio increases the reuse of the same off-policy data, making the preservation of learning capacity, control of Q-target overestimation, and resistance to overfitting relevant even with a strong backbone. ScaleAC is designed to address these training challenges, thereby complementing modern architectural improvements at high replay ratios.

Following the reviewer's suggestion, we extended the SimbaV2 experiments to RR16 and compared vanilla SimbaV2 with ScaleAC + SimbaV2 at the same RR16 and interaction budget. The AUC results below are means ± 95% confidence intervals across six seeds.

TaskComparison horizonSimbaV2 (RR=16)ScaleAC + SimbaV2 (RR=16)AUC change
Hopper-v5100k2421 ± 2812646 ± 275+9.3%
Ant-v5300k1752 ± 9003012 ± 864+71.9%
Walker2d-v5300k3543 ± 4734528 ± 322+27.8%
Humanoid-v5300k5074 ± 2575594 ± 315+10.3%

At RR16, ScaleAC + SimbaV2 has higher AUC means on all four tasks, with improvements of 9.3% on Hopper, 71.9% on Ant-v5, 27.8% on Walker2d, and 10.3% on Humanoid. These results support the value of combining ScaleAC with a modern architecture when using a large replay ratio for more aggressive data reuse.

[W3] "Missing recent baselines"

XQC and FlashSAC introduce improvements that are potentially complementary to ScaleAC. XQC [1] improves critic optimization by combining batch normalization, weight normalization, and a distributional cross-entropy loss to obtain a better-conditioned optimization problem. FlashSAC [2] improves training efficiency by combining fewer gradient updates with larger models and higher data throughput, and stabilizes critic learning through explicit control of weight, feature, and gradient norms. Their architectural and optimization mechanisms may complement ScaleAC's combination of soft resets, randomized target minimization, and data augmentation for effective learning at high replay ratios.

Following the reviewer's suggestion, we added experimental comparisons with XQC and FlashSAC on four MuJoCo tasks, using six seeds and matched interaction budgets. We also include vanilla SimbaV2 at RR1 as a reference. The table reports AUC as a metric for sample efficiency, with means ± 95% confidence intervals across six seeds, together with the relative AUC changes of ScaleAC + SimbaV2 against vanilla SimbaV2, XQC, and FlashSAC:

TaskScaleAC + SimbaV2 (RR4)SimbaV2 (RR1)XQCFlashSACAUC vs SimbaV2AUC vs XQCAUC vs FlashSAC
Hopper-v52276 ± 621099 ± 4352162 ± 2502020 ± 173+107.0%+5.3%+12.7%
Ant-v53505 ± 6772832 ± 6013570 ± 10103151 ± 773+23.7%-1.8%+11.2%
Walker2d-v54171 ± 4623863 ± 3703793 ± 5143884 ± 227+8.0%+10.0%+7.4%
Humanoid-v54948 ± 4293607 ± 3114781 ± 5506108 ± 307+37.2%+3.5%-19.0%

We use SimbaV2 at RR1, XQC at RR2, and FlashSAC at its default single-environment RR1, while ScaleAC + SimbaV2 uses RR4.

The results show faster learning for ScaleAC + SimbaV2 relative to vanilla SimbaV2 RR1, with higher AUC means on all four tasks. ScaleAC + SimbaV2 also has higher AUC means than each of XQC and FlashSAC on three of four tasks. These results suggest that ScaleAC remains beneficial when environment interactions are limited, with its gains reflected in higher AUC on several tasks. Overall, ScaleAC + SimbaV2 achieves comparable sample-efficiency performance to XQC and FlashSAC on these tasks, with the relative advantages depending on the task and comparison method.

We will include a discussion of XQC and FlashSAC in the revision. Furthermore, given the potential complementarity of their architectural and optimization improvements with ScaleAC's training recipe, we expect that carefully combining these methods could further improve sample efficiency and training stability. We view this as a promising direction for future work.

We hope the matched-RR and longer-horizon results (W1), the component discussion and RR16 evaluation (W2), and the recent-baseline comparisons (W3) clarify the benefits and scope of the extension. Thank you again for the constructive review.

References

[1] Palenicek et al. XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning. ICLR 2026. arXiv:2509.25174.

[2] Kim et al. FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control. 2026. arXiv:2604.04539.

AuthorsOpenReview ↗

Thank You for the Helpful Feedback

Dear Reviewer RhMC,

Thank you for your thoughtful review. Your feedback has been very helpful in strengthening our evaluation and clarifying the contribution. We have posted additional matched-RR comparisons, higher-RR results, and comparisons with XQC and FlashSAC. We wanted to check whether our responses address your main concerns, or whether there is any point we could clarify further during the discussion.

We appreciate your time and consideration.

Best regards,
The Authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

Theoretical Scope, Computational Tradeoffs, and Additional Controls

We thank the reviewer for the constructive comments. We clarify the scope of the theoretical analysis, discuss the interaction-efficiency and computation tradeoff, and provide matched baseline and higher-RR comparisons below.

[W1 + Q1] "Does the fixed-input assumption of Theorem A.1 still hold later in training?"

The fixed-input assumption is a simplifying approximation for analyzing repeated updates on the same input. It is most plausible early in training, when the replay buffer contains limited experience and similar inputs are reused frequently. As the replay buffer becomes more diverse and the learned representations change, this assumption becomes less representative of the full training process. Under the fixed-input assumption, Theorem A.1 helps explain how repeated gradient updates, especially at high replay ratios, can lead to neuron inactivity.

To examine whether dormancy remains relevant as training progresses, we analyzed the dormant neuron rate of SAC at RR128 on four MuJoCo tasks, averaged over six seeds at each checkpoint:

Task (SAC RR128)6k10k20k40k60k80k100k200k300k
Hopper-v510.6%14.2%19.5%23.2%29.6%39.2%43.7%NANA
Ant-v50.3%4.4%18.5%20.0%21.4%23.3%24.6%30.6%31.5%
Walker2d-v53.1%5.3%10.2%14.5%17.4%19.9%23.6%40.1%50.7%
Humanoid-v50.1%0.1%2.6%14.0%15.4%16.8%18.7%28.1%31.6%

The results show faster growth in the mean dormant neuron rate during the earlier training interval. On Ant, Walker2d, and Humanoid, the average increases per 10k environment steps are approximately 2.6, 2.2, and 2.0 percentage points over 6k–100k, compared with 0.3, 1.4, and 0.6 percentage points over 100k–300k, respectively. Hopper also exhibits an early increase. This pattern is consistent with the intuition that frequent reuse of similar inputs early in training can contribute to the development of dormancy. We will clarify the fixed-input assumption and the scope of Theorem A.1 in the revision.

[W2] "The wall-clock overhead vs. sample-efficiency tradeoff needs more prominence in the main text."

We agree that improved sample efficiency should not be interpreted as improved overall efficiency. ScaleAC reduces the environment interactions needed for learning at the cost of additional computation. The substantial wall-clock overhead at high replay ratios should therefore be stated prominently in the main text.

ScaleAC is intended for settings where environment interactions are limited or expensive. When simulation is inexpensive or wall-clock time is the primary constraint, a lower replay ratio may be preferable. In the revision, we will bring the conclusions and discussion of the wall-clock comparison from Appendix J into the main text to make the computational tradeoff clear.

[W3 + Q3] "Are the ReDo-SAC baselines in Figure 3 also using in-target random minimization and data augmentation?"

The ReDo-SAC baseline in Figure 3 does not use randomized target minimization or random amplitude scaling. Following the reviewer's suggestion, we additionally compare ReDo with randomized target minimization and augmentation, using the same RR32, ten-critic ensemble, M=2 target minimum, and RAS range as ScaleAC.

The comparison covers all four MuJoCo tasks, using six seeds and matched interaction budgets. The table reports AUC means ± 95% confidence intervals. AUC is the normalized integral of the return curve to measure the overall sample efficiency.

TaskBudgetReDo-SACReDo + RM + RASScaleAC (S&P + RM + RAS)
Hopper-v5100k147 ± 191065 ± 7782350 ± 398
Ant-v5300k989 ± 42257 ± 13343657 ± 199
Walker2d-v5300k105 ± 122951 ± 7193571 ± 538
Humanoid-v5300k198 ± 1197 ± 22874 ± 608

Adding randomized target minimization and RAS improves ReDo's AUC means on Hopper, Ant, and Walker2d. Meanwhile, ScaleAC achieves higher AUC means on all four tasks under these matched settings.

We will include the complete four-task comparison in the revision and clarify the configurations of the standalone ReDo-SAC baseline and the ReDo + RM + RAS control in the experimental description.

[W4] "The SimbaV2 extension uses only replay ratio 4."

RR4 was the initial setting used to evaluate the extension to SimbaV2. Following the reviewer's suggestion, we extended the experiments to RR16 and compared vanilla SimbaV2 with ScaleAC + SimbaV2 at the same replay ratio and interaction budget. We also include vanilla SimbaV2 at its default RR1 as a reference.

The comparison covers four MuJoCo tasks, with six seeds per method. The table reports AUC means ± 95% confidence intervals:

TaskComparison horizonSimbaV2 (RR=1)SimbaV2 (RR=16)ScaleAC + SimbaV2 (RR=16)AUC vs RR16
Hopper-v5100k1099 ± 4352421 ± 2812646 ± 275+9.3%
Ant-v5300k2832 ± 6011752 ± 9003012 ± 864+71.9%
Walker2d-v5300k3863 ± 3703543 ± 4734528 ± 322+27.8%
Humanoid-v5300k3607 ± 3115074 ± 2575594 ± 315+10.3%

At RR16, ScaleAC + SimbaV2 has higher AUC means on all four tasks, with improvements of 9.3% on Hopper, 71.9% on Ant, 27.8% on Walker2d, and 10.3% on Humanoid. These results show that the extension can operate beyond RR4 and improve interaction efficiency relative to vanilla SimbaV2 at the same RR.

We will include these higher-RR comparisons in the revision.

[Q2] "In settings where environment steps are not the bottleneck (e.g., simulation), does ScaleAC's wall-clock overhead make it inapplicable? How should the replay ratio be chosen?"

ScaleAC is not inherently inapplicable when environment steps are inexpensive. However, its additional computation may outweigh the benefit of improved sample efficiency when wall-clock time is the primary constraint. In such settings, high-RR ScaleAC may be less suitable than a lower-RR alternative; our sample-efficiency results do not establish a wall-clock advantage.

The replay ratio should therefore be chosen according to the interaction and computation budgets. When environment interactions are expensive or limited, a higher RR can help make better use of the collected data. When interactions are inexpensive, we recommend starting with a low RR and increasing it only if the performance gain justifies the additional training time under the available wall-clock budget.

We hope the clarified theoretical scope (W1/Q1), the explicit computation tradeoff (W2/Q2), the matched reset controls (W3/Q3), and the higher-RR results (W4) address the concerns. Thank you again for the careful review.

AuthorsOpenReview ↗

Summary of Rebuttal Clarifications and Additional Evidence

Dear Area Chair and Reviewers,

We sincerely appreciate your time and careful assessment of our work. We would like to summarize the main clarifications and additional evidence provided during the discussion.

  1. [Reviewer RhMC: Fair comparisons and recent baselines] We added matched-replay-ratio comparisons between vanilla SimbaV2 and ScaleAC + SimbaV2 at RR4 and RR16 on four MuJoCo tasks, using six seeds and matched interaction budgets. ScaleAC + SimbaV2 achieves higher mean AUC on all four tasks at both replay ratios. We also compared ScaleAC + SimbaV2 with XQC and FlashSAC. The results show task-dependent advantages and suggest that combining their complementary techniques with ScaleAC is a promising direction for future study.

  2. [Reviewer dZwt: Theoretical scope and computational tradeoffs] We clarified the fixed-input assumption and reported dormant-neuron measurements throughout training. We also distinguished sample efficiency from overall computational efficiency: high replay ratios require additional computation, and their suitability depends on the relative costs of environment interaction and training. We will make this tradeoff more prominent in the main text.

  3. [Reviewer k4hU: Experimental controls and theoretical correction] We added matched-RR visual comparisons and actor-update sensitivity experiments. We corrected the stored-initialization argument in Theorems 3.2 and A.2, and added a separate result for fresh independent initialization. The changes to Theorems 3.2 and A.2 are marked in red in the revised manuscript. In response to the remaining concern about mechanistic insight, our follow-up explains how parameter retention, value-estimation control, and augmentation of reused data motivate the integration of ScaleAC’s components.

We appreciate the constructive discussion and hope these clarifications and additional results help the Area Chair and reviewers assess the contribution and its scope.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.