- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ScaleAC to address neuron dormancy and training instability at high replay ratios by combining soft network resets, in-target random minimization, and data augmentation. Experiments on continuous control benchmarks demonstrate improved sample efficiency at replay ratios up to 256.
Critical strengths
- The experiments cover a broad range of tasks and standard baselines, with ablations supporting the contribution of each component.
- On some challenging tasks, increasing the replay ratio to 128 or 256 improves sample efficiency, which is an interesting empirical result.
- The proposed algorithmic recipe is simple and easy to understand.
Critical weaknesses
-
Limited conceptual novelty. The connection between replay ratio and neuron dormancy has already been investigated in prior work (e.g., Sokar et al., 2023), and all three components have established precedents. The main contribution lies in their effective combination for continuous control at higher replay ratios. While this contribution is somewhat nontrivial, I still find that this paper offers little new insight to the community.
-
Insufficient controls in visual RL experiments. ScaleAC uses RR=8, while DrQ-v2 and DrM use RR=0.5 and Adaptive RR increases only to 2. Matched-RR comparisons or baseline RR sweeps are needed to isolate the contribution of the proposed recipe from additional gradient updates.
A minor issue with Theorems 3.1 and 3.2: both theoretical analyses define dormancy as , where . However, the actual criterion in Definition 2.1 is
\frac{\mathbb E_{x\sim D}[|\rho_j^l(x)|]} {\frac{1}{N_l}\sum_{k=1}^{N_l}\mathbb E_{x\sim D}[|\rho_k^l(x)|]}.$$ It is easy to see that these two criteria are related: $$\mathbb E_{x\sim D}[\rho(x)]=P(z>0)\cdot\mathbb E[z\mid z>0].$$ If a ReLU neuron consistently outputs zero, its contribution will also tend to be low. However, there is still a gap in the theoretical analysis. Moreover, there is an error in Theorem 3.2: if $z_0$ always comes from the original initialization, then using $z_0\sim\mathcal N(\mu,\sigma^2)$ in the proof is not justified, since the relevant distribution is actually the conditional distribution $$z_0\mid(z_t\leq0).$$ In fact, we can construct a complete counterexample. Consider a scalar ReLU neuron: $$x=1,\qquad w_0\sim\mathcal N(0,1).$$ Let the loss be $$L(w)=\frac12\operatorname{ReLU}(w)^2.$$ Performing one ordinary SGD step with learning rate $\eta=2$ gives $$w_t= \begin{cases} -w_0,&w_0>0,\\ w_0,&w_0\leq0, \end{cases} =-|w_0|\leq0.$$ The neuron is now certainly dormant. Applying the reset with the paper’s default $\alpha=0.8$ gives $$w_t'= \begin{cases} -0.6w_0<0,&w_0>0,\\ w_0\leq0,&w_0\leq0. \end{cases}$$ Therefore, $$\boxed{P(z_t'>0\mid z_t\leq0)=0.}$$ This directly contradicts the paper’s claim of a strictly positive probability. The learning rate of 2 is only chosen to construct a simple counterexample. Since the theorem does not exclude this training history, the counterexample is sufficient to show that the current statement does not hold. The solution is quite simple: to draw a fresh initialization $z_0'$ each time. Since it is independent of $z_t$, this would allow Theorem 3.2 to hold. If the authors provide the missing experiments and correct the theoretical errors, I would consider raising my score, although I would still have concerns about the paper’s limited novelty.Questions for authors
- Can you provide matched-RR comparisons for the visual RL experiments?
- Why is the actor updated every 20 critic updates in MuJoCo but after every critic update in DMC? What motivates this difference, and how sensitive are the results to the actor update frequency?
7 comments
Follow-up
[W3a] "The gap between the theoretical and empirical definitions of dormancy."
Complete inactivity () is a stricter case of the dormancy criterion in Definition 2.1. If a ReLU neuron is inactive almost surely under the input distribution, its expected activation is zero, corresponding to . However, inactivity on a fixed input does not imply inactivity across the distribution. In the newly uploaded manuscript, we have clarified the scope of Theorems 3.1 and 3.2 in Appendix A, with the added explanation marked in red.
[W3b] "The conditional-probability error in Theorem 3.2."
We thank the reviewer for the counterexample. The stored initialization and the trained parameters are dependent, so the original Gaussian probability argument is not valid. We have retained the implemented reset rule and replaced the original positive-probability claim with a one-reset reactivation condition for a sufficiently small interpolation coefficient.
Revised Theorem 3.2. Under an arbitrary but fixed input to the neuron, consider a ReLU neuron with and current pre-activation . Apply Shrink & Perturb: Then the neuron is reactivated if and only if For any finite and , this interval is nonempty. Therefore, there always exists a sufficiently small such that one Shrink & Perturb update reactivates the neuron.
Proof. From the update rule,
=\alpha z_t+(1-\alpha)z_0.$$ Assume the neuron is dormant at step $t$, i.e., $z_t\le0$. Then the condition $z_t'>0$ is equivalent to $$z_0>-\frac{\alpha}{1-\alpha}z_t.$$ Since $z_t\le0$, we have $-z_t=|z_t|$. Using $1-\alpha>0$ and $z_0+|z_t|>0$, we obtain $$\begin{aligned} z_t'>0 &\quad\Longleftrightarrow\quad z_0>\frac{\alpha}{1-\alpha}|z_t| \\ &\quad\Longleftrightarrow\quad (1-\alpha)z_0>\alpha|z_t| \\ &\quad\Longleftrightarrow\quad z_0>\alpha(z_0+|z_t|) \\ &\quad\Longleftrightarrow\quad \alpha<\frac{z_0}{z_0+|z_t|}. \end{aligned}$$ The boundary is strict: equality gives $z_t'=0$, which remains inactive under ReLU. In the positive-initialization branch of the reviewer's example, $z_t=-z_0$, so reactivation requires $\alpha<1/2$. The default $\alpha=0.8$ does not satisfy this condition, consistent with the counterexample. Choosing any $0<\alpha<1/2$ instead reactivates the neuron in one reset. For example, choosing $\alpha=0.4$ gives $z_t'=0.4(-z_0)+0.6z_0=0.2z_0>0$. We also agree that a fresh independent Gaussian initialization with nonzero variance restores the one-reset positive-probability argument. In the revised manuscript, we have updated Theorems 3.2 and A.2 to state the explicit bound on $\alpha$, updated the proof, and added the fresh independent Gaussian initialization result as a separate proposition in Appendix A. These changes are marked in red in the revised manuscript. **[Q2] "Why is the actor updated every 20 critic updates in MuJoCo but after every critic update in DMC? What motivates this difference, and how sensitive are the results to the actor update frequency?"** We retained the default actor-update intervals of the original codebases: 20 in REDQ (PyTorch) for MuJoCo and 1 in SR-SAC (JAX) for DMC. These settings were inherited from the original implementations rather than retuned for ScaleAC. The actor-update interval is measured in critic gradient updates. Following the reviewer's suggestion, we additionally evaluated the alternative interval while keeping the other components unchanged: | RR64 task | Codebase | Original actor interval | Actor interval 1 | Actor interval 20 | | --- | --- | --- | --- | --- | | Hopper-v5 | REDQ (PyTorch) | 20 | 3398.5 ± 145.2 | 3265.5 ± 97.8 | | dog-run | SR-SAC (JAX) | 1 | 164.3 ± 93.4 | 112.8 ± 119.9 | | humanoid-run | SR-SAC (JAX) | 1 | 34.8 ± 85.4 | 1.5 ± 0.1 | The table reports final returns as means ± 95% confidence intervals across six seeds per method. On Hopper-v5, the final-return means are close. On dog-run and humanoid-run, interval 20 has lower final-return means than the original interval 1. The humanoid-run results remain weak under both settings, although the default actor interval of 1 yields a higher final-return mean. These results indicate sensitivity to the actor-update schedule, particularly on the DMC tasks. We will clarify the source of the actor-update settings in the revision. We hope the clarified empirical contribution (W1), the matched-RR visual controls (W2/Q1), the corrected theoretical statements (W3a/W3b), and the actor-schedule comparison (Q2) help assess the recipe on the evidence presented. Thank you again for the constructive discussion. **Reference** [1] Sokar, Agarwal, Castro, and Evci. *The Dormant Neuron Phenomenon in Deep Reinforcement Learning.* ICML 2023.Thank you for your detailed response and the substantial effort devoted to the additional experiments.
I. Experimental controls and theoretical correction.
The additional visual controls and actor-update sensitivity study strengthen the empirical evaluation and address my corresponding concerns. I also appreciate your acknowledgment and correction of the theoretical error.
II. Novelty and mechanistic insight.
The observation that ReDo substantially reduces dormancy yet learns poorly is potentially valuable. I would welcome a clearer explanation of why reducing dormancy does not translate into improved performance, and how the proposed combination addresses the remaining difficulties. While the recipe is empirically effective, its components are established techniques, and the additional mechanistic understanding remains somewhat limited. Explaining this observation more deeply could strengthen the contribution.
I recognize the thorough empirical validation and appreciate your careful responses(experiments are indeed solid). My remaining reservation concerns the extent of the contribution: I found the new insights into what fundamentally matters for learning at high replay ratios relatively limited. As an academic work, I consider this paper borderline.
Contribution Clarification, Theoretical Corrections, and Matched Comparisons
We thank the reviewer for the detailed review and the explicit counterexample. We have completed RR8 visual controls and actor-update sensitivity experiments, and we correct the stored-initialization probability argument below.
[W1] "Limited conceptual novelty."
Our work investigates neuron dormancy and its mitigation in continuous control at high replay ratios, with experiments extending to RR256. Sokar et al. [1] studied the relationship between replay ratio and neuron dormancy in DQN at small replay ratios of 0.25, 0.5, 1, and 2. Building on this finding, we further examine whether reducing dormancy is sufficient to sustain effective learning under substantially more intensive reuse of experience.
A key observation in this regime is that reducing the dormant neuron rate alone is insufficient. In our matched-RR32 experiments, ReDo-SAC reduces the dormant neuron rate to nearly zero but learns poorly on all four MuJoCo tasks. This motivates a training approach that addresses dormancy while also retaining learned knowledge and stabilizing repeated updates. ScaleAC therefore uses soft resets to mitigate dormancy while retaining learned parameters. It also incorporates randomized target minimization and data augmentation for further enhancement.
Our additional RR32 comparisons further examine the choice of reset mechanism within this combination. We combine ReDo with randomized target minimization (RM) and random amplitude scaling (RAS), using the same ten-critic ensemble, M=2 target minimum, and RAS range as ScaleAC. The comparison covers four MuJoCo tasks, using six seeds per method and matched interaction budgets. The table reports AUC (the normalized integral of the return curve) as a metric for sample efficiency, with means ± 95% confidence intervals:
Adding RM and RAS improves ReDo's AUC means on Hopper, Ant, and Walker2d. ScaleAC achieves higher AUC means than ReDo + RM + RAS on all four tasks, supporting the use of soft resets as part of the combined approach to high-replay-ratio training.
Together, these results support ScaleAC’s integration of these components to sustain effective learning at high replay ratios.
[W2 + Q1] "Can you provide matched-RR comparisons for the visual RL experiments?"
Following the reviewer's suggestion, we added visual RL comparisons with DrQ-v2 and DrM at the same RR8 as ScaleAC. We also increased the upper bound of Adaptive RR to 8. Adaptive RR retains its adaptive schedule, so this comparison matches its maximum replay ratio.
The comparison covers three tasks, using six seeds per method and matched interaction budgets. The table reports AUC (the normalized integral of the return curve) as a metric for sample efficiency, with means ± 95% confidence intervals:
ScaleAC has higher AUC means than all three baselines on hopper-hop and reacher-hard. On quadruped-run, ScaleAC has a higher AUC mean than DrQ-v2 and Adaptive RR, while DrM has the highest mean. These results show that ScaleAC's sample-efficiency benefits, especially on hopper-hop and reacher-hard, remain when DrQ-v2 and DrM use the same replay ratio.
We will clarify the replay-ratio settings of the visual RL baselines.