- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OPAIR addresses plasticity loss in deep RL by improving neuron repopulation. It follows ReDo’s release rule and fills released slots with paired donor copies assigned opposite outgoing weights, preserving the post-release function while enabling incoming-parameter gradients at fixed network size. Donor selection maximizes a first-order feature-rescaling criterion within the candidate set. Across 17 Atari and 10 MuJoCo tasks with five seeds, OPAIR reports the highest mean final return compared with no recycling, ReDo, and ReGraMa.
Critical strengths
The paper addresses a meaningful question—how to enable recycled neurons to learn rather than merely reset them—and is clearly written and well organized. OPAIR provides a simple fixed-capacity solution to the initial gradient blockage caused by zero outgoing weights. It preserves the post-release function and leaves the donor intact while allowing its copies to receive opposite incoming-parameter gradients. The cancellation and donor-selection arguments are clear within their stated scope.
The experiments cover different RL algorithms and architectures across Atari and MuJoCo, with five seeds per task. Comparisons with alternative donor-selection rules and improvements over OPAIR-Zero support the complete method’s value beyond simply copying active features. Full learning curves and hyperparameter studies further strengthen the empirical evidence, although the highest mean return on every task should not be interpreted as statistical significance in every comparison.
Critical weaknesses
1. Longer re-reset intervals do not directly demonstrate useful learning. Section 4.3 interprets longer re-reset intervals as evidence that recycled neurons re-enter optimization. However, copied neurons may remain active because they inherit an active donor’s features; OPAIR-Zero also substantially extends these intervals. This statistic alone therefore does not establish how effectively recycled neurons contribute to target fitting. A controlled measurement of post-recycling loss reduction, or a narrower interpretation of this statistic, would strengthen the analysis.
2. The ablations do not fully isolate the reasons for improvement. As the authors acknowledge, OPAIR-Same changes both the forward output and the copies’ update symmetry, so it cannot isolate the benefit of function preservation. Likewise, capping baseline resets at 2% does not establish that their actual reset counts match OPAIR’s. Without that comparison, fewer resets cannot be ruled out as a contributor. The ablations support the complete method, but the claims about why it works should be qualified.
Questions for authors
-
Starting from the same pre-recycling checkpoint, could you compare short-horizon loss reduction under OPAIR and OPAIR-Zero on fixed replay data with fixed Bellman targets, using matched minibatches and update budgets? How does any fitting benefit relate to the longer re-reset intervals?
-
Given that OPAIR-Same changes both the forward output and the copies’ update symmetry, is there an informative control or analysis that helps distinguish their contributions? A function-preserving same-sign construction would require compensation elsewhere, potentially changing the donor’s parameters and gradients; please account for those differences if using such a control. If these effects cannot be cleanly separated, please qualify the interpretation of the existing ablation accordingly.
-
How do actual reset counts over training compare between the 2%-capped baselines and OPAIR? If they remain substantially different, please qualify the claim that fewer resets have been ruled out as a contributor.
-
Are donor scores computed on , as in Appendix A.6, or on the pre-release network? If the latter, how much does masking change donor rankings on a representative batch?
2 comments
Responses to Reviewer LAHy 2/2
W2. Capping baseline resets at 2% does not establish that their actual reset counts match OPAIR's. Without that comparison, fewer resets cannot be ruled out as a contributor.
Q3. How do actual reset counts over training compare between the 2%-capped baselines and OPAIR? If they remain substantially different, please qualify the claim that fewer resets have been ruled out as a contributor.
To further control for differences in reset budgets, we conduct a stricter comparison by matching the number of resets in each layer at every recycling event to OPAIR, ensuring identical cumulative reset counts. The results are shown below (5 seeds, mean ± std).
| Task | ReDo (aligned) | ReGraMa (aligned) | OPAIR |
|---|---|---|---|
| Asterix | 1456.8 ± 134.8 | 1428.0 ± 336.5 | 3237.1 ± 180.3 |
| Ant | 6789.2 ± 51.4 | 6624.1 ± 325.3 | 7064.4 ± 42.4 |
| Walker2d | 4715.6 ± 237.5 | 4573.0 ± 175.9 | 5238.7 ± 183.8 |
Under the same reset budget, OPAIR still outperforms both baselines on all three tasks, showing that its advantage remains when the number of reset neurons is matched. The matched-reset settings and results are provided in Appendix B.5.
Q4: Are donor scores computed on the released network or the pre-release network?
Thank you for the careful check. Donor scores are computed on the released network , consistent with the derivation in Appendix A.6. Since zero-pair repopulation preserves , the next-layer preactivation gradients used in Eqs. (41)–(44) coincide with those on . We have stated this implementation detail in Section 3.2 and Appendix B.1.
References
[1] Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan. Measure Gradients, Not Activations! Enhancing Neuronal Activity in Deep Reinforcement Learning. NeurIPS, 2025.
[2] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz. Importance Estimation for Neural Network Pruning. CVPR, 2019.
Responses to Reviewer LAHy 1/2
We appreciate the reviewer's detailed review and valuable feedback. We provide our responses to the specific questions below.
Thank you for the suggestion. We have narrowed the interpretation of the re-reset interval in Section 4.3 to the activity of recycled neurons.
We add experiments to further measure the contribution of recycled neurons to target fitting. Starting from the same pre-recycling checkpoint, we construct two branches with the same released slots, donors, and incoming initialization. OPAIR uses nonzero, oppositely signed outgoing vectors for the copies, while OPAIR-Zero sets them to zero. Both branches continue normal training for one recycling period. Immediately before the next recycling event, we measure the following two metrics for the neurons recycled at the start of that period:
Gradient flow
Functional contribution (×10⁻⁴)
5 seeds, mean ± standard error.
Compared with OPAIR-Zero, the recycled neurons of OPAIR achieve gradient flow closer to 1 and higher functional contribution on all three tasks. This shows that they keep learning while remaining active and contribute more to target fitting. The details and results are provided in Appendices E.1 and E.3.
To better distinguish the contributions of function preservation and opposite updates, we introduce OPAIR-Perturbed. The three variants are compared below:
OPAIR-Perturbed retains OPAIR's opposite-sign trainable pair but introduces the same initial forward perturbation as OPAIR-Same through a frozen donor feature, whose contribution decays linearly to zero within one recycling period. Comparing it with OPAIR examines the effect of function preservation, while comparing it with OPAIR-Same helps distinguish the contribution of opposite-gradient initialization.
To exclude the effect of the number of resets, OPAIR-Same and OPAIR-Perturbed reset, in each layer and at each recycling event, the same number of neurons that OPAIR actually resets.
5 seeds, mean ± std.
OPAIR-Perturbed outperforms OPAIR-Same, showing the benefit of opposite updates, and OPAIR outperforms OPAIR-Perturbed, showing the benefit of function preservation. We have added the control settings and analysis to Appendix B.5.