acceptodds
Under review as a conference paper at ICLR 2027

OPAIR: Objective-Guided Capacity Repopulation for Neuron Recycling

Abstract

Loss of plasticity limits the ability of deep reinforcement learning agents to keep learning, and neuron recycling mitigates it by periodically resetting under-utilized neurons. Existing methods focus on which neurons to release and reinitialize them with a rule independent of the training objective that zeroes their outgoing weights. The recycled neurons thus receive little learning signal from the objective and return to dormancy before they contribute to reducing it, which leaves the released capacity unused. To enable the released capacity to learn from the objective, we revisit neuron recycling as a two-stage procedure of release and repopulation and propose Objective-guided PAIRed neuron recycling (OPAIR). OPAIR adopts the activation-based release rule of ReDo, which we prove causes only a small objective perturbation, and replaces each pair of released neurons with two copies of an unreleased neuron, chosen by the objective and assigned opposite outgoing weights. This paired construction preserves the network function, and the objective-guided choice maximizes their objective-aligned learning signal along feature-rescaling directions among the candidate donors. Across 27 Atari and MuJoCo tasks, OPAIR achieves the highest mean final return on every task, and mechanism analyses confirm that both the paired construction and the objective-guided selection of copied neurons contribute.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning, neuron recycling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LAHyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OPAIR addresses plasticity loss in deep RL by improving neuron repopulation. It follows ReDo’s release rule and fills released slots with paired donor copies assigned opposite outgoing weights, preserving the post-release function while enabling incoming-parameter gradients at fixed network size. Donor selection maximizes a first-order feature-rescaling criterion within the candidate set. Across 17 Atari and 10 MuJoCo tasks with five seeds, OPAIR reports the highest mean final return compared with no recycling, ReDo, and ReGraMa.

Critical strengths

The paper addresses a meaningful question—how to enable recycled neurons to learn rather than merely reset them—and is clearly written and well organized. OPAIR provides a simple fixed-capacity solution to the initial gradient blockage caused by zero outgoing weights. It preserves the post-release function and leaves the donor intact while allowing its copies to receive opposite incoming-parameter gradients. The cancellation and donor-selection arguments are clear within their stated scope.

The experiments cover different RL algorithms and architectures across Atari and MuJoCo, with five seeds per task. Comparisons with alternative donor-selection rules and improvements over OPAIR-Zero support the complete method’s value beyond simply copying active features. Full learning curves and hyperparameter studies further strengthen the empirical evidence, although the highest mean return on every task should not be interpreted as statistical significance in every comparison.

Critical weaknesses

1. Longer re-reset intervals do not directly demonstrate useful learning. Section 4.3 interprets longer re-reset intervals as evidence that recycled neurons re-enter optimization. However, copied neurons may remain active because they inherit an active donor’s features; OPAIR-Zero also substantially extends these intervals. This statistic alone therefore does not establish how effectively recycled neurons contribute to target fitting. A controlled measurement of post-recycling loss reduction, or a narrower interpretation of this statistic, would strengthen the analysis.

2. The ablations do not fully isolate the reasons for improvement. As the authors acknowledge, OPAIR-Same changes both the forward output and the copies’ update symmetry, so it cannot isolate the benefit of function preservation. Likewise, capping baseline resets at 2% does not establish that their actual reset counts match OPAIR’s. Without that comparison, fewer resets cannot be ruled out as a contributor. The ablations support the complete method, but the claims about why it works should be qualified.

Questions for authors
  1. Starting from the same pre-recycling checkpoint, could you compare short-horizon loss reduction under OPAIR and OPAIR-Zero on fixed replay data with fixed Bellman targets, using matched minibatches and update budgets? How does any fitting benefit relate to the longer re-reset intervals?

  2. Given that OPAIR-Same changes both the forward output and the copies’ update symmetry, is there an informative control or analysis that helps distinguish their contributions? A function-preserving same-sign construction would require compensation elsewhere, potentially changing the donor’s parameters and gradients; please account for those differences if using such a control. If these effects cannot be cleanly separated, please qualify the interpretation of the existing ablation accordingly.

  3. How do actual reset counts over training compare between the 2%-capped baselines and OPAIR? If they remain substantially different, please qualify the claim that fewer resets have been ruled out as a contributor.

  4. Are donor scores computed on FmaskF_{\mathrm{mask}}, as in Appendix A.6, or on the pre-release network? If the latter, how much does masking change donor rankings on a representative batch?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer LAHy 1/2

We appreciate the reviewer's detailed review and valuable feedback. We provide our responses to the specific questions below.


W1 & Q1: How do recycled neurons contribute to target fitting? More analyses are needed.

Thank you for the suggestion. We have narrowed the interpretation of the re-reset interval in Section 4.3 to the activity of recycled neurons.

We add experiments to further measure the contribution of recycled neurons to target fitting. Starting from the same pre-recycling checkpoint, we construct two branches with the same released slots, donors, and incoming initialization. OPAIR uses nonzero, oppositely signed outgoing vectors for the copies, while OPAIR-Zero sets them to zero. Both branches continue normal training for one recycling period. Immediately before the next recycling event, we measure the following two metrics for the neurons recycled at the start of that period:

  • Gradient flow (GraMa [1]): the activation-gradient magnitude of recycled neurons relative to the layer average. Closer to 1 is better.
  • Functional contribution [2]: the relative increase in TD loss after zeroing out recycled neurons. Higher is better.

Gradient flow

TaskOPAIR-ZeroOPAIR
Asterix0.043 ± 0.0020.647 ± 0.010
Ant0.098 ± 0.0190.421 ± 0.012
Walker2d0.103 ± 0.0580.523 ± 0.017

Functional contribution (×10⁻⁴)

TaskOPAIR-ZeroOPAIR
Asterix0.172 ± 0.0910.935 ± 0.620
Ant8.504 ± 1.99936.709 ± 3.310
Walker2d3.936 ± 2.69015.168 ± 1.524

5 seeds, mean ± standard error.

Compared with OPAIR-Zero, the recycled neurons of OPAIR achieve gradient flow closer to 1 and higher functional contribution on all three tasks. This shows that they keep learning while remaining active and contribute more to target fitting. The details and results are provided in Appendices E.1 and E.3.


W2 & Q2: Can you separate and analyze the individual contributions of function preservation and opposite updates in the OPAIR-Same ablation?

To better distinguish the contributions of function preservation and opposite updates, we introduce OPAIR-Perturbed. The three variants are compared below:

PropertyOPAIR-SameOPAIR-PerturbedOPAIR
Donor-based initialization✓✓✓
Function preservation at recycling✗✗✓
Opposite incoming gradients at initialization✗✓✓
Frozen, decaying perturbation✗✓✗

OPAIR-Perturbed retains OPAIR's opposite-sign trainable pair but introduces the same initial forward perturbation as OPAIR-Same through a frozen donor feature, whose contribution decays linearly to zero within one recycling period. Comparing it with OPAIR examines the effect of function preservation, while comparing it with OPAIR-Same helps distinguish the contribution of opposite-gradient initialization.

To exclude the effect of the number of resets, OPAIR-Same and OPAIR-Perturbed reset, in each layer and at each recycling event, the same number of neurons that OPAIR actually resets.

TaskOPAIR-Same (aligned)OPAIR-Perturbed (aligned)OPAIR
Asterix1265 ± 1582326 ± 2193237 ± 180
Ant6194 ± 1086665 ± 1607064 ± 42
Walker2d4266 ± 2414648 ± 2965239 ± 184

5 seeds, mean ± std.

OPAIR-Perturbed outperforms OPAIR-Same, showing the benefit of opposite updates, and OPAIR outperforms OPAIR-Perturbed, showing the benefit of function preservation. We have added the control settings and analysis to Appendix B.5.

AuthorsOpenReview ↗

Responses to Reviewer LAHy 2/2


W2. Capping baseline resets at 2% does not establish that their actual reset counts match OPAIR's. Without that comparison, fewer resets cannot be ruled out as a contributor.

Q3. How do actual reset counts over training compare between the 2%-capped baselines and OPAIR? If they remain substantially different, please qualify the claim that fewer resets have been ruled out as a contributor.

To further control for differences in reset budgets, we conduct a stricter comparison by matching the number of resets in each layer at every recycling event to OPAIR, ensuring identical cumulative reset counts. The results are shown below (5 seeds, mean ± std).

TaskReDo (aligned)ReGraMa (aligned)OPAIR
Asterix1456.8 ± 134.81428.0 ± 336.53237.1 ± 180.3
Ant6789.2 ± 51.46624.1 ± 325.37064.4 ± 42.4
Walker2d4715.6 ± 237.54573.0 ± 175.95238.7 ± 183.8

Under the same reset budget, OPAIR still outperforms both baselines on all three tasks, showing that its advantage remains when the number of reset neurons is matched. The matched-reset settings and results are provided in Appendix B.5.


Q4: Are donor scores computed on the released network or the pre-release network?

Thank you for the careful check. Donor scores are computed on the released network FmaskF_{\mathrm{mask}}, consistent with the derivation in Appendix A.6. Since zero-pair repopulation preserves Frec=FmaskF_{\mathrm{rec}}=F_{\mathrm{mask}}, the next-layer preactivation gradients used in Eqs. (41)–(44) coincide with those on FmaskF_{\mathrm{mask}}. We have stated this implementation detail in Section 3.2 and Appendix B.1.


References

[1] Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan. Measure Gradients, Not Activations! Enhancing Neuronal Activity in Deep Reinforcement Learning. NeurIPS, 2025.

[2] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz. Importance Estimation for Neural Network Pruning. CVPR, 2019.

Reviewer_f7rtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses loss of plasticity in deep reinforcement learning by separating neuron recycling into capacity release and repopulation. OPAIR retains ReDo's activation-based release strategy but introduces objective-guided donor selection and paired neuron initialization with opposite outgoing weights. This construction preserves the masked network's function while providing nonzero gradients to recycled neurons.

Critical strengths
  1. It is a well-motivated problem and meaningful methodological contribution.
  2. The function-preserving construction is an effective idea for the downstream task.
  3. The results show strong empirical performance across diverse environments.
  4. The donor-selection comparisons, OPAIR-Zero and OPAIR-Same controls, reset-budget experiments, hyperparameter studies, and complete learning curves strengthen the empirical contribution.
Critical weaknesses
  1. Section 3.1 does not establish that paired repopulation is necessary for function preservation. The claim requires additional assumptions about downstream mappings. So, it is an overstated theoretical claim.
  2. Equations (7)–(13) consider feature-rescaling directions rather than actual Adam updates. The optimality guarantee also applies only to the restricted donor-selection space. SO, it has limited optimization justification.
  3. Longer re-reset intervals do not necessarily indicate improved plasticity. Direct measurements of gradient flow, feature diversity, and functional contribution are missing. The plasticity analysis is insufficiently done.
  4. ReDo and OPAIR use different scoring batch sizes. The reset-budget ablation does not match cumulative resets. Comparisons with additional plasticity methods are also missing. The author should compare the baseline in a more fairer way.
  5. The paper lacks computational overhead measurements and aggregate statistical significance analysis. These are important for evaluating practical benefits. The paper needs more efficiency and statistical analysis.
  6. Experiments focus on final returns in fixed environments. Explicit non-stationary benchmarks are needed to substantiate long-term plasticity improvements. More non-stationary evaluation may help to boost the credibility of the work.
Questions for authors
  1. Under what assumptions is paired repopulation necessary for function preservation with nonzero learning signals?
  2. Do recycled neuron pairs learn distinct and useful representations, beyond maintaining longer re-reset intervals?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer f7rt 1/3

We thank the reviewer for the thoughtful comments and helpful suggestions. We address each point in the following.


W1: Section 3.1 makes an overstated theoretical claim about the necessity of paired repopulation for function preservation.

We thank the reviewer for pointing out that our necessity claim was too broad. Our intended argument is that zero-pair repopulation provides a sufficient construction for preserving the network function while allowing nonzero learning signals.

Specifically, the contributions of the two copies cancel exactly at the next-layer preactivations, independently of the downstream mappings, as shown in Eq. (10) and Appendix A.6. We have revised Section 3.1 to clarify this distinction and remove the universal necessity claim under Eq. (8). The assumptions and formal necessity statement are provided in Appendix A.8.


Q1: What assumptions make paired repopulation necessary for function preservation with nonzero learning signals?

We formalize the assumptions and the resulting necessity statement in the following lemma.

Lemma. Consider repopulation in a hidden layer ℓ\ell that modifies only the released slots SS, leaving all other parameters unchanged. Let hs(x)≥0h_s(x)\geq 0 denote the activation of slot ss and vsv_s its outgoing vector. Suppose the repopulation exactly preserves the next-layer preactivations, i.e.,

∑s∈Shs(x)vs=0,∀x.\sum_{s\in S} h_s(x)v_s = 0, \quad \forall x.

Then:

(i) Any slot satisfying hs(x)vs≡0h_s(x)v_s\equiv 0 contributes zero to the first-order objective reduction in Eq. (7). Consequently, a positive reduction requires at least two slots with nonzero, canceling forward contributions.

(ii) If exactly two slots s1,s2s_1,s_2 have nonzero forward contributions, then there exists a constant c>0c>0 such that

hs2(x)=c hs1(x),vs2=−1cvs1.h_{s_2}(x)=c\,h_{s_1}(x), \qquad v_{s_2}=-\frac{1}{c}v_{s_1}.

Thus, the two slots form a zero-pair structure up to positive feature rescaling.

The zero-pair construction in Eq. (10), which uses identical incoming parameters and opposite outgoing vectors, provides a parameter-level realization of this structure. The result does not require any injectivity assumption on the downstream mapping. We have added the lemma and its proof to Appendix A.8.


W2: How does feature rescaling relate to Adam? The scope of the theoretical guarantee needs clarification.

Analyzing learning signals along simplified optimization while using practical optimizers is a common approach in the literature [1, 2].

Our analysis derives an objective-guided donor-selection criterion with an optimality guarantee along feature-rescaling directions within the candidate set. Its practical effectiveness under Adam is further supported by the donor-selection ablations in Table 3, where OPAIR achieves the highest mean return across all seven tasks.

We have clarified the scope of the theoretical guarantee and its relation to Adam in Appendix A.4.

AuthorsOpenReview ↗

Responses to Reviewer f7rt 2/3


W3 & Q2: More plasticity analyses of recycled neuron pairs are needed.

To directly measure whether recycled neurons are learning and whether what they learn is useful, we add two measurements, gradient flow and functional contribution following your suggestion.

  • Gradient flow (GraMa [3]): the activation-gradient magnitude of recycled neurons relative to the layer average. Closer to 1 is better.
  • Functional contribution [4]: the relative increase in TD loss after zeroing out recycled neurons before the next recycling event. Higher is better.

We track both metrics over a full recycling period to capture post-recycling learning. Specifically, each method applies its own recycling procedure and continues training for one period. Immediately before the next recycling event, we measure both metrics for the neurons recycled at the start of that period.

Gradient flow

TaskReDoReGraMaOPAIR
Asterix0.080 ± 0.0040.042 ± 0.0010.647 ± 0.010
Ant0.043 ± 0.0040.076 ± 0.0010.421 ± 0.012
Walker2d0.054 ± 0.0040.051 ± 0.0020.523 ± 0.017

Functional contribution (×10⁻⁴)

TaskReDoReGraMaOPAIR
Asterix−0.039 ± 0.0010.063 ± 0.0490.935 ± 0.620
Ant0.021 ± 0.0040.461 ± 0.06036.709 ± 3.310
Walker2d0.022 ± 0.0030.497 ± 0.04015.168 ± 1.524

5 seeds, mean ± standard error.

OPAIR achieves the gradient flow closest to 1 and the highest functional contribution. These measurements show that the recycled neurons of OPAIR regain plasticity and learn useful representations.

Feature diversity

We also measure the effective feature rank (srank [6]) of features contributing to the network output. The results are averaged over the last 10% of training (5 seeds, mean ± std).

TaskBaseReDoReGraMaOPAIR
Asterix287.79 ± 3.8156.75 ± 7.50155.21 ± 3.63269.16 ± 8.09
Ant235.02 ± 2.59219.49 ± 7.48200.55 ± 6.61235.10 ± 2.80
Walker2d229.96 ± 1.37156.98 ± 9.65200.77 ± 4.26231.14 ± 1.41

Across all three tasks, OPAIR achieves a higher srank than ReDo and ReGraMa while remaining comparable to Base. This shows that OPAIR maintains feature diversity. The details and results are provided in Appendices E.1, E.2, and E.4.


W4.1 ReDo and OPAIR use different scoring batch sizes.

We thank you for the careful check. The scoring batch size of 64 for ReDo on Atari in Appendix B.1 is a typo. In our experiments, all methods use a scoring batch size of 128. We have corrected the scoring batch size in Appendix B.1.


W4.2 The reset-budget ablation does not match cumulative resets.

We add an experiment that aligns the reset timing and the number of resets at each recycling event with OPAIR, so that the cumulative resets of ReDo and ReGraMa exactly match those of OPAIR (5 seeds, mean ± std). The results are shown below.

TaskReDo (aligned)ReGraMa (aligned)OPAIR
Asterix1456.8 ± 134.81428.0 ± 336.53237.1 ± 180.3
Ant6789.2 ± 51.46624.1 ± 325.37064.4 ± 42.4
Walker2d4715.6 ± 237.54573.0 ± 175.95238.7 ± 183.8

Under the same reset budget, OPAIR outperforms both baselines on all three tasks. The matched-reset settings and results are provided in Appendix B.5.


W4.3 Comparisons with additional plasticity methods are missing.

We add Continual Backprop (CBP [5]) as an additional baseline, using the official implementation and default hyperparameters. The results are shown below (5 seeds, mean ± std).

TaskCBPOPAIR
Asterix2448.5 ± 270.53237.1 ± 180.3
BeamRider5791.8 ± 146.67222.7 ± 150.8
DemonAttack2481.1 ± 352.87613.3 ± 603.8
Seaquest2909.5 ± 95.43788.9 ± 243.4
Ant5862.3 ± 34.77064.4 ± 42.4
Walker2d4616.8 ± 142.65238.7 ± 183.8
Humanoid5298.6 ± 57.15380.7 ± 32.1

We have added these results to Appendix B.10.

AuthorsOpenReview ↗

Responses to Reviewer f7rt 3/3


W5.1: Lack of computational overhead measurements

We train on the same hardware and measure the average cost (over 50K updates) of a single recycling event, split into scoring (computing neuron scores and selecting neurons) and replacement (rewriting weights and clearing Adam moments).

TaskMethodScoring cost (ms)Replacement cost (ms)
AsterixReDo6.0320.15
AsterixReGraMa6.9019.99
AsterixOPAIR6.6732.11
Walker2dReDo14.3844.97
Walker2dReGraMa14.2445.13
Walker2dOPAIR14.7351.95

The three methods have similar scoring costs. The measurement protocol and results are provided in Appendix B.1.


W5.2 Statistical significance analysis is missing.

Based on the results in Table 1 and Table 2, we add an aggregate statistical significance analysis. For each task, we compare OPAIR with the strongest baseline by mean using a one-sided Welch t-test, and apply Holm correction for multiple comparisons across the 27 tasks. OPAIR significantly outperforms the strongest baseline on 26 of the 27 tasks (p < 0.05 after correction). The results are provided in Appendix B.9.


W6. More non-stationary evaluation may help to boost the credibility of the work.

We add a variant of SlipperyAnt [5], in which the ground friction coefficient changes every 250K steps, and train for 2M steps to further evaluate OPAIR on an explicit non-stationary benchmark. The final returns are shown below (5 seeds, mean ± std).

BaseReDoReGraMaOPAIR
4848.46 ± 1979.474749.52 ± 1612.373701.36 ± 1509.865943.89 ± 1862.94

OPAIR achieves the highest final return. The large standard deviations come from the different friction sequences across seeds. We have added this experiment to Appendix B.11.


References

[1] Edo Cohen-Karlik, Avichai Ben David, Nadav Cohen, Amir Globerson. On the Implicit Bias of Gradient Descent for Temporal Extrapolation. AISTATS, 2022.

[2] Etienne Boursier, Nicolas Flammarion. Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks. ICML, 2025.

[3] Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan. Measure Gradients, Not Activations! Enhancing Neuronal Activity in Deep Reinforcement Learning. NeurIPS, 2025.

[4] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz. Importance Estimation for Neural Network Pruning. CVPR, 2019.

[5] Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, Richard S. Sutton. Loss of Plasticity in Deep Continual Learning. Nature, 2024.

[6] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, Sergey Levine. Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning. ICLR, 2021.

Reviewer_dRBXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies neural recycling methods which address the plasticity loss problem in neural network training. The authors decompose neural recycling into two stages, namely release and repopulation. While existing methods are effective during the release stage, their objective-agnostic heuristics for repopulation are ineffective. One consequence is that the recycled neurons struggle to contribute to the reduction of the loss function and often recycled again in later iterations. To address this weakness, the authors propose OPAIR, an objective-guided paired neuron recycling method, which forks incoming weights of top contributing neurons to recycled neurons to maximize their utilities. To preserve the function parameterized by the neural network, OPAIR pairs recycled neurons and assigns them identical incoming weights and opposing outgoing weights. Empirical results in Atari and Mujoco demonstrate improved performance over existing methods and confirm that the OPAIR mitigates repeated recycling of the same neurons.

Critical strengths

I find the two-stage decomposition of neural recycling methods inspiring. While most existing work focuses on identifying dormant neurons, this paper takes a different angle and asks how to make the recycled neurons useful. This reformulation provides a new framework for understanding neuron recycling methods. It asks an interesting question that can benefit the community by inspiring follow-up studies.

The proposed method, OPAIR, is neat. OPAIR significantly reduces the search space by turning a continuous weights search problem into a discrete neuron selection problem. The idea of assigning opposing outgoing weights to pairs of neurons is clever.

Critical weaknesses

Risks of OPAIR are not well studied. Copying weights from existing neurons introduces correlated neurons, which may reduce the capacity of the neural network. It is critical to discuss risks like this and study them empirically. For example, we can track statistics like the feature rank to understand the effective capacity of the neural network through training.

Empirical performance is overclaimed. While OPAIR demonstrates dominating performance in Table 1 and Table 2, a closer look at the learning curves in Figure 2 and Figure 3 unveils that OPAIR is only marginally better than the baselines in most environments (e.g., DoubleDunk, Enduro, RoadRunner, Hopper, Humanoid). The cherry-picked numbers in the main text are misleading.

Questions for authors
  1. Do you think OPAIR has the risk of introducing too many correlated features? If so, can you share your thoughts on how to address it?

  2. In Section 4.4, one of the baselines is OPAIR-Same. If I understand correctly, OPAIR-Same introduces perfectly correlated neurons by design, which effectively reduces the capacity of the network. Yet this variant outperforms Base, Redo, and ReGraMa. These results make me wonder if the selected environments are suitable for evaluating methods which address plasticity loss. Can the authors explain these results in more detail and justify the validity of such evaluations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dRBX

We appreciate the reviewer's careful evaluation and constructive feedback. Our responses are provided below.


W1&Q1: How does OPAIR address the risk of correlated features from donor copying? Are there any statistics to track effective capacity?

We agree that copying a donor initially introduces identical features, which may reduce feature diversity if the copies remain correlated.

OPAIR mitigates this risk by assigning oppositely signed outgoing vectors to the two copies. Their incoming weights therefore receive opposite gradients immediately after repopulation, allowing the copies to develop different features during subsequent training.

To examine whether donor copying leads to a reduction in effective network capacity, we follow [1] and measure the effective feature rank (srank) of features contributing to the network output. The table below reports the average over the last 10% of training (5 seeds, mean ± std).

TaskBaseReDoReGraMaOPAIR
Asterix287.79 ± 3.8156.75 ± 7.50155.21 ± 3.63269.16 ± 8.09
Seaquest321.08 ± 9.1891.20 ± 9.2581.85 ± 16.02293.69 ± 12.36
Ant235.02 ± 2.59219.49 ± 7.48200.55 ± 6.61235.10 ± 2.80
Walker2d229.96 ± 1.37156.98 ± 9.65200.77 ± 4.26231.14 ± 1.41

The srank of OPAIR is close to that of Base, which performs no recycling, on all four tasks, and higher than those of ReDo and ReGraMa. This shows that OPAIR mitigates the risk of too many correlated features. We have added these results to Appendix E.4.


Q2: Why does OPAIR-Same outperform the baselines? Are the selected environments suitable for evaluating plasticity loss?

The competitive performance of OPAIR-Same in the original comparison does not necessarily indicate that these environments are insensitive to plasticity loss. An important factor is the number of neurons recycled during training. Different methods follow different recycling dynamics, and frequent resets can disrupt learned computation, potentially affecting their relative performance.

To examine this factor, we align the number of resets per layer and per recycling event for ReDo, ReGraMa, and OPAIR-Same with OPAIR. The results are shown below (5 seeds, mean ± std).

MethodAsterixAntWalker2d
Base1403.9 ± 179.86723.3 ± 139.24659.2 ± 112.9
ReDo (aligned)1456.8 ± 134.86789.2 ± 51.44715.6 ± 237.5
ReGraMa (aligned)1428.0 ± 336.56624.1 ± 325.34573.0 ± 175.9
OPAIR-Same (aligned)1264.6 ± 158.36193.5 ± 107.74266.1 ± 240.5
OPAIR3237.1 ± 180.37064.4 ± 42.45238.7 ± 183.8

Under matched reset budgets, OPAIR-Same no longer outperforms Base, ReDo, or ReGraMa on any of the three tasks, while OPAIR consistently achieves the highest return. OPAIR-Same inherits OPAIR's donor-copying mechanism, which initializes recycled neurons with active features and reduces repeated resets. This explains its good performance despite introducing correlated features. The matched-budget results further demonstrate that the selected environments can distinguish the effectiveness of different repopulation strategies. The matched-reset settings and results are provided in Appendix B.5.


W2.1: Are OPAIR's performance gains over the baselines significant?

We thank you for carefully reading. Table 1 and Table 2 report the values at the last evaluation point of each curve in Figure 2 and Figure 3.

For a more precise comparison, we run significance tests on the five tasks you mentioned, and OPAIR significantly outperforms the strongest baseline by mean on all of them (one-sided Welch t-test, p < 0.05 after Holm correction). Across all 27 tasks, OPAIR significantly outperforms the strongest baseline on 26. The results are provided in Appendix B.9.


W2.2 The cherry-picked numbers in the main text are misleading.

We thank the reviewer for raising this concern. We would like to clarify that Tables 1 and 2 report the results of all 27 tasks at the final evaluation checkpoint, with all methods evaluated using the same protocol and five seeds. No tasks or evaluation checkpoints were selectively chosen. The evaluation and reporting protocols are detailed in Appendix B.1.


References

[1] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, Sergey Levine. Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning. ICLR, 2021.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers

Dear Reviewers,

We thank all reviewers for the positive feedback and constructive suggestions. We have conducted additional experiments and revised the manuscript to address the questions and concerns raised in the reviews. The main updates are summarized below.

  1. Correlated features: We measure effective feature rank (srank) to assess whether donor copying reduces feature diversity, as discussed in our response to Reviewer dRBX (W1/Q1).

  2. Ablation on Reset Budgets: We match reset counts per layer and recycling event on baselines under equal reset budgets, as discussed in our response to Reviewer dRBX (Q2), Reviewer f7rt (W4.2), Reviewer LAHy (W2/Q3).

  3. Learning of recycled neurons: We measure gradient flow and functional contribution to assess whether recycled neurons participate in subsequent learning and contribute to fitting the training objective, as discussed in our response to Reviewer f7rt (W3/Q2), Reviewer LAHy (W1/Q1).

  4. Function preservation and opposite updates: We introduce OPAIR-Perturbed to disentangle the effects of function preservation and opposite updates under matched reset budgets, as discussed in our response to Reviewer LAHy (W2/Q2).

  5. Additional evaluations: We add comparisons with Continual Backprop (CBP), experiments in the non-stationary SlipperyAnt environment, computational overhead measurements, and statistical significance tests, as discussed in our responses to Reviewer f7rt (W4.3, W5.1, W5.2, W6).

  6. Theoretical clarification: We clarify the assumptions underlying paired repopulation and the scope of the donor-selection optimality guarantee, as discussed in our responses to Reviewer f7rt (W1, Q1, W2).

The Authors

AuthorsOpenReview ↗

Do Our Responses Address Your Concerns?

Dear Reviewers,

Thank you once again for your constructive reviews.

With less than 24 hours remaining in the discussion period, we were wondering whether our responses have adequately addressed your concerns or whether any points require further clarification. We would be happy to provide additional explanations if needed.

We appreciate your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.