acceptodds
Under review as a conference paper at ICLR 2027

Stagnant Neurons: Understanding and Restoring Plasticity in Multi-Agent Reinforcement Learning

Abstract

Multi-Agent Reinforcement Learning (MARL) methods can suffer from loss of plasticity, gradually becoming harder to adapt under sequential task changes. We identify stagnant neurons as a recurring neuron-level pattern associated with im- paired adaptation. In value-factorization methods, these units exhibit disproportion- ately small gradient-based update activity relative to parameter scale and layer-wise peers. Existing plasticity interventions are not designed to identify and repair this pattern directly. We therefore propose Knowledge-retentive Neuron-level PlastIcity Focusing InjEction (KNIFE), a diagnosis-conditioned, neuron-local plasticity re- allocation method that exactly preserves the network function at the intervention boundary. RUA identifies impaired units, local surgery restores trainable capacity, and decay with pruning recycles temporary capacity. Across SMACv2, SMAC, Predator–Prey, matrix games, and MEAL, KNIFE consistently outperforms the compared plasticity interventions. Beyond value factorization, KNIFE also im- proves adaptation with MAPPO, MADDPG, and DGN, spanning policy-gradient, actor–critic, and graph-based MARL learners. Additional single-agent PPO results demonstrate that the intervention is not specific to value-factorization training.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Plasticity Loss

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4QpHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies loss of plasticity during sequential training in multi-agent reinforcement learning. It identifies “stagnant neurons,” defined as neurons with low relative update activity (RUA), and proposes KNIFE, a method that periodically detects stagnant and volatile neurons and applies a local surgery to them. The surgery keeps the network function unchanged at the time of intervention while introducing a fresh trainable branch. The frozen branches are later decayed and pruned. The method is evaluated across several MARL benchmarks and algorithms, as well as a single-agent PPO setting, and generally improves adaptation over existing plasticity interventions.

Critical strengths
  • The paper studies an interesting and important problem: loss of plasticity under sequential training, and it provides a neuron-level diagnostic that appears strongly related to poor adaptation.
  • The experimental evaluation is broad, covering several environments, value-factorization methods, other MARL algorithms, and a single-agent PPO experiment.
  • The ablations are fairly thorough. In particular, the paper separately studies the effect of the neuron-selection rule and the repair procedure, and shows that RUA-based targeting and the proposed surgery both matter.
Critical weaknesses
  • It is still unclear whether stagnant neurons are a cause of plasticity loss or mainly a symptom of some broader optimization failure. The intervention experiments show that they are useful targets, and artificially freezing gradients hurts learning, but this does not fully explain the causal role of naturally occurring low-RUA neurons.
  • I find the MARL framing questionable. Nothing in RUA or the KNIFE surgery seems inherently specific to multi-agent learning. The same procedure can in principle be applied to ordinary continual learning or even more general neural-network training, and the paper itself shows that it improves single-agent PPO. The MARL-specific parts are mainly where the authors choose to study the phenomenon and the value-factorization-specific KI++ analysis, rather than the method itself. I think the paper should be much clearer about what, if anything, is fundamentally specific to MARL.
  • KNIFE requires fairly low-level changes to the network: neuron-level statistics, dynamic branch insertion, freezing, decay, and pruning. The reported runtime and memory results are encouraging, but it is not clear how easy this machinery would be to integrate into larger or more optimized architectures and training systems.
  • The theoretical result seems somewhat limited. KI++ guarantees that the network function is unchanged exactly at the intervention point, but this follows from the construction explicitly cancelling the newly added branch. The result therefore does not explain why KNIFE improves plasticity or optimization after the intervention.
Questions for authors
  1. Does task order matter? The main sequential experiments use fixed task sequences. Would the amount of stagnation and the benefit of KNIFE remain similar if the same tasks were presented in a different order?
  2. What causes neurons to become stagnant in the first place? The appendix shows that more frequent task switching and target-network updates are associated with more stagnation, but this does not yet give a clear mechanism for why particular neurons enter and remain in a low-RUA state.
  3. Why does the surgery bring a neuron out of stagnation, mechanistically? The active branch is freshly initialized and trainable while the other branches preserve the current function, but it is not clear why this changes the optimization dynamics in the desired way. What happens to the gradients of the active branch immediately after surgery, and why should it escape the same low-update regime that affected the original neuron?
  4. Why is it necessary to repair volatile neurons as well? The paper shows that repairing both stagnant and volatile neurons performs better than repairing stagnant neurons alone, but the role of volatile neurons is less clear. Why should high-RUA neurons indicate a plasticity problem, and how does repairing them prevent later stagnation?

Minor comment

The fonts and legends in several figures are too small, especially in the dense multi-panel plots, which makes them difficult to read at normal paper scale.

8 comments

W1. It is still unclear whether stagnant neurons are a cause of plasticity loss or mainly a symptom of some broader optimization failure. The intervention experiments show that they are useful targets, and artificially freezing gradients hurts learning, but this does not fully explain the causal role of naturally occurring low-RUA neurons.

We agree with the reviewer. We do not claim that stagnant neurons are the sole cause of plasticity loss. Rather, they may be both a manifestation of broader optimization problems and a contributing factor to impaired plasticity.

Our intervention experiments provide evidence beyond correlation. In Figure 3(c) and Appendix D.2, Table A7, repairing different proportions of naturally occurring stagnant neurons (0%–100%) progressively reduces matrix approximation bias during recovery (from 15.19 ± 3.84 to 0.58 ± 0.17 at R8). Furthermore, Appendix D.8 (Table A26) shows that RUA-based selection outperforms alternative selection criteria and random selection under the same repair mechanism.

Together with the gradient-suppression experiments in Appendix D.2 (Figure A4), these results suggest that stagnant neurons are not merely passive indicators of optimization failure, but may also contribute to plasticity loss. We therefore view them as actionable targets for plasticity restoration, while acknowledging that their precise causal role requires further investigation.

W2. I find the MARL framing questionable. Nothing in RUA or the KNIFE surgery seems inherently specific to multi-agent learning. The same procedure can in principle be applied to ordinary continual learning or even more general neural-network training, and the paper itself shows that it improves single-agent PPO. The MARL-specific parts are mainly where the authors choose to study the phenomenon and the value-factorization-specific KI++ analysis, rather than the method itself. I think the paper should be much clearer about what, if anything, is fundamentally specific to MARL.

A key finding of our study is that stagnant neurons (neuron with low RUA) are substantially more prevalent in mixing networks than in individual agent networks (Figure 3(a) and Appendix D.2, Figure A6). The mixing network is a distinctive component of value-factorization methods in cooperative MARL, where it combines individual agents' utilities to represent the joint action-value function. This observation motivates our focus on mixing networks as the primary intervention targets in value-factorization MARL. Furthermore, our KI++ analysis addresses the function-preservation requirements specific to value-factorization MARL architectures.

We have also evaluated KNIFE on MAPPO, MADDPG, and DGN (main-text Table 2; Appendix D.5, Table A23), demonstrating its effectiveness beyond value factorization. The additional single-agent PPO experiment (Appendix D.4.3, Table A21) provides preliminary evidence that the underlying intervention can transfer beyond MARL. These results demonstrate the broader applicability of our approach, while our primary contribution remains the investigation and restoration of plasticity in MARL. We have clarified this distinction and the MARL-specific motivation in the revised manuscript.

W3. KNIFE requires fairly low-level changes to the network: neuron-level statistics, dynamic branch insertion, freezing, decay, and pruning. The reported runtime and memory results are encouraging, but it is not clear how easy this machinery would be to integrate into larger or more optimized architectures and training systems.

To further evaluate its scalability, we conducted additional experiments using a QMIX network with 10 additional layers.

As shown in the new results, KNIFE reduces the Recovery Mean Bias from 19.42 ± 5.37 to 4.85 ± 5.07. At the final recovery checkpoint (R8), the bias decreases from 13.41 ± 14.86 to 0.06 ± 0.05. These results demonstrate that KNIFE remains effective in substantially deeper networks.

MethodT0T1T2T3T4T5T6T7T8R1R2R3R4R5R6R7R8Recovery Mean Bias
QMIX0.444 ± 0.34021.467 ± 9.01326.356 ± 11.16330.998 ± 24.78624.947 ± 16.29038.052 ± 2.21337.489 ± 2.57541.894 ± 0.89762.290 ± 1.99222.098 ± 0.04622.146 ± 0.07422.120 ± 0.04622.110 ± 0.04719.172 ± 8.18115.531 ± 12.20314.504 ± 13.26613.411 ± 14.86419.420 ± 5.369
KNIFE0.550 ± 0.41821.442 ± 9.33225.906 ± 8.03129.556 ± 27.57528.406 ± 17.25337.067 ± 4.79337.212 ± 3.82042.078 ± 0.69256.889 ± 12.04214.675 ± 12.7476.322 ± 11.3904.561 ± 12.2183.614 ± 9.8130.052 ± 0.0220.062 ± 0.0330.062 ± 0.0710.061 ± 0.0474.849 ± 5.074

In terms of computational overhead, KNIFE requires 77.92 minutes of training time, compared with 67.76 minutes for QMIX, corresponding to an additional overhead of approximately 15%. This indicates that KNIFE achieves substantial improvements in recovery performance with moderate additional training time in the evaluated deeper architecture.

MethodTime(min)
QMIX67.76
KNIFE77.92

We acknowledge that integration into highly optimized training systems may require framework-specific modifications, which we leave for future work.

W4. The theoretical result seems somewhat limited. KI++ guarantees that the network function is unchanged exactly at the intervention point, but this follows from the construction explicitly cancelling the newly added branch. The result therefore does not explain why KNIFE improves plasticity or optimization after the intervention.

KNIFE does not attempt to reactivate the original stagnant neurons. Instead, it introduces freshly initialized and trainable active branches, while the knowledge and compensation branches preserve the original network function and remain frozen.

To examine the optimization dynamics, we measured the gradient norms, weight norms, and RUA of these neurons. Here, Original Stagnant Neurons refer to the neurons detected by RUA before KNIFE intervention, while Active Branches refer to the newly introduced trainable neurons after KNIFE. All reported metrics are averaged across the corresponding neuron groups.

As shown in the following table, the original stagnant neurons exhibit extremely low gradient activity (1.445 × 10⁻⁵) and RUA (2.582 × 10⁻⁵). In contrast, the active branches achieve a gradient norm of 0.818 and an RUA of 1.994 after the first update. These values remain substantial after the second update (0.600 and 1.486, respectively).

MetricOriginal Stagnant Neuron (Before Freezing)Active Branch (After KNIFE 1st Update)Active Branch (After KNIFE 2nd Update)
Gradient Norm1.445×10−5±2.594×10−51.445 \times 10^{-5} \pm 2.594 \times 10^{-5}0.818±0.3560.818 \pm 0.3560.600±0.2680.600 \pm 0.268
Weight Norm0.548±0.0300.548 \pm 0.0300.555±0.0220.555 \pm 0.0220.555±0.0220.555 \pm 0.022
RUA2.582×10−5±4.356×10−52.582 \times 10^{-5} \pm 4.356 \times 10^{-5}1.994±0.7551.994 \pm 0.7551.486±0.6061.486 \pm 0.606

Although the average weight norms appear identical after rounding, their full-precision values are 0.555121 ± 0.021822 and 0.555125 ± 0.021832, respectively. The average per-neuron parameter displacement between these two measurements is 8.16 × 10⁻⁴, confirming that actual parameter updates occur. This is because weight norm measures only vector magnitude, whereas parameter displacement measures changes in the entire weight vector.

These results suggest that fresh initialization provides new trainable capacity with substantially higher gradient activity, allowing learning to proceed through the active branches rather than relying on the original stagnant neurons. However, active branches may themselves become stagnant during subsequent training. We have clarified this mechanism and acknowledged the need for further investigation into the long-term maintenance of plasticity.

Reviewer_4DU2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates loss of plasticity in multi-agent reinforcement learning (MARL), focusing initially on value-factorization methods. The authors identify stagnant neurons, characterized by low gradient-based update activity relative to their incoming parameter norms and layer-wise peers. They introduce Relative Update Activity (RUA) to detect these neurons and report that stagnation is particularly prevalent in mixing networks and is associated with impaired adaptation under sequential task changes.

To address this phenomenon, the paper proposes KNIFE, a neuron-level plasticity intervention combining RUA-based identification, function-preserving neuron surgery, and decay-based pruning. The intervention replaces selected neurons with frozen knowledge and compensation branches and a newly initialized trainable branch. The authors establish that the injection operation preserves the network function, and consequently action-value orderings, at the intervention boundary under specified architectural assumptions. The method is evaluated on modified SMAC/SMACv2 environments, Predator–Prey, matrix games, and MEAL, with comparisons against existing plasticity interventions.

Critical strengths

The paper evaluates KNIFE across several MARL benchmarks and learning architectures, including value-factorization methods, MAPPO, MADDPG, and DGN, with an additional single-agent PPO experiment. The improvements are particularly convincing in the sequential matrix-game setting, where existing methods experience severe re-adaptation difficulties. The higher-seed experiments strengthen these findings. These experiments provide substantial evidence that the intervention is effective in the evaluated settings, making the empirical contribution meaningful even if the underlying mechanism is not yet fully understood.

The paper goes beyond documenting performance degradation by proposing a neuron-level diagnostic and examining its relationship with learning dynamics. Results indicate that RUA-selected neurons are more effective intervention targets than several alternatives under the evaluated conditions, and that KNIFE's repair operation provides benefits when intervention targets are held fixed. These experiments make the proposed diagnostic more compelling than a purely correlational observation and provide an interesting direction for understanding and mitigating plasticity loss.

Critical weaknesses

The central diagnostic, RUA, measures gradient magnitude relative to incoming parameter magnitude and layer-wise peers. However, this quantity is not necessarily equivalent to the effective parameter updates produced by an adaptive optimizer, nor does low relative gradient activity necessarily imply impaired learning capacity. Its dependence on parameter scale also raises questions about whether functionally equivalent parameterizations could lead to different neuron classifications. Although the intervention studies establish that RUA identifies useful targets, they do not fully establish that the selected neurons are intrinsically responsible for plasticity loss. In particular, improvements from refreshing selected neurons could also reflect general optimization benefits rather than the correction of a specific underlying pathology.

I think this distinction matters because identifying stagnant neurons as a mechanism of plasticity loss is a central scientific contribution of the paper. Stronger validation against actual optimizer-induced updates or parameterization-aware diagnostics would help establish whether RUA captures impaired plasticity rather than an optimization-related proxy.

Second, the function-preservation guarantee follows from the cancellation between the active and compensation branches at injection. Although this is a useful construction, the KI++ guarantee is an immediate consequence of preserving the entire network function and applies only before subsequent optimization and decay. It therefore does not, by itself, establish that cooperative knowledge or the architectural constraints of value-factorization methods are preserved throughout the intervention. It is unclear how the signed active and compensation contributions are reconciled with the monotonicity constraints of methods such as QMIX, or whether these constraints remain satisfied during the subsequent optimization and pruning process.

There is also a concrete numerical issue in the mechanistic evaluation. In Table A6, the reported Mean recovery biases do not equal the arithmetic averages of the displayed R1–R8 values. For example, the reported means for the 0% and 100% repair conditions are 18.33 and 3.94, whereas averaging the eight displayed checkpoints gives approximately 16.81 and 1.32, respectively. The qualitative improvement remains visible, but the aggregation procedure underlying a central experimental claim is unclear. Am i mising something here?

The plots look somewhat crowded. I would suggest simplifying them to improve readability. For example, there is no need to repeat the legend in every subplot when the same methods are shown. Similarly, shared y-axis labels could help reduce redundancy and make the figures cleaner and easier to interpret.

Questions for authors
  1. Why is the gradient-to-parameter norm ratio an appropriate measure of neuron-level plasticity under the optimizer used in the experiments? Would the reported neuron classifications and intervention advantages persist when using actual optimizer-induced parameter updates or function-preserving reparameterizations?

  2. How are the active and compensation branches implemented while respecting the nonnegative mixing-weight constraints in QMIX? Does the implementation preserve monotonicity and IGM throughout optimization, decay, and pruning, or only immediately after injection?

  3. How are the Mean values and corresponding confidence intervals in Table A6 calculated? The reported values do not match the arithmetic means of the displayed R1–R8 checkpoints. Were additional checkpoints or a different aggregation procedure used?

  4. Can the authors clarify whether KNIFE has been compared with a random-selection intervention matched for the number of repaired neurons, injection frequency, and repair mechanism? The existing controlled ablations are useful, but this comparison would help determine whether the benefits specifically arise from identifying low-RUA neurons rather than periodically introducing fresh trainable capacity.

Please also refer to the Critical Weaknesses section for the main questions and concerns I would like the authors to address.

10 comments

W1. The central diagnostic, RUA, measures gradient magnitude relative to incoming parameter magnitude and layer-wise peers. However, this quantity is not necessarily equivalent to the effective parameter updates produced by an adaptive optimizer, nor does low relative gradient activity necessarily imply impaired learning capacity.

We agree that RUA is not exactly equivalent to actual optimizer updates. To examine their relationship, we compare gradient-based RUA (we use RMSProp in KNIFE) with actual-update-based activity using Spearman correlation ρ\rho (−1≤ρ≤1-1 \leq \rho \leq 1) and stagnant-neuron overlap.

MetricT0T1T2T3T4T5T6T7T8R1R2R3R4R5R6R7R8
Spearman ρ\rho0.963 ± 0.0110.976 ± 0.0110.972 ± 0.0090.967 ± 0.0120.971 ± 0.0120.969 ± 0.0150.953 ± 0.0320.978 ± 0.0110.955 ± 0.0430.834 ± 0.1160.812 ± 0.1080.805 ± 0.1040.809 ± 0.0990.800 ± 0.1060.834 ± 0.0990.830 ± 0.1040.825 ± 0.099
Stagnant Neuron Overlap0.990 ± 0.0121.000 ± 0.0001.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.994 ± 0.0080.982 ± 0.0310.997 ± 0.0060.948 ± 0.0620.813 ± 0.0940.790 ± 0.0950.812 ± 0.0910.830 ± 0.0870.827 ± 0.0920.823 ± 0.0960.823 ± 0.0960.830 ± 0.096

During normal training (T0–T8), the two metrics show strong agreement, with correlations of 0.953–0.978 and overlaps of 0.948–1.000. During recovery (R1–R8), their agreement decreases but remains substantial, with correlations of 0.800–0.834 and overlaps of 0.790–0.830 (Appendix D.2.1, Table A10). This supports interpreting RUA as a useful proxy for relative update activity, not as an exact measure of optimizer updates.

MethodT0T1T2T3T4T5T6T7T8R1R2R3R4R5R6R7R8Recovery Mean Bias ↓
QMIX0.52 ± 0.1816.96 ± 2.6317.65 ± 2.481.59 ± 0.3513.27 ± 0.2323.66 ± 4.4038.95 ± 1.1021.13 ± 4.3157.26 ± 2.1918.73 ± 2.9618.48 ± 2.9918.79 ± 3.0118.22 ± 2.9514.95 ± 3.7115.12 ± 3.8014.99 ± 3.7915.19 ± 3.8418.33 ± 2.86
KNIFE (Non-adaptive)0.39 ± 0.1322.36 ± 4.7024.76 ± 4.411.49 ± 0.7614.41 ± 2.6327.82 ± 5.3134.40 ± 2.7128.33 ± 5.1250.85 ± 4.994.56 ± 3.623.21 ± 3.082.68 ± 3.141.95 ± 2.411.42 ± 2.281.38 ± 2.291.41 ± 2.311.41 ± 2.284.75 ± 2.29
KNIFE0.77 ± 0.2316.82 ± 2.6717.60 ± 2.491.59 ± 0.3713.66 ± 0.3424.31 ± 4.7735.97 ± 0.8528.92 ± 4.3947.15 ± 4.444.44 ± 2.913.05 ± 1.840.63 ± 0.210.33 ± 0.100.38 ± 0.130.55 ± 0.200.56 ± 0.170.58 ± 0.173.94 ± 0.82

We also evaluate an actual-update-based variant, labeled KNIFE (Non-adaptive) in the manuscript. Both variants substantially reduce recovery mean matrix bias compared with QMIX (18.33), achieving 4.75 and 3.94, respectively. The gradient-based RUA achieves a lower mean bias in this setting (Appendix D.2.1, Table A11). Note that R1–R8 report the matrix bias at the final evaluation checkpoint of each recovery stage, whereas Recovery Mean Bias is calculated by averaging all evaluation records throughout the entire recovery period, including intermediate checkpoints within each stage.

MethodT0T1T2T3T4T5T6T7T8R1R2R3R4R5R6R7R8Recovery Mean Bias ↓
SGD + QMIX0.677 ± 0.02229.189 ± 8.12333.198 ± 0.31646.210 ± 0.26113.852 ± 0.83636.183 ± 1.24132.603 ± 2.36031.346 ± 9.18851.768 ± 10.80615.554 ± 9.67915.505 ± 9.80313.359 ± 11.94313.283 ± 12.04612.695 ± 11.5849.008 ± 11.9289.017 ± 11.8808.986 ± 11.97113.412 ± 9.460
SGD + KNIFE0.797 ± 0.05032.790 ± 0.35533.025 ± 0.19846.275 ± 0.32513.242 ± 0.44035.799 ± 0.75031.467 ± 0.45133.653 ± 0.45855.544 ± 4.3894.850 ± 8.8700.656 ± 0.3830.433 ± 0.2450.378 ± 0.2100.322 ± 0.2570.279 ± 0.1470.386 ± 0.3710.197 ± 0.1682.609 ± 0.951
Adam + QMIX0.024 ± 0.01217.014 ± 8.94427.953 ± 13.0520.064 ± 0.08913.413 ± 0.56932.052 ± 12.25630.364 ± 13.04428.280 ± 13.09750.068 ± 11.5228.937 ± 12.0778.925 ± 12.1608.907 ± 12.1348.881 ± 12.1488.860 ± 12.0538.861 ± 12.1204.607 ± 9.8674.576 ± 9.9109.228 ± 10.508
Adam + KNIFE0.520 ± 0.14817.072 ± 8.63917.867 ± 8.8070.546 ± 0.45012.771 ± 0.07915.953 ± 11.49523.955 ± 12.45720.842 ± 12.39136.640 ± 5.5360.825 ± 0.7190.491 ± 0.0970.787 ± 1.0020.752 ± 0.8670.321 ± 0.1840.373 ± 0.2020.351 ± 0.3880.333 ± 0.2872.508 ± 0.682

Furthermore, KNIFE consistently improves performance with both SGD and Adam, reducing recovery mean bias from 13.41 to 2.61 and from 9.23 to 2.51, respectively (Appendix D.2.1, Table A12). We have clarified the distinction between gradient-based activity and actual optimizer updates in the revised manuscript.

W2. Its dependence on parameter scale also raises questions about whether functionally equivalent parameterizations could lead to different neuron classifications. Although the intervention studies establish that RUA identifies useful targets, they do not fully establish that the selected neurons are intrinsically responsible for plasticity loss. In particular, improvements from refreshing selected neurons could also reflect general optimization benefits rather than the correction of a specific underlying pathology.

We agree with the reviewer that stagnation neuron is not the only source of plasticity loss. Stagnation neurons could be just a sympton of plasticity loss. We have added such discussion in the revised PDF.

We conducted additional experiments on function-preserving reparameterizations of QMIX networks. Specifically, for a ReLU hidden neuron, we scaled its incoming weights and bias by a positive factor cc, while inversely scaling its outgoing weights:

$W_{\mathrm{in}}'=cW_{\mathrm{in}},\quad b'=cb,\quad W_{\mathrm{out}}'=\frac{W_{\mathrm{out}}}{c}.$ Due to the positive homogeneity of ReLU, this transformation preserves the network's input-output function while changing its parameter and gradient magnitudes. The results are depicted in the following table. The output remains zero across all tested scaling factors, confirming that the network function is unchanged. However, RUA values change with parameter scaling, and the overlap of identified stagnant neurons decreases from 1.00 to approximately 0.71.

cOutput DifferenceRUA Change (after/before)Stagnant Neuron Overlap (Jaccard)
10 ± 01.0000 ± 01.0000 ± 0.0000
20 ± 00.2500 ± 00.8758 ± 0.0424
30 ± 00.1111 ± 00.8246 ± 0.0274
40 ± 00.0625 ± 00.7060 ± 0.0476

This is because RUA measures relative update activity based on parameter and gradient magnitudes. It characterizes neuron-level learning activity under a given parameterization, rather than an intrinsic property of the represented function. Accordingly, our use of RUA focuses on identifying low-update neurons within the training parameterization, rather than maintaining identical neuron rankings across arbitrary equivalent parameterizations.

To evaluate the practical robustness of KNIFE, we further apply it under four scaling factors (c=1,2,3,4)(c=1,2,3,4). Despite changes in neuron classifications, KNIFE consistently achieves lower recovery mean matrix bias than QMIX across all four settings (3.94, 4.55, 4.66, and 6.51, respectively, versus 18.33, 15.43, 15.93, and 13.49). These results demonstrate that KNIFE remains effective under the tested function-preserving reparameterizations.

cmethodR1R8Recovery Mean Bias
1QMIX18.73 ± 2.9615.19 ± 3.8418.33 ± 2.86
1KNIFE4.44 ± 2.910.58 ± 0.173.94 ± 0.82
2QMIX17.58 ± 7.6113.36 ± 9.0215.43 ± 7.05
2KNIFE6.64 ± 7.640.49 ± 0.154.55 ± 1.44
3QMIX18.80 ± 5.4612.63 ± 9.8215.93 ± 6.87
3KNIFE6.29 ± 7.920.52 ± 0.324.66 ± 1.86
4QMIX15.50 ± 7.3511.58 ± 7.0513.49 ± 6.09
4KNIFE6.81 ± 5.180.64 ± 0.146.51 ± 1.63
Reviewer_oLrpOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

While maintaining the balance of plasticity and stability is a well studied problem in a single agent setting, it is less so in a multi-agent RL environment. This work aims to address this gap, focusing using the lens of plasticity. The authors rely primarily on value-factorization methods for analysis, which reveal that stagnant neurons, where gradient-based update activity is minimal compared to other neurons within the same layer. Based on their findings, the authors proposed KNIFE, which is a method that first identifies stagnant neurons to find neurons that have minimal gradient updates with respect to other neurons of the same layer, and reset the parameters during injection phase, and weights that are closed to zero are pruned. The authors evaluated using various multi-agent games, ranging from Starcraft, to one-step matrix and a continual multi-agent RL benchmark (MEAL). Results from various settings in Starcraft demonstrated that KNIFE not only results in higher win rates, but also leads to lower amounts of stagnant neurons, which is not surprising. Results from the one-step matrix environment and MEAL benchmark also seem to suggest that KNIFE achieve better performance.

Critical strengths

Interesting research question on the loss of plasticity in the multi-agent setting. The writing is ok, but can be significantly improved. I appreciate that the authors try to clarify the equations being used in the main paper and a motivating figure to assist the explanation of their approach (Figure 5). The experiment settings seem interesting but could be explained or studied more in depth to demonstrate the effectiveness of preserving or improving plasticity.

Critical weaknesses
  1. Approaches used in the main paper are not explained or defined. For example, what are Value Factorisation methods, QMIX, QPlex? At least a sentence to briefly describe each approach is required, since these are baselines. Same for GraMa neurons.
  2. Citations were all messed up. (For example. Line 38-39, 53, 101,..)
  3. Lines 57-61 are poorly written and incoherent. They also feel AI-generated.
  4. The writing has to be a lot better. Basic writing and communication standards must be met. Right now, the text is incoherent and full of mistakes.
  • Eg. Line 62 : We theoretically show -> We show using theoretical analyses.
  1. What is O and Omega (Line 78)?
  2. What are the best agents mentioned in Line 102?
  3. Quality of figures should be improved as well. For example, in Figure 2, the colours choices are questionable. Why blue curves on blue background? This makes the standard error or standard deviation bars hard to see.
  4. Figure 5 should have been placed much earlier in the paper. Also, a figure of other approaches for comparison would be welcomed; even in appendix.
Questions for authors

As mentioned, the writing can be significantly improved. For example:

  1. Why is this question, "loss of plasticity for multi-agents" important? There was no plots to support the claim that perhaps, in multi-agent setting, we see a similar loss of plasticity as in a single-agent setting, or perhaps even more severe loss of plasticity. It wasn't clear to me the main motivation behind this paper and without this evidence, it is hard to justify why should this paper even exist in the first place.

  2. What happened to Stability? Given that stability is the other side of the coin, was this investigated [1-2]? Typically for RL agents to do well, they would have to strike a balance between plasticity and stability. It is ok if in scenarios that the authors explored, most of the deficiencies were caused by a loss of plasticity.

  3. What are Value Factorisation methods? Why choose them rather than other methods? Would other methods have reveal similar or same results?

  4. Do you have any insights on why stagnation seems to increase in Figure 3a?

  5. How was alpha determined or defined in Definition 2? Why is 0.25 a good threshold?

  6. I am also curious to know, what are the distribution of different weight magnitude and their corresponding RUA, corresponding to Line 252.

  7. What information does stagnant neuron provides that dormant neurons and GraMa neurons do not? (line 280)

References: [1] Parisi, German I., et al. "Continual lifelong learning with neural networks: A review." Neural networks 113 (2019): 54-71.

[2] Khetarpal, Khimya, et al. "Towards continual reinforcement learning: A review and perspectives." Journal of Artificial Intelligence Research 75 (2022): 1401-1476.

9 comments

W1. Approaches used in the main paper are not explained or defined. For example, what are Value Factorisation methods, QMIX, QPlex? At least a sentence to briefly describe each approach is required, since these are baselines. Same for GraMa neurons.

Thanks for your time and valuable comments.

Section 2.2 (Lines 94–105) had introduced value factorization and QMIX, including individual utility estimation, value mixing, and the monotonicity constraint. Appendix A.1 (Lines 709–720) further provides the formal definition of the Individual-Global-Max (IGM) principle of value factorization methods. We agree that QPLEX and GraMa would benefit from further explanation. In the revised manuscript, we have expanded Section 2.2 to introduce QPLEX and clarified the distinction between GraMa and RUA in Section 4.2 (“Difference from Other Neuron Criteria”). We have also added a comparison table (Appendix Table A1) summarizing the selection criteria and intervention mechanisms of related methods, including ReGraMa.

For convenience, we provide brief descriptions of the requested concepts below:

  • Value Factorization: A class of MARL methods that decomposes the joint state-action value function into individual agent utilities, through a mixing network. Agents execute theirs actions through using agent networks (individual agent utilities). The mixing network model the joint state-action value function of the multiple agents, whereas an agent network models an individual agent utility.

  • QMIX : A popular MARL value-factorization method that employs a mixing network with non-negative weights to enforce monotonicity among its joint action-value function and individual agent utilities.

  • QPLEX : A value-factorization method whose mixing network uses a duplex dueling architecture. It has the same agent network as QMIX.

  • GraMa Neurons : Neurons identified by low normalized gradient magnitude, indicating weak gradient-based activity. Unlike RUA, this criterion does not account for the ratio between gradient magnitude and incoming parameter magnitude.

W2. Citations were all messed up. (For example. Line 38-39, 53, 101,..)

We have revised all the citations in the revised PDF.

W3. Lines 57-61 are poorly written and incoherent. They also feel AI-generated.

We have revised the origin text.

Original Text:

"KNIFE performs diagnosis-conditioned, neuron-local plasticity reallocation while exactly preserving the network function at the intervention boundary: RUA identifies where plasticity is impaired, local surgery restores trainable capacity, and per-update decay with pruning recycles the temporary capacity. The surgery is realized by knowledge, active, and compensation branches; the knowledge and compensation branches are frozen, while the active branch is re-initialized and trainable. The algorithmic contribution lies in integrating diagnosis, local repair, and recycling rather than in any single branch in isolation"

Revised Text:

"KNIFE applies a knowledge-invariant neuron surgery. In the injection stage, KNIFE replaces each stagnant/volatile neuron with three neurons (knowledge, active, compensation) so that the combined output matches the original at the intervention boundary. The knowledge and compensation are frozen while the active neuron is re-initialized to restore plasticity."

W4. The writing has to be a lot better. Basic writing and communication standards must be met. Right now, the text is incoherent and full of mistakes. Eg. Line 62 : We theoretically show -> We show using theoretical analyses.

Thank you for your valuable suggestion. The statement identified at Line 62 in the earlier draft appears in the revised manuscript in Section 1 (Lines 66–70). Theorem 1 in Appendix B (Lines 916–971) establishes that KNIFE’s injection operation satisfies the KI++ principle under the specified architectural conditions. The proof first establishes that the mixing function remains unchanged immediately after injection and then derives the preservation of joint action-value orderings.

We agree that the wording can be made more precise and have replaced "We theoretically show" with "We show using theoretical analyses" to explicitly reflect the formal nature of this result. We have also conducted an additional language-editing pass throughout the manuscript, using a grammar-checking tool to improve grammatical accuracy, clarity, and readability.

W5. What is O and Omega (Line 78)?

In the Dec-POMDP formulation in Section 2.1 (Lines 76–86), OiO_i denotes the local observation space of agent ii, while Ωi\Omega_i denotes its observation function, which specifies the probability distribution of observations conditioned on the environment state.

We have revised the corresponding Dec-POMDP definition to clarify their meanings.

W6. What are the best agents mentioned in Line 102?

The phrase "best agents" refers to a general characterization of continual learners in Abel et al. (2023), rather than any specific agents or algorithms. This statement serves only as background context and is not related to the proposed method. To avoid ambiguity, we have revised the sentence from "Abel et al. Abel et al.(2023) provide a general definition of continual reinforcement learning (CRL) in which the best agents never stop learning." to "Abel et al. (2023) describe continual reinforcement learning (CRL) broadly as a setting in which agents never stop learning over time."

W7. Quality of figures should be improved as well. For example, in Figure 2, the colours choices are questionable. Why blue curves on blue background? This makes the standard error or standard deviation bars hard to see.

We agree that the current color choices and background shading in certain figures, particularly Figure 2, reduce the visibility of performance curves and their confidence intervals.

We have improved the figures by adopting a more distinguishable color palette, reducing the opacity of background task-phase shading, and adjusting the contrast between performance curves and confidence intervals. We have also enlarged labels and legends where necessary to ensure readability at the standard paper scale.

To further assess the readability of the revised figures, we asked 10 researchers to review them. Their feedback indicated that the revised figures were highly readable and clearly presented the experimental results.

These changes improve the clarity of the visual presentation without altering the underlying experimental results.

W8. Figure 5 should have been placed much earlier in the paper. Also, a figure of other approaches for comparison would be welcomed; even in appendix.

Thank you for the suggestion. We agree that Figure 5 is important for understanding KNIFE. In the revised manuscript, we keep it close to the corresponding mathematical formulation so that the illustration can be read together with the equations describing the injection mechanism.

To make the comparison with prior approaches clearer, we have also expanded the appendix by adding both a comparison table summarizing the main differences between KNIFE, PI, ReDo, ReBorn, and ReGraMa, and a comparison figure illustrating the intervention mechanisms of KNIFE, ReDo, ReBorn, and ReGraMa. These additions provide a more direct visual comparison with existing methods.

Q1. Why is this question, "loss of plasticity for multi-agents" important? There was no plots to support the claim that perhaps, in multi-agent setting, we see a similar loss of plasticity as in a single-agent setting, or perhaps even more severe loss of plasticity. It wasn't clear to me the main motivation behind this paper and without this evidence, it is hard to justify why should this paper even exist in the first place.

Thanks for your suggestion. A central contribution of our work is to demonstrate that loss of plasticity also exists in MARL, particularly in value-factorization MARL, a major family of cooperative MARL methods. Unlike single-agent RL, value-factorization MARL consists of multiple agent networks and a mixing network that combines their outputs. These distinct components introduce plasticity challenges that are not present in standard single-agent architectures.

Our experiments provide direct evidence. Figure 1 shows degraded re-adaptation for QMIX in the sequential matrix game, while Figure 2 shows similar degradation for QMIX, QPLEX, and DMIX in modified SMACv2. Figure 3(a) and Appendix D.2, Figure A6, show that stagnant neurons are more prevalent in mixing networks than in agent networks. Because mixing networks are distinctive components of value-factorization architectures, these findings reveal an underexplored manifestation of plasticity loss in cooperative MARL and motivate interventions that account for this architecture.

Q2. What happened to Stability? Given that stability is the other side of the coin, was this investigated [1-2]? Typically for RL agents to do well, they would have to strike a balance between plasticity and stability. It is ok if in scenarios that the authors explored, most of the deficiencies were caused by a loss of plasticity.

We thank the reviewer for raising this important question. We agree that stability and plasticity are two complementary objectives in continual learning [1,2]. Previous studies have explored this trade-off through knowledge consolidation and experience-based mechanisms. For example, Progress & Compress [3] preserves previously acquired knowledge while learning new tasks, while Neuroplastic Expansion [4] incorporates neuron consolidation via experience review to balance stability and plasticity.

In our work, KNIFE provides a form of stability at the intervention boundary through function-preserving neuron surgery. Specifically, the knowledge branch retains the original parameters, while the active and compensation branches initially cancel each other's contributions. As formally established in Theorem 1 (Appendix B), this construction preserves the network function immediately after injection. Subsequent optimization and decay are intentionally allowed to modify the learned function to facilitate adaptation.

We acknowledge that preserving the network function at injection does not guarantee long-term retention of previously learned tasks. Our primary objective is to maintain the ability to efficiently adapt to the currently active task, rather than simultaneously preserve performance on all previous tasks.

This focus is consistent with a seminal work [5], which focus on maintaining plasticity rather than stability. Following this study, we focus on understanding and mitigating loss of plasticity in MARL, rather than providing a comprehensive solution to the stability–plasticity dilemma. We have also explicitly acknowledged this scope limitation in the revised manuscript's Limitations section.

[1] Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural Networks, 113, 54–71.

[2] Khetarpal, K., Riemer, M., Rish, I., and Precup, D. (2022). Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75, 1401–1476.

[3] Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. (2018). Progress & Compress: A scalable framework for continual learning. Proceedings of the 35th International Conference on Machine Learning (ICML), PMLR 80, 4528–4537.

[4] Liu, J., Obando-Ceron, J. S., Courville, A., and Pan, L. (2025). Neuroplastic Expansion in Deep Reinforcement Learning. International Conference on Learning Representations (ICLR).

[5] Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. (2024). Loss of plasticity in deep continual learning. Nature, 632(8026), 768–774.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.