- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies loss of plasticity during sequential training in multi-agent reinforcement learning. It identifies “stagnant neurons,” defined as neurons with low relative update activity (RUA), and proposes KNIFE, a method that periodically detects stagnant and volatile neurons and applies a local surgery to them. The surgery keeps the network function unchanged at the time of intervention while introducing a fresh trainable branch. The frozen branches are later decayed and pruned. The method is evaluated across several MARL benchmarks and algorithms, as well as a single-agent PPO setting, and generally improves adaptation over existing plasticity interventions.
Critical strengths
- The paper studies an interesting and important problem: loss of plasticity under sequential training, and it provides a neuron-level diagnostic that appears strongly related to poor adaptation.
- The experimental evaluation is broad, covering several environments, value-factorization methods, other MARL algorithms, and a single-agent PPO experiment.
- The ablations are fairly thorough. In particular, the paper separately studies the effect of the neuron-selection rule and the repair procedure, and shows that RUA-based targeting and the proposed surgery both matter.
Critical weaknesses
- It is still unclear whether stagnant neurons are a cause of plasticity loss or mainly a symptom of some broader optimization failure. The intervention experiments show that they are useful targets, and artificially freezing gradients hurts learning, but this does not fully explain the causal role of naturally occurring low-RUA neurons.
- I find the MARL framing questionable. Nothing in RUA or the KNIFE surgery seems inherently specific to multi-agent learning. The same procedure can in principle be applied to ordinary continual learning or even more general neural-network training, and the paper itself shows that it improves single-agent PPO. The MARL-specific parts are mainly where the authors choose to study the phenomenon and the value-factorization-specific KI++ analysis, rather than the method itself. I think the paper should be much clearer about what, if anything, is fundamentally specific to MARL.
- KNIFE requires fairly low-level changes to the network: neuron-level statistics, dynamic branch insertion, freezing, decay, and pruning. The reported runtime and memory results are encouraging, but it is not clear how easy this machinery would be to integrate into larger or more optimized architectures and training systems.
- The theoretical result seems somewhat limited. KI++ guarantees that the network function is unchanged exactly at the intervention point, but this follows from the construction explicitly cancelling the newly added branch. The result therefore does not explain why KNIFE improves plasticity or optimization after the intervention.
Questions for authors
- Does task order matter? The main sequential experiments use fixed task sequences. Would the amount of stagnation and the benefit of KNIFE remain similar if the same tasks were presented in a different order?
- What causes neurons to become stagnant in the first place? The appendix shows that more frequent task switching and target-network updates are associated with more stagnation, but this does not yet give a clear mechanism for why particular neurons enter and remain in a low-RUA state.
- Why does the surgery bring a neuron out of stagnation, mechanistically? The active branch is freshly initialized and trainable while the other branches preserve the current function, but it is not clear why this changes the optimization dynamics in the desired way. What happens to the gradients of the active branch immediately after surgery, and why should it escape the same low-update regime that affected the original neuron?
- Why is it necessary to repair volatile neurons as well? The paper shows that repairing both stagnant and volatile neurons performs better than repairing stagnant neurons alone, but the role of volatile neurons is less clear. Why should high-RUA neurons indicate a plasticity problem, and how does repairing them prevent later stagnation?
Minor comment
The fonts and legends in several figures are too small, especially in the dense multi-panel plots, which makes them difficult to read at normal paper scale.
8 comments
W3. KNIFE requires fairly low-level changes to the network: neuron-level statistics, dynamic branch insertion, freezing, decay, and pruning. The reported runtime and memory results are encouraging, but it is not clear how easy this machinery would be to integrate into larger or more optimized architectures and training systems.
To further evaluate its scalability, we conducted additional experiments using a QMIX network with 10 additional layers.
As shown in the new results, KNIFE reduces the Recovery Mean Bias from 19.42 ± 5.37 to 4.85 ± 5.07. At the final recovery checkpoint (R8), the bias decreases from 13.41 ± 14.86 to 0.06 ± 0.05. These results demonstrate that KNIFE remains effective in substantially deeper networks.
| Method | T0 | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | Recovery Mean Bias |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| QMIX | 0.444 ± 0.340 | 21.467 ± 9.013 | 26.356 ± 11.163 | 30.998 ± 24.786 | 24.947 ± 16.290 | 38.052 ± 2.213 | 37.489 ± 2.575 | 41.894 ± 0.897 | 62.290 ± 1.992 | 22.098 ± 0.046 | 22.146 ± 0.074 | 22.120 ± 0.046 | 22.110 ± 0.047 | 19.172 ± 8.181 | 15.531 ± 12.203 | 14.504 ± 13.266 | 13.411 ± 14.864 | 19.420 ± 5.369 |
| KNIFE | 0.550 ± 0.418 | 21.442 ± 9.332 | 25.906 ± 8.031 | 29.556 ± 27.575 | 28.406 ± 17.253 | 37.067 ± 4.793 | 37.212 ± 3.820 | 42.078 ± 0.692 | 56.889 ± 12.042 | 14.675 ± 12.747 | 6.322 ± 11.390 | 4.561 ± 12.218 | 3.614 ± 9.813 | 0.052 ± 0.022 | 0.062 ± 0.033 | 0.062 ± 0.071 | 0.061 ± 0.047 | 4.849 ± 5.074 |
In terms of computational overhead, KNIFE requires 77.92 minutes of training time, compared with 67.76 minutes for QMIX, corresponding to an additional overhead of approximately 15%. This indicates that KNIFE achieves substantial improvements in recovery performance with moderate additional training time in the evaluated deeper architecture.
| Method | Time(min) |
|---|---|
| QMIX | 67.76 |
| KNIFE | 77.92 |
We acknowledge that integration into highly optimized training systems may require framework-specific modifications, which we leave for future work.
W4. The theoretical result seems somewhat limited. KI++ guarantees that the network function is unchanged exactly at the intervention point, but this follows from the construction explicitly cancelling the newly added branch. The result therefore does not explain why KNIFE improves plasticity or optimization after the intervention.
KNIFE does not attempt to reactivate the original stagnant neurons. Instead, it introduces freshly initialized and trainable active branches, while the knowledge and compensation branches preserve the original network function and remain frozen.
To examine the optimization dynamics, we measured the gradient norms, weight norms, and RUA of these neurons. Here, Original Stagnant Neurons refer to the neurons detected by RUA before KNIFE intervention, while Active Branches refer to the newly introduced trainable neurons after KNIFE. All reported metrics are averaged across the corresponding neuron groups.
As shown in the following table, the original stagnant neurons exhibit extremely low gradient activity (1.445 × 10⁻⁵) and RUA (2.582 × 10⁻⁵). In contrast, the active branches achieve a gradient norm of 0.818 and an RUA of 1.994 after the first update. These values remain substantial after the second update (0.600 and 1.486, respectively).
| Metric | Original Stagnant Neuron (Before Freezing) | Active Branch (After KNIFE 1st Update) | Active Branch (After KNIFE 2nd Update) |
|---|---|---|---|
| Gradient Norm | |||
| Weight Norm | |||
| RUA |
Although the average weight norms appear identical after rounding, their full-precision values are 0.555121 ± 0.021822 and 0.555125 ± 0.021832, respectively. The average per-neuron parameter displacement between these two measurements is 8.16 × 10⁻⁴, confirming that actual parameter updates occur. This is because weight norm measures only vector magnitude, whereas parameter displacement measures changes in the entire weight vector.
These results suggest that fresh initialization provides new trainable capacity with substantially higher gradient activity, allowing learning to proceed through the active branches rather than relying on the original stagnant neurons. However, active branches may themselves become stagnant during subsequent training. We have clarified this mechanism and acknowledged the need for further investigation into the long-term maintenance of plasticity.
We agree with the reviewer. We do not claim that stagnant neurons are the sole cause of plasticity loss. Rather, they may be both a manifestation of broader optimization problems and a contributing factor to impaired plasticity.
Our intervention experiments provide evidence beyond correlation. In Figure 3(c) and Appendix D.2, Table A7, repairing different proportions of naturally occurring stagnant neurons (0%–100%) progressively reduces matrix approximation bias during recovery (from 15.19 ± 3.84 to 0.58 ± 0.17 at R8). Furthermore, Appendix D.8 (Table A26) shows that RUA-based selection outperforms alternative selection criteria and random selection under the same repair mechanism.
Together with the gradient-suppression experiments in Appendix D.2 (Figure A4), these results suggest that stagnant neurons are not merely passive indicators of optimization failure, but may also contribute to plasticity loss. We therefore view them as actionable targets for plasticity restoration, while acknowledging that their precise causal role requires further investigation.
A key finding of our study is that stagnant neurons (neuron with low RUA) are substantially more prevalent in mixing networks than in individual agent networks (Figure 3(a) and Appendix D.2, Figure A6). The mixing network is a distinctive component of value-factorization methods in cooperative MARL, where it combines individual agents' utilities to represent the joint action-value function. This observation motivates our focus on mixing networks as the primary intervention targets in value-factorization MARL. Furthermore, our KI++ analysis addresses the function-preservation requirements specific to value-factorization MARL architectures.
We have also evaluated KNIFE on MAPPO, MADDPG, and DGN (main-text Table 2; Appendix D.5, Table A23), demonstrating its effectiveness beyond value factorization. The additional single-agent PPO experiment (Appendix D.4.3, Table A21) provides preliminary evidence that the underlying intervention can transfer beyond MARL. These results demonstrate the broader applicability of our approach, while our primary contribution remains the investigation and restoration of plasticity in MARL. We have clarified this distinction and the MARL-specific motivation in the revised manuscript.