- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates two dynamic structural editing operations, pruning and growth, within the context of structural plasticity, highlighting that they are not symmetric during the optimization process. Unlike pruning, which removes neurons that have already accumulated a training history, growth introduces new capacity into a partially trained network. The final quality of the newly added structure and its ability to function effectively within the ongoing optimization process are distinct issues.
Critical strengths
- The authors show that while both growth and pruning can yield final sparse structures of comparable quality, they result in different training trajectories. Further diagnostic analysis reveals that although newly added units are activated normally during the forward pass, they receive relatively weak gradients during the backward pass.
- The authors find that the locations where growth occurs significantly influence network representations and the capacity for adaptation to subsequent tasks; while improving gradient propagation for new units may not necessarily boost current accuracy, it can enhance future adaptability in certain continual learning scenarios.
- The experiments also suggest that the effectiveness of growth is contingent upon factors such as the frequency of structural editing, newborn-unit trainability, and the rate at which tasks evolve.
Critical weaknesses
- The causal evidence for "Backward Starvation" is still incomplete. The authors show that newly added neurons receive weaker backward gradients than existing neurons. However, it is not yet clear whether these weaker gradients actually prevent new neurons from integrating into the network and reduce their ability to adapt later. A weaker gradient does not necessarily mean less effective learning. Some of the paper's own findings make this connection harder to establish. Table B.9 shows that gradient integration metrics do not reliably predict linear-probe accuracy. Figure B.5 also reports a negative correlation between gradient signals and current accuracy (Spearman ρ = −0.62). Although the authors conduct several interventions, these change other aspects of optimization as well. It is therefore difficult to determine how much of the observed performance difference is specifically caused by "Backward Starvation."
- GROW and PRUNE differ in their training histories. GROW and PRUNE do not simply differ in how neurons are added or removed. They also expose the network to different amounts of capacity during training. Figure B.2 shows that the average active-head capacity is 38.2% for GROW and 65.0% for PRUNE. The PRUNE-FT experiment helps address the effect of weight regression, but it does not fully account for this difference in available capacity. As a result, the performance gap may reflect both the integration of new neurons and the different training histories. A more closely matched comparison would help separate these effects.
- The evidence for "future adaptability" covers a limited setting. In section 4.3.1 (Figure 4), future adaptability is mainly evaluated by permuting CIFAR-100 labels. The input images and visual categories remain unchanged, so this experiment primarily tests how well a model can reuse its existing representations under a different label assignment. This is a useful measure of adaptability, but it does not directly test whether the model can learn genuinely new features or knowledge. The continual-learning experiments in §6 provide a broader evaluation, yet they do not clearly establish that "Backward Starvation" is the mechanism responsible for performance differences across tasks. The broader applicability of this explanation remains uncertain.
- Some experimental results need clarification. Table B.8 reports a final future accuracy of 43.3% for MID+LATE GROW, while the text before Figure B.9 reports 44.8%. A similar discrepancy appears for LATE GROW, with values of 46.6% and 47.2%. Since these results appear to follow the same evaluation protocol, it would be helpful for the authors to check the original records and explain the differences.
- Table caption placement does not conform to the template specifications. The official ICLR 2027 template requires that table numbers and captions be placed above the tables. However, tables in this paper follow the opposite layout: the caption is placed below the table.
Questions for authors
Please address the concerns raised in the Weaknesses section, particularly the causal evidence for the proposed backward-starvation mechanism, the confounding effects of unequal capacity exposure, the scope of the future-adaptability evaluation, and the inconsistencies in the reported results.
We thank the reviewer for the feedback.
Specifically around the distinction between the observed backward-gradient asymmetry and its possible causal consequences. We appreciate such comments as it helps clarify the overall interpretation, and limitations of our findings.
In response, we have:
CW 1
We acknowledge that the interventions in Section 5 affect multiple aspects of optimization. However, we characterize backward starvation as the persistent disparity in gradient signals between newborn and previously active units, and we do not directly simply establish that weaker gradients alone are the main cause of poor integration or of reduced future adaptability. We have clarified this by revising the wording in Sections 1, 4.2, 5, 6, and the conclusion.
Also, our Rand. Smooth-Leaky activation intervention on Split-CIFAR100 brings newborn post-birth gradients closer to previously active units levels while improving Grow's CL performance (Figs. 5–6). This does not isolate gradient magnitude as the causal factor, but it is consistent with our hypothesis providing further supporting evidence.
CW 2
We agree that Grow and Prune undergo different capacity histories, but this is an inherent consequence of their respective methods: one introduces units and the other removes them.
This asymmetry would be generally expected to favor Prune, due to having more active capacity throughout training, but despite this, Grow reaches comparable final accuracy. This asymmetry is part of what interests us most about studying these two differing approaches.
We have revised the draft and clarified this distinction specifically in Sections 4.1 - 4.2 and Appendix B.4.2.
The asymmetry in Appendix B.4.2 (PRUNE-FT) does not claim that these controls eliminate differences in the capacity histories. It’s just isolates the effect of IMP rewinding, which has a modest impact on performance. The central observation of backward starvation is not meant to rely on the cross-method comparison. Figure 2 compares newborn and incumbent units within the same Grow network at various checkpoints and demonstrates that new units can be forward-active but still receive weaker backward signal. Therefore, the capacity history does not account for this specific newborn-incumbent asymmetry.
CW 3
We agree that the class-permutation experiment in Section 4.3.1 is designed to focus on a constrained functional remapping, rather than on learning entirely new visual features. This is intentional: label-remapping experiments have been used in prior work to isolate adaptation to changing input–output relationships while keeping the visual categories fixed [1, 2, 3]. We cite and clarify this in Section 4.3.1.
Also, in Section 6 we provide evidence that the behavior of Grow depends on the CL regime and the trainability of newborn units. This is shown by extending the evaluation beyond label remapping through class-incremental Split-CIFAR100, non-repeating class pairs in Continual ImageNet, and CIFAR 5+1, where new visual classes are encountered sequentially.
However, we acknowledge that they do not independently establish backward starvation as the causal mechanism underlying all performance differences. We have clarified this distinction in the revised manuscript.
References
[1] Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., ... & Yun, C. (2023). Plastic: Improving input and label plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 62270-62295.
[2] Elsayed, M., & Mahmood, A. R. (2024, May). Addressing loss of plasticity and catastrophic forgetting in continual learning. In International Conference on Learning Representations (Vol. 2024, pp. 33399-33430).
[3] Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, Sharan Vaswani Proceedings of The 4th Conference on Lifelong Learning Agents, PMLR 330:418-444, 2026.
CW 4
We thank the reviewer for catching this error. The discrepancy arose because Table B.8 was not updated following our final experimental run, whereas the text accompanying Figure B.9 already reflects the correct results. We have corrected the affected entries in Table B.8 to match the final run. These corrections do not alter the conclusions of the analysis.