acceptodds
Under review as a conference paper at ICLR 2027

On the Stability of Growth in Structural Plasticity: Forward-Active yet Backward-Starved

Abstract

Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization. In contrast, a model can also be adapted by editing its structure during training, for example by pruning existing hidden units or growing new ones; however, growth is not simply the inverse of pruning. Pruning selects among units trained from initialization, whereas growth inserts new capacity into an already specialized optimization trajectory. We isolate this insertion problem and show that newborn units can be forward-active yet backward-starved, receiving substantially weaker gradient signal than incumbent units. This asymmetry is weak in small MLPs but emerges in more challenging convolutional settings, where Grow reaches competitive final architectures despite weaker trajectory-level performance. Extending structural edits to residual networks reveals a strong dependence on edit location: head-localized growth produces concentrated but task-aligned representations and faster bounded label remapping, whereas growth throughout the residual hierarchy yields lower effective feature support, and a lower bounded adaptation endpoint. Interventions targeting optimizer state, insertion, selection, and trainability improve newborn integration, but not necessarily final subnetwork quality. Across continual-learning benchmarks, growth is most competitive when newborn units have sufficient time and trainability to integrate. Therefore, Grow should be evaluated not only by the architecture it ultimately discovers, but by whether late-arriving capacity becomes usable before a new structural change occurs during training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Grow, Prune, Continual Learning, IID, structural plasticity, loss of plasticity, network architecture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eAevOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates two dynamic structural editing operations, pruning and growth, within the context of structural plasticity, highlighting that they are not symmetric during the optimization process. Unlike pruning, which removes neurons that have already accumulated a training history, growth introduces new capacity into a partially trained network. The final quality of the newly added structure and its ability to function effectively within the ongoing optimization process are distinct issues.

Critical strengths
  1. The authors show that while both growth and pruning can yield final sparse structures of comparable quality, they result in different training trajectories. Further diagnostic analysis reveals that although newly added units are activated normally during the forward pass, they receive relatively weak gradients during the backward pass.
  2. The authors find that the locations where growth occurs significantly influence network representations and the capacity for adaptation to subsequent tasks; while improving gradient propagation for new units may not necessarily boost current accuracy, it can enhance future adaptability in certain continual learning scenarios.
  3. The experiments also suggest that the effectiveness of growth is contingent upon factors such as the frequency of structural editing, newborn-unit trainability, and the rate at which tasks evolve.
Critical weaknesses
  1. The causal evidence for "Backward Starvation" is still incomplete. The authors show that newly added neurons receive weaker backward gradients than existing neurons. However, it is not yet clear whether these weaker gradients actually prevent new neurons from integrating into the network and reduce their ability to adapt later. A weaker gradient does not necessarily mean less effective learning. Some of the paper's own findings make this connection harder to establish. Table B.9 shows that gradient integration metrics do not reliably predict linear-probe accuracy. Figure B.5 also reports a negative correlation between gradient signals and current accuracy (Spearman ρ = −0.62). Although the authors conduct several interventions, these change other aspects of optimization as well. It is therefore difficult to determine how much of the observed performance difference is specifically caused by "Backward Starvation."
  2. GROW and PRUNE differ in their training histories. GROW and PRUNE do not simply differ in how neurons are added or removed. They also expose the network to different amounts of capacity during training. Figure B.2 shows that the average active-head capacity is 38.2% for GROW and 65.0% for PRUNE. The PRUNE-FT experiment helps address the effect of weight regression, but it does not fully account for this difference in available capacity. As a result, the performance gap may reflect both the integration of new neurons and the different training histories. A more closely matched comparison would help separate these effects.
  3. The evidence for "future adaptability" covers a limited setting. In section 4.3.1 (Figure 4), future adaptability is mainly evaluated by permuting CIFAR-100 labels. The input images and visual categories remain unchanged, so this experiment primarily tests how well a model can reuse its existing representations under a different label assignment. This is a useful measure of adaptability, but it does not directly test whether the model can learn genuinely new features or knowledge. The continual-learning experiments in §6 provide a broader evaluation, yet they do not clearly establish that "Backward Starvation" is the mechanism responsible for performance differences across tasks. The broader applicability of this explanation remains uncertain.
  4. Some experimental results need clarification. Table B.8 reports a final future accuracy of 43.3% for MID+LATE GROW, while the text before Figure B.9 reports 44.8%. A similar discrepancy appears for LATE GROW, with values of 46.6% and 47.2%. Since these results appear to follow the same evaluation protocol, it would be helpful for the authors to check the original records and explain the differences.
  5. Table caption placement does not conform to the template specifications. The official ICLR 2027 template requires that table numbers and captions be placed above the tables. However, tables in this paper follow the opposite layout: the caption is placed below the table.
Questions for authors

Please address the concerns raised in the Weaknesses section, particularly the causal evidence for the proposed backward-starvation mechanism, the confounding effects of unequal capacity exposure, the scope of the future-adaptability evaluation, and the inconsistencies in the reported results.

1 comment

We thank the reviewer for the feedback.

Specifically around the distinction between the observed backward-gradient asymmetry and its possible causal consequences. We appreciate such comments as it helps clarify the overall interpretation, and limitations of our findings.

In response, we have:

  • Revised Sections 1, 4.2, 5, 6, and the conclusion to make the distinction clearer, without changing our central findings.
  • We have clarified in Section 4.1 the asymmetry on capacity histories of Grow vs. Prune, strengthening our argument.
  • Clarified and cited new work supporting our experimental design choices of Section 4.3.1.
  • Clarified the different capacity histories of Grow and Prune, the scope of our future-adaptability evaluation, and the interpretation of the existing ablation controls.
  • Corrected the outdated values in Table B.8,
  • Adjusted the captions of tables to follow the conference template.

CW 1

We acknowledge that the interventions in Section 5 affect multiple aspects of optimization. However, we characterize backward starvation as the persistent disparity in gradient signals between newborn and previously active units, and we do not directly simply establish that weaker gradients alone are the main cause of poor integration or of reduced future adaptability. We have clarified this by revising the wording in Sections 1, 4.2, 5, 6, and the conclusion.

Also, our Rand. Smooth-Leaky activation intervention on Split-CIFAR100 brings newborn post-birth gradients closer to previously active units levels while improving Grow's CL performance (Figs. 5–6). This does not isolate gradient magnitude as the causal factor, but it is consistent with our hypothesis providing further supporting evidence.

CW 2

We agree that Grow and Prune undergo different capacity histories, but this is an inherent consequence of their respective methods: one introduces units and the other removes them.

This asymmetry would be generally expected to favor Prune, due to having more active capacity throughout training, but despite this, Grow reaches comparable final accuracy. This asymmetry is part of what interests us most about studying these two differing approaches.
We have revised the draft and clarified this distinction specifically in Sections 4.1 - 4.2 and Appendix B.4.2.

The asymmetry in Appendix B.4.2 (PRUNE-FT) does not claim that these controls eliminate differences in the capacity histories. It’s just isolates the effect of IMP rewinding, which has a modest impact on performance. The central observation of backward starvation is not meant to rely on the cross-method comparison. Figure 2 compares newborn and incumbent units within the same Grow network at various checkpoints and demonstrates that new units can be forward-active but still receive weaker backward signal. Therefore, the capacity history does not account for this specific newborn-incumbent asymmetry.

CW 3

We agree that the class-permutation experiment in Section 4.3.1 is designed to focus on a constrained functional remapping, rather than on learning entirely new visual features. This is intentional: label-remapping experiments have been used in prior work to isolate adaptation to changing input–output relationships while keeping the visual categories fixed [1, 2, 3]. We cite and clarify this in Section 4.3.1.

Also, in Section 6 we provide evidence that the behavior of Grow depends on the CL regime and the trainability of newborn units. This is shown by extending the evaluation beyond label remapping through class-incremental Split-CIFAR100, non-repeating class pairs in Continual ImageNet, and CIFAR 5+1, where new visual classes are encountered sequentially.

However, we acknowledge that they do not independently establish backward starvation as the causal mechanism underlying all performance differences. We have clarified this distinction in the revised manuscript.

References

[1] Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., ... & Yun, C. (2023). Plastic: Improving input and label plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 62270-62295.

[2] Elsayed, M., & Mahmood, A. R. (2024, May). Addressing loss of plasticity and catastrophic forgetting in continual learning. In International Conference on Learning Representations (Vol. 2024, pp. 33399-33430).

[3] Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, Sharan Vaswani Proceedings of The 4th Conference on Lifelong Learning Agents, PMLR 330:418-444, 2026.

CW 4

We thank the reviewer for catching this error. The discrepancy arose because Table B.8 was not updated following our final experimental run, whereas the text accompanying Figure B.9 already reflects the correct results. We have corrected the affected entries in Table B.8 to match the final run. These corrections do not alter the conclusions of the analysis.

Reviewer_pRmfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
3
Read review
Paper summary

Studies unit-level growth as a structural-plasticity primitive and compares it with IMP pruning. Finds newborn units forward-active but backward-starved: active at birth, weak gradients. Growth matches pruning on final retrained masks but differs in trajectory. In ResNets, the edit locus matters. Tests interventions on optimizer state, insertion, selection and activation, plus continual-learning benchmarks.

Critical strengths
  1. Clear, useful framing: growth judged by integration over time, not just final architecture.
  2. Well executed. Many controls, ablations, honest reporting of negative results.
  3. Effective rank, variance and probing analyses add real insight into what added capacity becomes.
  4. Bounded future-reachability probe: nice way to separate current utility from adaptability.
  5. Interventions well chosen and informative, especially the current-vs-future decoupling.
  6. Edit-locus analysis in ResNets is a valuable contribution.
Critical weaknesses
  1. Plasticity limited to scheduled insertion. No learned allocation of how much new capacity to use (cf. NAS, differentiable gating).
  2. Integration deficit may partly be an artifact of hard, schedule-driven insertion. Consider using synthetic data PCFGs ala Allen-Zhou or RHM to make sure.
  3. Baselines uneven: Dense with fixed learning rate, others tuned. Dense below sparse Grow in places.
  4. Small models, modest accuracy. Scale and generality unclear.
  5. No comparison to learned or soft structure-adaptation methods.
  6. Main text dense, key results spread across appendix.
Questions for authors

1,. Would learned soft gates on newborn units remove the gradient deficit? 2. Can growth timing or locus be learned rather than fixed? 3. Does the deficit persist after normalizing by outgoing-weight norm? 4. How much of PRUNE’s continual advantage comes from rewinding? 5. Do conclusions hold at larger scale or in transformers? -- Consider using synthetic PCFG data as a quick and dirty way to answer some of these questions

2 comments

We thank the reviewer for the feedback and suggestions.

In response, we have:

  • Clarified the hyperparameter tuning procedure in Appendix A.2.
  • Improved the cross-references.
  • Clarified gradient measurements, limitations of Prune-FT control, and the scope of our conclusions.

We believe these clarifications improve the understanding of the experimental design progression.


CW 1

We acknowledge this limitation. We attempted to limit our hypothesis to isolating the optimization consequences of inserting new capacity into an already-trained network, rather than to propose an end-to-end learned architecture-allocation policy. But the latter is an excellent suggestion for an area of future work.

CW 2

We agree with the reviewer. Thus, why we studied it through controlled interventions. For example: Appendix B.2.2 examines different unit-selection criteria, Appendix B.3 investigates alternative capacity allocations, Appendix B.4.8 varies growth frequency, and Section 5 introduces Net2Wider.

CW 3

We'd like to clarify that all methods, including Dense, are independently tuned over the same optimizer and learning-rate search space in Sections 4 and 5, as detailed in Appendix A.2. In Section 6, Dense uses learning rates previously established through hyperparameter tuning on this architecture and dataset in prior literature, while Grow and Prune are independently tuned. We clarify this in App. A.2, and in Figure 5.

CW 4

We agree with this limitation. Our experiments are designed to characterize insertion-time optimization under controlled structural interventions, rather than maximize benchmark accuracy. Thus, we design a progression of controlled experiments from MLPs to ConvNet and to residual network to examine how the phenomenon changes with architectural complexity. We identify extensions in Section 7.

CW 5

We acknowledge this limitation. Such comparisons could introduce additional mechanisms for controlling structural allocation. That is the reason we use specific targeted interventions, such as GradMax or Net2Wider, that allow us to individually study the aspects of the integration bottleneck.

CW 6

We recognize that several controls relevant to interpreting the main findings could be more visible. We have revised the cross-references in Section 4.3. We hope that the issue has been partially mitigated with more clarity now with the use of the extra page


Questions For Authors:

Q1

This is an interesting empirical question for a future work extension. We discussed such in the Conclusion & Future Work section 7. Now, if we were to hypothesize about it, we could think that some kind of soft gating could mitigate the integration by controlling the progress at which the new units contribute to an existing representation, acting as a balance for abrupt insertion. But, if those gates are initialized near zero they could instead increase the gradient deficit and attenuate the backward signal to the new inserted units. Thus empirical work in future studies may be particularly helpful here.

Q2

Yes, learning when and where to grow is explicitly identified as an excellent area for future work in Section 7. Here, we demonstrate that both dimensions influence integration: Appendix B.4.8 investigates growth frequency under a fixed training budget, while Section 4.3 establishes that edit locus affects representational quality and future adaptability.

Q3

In our analysis, we measure the pre-activation gradient magnitude received by newborn units relative to previously active units, and it captures the backward disadvantage of newborn units regardless of differences in outgoing-weight norms or downstream credit assignment. Normalizing could help disentangle these contributions but it is not necessary to establish the forward-backward asymmetry. We clarify this in Section 4.2.

Q4

We partially address the effect of rewinding in Appendix B.4.2, where we compare the standard rewind-based Prune-IMP with Prune-FT, which follows the same pruning schedule without rewinding. Using a ResNet architecture, looking at the HEAD, and removing rewinding changes trajectory-average accuracy by only +0.46 percentage points and final accuracy by approximately -0.03 points at 50% compactness when evaluating in CIFAR-100. So, this already suggests that rewinding is not the primary cause of any observed difference in this setting. We mention this specifically in Section 4.1. However, with this ablation we cannot directly quantify the contribution of rewinding to Prune’s advantage under repeated continual-learning shifts. We clarify this in Appendix B.4.2

Q5

We appreciate the suggestion. Our current experiments investigate the phenomenon across many architectures with different structural edit loci and CL regimes. Extending this study further to large models or transformers would expand the scope and generalizability in future works.

I thank the authors for the clarifications. I was expecting a more thorough revision of the paper. I will retain my score although the lack of a more careful revision has probably weakened the quality of the paper compared to other submissions.

Reviewer_8QzBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates failure modes of the growth strategy for training sparse neural networks. It argues that optimization difficulties for newly inserted neurons make growth strategy perform worse during training compared to the prune strategy. The performance gap does not observed in simple architectures like MLP but emerges in more complex architectures like CNN. Lastly, the paper applies number of interventions to tackle the optimization difficulties and reveals which intervention is the most effective. In result, applying randomized smooth leaky activation function largely improves performance, while other interventions remain similar.

Critical strengths

None

Critical weaknesses
  1. Limited novelty.
  • The paper points out three culprits of optimization difficulties in growth strategy in L055. However, those problems are already have been considered in previous works. For example, [1-2] zero out outgoing weights of new neurons to prevent perturbation in input-output mapping. [3] initialize new neurons to maximize their gradients to prevent weak learning signal. The paper ignores such important related works and merely revisits solved problems. Those methods should be discussed in the paper and also be included as baselines.
  1. Limited significance.
  • The problem that the paper aims to solve is not significant. While the paper argues that the optimization difficulties in growth strategy impair performance. However in Figure 1 the growth strategy shows significantly better performance than prune or dense baselines. This contradicts with paper's claim and makes readers suspicious whether the problem is worth addressing.
  1. The paper is extremely unreadable.
  • Lots of coined terms are used in the paper (optimization-symmetric, GROW–PRUNE gap, in-place structural-edit, trajectory-level performance, warm-start editing process, initial structural plasticity, realized structural plasticity, bounded future reachability, class-permutation remapping probe, edit-time, and more), without sufficient explanation or definition. The paper is largely written by these coined words and ignores frequently used jargon in the field, which makes the paper extremely difficult to follow.
  • The purpose and lesson of each experiment is poorly described. The paper largely focuses on explaining details and results of experiments rather than what motivates them. Also justification of such experiment design is very weak. The paper should explain why the experiments are needed and why such experiment design is reasonable, especially in section 4.2 and 4.3.
  • Overall, I believe the paper is largely written by LLMs without sufficiently being polished by the authors.

References

[1] Dohare, Shibhansh, et al. "Loss of plasticity in deep continual learning." Nature 632.8026 (2024): 768-774.

[2] Evci, Utku, et al. "Rigging the lottery: Making all tickets winners." International conference on machine learning. PMLR, 2020.

[3] Evci, Utku, et al. "Gradmax: Growing neural networks using gradient information." arXiv preprint arXiv:2201.05125 (2022).

Questions for authors

See weaknesses.

1 comment

We thank the reviewer for taking the time to evaluate the paper and for the feedback provided.

In response, we have:

  • Revised and simplified terminology in the abstract, Sections 1, 2, 4.2, and 4.2 and the interpretation of Figure 1.
  • Clarified the transitions between experiments in Section 4.
  • Clarified the scope of our GradMax-style intervention.

We believe these revisions make the motivation, experimental progression, and conclusions of the paper clearer.


CW1: Limited novelty

The reviewer mentions that the paper ignores all three works: Dohare et al. (2024), Evci et al. (2020), and GradMax (Evci et al., 2022), and that the mechanisms are not discussed or included in the paper.

We refer the reviewer to Sections 2 and 5, where all three papers are cited. In particular, our experimentation in Section 5 includes an approach that is a slight variation on one of the named omissions: GradMax (Evci et al., 2022). We additionally compare activation-based and gradient-based recruitment in Appendix B.2.2, Table B.5. We also include other prior works along that may speak to the comments on function-preserving insertion, namely Net2Wider (Chen et al. 2016) in Section 5.

CW2: Limited Significance

1. Significance of the problem

Reviewer: "The problem that the paper aims to solve is not significant."

While we realize that significance of a line of research is a subjective assessment and can agree to disagree, we hope that the wealth of recent and prior works addressing the challenges of introducing and integrating new capacity into networks helps to speak to the potential interest in our topic of study.

We particularly believe that it is increasingly relevant to resource-efficient neural networks and continual learning [1,2,3]. Even more recently, Kong & Sutton (2026) [4] explicitly investigated growing and elastic neural networks as promising mechanisms for sustaining plasticity in online continual learning.

References:

[1] Evci, U., van Merrienboer, B., Unterthiner, T., Vladymyrov, M., & Pedregosa, F. (2022). Gradmax: Growing neural networks using gradient information. arXiv preprint arXiv:2201.05125.

[2] Zhao, Y., Cao, J., Saxena, D., Liu, X., Song, C., Yuan, B., & McCann, J. (2024). Overcoming growth-induced forgetting in task-agnostic continual learning. arXiv preprint arXiv:2408.10566.

[3] Liu, J., Obando Ceron, J. S., Courville, A., & Pan, L. (2025, May). Neuroplastic expansion in deep reinforcement learning. In International Conference on Learning Representations (Vol. 2025, pp. 90056-90078).

[4] Kong, J. M., & Sutton, R. S. (2026). Plasticity of Growing and Elastic Neural Networks in Online Continual Learning. arXiv preprint arXiv:2608.01475.

2. Interpretation of Figure 1

Reviewer: "However in Figure 1 the growth strategy shows significantly better performance than prune or dense baselines."

We agree that our original wording could have led to misinterpretation.

Our results in Figure 1 show that Grow outperforms Prune in final in-place accuracy across compactness levels (54.34% vs. 50.23% at 50% compactness), while their winning-ticket accuracies remain statistically indistinguishable.

Our mechanistic findings are distinct. Despite Grow's strong final performance, newborn units receive substantially weaker gradients than previously active units. That is the reason that we do not interpret backward starvation as evidence of inferior overall ConvNet performance, but as a local integration disadvantage whose consequences depend on the structural-edit process.

We made revisions to the Abstract, Section 2 (bullet point 1), and Section 4.2.

CW3 The paper is extremely unreadable

Reviewer: "Lots of coined terms are used in the paper ..."

We thank the reviewer for raising this concern. While several of the terms identified are mean to be descriptive in nature rather than formally proposed technical concepts, other terms are explicitly defined in the paper, including in-place structural-edit (Section 3), initial and realized structural plasticity (Section 4.3), and bounded future reachability (Section 4.3.1). This last one is accompanied by a mathematical definition and a controlled experimental protocol. We appreciate the cognitive burden of any extra jargon and recognize that some terminology could be simplified. Therefore, we have improved clarity without sacrificing technical precision in Sections 1, 4.2, and 4.3.

Reviewer: ""The purpose and lesson of each experiment is poorly described."

We have revised Sections 4.2 and 4.3 to better motivate the individual experiments before presenting their results. Specifically, the logical flow of the paper progresses by first exploring structural-edit performance, which leads up to investigating newborn integration. Then, we ask what the representational consequences of this are. And finally, we try to suggest the practical applications of work by examining the impact on future adaptability.

Comments on the paper

1 comment
AuthorsOpenReview ↗

AC Summary Message

Dear Area Chair, We’d like to briefly summarize the changes we have made in response to the reviewers' feedback.

  • Revised several sections to improve clarity and make our conclusions more precise. Those include Sections 4.1–4.3, the interpretation of Grow’s stronger final performance in Figure 1, and our discussion around backward starvation.
  • Clarified the different capacity histories of Grow and Prune, and the scope and limitations of our existing ablations.
  • Clarified hyperparameter tuning, and scope of our interventions in Section 5, and 6 and the Appendix A.6.
  • Improved cross-references to relevant appendix experiments.
  • Expanded the discussion of related work with additional references to recent research on network growth and plasticity.
  • Clarified limitations of learned allocation methods and larger architectures.
  • Addressed smaller issues, including terminology and presentation, outdated entries in Table B.8, and table-caption formatting.

We have taken the feedback seriously and made revisions where we believe they improve the paper. Most importantly, the central experimental findings remain unchanged.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.