On the Stability of Growth in Structural Plasticity: Forward-Active yet Backward-Starved

Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization. In contrast, a model can also be adapted by editing its structure during training, for example by pruning existing hidden units or growing new ones; however, growth is not simply the inverse of pruning. Pruning selects among units trained from initialization, whereas growth inserts new capacity into an already specialized optimization trajectory. We isolate this insertion problem and show that newborn units can be forward-active yet backward-starved, receiving substantially weaker gradient signal than incumbent units. This asymmetry is weak in small MLPs but emerges in more challenging convolutional settings, where Grow reaches competitive final architectures despite weaker trajectory-level performance. Extending structural edits to residual networks reveals a strong dependence on edit location: head-localized growth produces concentrated but task-aligned representations and faster bounded label remapping, whereas growth throughout the residual hierarchy yields lower effective feature support, and a lower bounded adaptation endpoint. Interventions targeting optimizer state, insertion, selection, and trainability improve newborn integration, but not necessarily final subnetwork quality. Across continual-learning benchmarks, growth is most competitive when newborn units have sufficient time and trainability to integrate. Therefore, Grow should be evaluated not only by the architecture it ultimately discovers, but by whether late-arriving capacity becomes usable before a new structural change occurs during training.