acceptodds
Under review as a conference paper at ICLR 2027

What Does Model Growth Really Add? Functional Capacity that Survives Continual Learning

10 views

Abstract

Model growth adds parameters to a trained checkpoint and continues training, aiming to increase capacity while reusing the existing model instead of training from scratch. But does growing a trained model actually create new functional capacity that survives subsequent training? Parameter count alone cannot answer this question, and “capacity” is itself ambiguous; it can refer to parameter count, effective dimensionality, or the ability to represent new functions. We measure it directly, as the number of new, functionally independent directions a growth step adds, formalized as the rank increment of the model's functional Jacobian and estimated matrix-free at scale. For zero-gated function-preserving growth, this increment is exactly under a transversality condition, i.e. . Continually training a Transformer grown from to parameters across three domains, we find these added directions persist, with all remaining independent after continual learning, even when the model catastrophically forgets (perplexity ) or we deliberately destroy the old function (). Geometric capacity and retained function are therefore decoupled. The functional capacity introduced by growth persists, while the previously learned function is not retained. A second, geometrically distinct mechanism, Net2Net, reaches the same -dimensional capacity from directions that are dormant at initialization () and then retains it, so the decoupling is not specific to the zero-gate parameterization. The implication is precise. Catastrophic forgetting need not reflect a loss of functional capacity, and preserving dimensionality alone is insufficient to preserve the learned function.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model growth, continual learning, catastrophic forgetting, functional Jacobian rank, function-preserving growth, network expansion, effective rank, capacity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qXZZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a geometric analysis of model growth and its persistence under continual learning, measured through functional-Jacobian rank.

Critical strengths
  1. The paper is clearly written, carefully scoped.
  2. It reports per-seed values, a stress test on the old-span sampling budget, and a cautionary note about a coarse proxy that produced a spurious collapse.
  3. The matrix-free JVP pipeline is sound engineering, and the reproducibility and LLM-use statements are thorough.
Critical weaknesses
  1. The central empirical result is close to guaranteed by construction, and the paper’s own theory predicts it. The measured object is the matrix of K = 36 vectors ∂f/∂α_i \in R^{12.9M}, each equal to the downstream-propagated output of block i. In such a high-dimensional space, 36 nonzero vectors produced by distinct, non-identical blocks are almost surely linearly independent at a relative tolerance of 1e-3. Hard rank drops below K only if a block’s output is exactly zero or two block outputs are almost exactly collinear. Neither is expected from training on any objective, including a “conflicting” one. Theorem 2 already establishes that rank and function are independent, so observing 36 -> 36 while perplexity changes confirms a near-tautology rather than revealing an empirical decoupling. The paper should either identify a training regime in which this rank could plausibly fall, and show it does not, or reframe the result accordingly. E2 (zeroing output projections makes columns exactly zero, so rank drops by exactly that count) is also tautological. It validates the code but gives “transversality has bite” no real support.
  2. The transversality test is very weak relative to what Theorem 1 requires. Theorem 1 concerns col(J_old), the column space of the full Jacobian over roughly 253M old parameters, whose rank on the probe is likely in the millions. The test projects onto the span of only 64–1024 random old directions. In a 12.9M-dimensional output space, 36 vectors are trivially independent of a ≤1024-dimensional random subspace. Table 4 actually shows the minimum principal angle falling steadily with m (64° → 42.5° at seq = 16), which is consistent with the gate columns being substantially or entirely inside the full old span. The claim that residual = 36 is “invariant to how much of it we sample” extrapolates from m ≤ 1024 to m ~ rank(J_old) ~ 10^6–10^7. As it stands, the evidence for transversality, and so for “genuinely added capacity rather than a re-spanning,” is not convincing. Short probes (e.g., seq = 4–8, small output dimension) where m can approach rank(J_old) would make this test meaningful.
  3. Statements such as “catastrophic forgetting need not reflect a loss of functional capacity” and “forgetting is capacity reuse rather than loss” do not follow from the measurements. The work this paper contrasts with, the plasticity literature [1], measures representation rank, dormant units, and the ability to learn new tasks, so the paper never directly contests it. A real test would measure plasticity (e.g., the learning speed on a new domain) alongside rank and show where the two diverge.
  4. The hard rank is reported as “probe-independent,” but this is presumably an empirical statement at τ = 1e-3. Sensitivity to τ (e.g., 1e-2, 1e-4) and the singular-value spectra should be shown, particularly the smallest σ_i/σ_max for the 36 columns.

[1] Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632, 2024.

Questions for authors
  1. Do the results change with interleaved rather than appended insertion of new blocks?
2 comments

We thank the reviewer for this interesting question. Our experiments focus on appended insertion, and we have not systematically compared it with interleaved insertion.

Under zero-gate initialization, both strategies preserve the original model function. However, insertion location affects intermediate representations and downstream Jacobian propagation, potentially changing the rank, orientation, and evolution of the gate-Jacobian subspace. Our theoretical framework applies to either strategy, subject to its independence assumptions, but does not guarantee identical empirical behavior. Investigating insertion placement is a valuable direction for future work.

We also address the concerns raised in the weakness section:

  1. We agree that full column rank is often expected for generic vectors in high-dimensional spaces. However, high dimensionality does not guarantee independence, as Jacobian columns are structured derivatives of a shared network, not independently sampled random vectors. Rank deficiency is not restricted to zero or pairwise-collinear columns; linear dependence can arise among multiple nonzero, pairwise non-collinear directions. The number of gates provides an upper bound, not a guarantee of full rank. Theorem 2 establishes the possibility of decoupling between Jacobian rank and retained function value; it does not predict their evolution under particular training trajectories. Our experiments examine this behavior under continual learning, forgetting, and deliberate function degradation.

  2. Theorem 1 explicitly conditions rank additivity on independence from the complete old-parameter Jacobian. Appendix B (Table 4) extends the sampled-direction diagnostic from 64 to 1,024 old-parameter directions. We acknowledge that finite sampling does not establish full transversality, particularly as the minimum principal angle decreases with increasing sampling budget. Our empirical focus is the rank and evolution of a controlled gate-Jacobian subspace, not exhaustive full-model rank expansion. Even if the old Jacobian spans the finite probe-output space, this does not preclude greater global expressivity after growth. We will clarify these distinctions.

  3. We agree that forgetting and loss of plasticity are distinct. However, our three-domain continual-learning experiment (A → B → C) provides behavioral evidence of continued adaptation alongside the Jacobian measurements. Our results show that retained functionality can deteriorate while the measured gate-Jacobian rank remains unchanged, providing empirical evidence that these quantities need not vary together. We do not claim that this establishes preserved learning efficiency or that gate rank predicts plasticity. Systematic adaptation-speed comparisons would be needed to investigate that stronger relationship.

  4. We agree that numerical rank depends on the relative threshold and singular-value conditioning. Our conclusions use the stated tolerance of 1e-3 and the evaluated probe configurations. We will clarify that probe independence is an empirical observation within these settings, not a universal guarantee. Additional threshold sensitivity and normalized singular-value spectra would strengthen this assessment.

Our contribution is to distinguish parameter count, local functional dimensionality, and retained function value, supported by scalable matrix-free JVP/QR diagnostics and empirical analysis of their evolution during continual learning. We believe these results provide a useful framework for distinguishing changes in local functional geometry from changes in retained model behavior.

I sincerely thank the authors for their response, but my main concerns remain. I accept that linear dependence can arise beyond zero or pairwise-collinear columns. However, the rebuttal offer no evidence that the gate-column rank could plausibly fall in any tested regime: no singular-value spectra, no threshold sensitivity, and no control where degradation is possible. On transversality, the authors concede that finite sampling cannot establish independence from the full old-parameter span, and that the minimum principal angle shrinks as more of it is sampled. This undermines the rank-additivity and “genuinely added capacity” claim that frame the paper. With only clarifications promised and no new results, the contribution narrows to the observation that a specific K-column subspace keeps its dimension while the function changes. Overall, I maintain my score.

Reviewer_swwZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what model growth really adds to a trained model. The authors add 36 new layers to a trained language model, each with an on/off knob (a gate) that starts at zero, so the model initially behaves exactly as before. They then count how many independent ways the new layers give the model to change its output, and find exactly 36, one per knob. They keep training the model on new data, and the count stays at 36 even when the model forgets what it learned before. From this, they conclude that forgetting is not caused by a lack of capacity.

Critical strengths
  • The question of what growth actually adds, beyond more parameters, is interesting.
  • The paper is clear about its scope.
Critical weaknesses
  • The main results are expected, and the measure does not track what growth actually adds. At the moment of growth, the number of added directions is at most the number of gates by construction, and it equals 36 whenever the new layers are not degenerate. That this count stays the same while the model forgets is expected, since the paper’s own Theorem 2 already shows that rank and the learned function can change independently. More importantly, once the gates open during training, the parameters inside the new layers also start to affect the output, so growth adds far more than 36 directions, yet the paper keeps counting only the 36 gate directions. Figure 1 further shows that these 36 directions after training are almost unrelated to the original ones, so what the paper calls “preserved” is only the number 36, not the capacity that growth introduced.
  • The independence check appears too weak, and the measure is not shown to be useful. The check described in the main text compares the new directions against only 64 sampled directions of the old model, in an output space of about 12.9 million dimensions, which is very unlikely to detect any overlap. In fact, the old model’s output embedding matrix alone can already change every one of these outputs independently on the probe batch, which would mean the old model already spans the whole output space and growth adds no truly new directions. Beyond this, the paper never shows that the measure predicts anything useful, such as how well the model learns new tasks, which the authors also acknowledge in the Future Work section.
Questions for authors
  • How many old directions are compared against in Appendix B? Does the result change if many more are used?
  • Does the old model already cover the whole output space on the test inputs? If so, how can growth add new directions?
  • Is there any evidence that this measure relates to model performance?
2 comments

We thank the reviewer for the constructive feedback and for recognizing the importance of the research question. We address the three questions below.

  1. The main-text diagnostic uses 64 sampled old-parameter directions. Appendix B (Table 4) extends the analysis to 64, 128, 256, 512, and 1,024 directions. The reported diagnostics remain consistent across the evaluated configurations, including the larger sampled subspaces. This provides an empirical stress test beyond the main-text comparison, although we acknowledge that finite sampling does not rigorously establish independence from the complete old-parameter Jacobian.

  2. Whether the old-parameter Jacobian spans the complete probe-output space depends on its rank and parameterization, which our sampled-direction experiments do not establish. Importantly, local tangent-space rank on a finite probe is distinct from global functional expressivity. Even if the old model spans the finite probe-output space, this does not imply that architectural growth cannot provide additional representational capabilities over the broader input distribution. Our experiments characterize the local dimensionality and evolution of the newly introduced gate-parameter directions. Theorem 1 explicitly conditions rank additivity on independence from the complete old-parameter Jacobian. We will clarify the distinction between gate-restricted dimensionality, full-model tangent-space rank, and global functional capacity.

  3. Our experiments examine the relationship between gate-Jacobian rank and retained functionality. They show that substantial forgetting or deliberate function degradation can coexist with preserved rank in the measured gate subspace. This provides empirical evidence of decoupling between local geometric dimensionality and retained function value, rather than predictive utility. We do not claim that gate-Jacobian rank predicts downstream performance, transfer, or adaptation ability. Establishing such predictive relationships is an important direction for future work.

On the other hand, we respectfully disagree with the characterization that our main results are merely expected or that the gate-restricted measurement is uninformative.

  1. Theoretical possibility does not imply empirical inevitability. Theorem 2 establishes that Jacobian rank and retained function value can vary independently, but does not guarantee rank preservation during continual learning. A 36-column Jacobian need not have rank 36. Our experiments examine whether rank collapse occurs under continual learning, catastrophic forgetting, and deliberate function degradation, providing empirical evidence beyond the theoretical possibility.

  2. Our measurements deliberately track a controlled functional subspace. As acknowledged in Corollary 1, additional block-weight directions become active after the gates open. However, measuring the same 36 gate parameters across checkpoints allows us to isolate their geometric evolution without conflating it with the larger active parameter space. Our goal is not to exhaustively characterize the full new-parameter Jacobian.

  3. Subspace rotation is a finding, not a contradiction. Figure 1 shows substantial rotation of the gate-Jacobian subspace during training. We do not claim preservation of the original functional directions. Rather, our analysis distinguishes local dimensionality, subspace orientation, and retained function value. Their different behaviors under forgetting demonstrate why these properties should not be conflated.

  4. The measurement has diagnostic value without established predictive utility. While predicting downstream performance would strengthen the work, our contribution is to characterize how selected functional directions evolve during continual learning. We do not claim that restricted rank alone determines global expressivity or adaptation ability.

We believe the theoretical decomposition, scalable matrix-free JVP/QR diagnostics, and empirical analysis of dimensionality, orientation, and retained functionality together constitute a meaningful contribution to understanding model growth.

Reviewer_swwZOpenReview ↗

response

Thanks to the authors for the response. It discusses the weaknesses I raised, but without new experiments, it mostly narrows the core claims rather than resolving the concerns. I will therefore keep my score.

Reviewer_2doCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes functional (parameter) Jacobian rank on a fixed probe set as a local measure of capacity added by function-preserving Transformer depth growth. It proves that rank increases by KK when KK new directions are independent of the full old-parameter Jacobian, then reports experiments intended to show 36 added directions, their persistence and rotation during continual learning, and similar behavior under a zero-output growth mechanism. The empirical validation uses selected gate-like columns and projections against sampled old-parameter directions.

Critical strengths

The conditional rank-additivity result is sound and the paper usefully distinguishes parameter count, function value, and local tangent-space dimension. The matrix-free JVP and QR diagnostics are potentially useful tools for studying selected Jacobian directions.

Critical weaknesses
  1. The experiments do not establish the premise needed for the headline rank claim. They project against only 50–1024 sampled old-parameter directions rather than the full old Jacobian, and after training they track only 36 selected gate columns rather than all live new-parameter directions. In the zero-output arm, the measured scalar gate is frozen and auxiliary, while the actual trainable output-projection directions are omitted.
  2. The supplementary reveals material configuration and reproducibility problems. Clean training uses exact clones with zero clone noise, for which the 36 insertion-time gate columns have rank at most 12, while the scripts reporting rank 36 silently rebuild a different model with the default clone noise 0.01. The shipped three-domain results also use non-clean defaults despite being described as clean, and the current three-domain entry point references a nonexistent enum member. These findings come from static inspection of the supplied code, configuration, and JSON; I did not execute the shipped code, load checkpoints, rerun training, or independently recompute Jacobians.
Questions for authors
  1. Can you compute or rigorously bound residual rank against the full old-parameter Jacobian and include all live new-parameter columns after training, rather than sampled old directions and 36 selected columns?
  2. Can you rerun the insertion-time diagnostic on the actual clean exact-clone model (clone_noise_std = 0) and reconcile the resulting rank-at-most-12 symmetry with the reported rank 36?
  3. For the zero-output arm, what rank is contributed by the actual trainable output-projection parameters, and why is a derivative with respect to a frozen auxiliary gate an appropriate measure of that model’s parameter-Jacobian capacity?
  4. Can you provide corrected executable code and genuinely clean three-domain results, with the exact run configuration and checkpoint provenance used for each reported diagnostic?
2 comments

We thank the reviewer for the feedback. For the questions raised by the reviewer,

We agree that computing the residual rank against the complete old-parameter Jacobian, together with all active new-parameter directions, would provide a stronger characterization of full-model tangent-space growth. However, we respectfully clarify that our empirical study is deliberately scoped to a fixed, interpretable family of 36 gate-Jacobian directions, rather than an exhaustive characterization of the full parameter Jacobian. The purpose of this restriction is to isolate and track the geometric evolution of the same parameter directions throughout continual learning, allowing us to examine their rank and changes in orientation independently of the larger set of block-weight directions that become active after training. Theorem 1 establishes a conditional rank-additivity result under the explicit assumption of independence from the full old-parameter Jacobian. Our sampled old-direction comparisons provide empirical diagnostics of this condition, rather than a rigorous verification of full transversality. Likewise, as acknowledged in Corollary 1, additional block-weight directions become active after the gates open. Measuring their complete contribution would address a broader question beyond the gate-restricted analysis undertaken here.

We believe the gate-restricted measurements remain meaningful independently of establishing full-model rank additivity, as they characterize the local dimensionality and evolution of a controlled set of parameter directions during continual learning. We will clarify the distinction between the conditional theoretical result, the gate-restricted empirical measurements, and the stronger full-Jacobian characterization proposed by the reviewer.

We want to clarify that the clean training experiments were executed with --clean-fp, which explicitly sets clone_noise_std = 0.0. The default value of 0.01 does not apply to these runs. We acknowledge that the reproduction guide should have documented this configuration more clearly.

We also distinguish the clean training runs from the separately constructed insertion-time diagnostic, which uses the default clone-noise configuration. Non-zero clone noise does not violate function preservation under zero-gate initialization. We agree that exact clones exhibit symmetry, yielding a gate-Jacobian rank of at most 12 at insertion, and will clarify the configurations associated with the reported rank measurements.

We want to clarify that E7 is a generality test of the geometric behavior observed under two function-preserving growth mechanisms, rather than an exhaustive measurement of their trainable-parameter Jacobians. The auxiliary-gate derivative provides a consistent branch-level functional diagnostic across both mechanisms, allowing us to track the dimensionality of the inserted branches' functional contributions during continual learning. In the zero-output mechanism, these directions are initially dormant and subsequently become active through training.

We agree that the output-projection weight Jacobian may contain additional functional directions. However, measuring those directions addresses a broader question than the branch-level geometric evolution studied in E7, which is not under the scope of this paper. We will clarify this distinction to avoid conflating the auxiliary-gate diagnostic with full parameter-Jacobian capacity.

Thank you for identifying these discrepancies. We reviewed again and confirmed that the released three-domain result files record the default training configuration (clone_noise_std=0.01, distill_lambda=0.5, and gate_warmup_eps=0.001), while the reproduction guide specifies --clean-fp. We acknowledge this inconsistency and will correct the experimental description and documentation. Importantly, the recorded runs remain function-preserving at insertion: the saved results report zero output difference after growth, consistent with zero-gate initialization. However, we agree that function preservation and the stricter clean configuration must be distinguished.

We also acknowledge the invalid enumeration reference in the published entry point and will correct it. The original checkpoints are no longer available for independent verification, so we cannot claim to have reproduced the stricter clean configuration or provide new clean-run measurements.

I hope the answers above can address part of your questions.

We thank the authors for their response to our comments. We will leave our rating unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.