acceptodds
Withdrawn from ICLR 2027

Growing in the Right Coordinates: Schur-Gated Function-Preserving Network Expansion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Expanding a trained network without changing its current predictions is useful when additional capacity is needed but restarting training is undesirable. In this setting, however, adding new parameters is not enough: the added directions should complement the inherited model, and they should be parameterized in coordinates that the optimizer can use effectively. Existing growth and conditioning methods typically address these two issues separately. We introduce Sketch-Schur-OST, a function-preserving network expansion method that treats representation and optimization geometry jointly. The key idea is to use a single conditional Schur metric to determine which candidate directions add information beyond the inherited tangent space, how they should be transported relative to the existing parameters, and how the resulting coordinates should be conditioned through a full SPD architectural gate. This produces a unified construction in which selection, redundancy removal, and conditioning are different parts of the same local geometric factorization. We show that the construction preserves the inherited function at insertion, separates support creation from conditioning, and yields an exact factorization of the corresponding damped calibration metric. We further develop a shared-sketch formulation and a row-space dual that make the method practical without materializing the full joint Jacobian–candidate design, while making explicit the conditions under which the sketch guarantees apply. Across controlled growth experiments, the unified construction consistently improves finite-budget optimization relative to matched ungated and sequential alternatives, with gains that persist across coordinate budgets, longer training horizons, adaptive optimization, heterogeneous candidate banks, and larger inherited models. The results also clarify the limits of the approach: aggressive undersketching can fall outside the certified regime, and stronger architectures may remain preferable when retraining from scratch is allowed. Overall, the paper argues that function-preserving growth is best viewed not only as a problem of adding capacity, but as a problem of adding useful directions in the right coordinates.

Primary area:
optimization
Keywords:
function-preserving network growth, neural network expansion, Schur complement, architectural preconditioning, neural tangent kernel, randomized sketching, optimization geometry, adaptive architecture, network morphism, implicit neural representations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dso6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a function-preserving network-expansion method that expands the network for continual training by considering useful directions. The main experiments are conducted in a very small-scale task setting using 6 images from BSDS500.

Critical strengths

None

Critical weaknesses
  1. The paper is poorly written and very hard to follow the flow as a reader. Some examples such as:
  • The abbreviation OST is used throughout the paper and i could not find anywhere that explains what OST stands for. To the best of my knowledge, it is not a common term in the field that does not need any explanation.
  • The variable N, P, and q is undefined in the introduction.
  • In Line 71, it says "SKETCH-SCHUR-OST uses the same Schur complement for all three decisions." And it is very unclear that what "three decisions" here refer to.
  • Putting the overview figure on page 6 and isn't mentioned in the main text anywhere in the paper.
  1. The motivation for growing network without changing its prediction is very weak and unjustified. The authors simply state that "restarting with a stronger architecture is not an available intervention", but never explicitly show the downstream use cases, and does not discuss why common alternatives such as efficient finetuning or distillation isn't a good choice.

  2. The experiments are poorly designed and far from realistic to support any reasonable contribution.

  • The main experiment is using a model with only 697 to 745 parameters, and it is only using six images from a small and outdated dataset (Line 304-307).
  • The baselines in the main exp results are all self-constructed. And closely related baselines such as [1, 2] are not compared.
  • Training loss is reported as one of the main metrics for comparison, which I believe have little to no value in this kind of limited data setting.

[1] Verbockhaven et al., "Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally." TMLR 2024.
[2] Wu et al., "Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks." NeurIPS 2020.

Questions for authors

See weaknesses.

Reviewer_RPPaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an approach, called SKETCH-SCHUR-OST, for function-preserving network expansion based on Schur complement. Compared to GradMax, the proposed method selects the features by taking the network's tangent space into account to project out the redundant features. To incorporate the new features with the existing ones, a dynamic parameter transport approach is used to update the old parameters to ensure no disruption to the network's original output. During training, pre-conditioning based on the Schur complement is used as "an architectural gate" to improve local optimization geometry. To reduce the memory usage of the second-order optimization, the proposed method uses one shared row sketch. Experimental results on BSDS500 shows that SKETCH-SCHUR-OST outperforms sequential and ungated baselines in AUC and MSE.

Critical strengths

The idea of considering the network's tangent space in the feature selection and update rules seems new. Their formulations based on Schur complement are well-motivated and enhanced with an improved complexity implementation by the shared row sketching or compression.

Critical weaknesses

It is not clear to the reader whether the SPD gated update would still have benefits after long training. If I understand it correctly, the gate bakes the second-order information at initialization into training to accelerate convergence and with extended long training the network should converge as good as the ungated one. Given that the gating may have a significant complexity overhead, it seems reasonable to use ungated updates. The proposed method relies on second-order statistics and they can be expensive to compute in large-scale problems even it is only needed in initialization. The technique proposed to address this issue, the shared row sketch, seems to require some heuristics about the data. It is not clear how the best trade-off is achieved and may require many trials and errors. Finally, I find the paper a bit hard to follow due to the disconnected descriptions. For example, in Section 3, it would be easier to follow if there are some descriptions following Theorem 2. On the other hand, Theorem 1 seems trivial and it doesn't seem to be the main result of the paper, instead, it is background knowledge.

Questions for authors
  1. Theorem 1 seems trivial. Is there a more intuitive example than the one in Theorem 2? What are the motivations behind making them theorems?
  2. Does the proposed approach scale with large datasets and architectures? Can it work with transformer models?
  3. For the row sketching, how to determine the best compression rate?
  4. Is there a more principled explanation on why the SPD gated update has diminishing benefits over many training steps?
  5. For the shared sketch, is there a significant approximation error? If the error is small, does it propagate to the updates and affect convergence?
Reviewer_Ekh3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a function-preserving network expansion method that adds new learnable parameters to a pretrained network without altering its original predictions. Existing approaches typically handle candidate selection, redundancy removal, and optimization conditioning as separate sequential steps. (main contribution) This paper unifies these steps using a single Schur-complement metric for conditional selection, ridge transport, and full-matrix SPD gating. To reduce the computational and memory overhead of the expansion procedure, the authors further introduce a shared-sketch approximation and a row-space dual formulation. Empirical results demonstrate that the proposed method performs well against the baselines in terms of optimization performance while reducing memory overhead through sketching.

Critical strengths

(S1) The proposed method is practical. It not only improves optimization efficiency but also reduces memory overhead through sketching, making network expansion more practical.

Critical weaknesses

(W1) This paper is difficult to follow. From this reviewer's perspective, the paper has significant room for improvement in terms of writing quality. For example, the introduction lacks sufficient explanations and definitions, making it difficult to follow the proposed concepts. Additionally, the method section should provide a high-level overview before delving into technical details. Therefore, this reviewer suggests that the authors further polish the manuscript during the revision.

  • (L047) Lack of explanation of spectral bias.
  • (L050, L053) Abrupt introduction of abbreviations (SPD, PINN, PIKAN, etc.).
  • (L110, L139, L199) Abrupt transitions into technical details without sufficient high-level explanations.
  • (Figure 1) The sentence at the bottom of Figure 1, "Sidebands propose support; Schur gain selects; ridge transport and the gate factor one damped metric," is not meaningful without further explanation.
  • (L485) The sentence summarizing the results, "The result is a scoped growth–preconditioning method with explicit sketch limits, not a universal architecture or optimizer claim," disrupts the flow of the manuscript. Moreover, it is the final sentence of the paper.
  • There are many other instances of poor writing quality that this reviewer cannot enumerate exhaustively.

(W2) The experimental scale is limited. The experiments are mainly conducted on MLP-based architectures. This reviewer suggests further evaluating the proposed method on more recent neural network architectures.

Questions for authors

See major weaknesses (W1, W2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.