acceptodds
Under review as a conference paper at ICLR 2027

Self-Partitioning Adaptive Widening Networks

Abstract

The design of neural architectures is commonly treated as a process separate from parameter optimization, which necessitates expensive retraining whenever the architecture is modified and frequently results in over-parameterized, memory-intensive models. To overcome these limitations, we introduce the Self-Partitioning Adaptive Widening Network (SPAWN), a novel framework that jointly learns both its architecture and its parameters by dynamically increasing its representational capacity during training. SPAWN starts from a single shallow predictor and incrementally builds a soft-gated mixture of experts by partitioning the input space where additional complexity is needed. In contrast to prior approaches that expand networks by adding neurons or layers, SPAWN increases capacity by recursively subdividing the input space, yielding a differentiable tree of affine experts trained end-to-end. This expansion process is governed by data-driven criteria that determine when to grow, where to split, and how to preserve the model’s outputs, enabling seamless optimization without retraining from scratch. Empirical evaluations on standard regression and classification benchmarks demonstrate that SPAWN automatically discovers compact architectures that match or surpass the predictive performance of substantially larger, fixed-capacity models, while using markedly fewer parameters.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Supervised Learning, Machine Learning, Growing Network, Adaptive Methods, Neural Architecture, MLP

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eW31OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPAWN (Self-Partitioning Adaptive Widening Network), an adaptive neural network framework that dynamically adjusts its architecture during training. SPAWN organizes affine experts into a soft-gated binary tree, determines when to expand based on training loss plateaus, selects experts for splitting using within-region loss heterogeneity, and employs output-preserving inheritance to maintain the model's predictions during expansion. Experiments across multiple classification and regression benchmarks demonstrate that SPAWN achieves competitive predictive performance with substantially fewer parameters.

Critical strengths
  1. SPAWN combines dynamic network growth with input-space partitioning, adaptively allocating model capacity through data-driven expansion criteria. Output-preserving inheritance allows optimization to continue smoothly after structural changes. The overall framework is well motivated and has practical potential.

  2. The paper evaluates SPAWN on multiple synthetic and real-world datasets, comparing it with MLPs, XGBoost, and existing network-growing methods. Results across multiple random seeds demonstrate the stability of the approach and its promising parameter efficiency.

Critical weaknesses
  1. The paper proposes dynamically partitioning the input space to adaptively grow model capacity. However, prior studies, such as Adaptive Neural Trees (ICML 2019)[1], have also explored dynamically expanding tree structures during training. Although SPAWN introduces distinctive expansion criteria and parameter inheritance mechanisms, the paper lacks direct experimental comparisons with these approaches, making it difficult to fully assess its advantages over existing adaptive tree-based models.

  2. The evaluation of practical computational efficiency remains limited.The paper primarily demonstrates the efficiency of SPAWN through parameter counts and predictive performance. However, the method requires multiple expansion and optimization cycles during training, potentially introducing additional computational overhead. A more comprehensive comparison of end-to-end training time and actual inference efficiency would help substantiate its overall computational efficiency advantages.

[1] Tanno, R., Arulkumaran, K., Alexander, D., Criminisi, A., & Nori, A. (2019). Adaptive Neural Trees. Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR 97, 6166–6175.

Questions for authors
  1. Can the authors further clarify the key advantages of SPAWN over existing methods such as Adaptive Neural Trees and provide more direct experimental comparisons?

  2. Given that SPAWN involves multiple expansion and optimization cycles, can the authors provide a more detailed analysis of its end-to-end training cost and inference efficiency?

Reviewer_meWBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The design of neural network architecture is often treated separately from parameter optimization, which means that whenever the architecture changes, costly retraining is required. The article proposes the Self-Partitioning Adaptive Widening Network (SPAWN), which dynamically increases network capacity during the training process to organically integrate parameter optimization with architecture design. The SPAWN technique improves model capacity by recursively subdividing the input space, thereby constructing a differentiable affine expert tree structure that supports end-to-end training. The effectiveness of this method has been experimentally verified.

Critical strengths
  1. This paper is an interesting work that can simultaneously add model architecture and model parameters during the training process. The Self-Partitioning Adaptive Widening Network autonomously constructs its network structure in a data-driven manner during the training phase.
  2. The experimental analysis is rigorous, the experimental protocol is fair and transparent, ten seeds and all methods share the same paired partition, and each baseline family is subjected to per dataset hyperparameter search
Critical weaknesses
  1. Insufficient motivation for the method: Existing neural network architecture search algorithms can dynamically search for model architecture based on task requirements. And there are already many neural network architectures and parameter generation methods based on large models, which can directly generate model architectures and parameters according to specific requirements, achieving efficient model generation. The method described in this article can only increase the model capacity unidirectionally, which limits its application.
  2. In terms of experimental results, the current methods compared are very traditional, and the datasets are all traditional small datasets that have not been validated on larger datasets. The existing performance improvement cannot directly indicate that adding both model architecture and knowledge has improved model performance. Suggest adding randomly generated model weights with the same architecture for comparative experiments.
  3. In terms of experimental analysis, appropriately increase the time and cost expenses of the method during the evolution process.
Questions for authors

Reference Weaknesses

1 comment

Thank you again for your review. As the discussion period ends soon, we would like to ask whether our response has addressed your concerns. If any point is still unclear, please let us know which one, and we will clarify it. If your concerns have been resolved, we would be grateful if you could consider raising your score.

Reviewer_zyTyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces SPAWN, a framework designed to dynamically enhance representation capacity during the training process by optimizing both network architecture and parameters. SPAWN increases capacity through the recursive subdivision of the input space, thereby constructing a differentiable tree of affine experts. Experiments conducted across multiple datasets validate the effectiveness of this framework.

Critical strengths

S1: The manuscript clearly motivates the proposed approach through a comparison with existing methods that expand networks by adding neurons. S2: Experiments in this manuscript are quite comprehensive, covering numerous datasets across two types of tasks, which provides support for the effectiveness of the proposed framework. S3: The method presented in the manuscript is easy to understand and achieves its objectives through a recursive tree of experts tailored to different datasets.

Critical weaknesses

W1: Figures in this manuscript contain little information, making it difficult to convey the design logic and core theoretical concepts of the proposed framework. W2: Content resembling footnotes appears on multiple pages of the manuscript, raising questions about whether this format is allowed. W3: The design of the methodology and the underlying logic in the manuscript are too simplistic, which limits its level of innovation and novelty.

Questions for authors

Q1: Are there additional experimental and theoretical proofs demonstrating the novelty and effectiveness of the framework? How are these demonstrated? Q2: Were title 2 and 3 generated by AI? What is the logic connecting the various sections?

2 comments

Thank you again for your review. As the discussion period ends soon, we would like to ask whether our response has addressed your concerns. If any point is still unclear, please let us know which one, and we will clarify it. If your concerns have been resolved, we would be grateful if you could consider raising your score.

Reviewer_zyTyOpenReview ↗

Standing by My Score

Thanks for the authors' response. However, I believe the work is not yet of sufficient quality for acceptance at ICLR, and I will stick to my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.