acceptodds
Under review as a conference paper at ICLR 2027

Topological Self-Regulation in Neural Networks

Abstract

A neural network learns how to use its channels during training, but where those channels sit is decided before the first gradient is taken, by convention rather than by the task. One part of a network can be starved of channels while another carries channels it barely uses. We ask whether training itself can revise this allocation. Topological Self-Regulation (TSR) treats channel width as a budgeted, revisable part of the model: silent candidate channels are attached throughout the network, the ordinary backward pass reveals which of them would be worth introducing and which incumbents contribute least, and capacity is transferred between them through edits that keep every dependent tensor consistent. A coupling calculus makes such edits valid across residual trunks, depthwise stacks, normalization state and optimizer moments, with exact accounting of the parameters they move. Across four image-classification families and four forecasting backbones, the discovered architectures match the dense reference at roughly fifteen percent fewer parameters in vision and at reductions reaching well over half the network in forecasting. Where the reduction is large, the task-directed allocation outperforms a random one of the same size, and pruning the same reference to the same count shows how much of that is reallocation rather than removal.

Primary area:
optimization
Keywords:
structural pruning, parameter-efficient training, optimization, neural architecture search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ARtmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Topological Self-Regulation (TSR), a method that redistributes model capacity within a constrained resource budget by adjusting channel widths during training. It evaluates the growth potential of candidate channels—whose output connections are set to zero—using their gradients, and subsequently adds or removes channels based on the importance of existing ones. The paper introduces a framework to consistently manage interdependent parameters, normalization states, and optimizer states throughout this adjustment process. In the contexts of image classification and time-series forecasting, the study compares "online" TSR—as well as models retrained from scratch using the discovered architectures—against dense models, pruned models of equivalent scale, and models with partially randomized channel allocations.

Critical strengths
  1. The paper presents a systematic approach to updating dependent parameters, normalization states, and optimizer states when adding or removing channels. This provides a useful foundation for implementing bidirectional structural modifications in complex networks.

  2. The paper separately evaluates online performance and the performance of the discovered architectures when retrained from scratch. This helps distinguish whether the performance depends on the structural editing trajectory during training or whether the discovered channel allocation itself provides value.

Critical weaknesses
  1. The effects of channel reallocation are not clearly disentangled from those of simple compression. It remains unclear whether the observed improvements arise from the proposed reallocation mechanism or merely from reducing the model size.

  2. The appendix is excessively fragmented, making it difficult to follow the paper's central claims and the evidence supporting them.

  3. The gap between the theoretical guarantees and the actual implementation, along with the associated limitations, should be explicitly discussed in the main text rather than being left to the appendix.

  4. The automatic adjustment mechanism requires a large number of hyperparameters, and it is unclear how robust the method is to their settings. A sensitivity analysis would help establish its practical reliability.

  5. The method appears to have limitations involving LayerNorm and attention mechanisms, which restrict its applicability to modern neural network architectures. This substantially limits the generality of the proposed approach.

  6. The experimental results are reported for only a single random seed, making it difficult to assess their statistical reliability and reproducibility.

  7. The paper lacks comparisons with recent pruning methods, making it difficult to assess the competitiveness of the proposed approach against current alternatives.

  8. Comparing methods solely based on the number of training epochs is insufficient, as the computational cost per epoch may differ substantially. A fair comparison should be conducted under matched computational budgets, or at least report actual training time and total computational cost (e.g., FLOPs).

Questions for authors
  1. Have the authors compared the proposed method against variants where growth is disabled or growth candidates are selected randomly, while keeping the training schedule and structural editing framework unchanged? How can the additional benefits of bidirectional channel reallocation and gradient-based growth selection be disentangled from the current results?

  2. Have the authors measured the total training costs of online TSR and architecture retraining C2 separately? Do the claimed efficiency advantages over pruning methods still hold in terms of actual training time or cumulative FLOPs when the computational overhead of candidate channel evaluation and structural editing is taken into account?

2 comments
AuthorsOpenReview ↗

Thank you! Response to Reviewer ARtm: growth ablation, allocation vs compression, cost, sensitivity, and OTOv2 comparison

Thank you for your review. Every analysis you asked for is now run.

Q1: growth off and random growth. Done with the same schedule, budget and framework, over 3 seeds. The table is in the general response.

  • Growth. We now present growth as optional. This does not weaken the method: the allocation retrains just as well with growth off, growth still helps the online ResNet-18 model (+0.37+0.37, 3/3), and dropping it removes the candidate bank, which makes TSR cheaper (Q2).
  • It is not compression. At the same size, a different allocation gives a different accuracy:
    • The submission already separates the two: matched-count pruning includes a from-scratch arm, and forecasting has a random-width control.
    • A ResNet-18 mis-allocated at full size loses 2.6 points.
    • At 85%, growth-off TSR retrains to −0.22-0.22 / −0.19-0.19 from dense. ℓ1\ell_1 pruning from scratch reaches −1.48-1.48 / −0.37-0.37.
    • In forecasting it beats random widths in 8 of 8 cells.

Q2: cost. Measured separately, as a multiple of the dense model:

FLOPsWall-clock (T4), ResNet-18 / VGG
Online TSR0.90–1.221.60 / 1.52
Prune + fine-tune1.23–1.351.39 / 1.39
C2 (retrain)1.70–2.70
TSR without the bank1.14 / 1.17
OTOv21.17 / 1.24
  • In FLOPs, online TSR beats fine-tuned pruning in all 6 measured cells.
  • Wall-clock is slower due to kernel tiling from the 4 dormant candidate channels, not the controller's computation: they make every width irregular, and 64→6864 \to 68 channels costs +67%+67\% time for +13%+13\% FLOPs.
  • With growth optional, the bank becomes dispensable, and TSR is then the cheapest. On MobileNetV2 that costs 0.66 points.
  • We drop the epoch-count framing. If a trained dense model already exists, fine-tuned pruning is cheaper.

W2, appendix. We respectfully see this differently. The appendix opens with a table of contents, and every section is cited from the main-text claim it supports. It holds proofs and per-cell detail, and each claim and its evidence are stated in the main text.

W3, theory gap. The main text already states it, in Secs. 1, 4, 5 and 9.

W4, sensitivity. The method is robust to its settings. We changed one setting at a time from the published values (H~=1\tilde{H} = 1, k=4k = 4, K=100K = 100, α=0.5\alpha = 0.5) on ResNet-18 / CIFAR-100. Similarity is the cosine between each run's widths and the centre run's.

SettingTop-1Δ\DeltaBirthsSimilarity
centre77.402611.00
H~=0.1\tilde{H} = 0.177.28−0.12-0.122500.94
H~=10\tilde{H} = 1077.62+0.22+0.221990.98
k=2k = 277.59+0.19+0.192650.99
k=8k = 877.46+0.06+0.062571.00
K=50K = 5077.55+0.15+0.156111.00
K=200K = 20077.55+0.15+0.1500.30
α=0.3\alpha = 0.377.31−0.09-0.092710.99
α=0.7\alpha = 0.777.44+0.04+0.042230.98
  • Every change stays within −0.12-0.12 to +0.22+0.22, the size of the dense model's own seed noise (0.20).
  • Seven of the eight give the same allocation.
  • At K=200K = 200 no growth cleared its cost, so the run was removal-only, at the same accuracy.

W5, LayerNorm. This is a limitation stated in the paper, and it is a theorem, not a bug: a new channel under LayerNorm cannot stay silent. dmodeld_{\text{model}} is one width shared by every block, so it sets overall size, not allocation. The per-block FFN widths (12\tfrac{1}{2} to 23\tfrac{2}{3} of each block's weights) are editable.

W6, seeds. The submission had repeats: MobileNetV2 Ref and C2, and the full Weather pipeline. New: discovery rerun, 3 seeds per arm.

W7, recent methods. The paper already had 5 pruning baselines, including DepGraph. New: OTOv2 (ICLR 2023) at the same count, 3 seeds:

TSRTSR without the bankOTOv2
ResNet-1877.5577.3577.20
VGG-16-BN73.2773.4673.18

The three are level within noise. Retrained, TSR's widths do at least as well.

We appreciate the authors’ efforts in providing additional experiments and clarifications. While these results address several of our initial concerns, they do not convincingly establish the distinct benefit of gradient-guided growth. In particular, retrained performance remains largely comparable when growth is disabled, and a direct comparison against random growth under the same editing framework has not been provided. Thus, the additional benefit of bidirectional reallocation over budgeted pruning remains unclear. For these reasons, we maintain our original evaluation.

Reviewer_pqTdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This draft proposes a new Neural Architecture Search (NAS) method, which applies on the fly, the first T runs, to update the architecture instead of an off-line setting.

The basic idea of the method is: When to remove existing channels based on first Taylor expansion and the parameter cost, Eq(6), which is widely used in model pruning(Molchanov et al., 2017, it is easy to google more). When increasing capacity (channels) based on Eq(4), a set of channels is added to each layer, but the output is zeroed(dormant). Then compute the gradient with respect to those dormant channels, the magnitude of the channels would be used as a criteria, lines 171-181. This method attempts to apply NAS while training, thus a time window is used to track the criteria to modify the architecture for the first T steps.

Generally the proposed methods are already widely used in previous pruning and NAS studies, the writing of the current draft requires improvement, the references seem to be a little out-dated.

Critical strengths
  1. Searching the best to modify a network architecture has a large impact in designing efficient models for edge devices.
Critical weaknesses

While discussing the weaknesses of this draft, I will also leave my questions accordingly in this section.

  1. (Writing), The writing of this draft is very confusing. The terms used seem to be arbitrarily defined or without introduction. First sentence in Section 3, “the coupling groups the calculus recovers, the tests that decide which of them may be edited, the two signals read at a dormant candidate and at an incumbent”. What are the calculus recovers? Which tests? Two two signals, what are they? It is highly recommended to define each term in advance.

  2. (Clarity problem) Line 105, “Let S index channel groups that..” is it better to introduce those channels are grouped into M groups? What is “admissible width”? The total number of channels from the original setup or after modification?

  3. (Clarity problem) Eq(1), C is the number of deployed parameters, what is B? I can only see it is in the Figure caption, budget?

  4. (Clarity problem) line 137, “where mb handles layouts”, from Figure 1, there exist M channels(correct me if I am wrong)? What is the b index?

  5. (Clarity problem), line 152, Condition (N) and Condition (R) ? Are those terms commonly used in the literature or are they only used in this draft without explanation. From the context, I can guess they are criteria when adding or removing channels, please correct me if I am wrong. The added channels can only affect LayerNorm or GroupNorm. And the added channels cannot change those hard-coded architectures(num of heads or head dimensions in a transformer block).

  6. (Clarity problem), Section 4, “FROM SENSITIVITY TO REALLOCATION”, again, I cannot find a clear definition for the term “sensitivity” in the context, is it the gradient? Line 166, “port coordinate s”, is this a newly invented name? What is the Us?

  7. (Writing), the whole section is trying to introduce the methodology, but it looks more like an engineering diagram. Figure 2 is not very informative, simply showing how networks are modified in common.Lines 257-260 seems to be a duplicate to paragraph lines 153-157. The following paragraph is more like implementation details of the size of the newly added channels and window-averaged measures.

  8. (Motivation), This is the most important weakness for this draft. The proposed method is not novel. The taylor-based removal has been widely used in model pruning and NAS. The gradient of dormant layers are also previously used in other NAS studies(Evci et al., ICLR 2022). The only difference is that this study intends to apply the unified budget for both adding and deleting. Similar design has also been used in previous studies [1], which is not cited in this draft.

  9. (Experiment Setup): Most of the references in this draft are from 2024 or earlier, no recent studies are discussed or compared to this draft. This is also a key factor that the proposed method may not be validated comprehensively.

[1] Świderski, Szymon, and Agnieszka Jastrzębska. "Data classification with dynamically growing and shrinking neural networks." Journal of Computational Science 91 (2025): 102660.

Questions for authors

Please see the above section.

2 comments
AuthorsOpenReview ↗

Thank you. Response to Reviewer pqTd: scope of TSR, edit schedule, contribution beyond known saliency signals, and notation

Thank you for your review. We would firstly like to point out four corrections.

  • TSR is not NAS. The skeleton is fixed. Only channel counts change, under a parameter budget.
  • It is not limited to the first TT steps. Edits run every 100 steps for the whole run; the last is at step 38,900 (Fig. 2c). Only the budget ceiling is annealed early.
  • Condition (N) works the other way. It forbids editing axes that LayerNorm or GroupNorm normalize, because a new channel would shift their statistics. Your reading of (R) is right.
  • Fig. 2 is real data: one measured run at four moments.

Novelty. We agree Taylor saliency and GradMax-style growth signals are known. The paper cites both and does not claim them. The contribution is:

  1. A coupling calculus. Exact edits in both directions across residual, depthwise, normalization and optimizer state. DepGraph covers removal only.
  2. A budgeted controller. Its allocation is stable (cosine 0.97–0.99) and matches OTOv2.
  3. A protocol. Retraining from scratch and random widths separate the allocation from the training path.

[1] is now cited, thank you. It decides edits by Monte Carlo tree search; TSR reads them from the gradient under one budget.

Recent work. We added OTOv2 (and ran it), Isomorphic Pruning (ECCV 2024), Yuan et al. (NeurIPS 2023) and [1].

Clarity. Most of these terms are defined where they are used:

  • Section 3's opening is a roadmap. Fig. 1's caption, beside it, names the groups, the tests and the two signals.
  • Conditions (N) and (R) are defined in the sentences right after they are introduced.
  • Sensitivity is the loss gradient at a channel's outgoing weights, its port (Sec. 1). Eq. 4, the first equation of Sec. 4, formalizes it.
  • The port and Us\mathcal{U}_s are defined in the sentence that uses them. Us\mathcal{U}_s is every place one port weight is used.
  • bb, ρb\rho_b and mbm_b are defined where they appear. bb is each place a group's index appears (layer input or output, norm buffer, optimizer state). mb>1m_b > 1 when one channel spans several positions. MM in Fig. 1 is the number of groups.

The revision fixes two points from your review: BB, the budget (a parameter count), is now named right after Eq. 1. Admissible widths get a sentence: S\mathcal{S} indexes the editable groups, nn holds their current widths, and N\mathcal{N} is the set of allowed width vectors. The second (N)/(R) paragraph is removed.

Reviewer_pqTdOpenReview ↗

The answers did not resolve the questions.

  1. "TSR is not NAS.", I consider NAS a general concept where one has computation budget(say FLOPs) then within that budget how one can design the best architectures. The main point is how to design such a budget-freidnt architecture, REGARDLESS if we need to drop the rest off-line or we can keep the weights. OTHERWISE, each "NAS" paper can be pulished again in the filed of "NOT REMOVING THE WEIGHT ONLY CHANGING THE COUNTING" as a new study. If ONLY changing the counting is a new topic, then it better shows convincing proof why this is a new topic instead of the well-established NAS or model pruning.
  2. "It is not limited to the first T steps." Then this might be a hyper parameters choice, thanks for the explanation.
  3. "Condition (N) works the other way." I suppose this is for my Q5, yes, this matches my impression but I didnot phrase it clearly, channel change affects those statistics.
  4. "Fig. 2 is real data", this is for Q7 I suppose? Yes, no doubt it could be for the real implementation how many params are "ignored", but this engineering implmentation details are not that necessary or informative, but this is not a critical point.
  5. "Novelty": The answers are more like repeating the writing without explanations. It adds "controller", "protocol" which then directly pops the claims. " Retraining from scratch and random widths separate the allocation from the training path." I was expecting a highlighted point here, other related works cannot do it from scratch or cannot use random widths? This type of sentences are not clear answers.
  6. "Clarity": "Most of these terms are defined where they are used", is it better to point which answers are for which questions for a better CLARITY? a) "Conditions (N) and (R) are defined", I already referenced the lines in my question, line 152 "Condition (N) excludes", this directly jumps into the paragraph without any context, this is the clarity problem raised above. b) "Sensitivity is the loss gradient", could you point to the line where it is defined in the draft?

Overall these answers did not clear the main motivation issue, and the references seem to be out-dated.

Reviewer_JeBmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Topological Self Regulation (TSR), a framework for reallocating channel widths during training under a resource budget. It attaches candidate channels with zero outgoing connections, preserving the current function while allowing outgoing gradients to indicate potential growth opportunities. These signals are combined with incumbent channel removal scores to propose structural edits. The authors also distinguish their implemented heuristic from a calibrated reference variant with conditional loss decrease guarantees. Their results suggest that compact allocationcs can retain much of the reference performance, although the advantage over alternative allocation methods is less clearly established.

Critical strengths
  1. The paper adresses practical issues that are often overlooked in network growth and pruning.
  2. The authors explicitly state that the descent certificate applies to a calibrated variant, not to the released heuristic.
  3. Comparing the online model with the same exported widths trained from scratch helps distinguish the value of the discovered allocation from inherited weights and the editing trajectory.
Critical weaknesses

None

Questions for authors

Do the seed repetitions repeat the entire discovery process or only retrain a fixed exported architecture?

1 comment
AuthorsOpenReview ↗

Thank you. Response to Reviewer JeBm: discovery rerun across seeds, allocation advantage, and growth

Thank you for your review!

Do the seed repeats rerun discovery? In the submission, vision did not (it retrained fixed widths) and forecasting did. The paper now says so.

New: discovery rerun with 3 seeds.

Similarity across seedsRetrained (C2)Dense
ResNet-180.99377.4277.63
VGG-16-BN0.99473.1873.46
MobileNetV20.96867.5968.08
  • The same layers change in every seed. For example, VGG widens its first convolution 2.62.6–3.8×3.8\times and shrinks the same five late ones.
  • The allocation is stable across hyperparameters too (≥0.94\geq 0.94 in 7 of 8 settings).

Advantage over other allocations. It is clear against ℓ1\ell_1 pruning (+0.26+0.26, 7 of 12 cells) and random widths (8 of 8). Against OTOv2 it is level, and we now claim parity there.

Growth. With growth off, the allocation retrains just as well, so we credit the budgeted controller and now present growth as optional.

Comments on the paper

1 comment

We thank all reviewers for their reviews. A revised version of the manuscript is uploaded with the presentation fixes. We conducted some new experiments and ablations, and summarize their results below:

  • Stability. We reran discovery with 3 seeds on ResNet-18, VGG-16-BN and MobileNetV2. The discovered widths agree across seeds (cosine 0.99 / 0.99 / 0.97).
  • Robustness. Eight hyperparameter changes move accuracy by −0.12-0.12 to +0.22+0.22, about the dense model's own seed noise.
  • Competitiveness. At the same parameter count, TSR is level with OTOv2 (ICLR 2023): +0.34+0.34 / +0.10+0.10. Retrained from scratch, our widths do at least as well.
  • Significance of Allocation. A ResNet-18 mis-allocated at full size loses 2.6 points. TSR's allocation beats ℓ1\ell_1 pruning retrained from scratch (+0.26+0.26 on average, 7 of 12 cells).

Growth

We ran growth ablations with 3 seeds. They demonstrate that the contribution is not significant, so we now present growth as optional. Growth on minus growth off, last-epoch top-1:

OnlineRetrained (C2)
ResNet-18+0.37+0.37 (3/3)+0.01+0.01
VGG-16-BN0.000.00−0.10-0.10
MobileNetV2+0.05+0.05+0.36+0.36

We would like to clarify that this does not weaken the method:

  • The allocation's value comes from the budgeted controller. With growth off, it retrains just as well.
  • Growth still improves the online ResNet-18 model (+0.37+0.37, 3/3).
  • Without growth, TSR drops its candidate bank and becomes cheaper (see Cost).

Cost

  • In FLOPs, online TSR is cheaper than prune + fine-tune: 0.900.90–1.22×1.22\times vs 1.231.23–1.35×1.35\times the dense model.
  • Wall-clock is slower (1.521.52–1.60×1.60\times vs 1.39×1.39\times) due to kernel tiling from the 4 dormant candidate channels at each layer, not the controller's computation: going from 64 to 68 channels costs +67%+67\% time for +13%+13\% FLOPs.
  • With growth optional, the bank becomes dispensable. TSR then costs 1.141.14–1.17×1.17\times, below fine-tuned pruning and OTOv2 (1.171.17–1.24×1.24\times).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.