- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Topological Self-Regulation (TSR), a method that redistributes model capacity within a constrained resource budget by adjusting channel widths during training. It evaluates the growth potential of candidate channels—whose output connections are set to zero—using their gradients, and subsequently adds or removes channels based on the importance of existing ones. The paper introduces a framework to consistently manage interdependent parameters, normalization states, and optimizer states throughout this adjustment process. In the contexts of image classification and time-series forecasting, the study compares "online" TSR—as well as models retrained from scratch using the discovered architectures—against dense models, pruned models of equivalent scale, and models with partially randomized channel allocations.
Critical strengths
-
The paper presents a systematic approach to updating dependent parameters, normalization states, and optimizer states when adding or removing channels. This provides a useful foundation for implementing bidirectional structural modifications in complex networks.
-
The paper separately evaluates online performance and the performance of the discovered architectures when retrained from scratch. This helps distinguish whether the performance depends on the structural editing trajectory during training or whether the discovered channel allocation itself provides value.
Critical weaknesses
-
The effects of channel reallocation are not clearly disentangled from those of simple compression. It remains unclear whether the observed improvements arise from the proposed reallocation mechanism or merely from reducing the model size.
-
The appendix is excessively fragmented, making it difficult to follow the paper's central claims and the evidence supporting them.
-
The gap between the theoretical guarantees and the actual implementation, along with the associated limitations, should be explicitly discussed in the main text rather than being left to the appendix.
-
The automatic adjustment mechanism requires a large number of hyperparameters, and it is unclear how robust the method is to their settings. A sensitivity analysis would help establish its practical reliability.
-
The method appears to have limitations involving LayerNorm and attention mechanisms, which restrict its applicability to modern neural network architectures. This substantially limits the generality of the proposed approach.
-
The experimental results are reported for only a single random seed, making it difficult to assess their statistical reliability and reproducibility.
-
The paper lacks comparisons with recent pruning methods, making it difficult to assess the competitiveness of the proposed approach against current alternatives.
-
Comparing methods solely based on the number of training epochs is insufficient, as the computational cost per epoch may differ substantially. A fair comparison should be conducted under matched computational budgets, or at least report actual training time and total computational cost (e.g., FLOPs).
Questions for authors
-
Have the authors compared the proposed method against variants where growth is disabled or growth candidates are selected randomly, while keeping the training schedule and structural editing framework unchanged? How can the additional benefits of bidirectional channel reallocation and gradient-based growth selection be disentangled from the current results?
-
Have the authors measured the total training costs of online TSR and architecture retraining C2 separately? Do the claimed efficiency advantages over pruning methods still hold in terms of actual training time or cumulative FLOPs when the computational overhead of candidate channel evaluation and structural editing is taken into account?
2 comments
We appreciate the authors’ efforts in providing additional experiments and clarifications. While these results address several of our initial concerns, they do not convincingly establish the distinct benefit of gradient-guided growth. In particular, retrained performance remains largely comparable when growth is disabled, and a direct comparison against random growth under the same editing framework has not been provided. Thus, the additional benefit of bidirectional reallocation over budgeted pruning remains unclear. For these reasons, we maintain our original evaluation.
Thank you! Response to Reviewer ARtm: growth ablation, allocation vs compression, cost, sensitivity, and OTOv2 comparison
Thank you for your review. Every analysis you asked for is now run.
Q1: growth off and random growth. Done with the same schedule, budget and framework, over 3 seeds. The table is in the general response.
Q2: cost. Measured separately, as a multiple of the dense model:
W2, appendix. We respectfully see this differently. The appendix opens with a table of contents, and every section is cited from the main-text claim it supports. It holds proofs and per-cell detail, and each claim and its evidence are stated in the main text.
W3, theory gap. The main text already states it, in Secs. 1, 4, 5 and 9.
W4, sensitivity. The method is robust to its settings. We changed one setting at a time from the published values (H~=1, k=4, K=100, α=0.5) on ResNet-18 / CIFAR-100. Similarity is the cosine between each run's widths and the centre run's.
W5, LayerNorm. This is a limitation stated in the paper, and it is a theorem, not a bug: a new channel under LayerNorm cannot stay silent. dmodel is one width shared by every block, so it sets overall size, not allocation. The per-block FFN widths (21 to 32 of each block's weights) are editable.
W6, seeds. The submission had repeats: MobileNetV2 Ref and C2, and the full Weather pipeline. New: discovery rerun, 3 seeds per arm.
W7, recent methods. The paper already had 5 pruning baselines, including DepGraph. New: OTOv2 (ICLR 2023) at the same count, 3 seeds:
The three are level within noise. Retrained, TSR's widths do at least as well.