acceptodds
Under review as a conference paper at ICLR 2027

Normalized Factorized Weight Generation for Nested Elastic Convolutional Networks

Abstract

How should a weight generator be parameterised so that one stored representation supports nested neural networks without making shared weights depend on the selected width or depth? We propose Spectrum-Shared Generation (SSG), a normalized factorized parameterisation for elastic convolutional networks. Each residual kernel is a CP map with layer-specific directions and network-wide component magnitudes. Normalising complete factor columns before channel selection makes every narrower generated kernel an exact prefix sub-tensor of wider ones; fixed layer identities similarly preserve the generated tensor assigned to a retained block under depth selection. The parameterisation is invariant to radial factor rescaling and admits explicit local partial-Jacobian bounds, while configuration-specific normalisers handle activation statistics outside the generator. In controlled comparisons, SSG reduces persistent learnable storage by – across CIFAR-10, CIFAR-100, and Tiny-ImageNet, with mean accuracy differences of – points from dense elastic supernets. On CIFAR-100, one 2.084M model serves eight configurations spanning in compute and reaches 76.67% at full capacity, 1.14 points above an OFA baseline retrained under our protocol while using less persistent learnable storage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Dynamic neural networks,Factorized Weight Generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_DtgxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Spectrum-Shared Generation (SSG), a normalized factorized weight generation method for nested elastic convolutional networks. The method uses CP factorization with layer-specific directions and a network-wide shared spectrum to generate convolutional weights. Through full-column normalization and fixed layer identities, SSG supports nested width and depth configurations while reducing persistent learnable parameter storage. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate substantial storage reduction with relatively small accuracy differences compared with dense elastic supernets.

Critical strengths
  1. The paper addresses a meaningful problem in elastic neural networks, namely reducing the storage cost of a single model that supports multiple network configurations. The motivation and overall framework are clearly presented, and the explanations of the proposed method are detailed. The authors also provide mathematical analyses to explain the structural properties of the proposed parameterization, which makes the method more understandable and convincing.
  2. The authors provide comparisons under different storage and computation budgets, together with detailed ablation studies to examine the contributions of different components. The supplementary material also includes analyses of different factorization strategies, rank settings, unseen configurations, and robustness under distribution shifts. These experiments provide useful evidence for the effectiveness and practical value of the proposed method.
Critical weaknesses

The authors report that SSG achieves 76.67% accuracy on CIFAR-100, outperforming their retrained OFA baseline by 1.14 points. However, Appendix M mentions some implementation differences between the main paper run and the controlled implementation. In the controlled implementation, SSG achieves 75.55% accuracy at the largest configuration, compared with 76.67% in the main paper run. Although the authors acknowledge these differences, it remains unclear how they affect the reported performance advantage over OFA. A clearer explanation would help make the experimental conclusions more convincing.

Questions for authors
  1. Could the authors further explain the implementation differences between the main paper run and the controlled implementation in Appendix M, and whether these differences affect the reported improvement over OFA?
  2. Are the best accuracies of the eight sub-networks in Table 1 obtained from the same shared checkpoint?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Dtgx

We thank the reviewer for the positive assessment; both questions are important. All numbers are final-checkpoint, 3-seed means.

W1/Q1: Do the implementation differences in Appendix M affect the improvement over OFA? To answer this, we trained OFA in the same code base, recipe and configuration space as SSG (keeping progressive shrinking, channel reorganisation and shared BN with recalibration):

MethodStorageMeanLargest
OFA11.22M73.12±0.2677.50±0.27
SSG2.08M72.33±0.2575.59±0.35
OFA, width 0.43 (equal storage)2.09M65.98±0.1672.70±0.33

SSG comes within 0.79 mean points of OFA with 5.4× less storage, and at equal storage it is 6.35 points higher.

Regarding Appendix M: the 76.67 of SSG comes from an earlier version of our training code, while the controlled experiments use the current code. Both are the same model with the same main settings; they differ in some code details (e.g., the distillation temperature) and in reporting (the original table takes each sub-network's best epoch). Re-training with the current code under the earlier settings gives 75.66±0.09 at the largest configuration. The original OFA row was likewise a single run, from a different training code base whose recipe did not fully match ours. The reviewer's concern is therefore valid: in the unified setting, the "+1.14 over OFA" claim does not hold. The revision replaces Table 1 with the same-code, multi-seed results above and makes the equal-storage comparison the headline.

Q2: Are the eight best accuracies in Table 1 from the same checkpoint? No; the original table reports each sub-network's best value over 200 epochs. We agree that a single checkpoint is the right protocol for a model serving several configurations. Using epoch 200, the mean of every method in the main table drops by only 0.10–0.16 and no comparison changes. All numbers in this rebuttal use the final checkpoint, the revision will do the same, and we will remove the inaccurate "within 0.01" statement from the appendix.

Dear Reviewer Dtgx,

Thank you again for your careful reading. Your question about the experimental protocol led to an important correction, and we are grateful for it. Under a single code base with three seeds and the final checkpoint, SSG is within 0.79 mean points of OFA while using 5.4× less storage, and 6.35 points higher at equal storage.

As the discussion deadline approaches, we would be happy to clarify anything further. Thank you for your time.

Reviewer_wvpaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Spectrum-Shared Generation (SSG), a parameterization for elastic convolutional neural networks based on normalized CP factorization. The method represents convolutional kernels using layer-specific factor directions and a shared set of component magnitudes. The authors argue that this construction enables exact prefix nesting across network widths and preserves the generated weights of retained layers when network depth changes. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet report approximately 5.0–6.1 X reductions in persistent learnable parameter storage compared with dense elastic supernets, with relatively small differences in mean accuracy.

Critical strengths
  1. Supporting multiple network configurations without storing a separate dense model for each configuration is important for resource-constrained deployment.
  2. Normalizing complete factor columns before channel slicing provides a straightforward mechanism for preserving exact prefix nesting of generated convolutional kernels.
Critical weaknesses

The title and overall framing risk overstating the novelty of the work. Elastic and nested neural networks are well-established research directions, as are low-rank convolutional compression, CP decomposition, and normalized factor parameterizations. The paper itself discusses these established lines of research, including slimmable networks, Once-for-All, and conventional tensor decomposition methods.

The main technical contribution appears to be the combination of normalized CP factors, shared component magnitudes, and fixed layer identities to preserve width nesting and depth-dependent weight identity. Although this combination is potentially useful, the paper does not convincingly establish that it represents a sufficiently substantial advance over applying conventional low-rank factorization to an elastic supernet.

Questions for authors
  1. What is the fundamental methodological novelty of SSG beyond established CP decomposition and low-rank factorization techniques?

  2. Given that Table 2 shows negligible differences between shared and per-layer spectra, what evidence demonstrates that spectrum sharing provides a meaningful benefit beyond marginal parameter savings?

  3. Why is there no controlled comparison against a per-layer normalized CP-factorized elastic network without spectrum sharing to isolate the proposed method's actual contribution?

  4. How would the reported compression gains change on modern architectures dominated by 1X1 convolutions, given that these layers remain largely uncompressed in the main experiments?

  5. Without evaluation on ImageNet or diverse modern backbones, what evidence supports the generalizability of the proposed approach beyond the relatively small-scale experimental settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wvpa

We thank the reviewer. The novelty concern led us to add the most direct controls. All numbers are final-checkpoint, 3-seed means unless noted.

W1/W2/Q1: Novelty beyond CP decomposition and low-rank factorisation. We agree that elastic networks, low-rank convolutions, CP decomposition and normalised factors are each established. Our contribution targets a problem none of them solves: making an elastic network's persistent storage smaller than its largest sub-network while keeping the nesting and weight identity that elastic inference relies on. Existing elastic methods store the largest sub-network, and applying conventional low rank to an elastic supernet is clearly worse (same code and objective):

Method (~2.09M)MeanLargest
SSG72.33±0.2575.59±0.35
Conventional CP elastic net (no normalisation, per-layer spectrum)71.65±0.1874.98±0.27
Post-hoc CP of a trained dense supernet + fine-tuning (1 seed)71.3774.47
Dense supernet narrowed to equal storage64.87±0.0872.64±0.05
OFA narrowed to equal storage65.98±0.1672.70±0.33

Other generators break nesting or collapse (Tensor Train 64.33, coordinate-based channel axes 49.44). The revision will delineate this contribution and its difference from conventional low rank more clearly in the introduction.

Q2: What does spectrum sharing contribute? Untying alone differs by 0.11 over 3 seeds, within noise, as the paper notes; the role of sharing is not parameter savings. Together with column normalisation it defines the parameterisation: normalisation separates atom magnitudes from directions, and sharing makes all layers use one set of magnitudes while each layer learns directions. Untying lets layer-wise magnitudes drift (std 0.05 → 0.23 decades), and removing both (conventional CP) costs 0.68. Trained dense layers have very similar normalised spectra (one mean spectrum explains 82.6% of the log-spectrum variance), an empirical basis for sharing one set of magnitudes.

Q3: Controlled per-layer normalised CP without sharing. This is the "Spectrum per layer" row of Table 2, whose name was unclear. We extended it to 3 seeds (72.22±0.27 vs 72.33±0.25), added the conventional CP and post-hoc baselines (table above), and will rename it "Per-layer normalized CP (no spectrum sharing)".

Q4: Architectures dominated by 1×1 convolutions. SSG's compression comes from the k×k spatial mode; on 1×1 kernels the generator reduces to matrix low rank. On ResNet-50 (bottleneck), 3×3 kernels are ~48% of convolution parameters, so generating only 3×3 gives 1.6×, and generating 1×1 as well does not beat plain width scaling. ResNet-50 is also highly width-redundant on CIFAR-100: width 0.70 costs the dense supernet only 1.33 mean points, versus 7.97 for ResNet-18 at 18% storage. We will scope our claims to k×k-dominated CNNs and report ImageNet-scale analytic storage ratios: about 1.6–4.9× for ResNet-50 and 1.0–1.8× for MobileNetV2, depending on whether and at which rank 1×1 layers are generated.

Q5: Generalisability without ImageNet. Within the claimed scope, the equal-storage gain holds on three datasets: +2.86 (CIFAR-10), +7.46 (CIFAR-100) and +8.90 (Tiny-ImageNet), growing with task difficulty, while the equal-compute gap to the dense supernet stays at 0.51–1.05. ResNet-50 marks the boundary of this scope. Eight-configuration elastic training on ImageNet-1K is not feasible within the discussion period, so we do not promise it.

Dear Reviewer wvpa,

Thank you again for your review. As the discussion deadline approaches, and understanding that this is a busy time, we just wanted to check whether our response has addressed your concerns.

Your comments on novelty led us to add the direct comparisons they called for: under the same code and ~2.09M budget, SSG reaches 72.33±0.25, compared with 71.65±0.18 for a conventional CP elastic network and 71.37 (single seed) for post-hoc CP. We have also extended the per-layer normalized CP comparison to three seeds.

We would be happy to discuss any remaining questions. Thank you for your time.

Reviewer_rSzzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose Spectrum-Shared Generation (SSG), a low-footprint, factorized parameterization for elastic CNNs, whose parameters and associated generator jointly support a family of models over varying network depths and widths. In contrast with most other recent weight generation approaches, SSG's generator is an analytic multilayer map rather than a black-box, and also maintains high-dimensional direction vectors on a per-layer basis to maintain high layer diversity.

SSG consistenty reduces the footprint of weight storage by 5x+ with minimal accuracy degradation on CIFAR-10/100 and Tiny Imagenet, and outperforms and OFA base on CIFAR-100 (Table 1, Figure 2), with depth combinations and per-stage width mixes densifying the Pareto front on both datasets (Figure 3).

Critical strengths
  • The paper addresses the problem of efficiently representing elastic CNN families with an analytic multilayer map, and demonstrates significant storage savings (5X+) with minimal degradation.
  • More generally, weight generators are an important research topic, considering the enormous size of SOTA models.
  • The paper is generally well written, and the experiments demonstrate that the approach is effective (Figure 2).
  • It is impressive and great to see that sandwich sampling during training is effective, particularly for variable depth (Figure 3).
Critical weaknesses
  • The manuscript references predominantly older work, and investigates the approach only for CNNs, and so may be of generally lower interest and significance.
  • As acknowledged by the authors, training the family is more expensive (line 428). This should be discussed and quantified. Ideally it would be great to make more efficient training of SSGs another contribution of the paper.
Questions for authors
  • Why was OFA chosen as your baseline as opposed to US-Net or Mutual-Net, which are stronger on CIFAR-100?
  • In table 1, can you clarify what the difference between "paper run" and R=256 is? Thanks.
  • FYI Line 187 "Equation equation"
2 comments
AuthorsOpenReview ↗

Response to Reviewer rSzz

We thank the reviewer for the positive assessment, in particular regarding sandwich training for elastic depth. All numbers are final-checkpoint, 3-seed means unless noted.

W1: Older references; CNNs only. We will add recent nested/elastic models, including MatFormer, SortedNet, Flextron and Matryoshka Representation Learning. SSG exploits the three-mode tensor structure of convolution kernels and its gain comes from the k×k spatial mode (General Response, point 4), so we scope the paper to CNNs. Without the spatial mode the same rule applies to linear layers (normalised low rank with shared magnitudes; full-column normalisation still guarantees nesting), which we list as future work.

W2: Quantify the cost of training the family. The cost comes mainly from elastic training itself (several configurations per step), which the dense supernet and OFA share; generation adds about 5% per step with equal memory (0.100 vs 0.094 s, 4.69 vs 4.70 GB). Following the suggestion, we tested cheaper training that updates the largest configuration plus k random others per step (the sampled runs use 1 seed):

Configurations per stepWall-clock (200 epochs)MeanLargest
8 (paper)1.18 h72.3375.59
40.78 h (−34%)72.3475.86
20.56 h (−53%)71.6376.31

Four configurations per step save about a third of the training time at no accuracy cost; we will add this and discuss the budget–accuracy trade-off.

Q1: Why OFA rather than US-Net or MutualNet? Our configuration space varies both width and depth, and OFA is the representative method supporting both; US-Net varies width only, and MutualNet adds input resolution. Like all existing elastic methods, they store the largest sub-network (about 11.2M for ResNet-18). Our "dense elastic supernet" is exactly the slimmable/US-Net construction (switchable BN, channel slicing, in-place distillation) extended to the same depth set and trained in the same code as SSG. At equal storage (~2.1M) it reaches 64.87 mean, OFA 65.98 and SSG 72.33 (General Response, point 1).

Q2: "paper run" vs. "R=256" in Table 1. Both are the same model (CP ResNet-18, R=256, 2.084M) trained with the same main settings (optimizer, learning rate, batch size, epochs, weight decay, distillation weight). "Paper run" comes from an earlier version of our training code and reports each sub-network's best epoch; "R=256" comes from the current code used for the controlled experiments in the appendix. The two code versions differ in some details (e.g., the distillation temperature). Re-training with the current code under the earlier settings gives 72.71±0.06 mean and 75.66±0.09 largest. To avoid mixing two sources, the revised Table 1 uses only the current code, 3 seeds and the final checkpoint.

Line 187 "Equation equation". Thank you; this came from the \eqref macro in the template and is fixed throughout.

Dear Reviewer rSzz,

Thank you again for your encouraging review. Your suggestion about training cost was very helpful: we found that training the largest plus three randomly sampled configurations per step reduces wall-clock time by about a third at the same mean accuracy. We have also clarified the difference between "paper run" and "R=256" in our response.

As the discussion deadline approaches, please let us know if anything remains unclear; we would be glad to explain further. Thank you for your time.

Reviewer_hrjJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spectrum-Shared Generation (SSG), a normalized factorized weight-generation framework for nested elastic convolutional networks. The central idea is to replace explicitly stored dense convolutional kernels with a compact CP-factorized representation that combines layer-specific directional factors and a network-wide shared magnitude spectrum. By normalizing complete factor columns before channel selection, the proposed framework guarantees exact prefix nesting across different network widths, while fixed layer identities preserve generated weights under depth selection. The authors also introduce configuration-specific normalization and classifier heads to accommodate different activation statistics. Several theoretical properties, including prefix nesting, depth-dependent weight identity, and local Jacobian bounds, are established. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate substantial reductions in persistent learnable parameters while maintaining competitive classification accuracy. On CIFAR-100, the proposed method achieves 76.67% accuracy using 2.084M parameters, outperforming the retrained OFA baseline by 1.14 percentage points while requiring approximately 5.4 times fewer learnable parameters.

Critical strengths
  1. The paper addresses an important limitation of existing elastic neural networks, namely that the storage requirements of a supernet are typically determined by its largest configuration. Reducing persistent model storage while preserving multiple deployment configurations is a practically relevant objective.
  2. The proposed SSG framework combines CP factorization with shared component magnitudes and layer-specific directional factors. This design provides a structured alternative to conventional dense supernets and black-box weight generators.
  3. The theoretical analysis establishes exact prefix nesting and preservation of generated-weight identities under depth selection. These properties provide a useful mathematical characterization of the proposed parameterization.
  4. The authors compare SSG against dense elastic supernets under matched architectures, training schedules, and configuration sets. The inclusion of equal-compute and approximately equal-storage comparisons makes the reported trade-offs more interpretable.
Critical weaknesses
  1. The proposed framework primarily combines established techniques, including CP tensor decomposition, weight normalization, cross-layer parameter sharing, and elastic network training. Although their integration is useful, the paper should more clearly explain which components represent fundamentally new methodological contributions. In particular, the exact prefix-nesting property appears to follow naturally from slicing a fixed generated tensor, rather than requiring a substantially new theoretical insight.
  2. The authors acknowledge that almost all parameter reduction comes from low-rank factorization rather than spectrum tying. Table 2 further shows that using independent spectra for different layers changes mean accuracy from 72.40% to 72.49%, with only 3,840 additional parameters. This raises an important question about whether network-wide spectrum sharing is essential to the proposed framework or primarily serves as an additional regularization constraint.
  3. Propositions 1 and 2 establish structural identities that follow directly from the parameterization and indexing strategy. Proposition 4 provides local partial-Jacobian bounds under nonzero factor norms, but does not establish improved optimization convergence, generalization, or robustness. The paper would benefit from a deeper theoretical investigation explaining why the proposed parameterization should preserve predictive performance under substantial compression.
Questions for authors
  1. Table 2 indicates that replacing the shared spectrum with independent layer-wise spectra results in almost identical accuracy and parameter counts. What is the essential advantage of spectrum sharing, and can the authors provide stronger empirical or theoretical evidence supporting its necessity?
  2. Since exact prefix nesting can be achieved by slicing a fixed dense tensor, what additional capability does the proposed normalized factorization provide beyond parameter compression? Could the authors clarify which theoretical properties are specific to SSG rather than generic consequences of fixed-weight slicing?
  3. How does the proposed method compare against a conventional CP-factorized elastic supernet without spectrum sharing, particularly when both methods use the same training objective and parameter budget?
  4. The experiments are primarily conducted on CIFAR-style ResNet-18. How well does SSG scale to ImageNet-1K or larger architectures, especially those containing substantial numbers of pointwise convolutions?
  5. The authors distinguish persistent learnable storage from optional dense-weight caches. Could they report actual deployment memory consumption, serialized checkpoint sizes, and weight-generation overhead for different deployment strategies?
  6. Can the authors provide training-time and peak GPU-memory comparisons with the dense elastic supernet and OFA under identical hardware and training configurations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hrjJ

We thank the reviewer for the detailed questions, which motivated most of the new experiments (see also the General Response). All numbers are final-checkpoint, 3-seed means unless noted.

W1/Q2: Nesting follows from slicing a fixed tensor; what does SSG add beyond compression? For a stored dense tensor, nesting is indeed trivial, and Props. 1–2 serve as correctness guarantees; we will state this explicitly. The cost of slicing a dense tensor is that storage equals the largest sub-network. SSG targets nesting and depth-wise weight identity at storage far below the largest sub-network, which generators do not provide automatically: normalising after slicing makes widths mutually rescaled; Tensor Train prefixes are no longer TTs of the same kernel (64.33 vs 68.15 for CP); coordinate-based channel axes collapse to 49.44; continuous layer coordinates change all later layers when a block is dropped. The payoff appears at equal storage (~2.1M): SSG is +6.35 over OFA and +7.46 over the dense supernet, and +8.90 on Tiny-ImageNet.

W2/Q1: Is spectrum sharing necessary? On ResNet-18, untying alone is within noise (72.22±0.27 vs 72.33±0.25) and compression comes mainly from low rank, as the paper states. Sharing is not about saving parameters; together with column normalisation it defines the parameterisation: normalisation separates atom magnitudes from directions, and the shared spectrum lets all layers use one set of magnitudes while each layer learns only directions. Untying lets layer-wise magnitudes drift (std of realised atom magnitudes 0.05 → 0.23 decades), and removing both sharing and normalisation (conventional CP) costs 0.68, about 3 std. Trained dense layers also have very similar normalised spectra (one mean spectrum explains 82.6% of the log-spectrum variance), which supports sharing one set of magnitudes. We will clarify this division of roles and report the untied result explicitly.

W3: Theory is shallow. Agreed: Props. 1–2 are structural, Prop. 4 is a local bound, and we claim no convergence or generalisation guarantee; the revision states this scope. Empirically: (i) the shared-spectrum prior above (82.6%); (ii) configurations differing only in width have aligned gradients (mean cosine +0.81 on the spectrum), since widths are restrictions of one generated tensor, whereas depth-only pairs conflict (−0.31), which independently motivates (d, ρ)-indexed normalisers; (iii) the storage formula ties the gain to k×k structure, consistent with ResNet-50 (Q4).

Q3: Conventional CP elastic supernet with the same objective and budget. At ~2.09M with identical code and objective: conventional CP 71.65±0.18, post-hoc CP 71.37 (1 seed), SSG 72.33±0.25 (table in the General Response).

Q4: ImageNet-1K and pointwise-heavy architectures. On ResNet-50, 3×3 kernels are ~48% of convolution parameters, so generating only 3×3 gives 1.6×, and generating 1×1 as well does not beat plain width scaling. ResNet-50 is also highly width-redundant on CIFAR-100 (width 0.70 costs only 1.33 mean points), leaving little room for any compression method. Analytic ImageNet-scale ratios are ~1.6× for ResNet-50 (3×3 only) and 1.0–1.8× for MobileNetV2. We will scope our claims to k×k-dominated CNNs. Eight-configuration elastic training on ImageNet-1K is not feasible within the discussion period, so we do not promise it.

Q5: Deployment memory, checkpoint size and generation overhead. Serialized checkpoint 8.4 MB (dense supernet 44.3 MB, OFA 42.9 MB). Generating all weights takes 3.5 ms and 13 MB of extra memory; the materialised dense cache is 41.9 MB, and inference is identical to a same-shape dense network (1.4–2.6 ms at batch 1 across the 8 configurations). Width is a prefix and depth a block subset, so switching configurations needs no regeneration.

Q6: Training time and peak memory. Same RTX 4090 and recipe: 0.100 / 0.094 / 0.094 s per step and 4.69 / 4.70 / 4.66 GB for SSG / dense supernet / OFA, i.e. about 5% slower with equal memory. Sampling 4 configurations per step reduces training time by 34% at unchanged accuracy.

Dear Reviewer hrjJ,

Thank you again for your thoughtful review and detailed questions, which shaped most of our additional experiments. As the discussion deadline approaches, and understanding that this is a busy period, we simply wanted to check whether our response has been helpful.

In brief, we compared SSG with a conventional CP elastic network and a post-hoc CP baseline under the same code and budget (Q3), measured training and deployment overhead on the same hardware (Q5/Q6), and added ResNet-50 results with a more clearly stated scope (Q4).

If any point remains unclear, we would be very glad to clarify it. Thank you for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank all reviewers. During the discussion period we ran about 40 additional full trainings. Unless stated otherwise, numbers are CIFAR-100, ResNet-18, the paper recipe, final checkpoint (epoch 200), and mean ± std over 3 seeds; "Mean" averages the 8 configurations.

1. OFA under identical code and recipe, and equal-storage comparison. We trained OFA in the same code base, recipe and configuration space as SSG, keeping progressive shrinking, channel reorganisation and shared BN with recalibration:

MethodStorageMeanLargest
OFA11.22M73.12±0.2677.50±0.27
Dense elastic supernet11.46M72.84±0.2976.92±0.26
SSG2.08M72.33±0.2575.59±0.35
OFA, width 0.43 (equal storage)2.09M65.98±0.1672.70±0.33
Dense, width 0.42 (equal storage)2.12M64.87±0.0872.64±0.05

SSG comes within 0.79 mean points of OFA with 5.4× less storage. At equal storage (~2.1M) it is +6.35 over OFA and +7.46 over the dense supernet, ahead on all 8 configurations; the equal-storage gain also holds on CIFAR-10 (+2.86) and Tiny-ImageNet (+8.90). The OFA row in the original Table 1 was a single run from a different training code base whose recipe and configuration space did not fully match ours, so "trained under the paper protocol" was inaccurate, and the "+1.14 over OFA" claim does not hold in this unified setting. The revision replaces Table 1 with the results above and makes the equal-storage comparison the headline.

2. Novelty: conventional low-rank elastic networks are clearly worse. Same code, objective and storage (~2.09M):

MethodMeanLargest
SSG (shared spectrum + column normalisation)72.33±0.2575.59±0.35
Per-layer spectrum only72.22±0.2775.52±0.34
No column normalisation only72.15±0.2475.37±0.29
Conventional CP elastic net (neither)71.65±0.1874.98±0.27
Post-hoc CP of a trained dense supernet + fine-tuning (best lr, 1 seed)71.3774.47

Column normalisation separates each atom's magnitude from its direction, and the shared spectrum makes all layers use one set of magnitudes while each layer learns only directions. Removing either alone stays within seed noise; removing both (conventional CP) costs 0.68 (about 3 std), and post-hoc factorisation is about 1 point lower even with a tuned learning rate.

3. Overhead (one RTX 4090, batch 256, 8 configurations). SSG / dense supernet / OFA: 0.100 / 0.094 / 0.094 s per step, 4.69 / 4.70 / 4.66 GB peak memory, 8.4 / 44.3 / 42.9 MB checkpoints. Generating all weights takes 3.5 ms and 13 MB. Training the largest plus 3 random configurations per step cuts wall-clock time by 34% (1.18 h → 0.78 h) at the same mean (72.34).

4. Scope. SSG's compression comes from k×k kernels: CP storage is R(Cin+Cout)+k2RR(C_{in}+C_{out})+k^2R versus k2CinCoutk^2C_{in}C_{out}, and for 1×1 convolutions the generator reduces to matrix low rank without the k2k^2 factor. On ResNet-50 (bottleneck, CIFAR-100), 3×3 kernels are ~48% of convolution parameters (ResNet-18: ~98%), so generating only 3×3 gives 1.6×, and generating 1×1 as well does not beat plain width scaling; raising the rank cap to 512 or using per-layer spectra changes accuracy by at most 0.2. ResNet-50 is also highly width-redundant on CIFAR-100: halving its parameters (width 0.70) costs the dense supernet only 1.33 mean points, versus 7.97 for ResNet-18 at 18% of its parameters. The equal-storage advantage therefore depends on the share of k×k convolutions and on width redundancy; we scope our claims to CNNs dominated by k×k convolutions and list pointwise generation as future work.

Revision summary. (i) Abstract, introduction and conclusion: same-code OFA results, equal-storage headline, scope to k×k-dominated CNNs. (ii) Table 1: same code, 3 seeds, final checkpoint; Table 2 adds the conventional CP and post-hoc baselines and renames "Spectrum per layer". (iii) New: overhead, sampled training, ResNet-50, ImageNet-scale analytic ratios, Tiny-ImageNet equal storage. (iv) Corrections: dense storage 11.501M → 11.464M (an unused block was counted; the 5.5× ratio is unchanged), equal-compute gap 0.36 → 0.51, the "within 0.01" best/final statement, and "Equation equation".

AuthorsOpenReview ↗

Summary of the author response

Dear Area Chair and Reviewers,

As the author–reviewer discussion closes, we briefly summarise the main concerns and how our response addresses them, for convenience in the next phase.

Protocol of the OFA comparison and checkpoint selection (Dtgx, rSzz). We retrained SSG, OFA and the dense elastic supernet in a single code base with three seeds, evaluated at the final checkpoint. In this setting the earlier "+1.14 over OFA" claim does not hold: SSG is within 0.79 mean points of OFA with 5.4× less storage, and 6.35 points higher at equal storage (~2.1M). We will correct the claim accordingly.

Novelty beyond conventional low-rank factorisation (hrjJ, wvpa). Under identical code and a ~2.09M budget, SSG reaches 72.33±0.25, compared with 71.65±0.18 for a conventional CP elastic network and 71.37 (single seed) for post-hoc CP of a trained dense supernet. Role of spectrum sharing (hrjJ, wvpa). Removing spectrum sharing or column normalisation alone stays within seed noise, whereas removing both costs 0.68 mean points. We clarify that compression comes from low rank, and that sharing and normalisation together define the parameterisation.

Training and deployment cost (hrjJ, rSzz). SSG adds about 5% time per step at equal peak memory, with an 8.4 MB checkpoint versus 44.3 MB for the dense supernet. Sampling four configurations per step reduces training time by 34% at the same mean accuracy. Scale and architectures (hrjJ, wvpa, rSzz). ResNet-50 experiments show that the gain depends on the share of k×k convolutions, and we scope our claims to CNNs dominated by k×k convolutions.

We sincerely thank the reviewers and the Area Chair for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.