- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Spectrum-Shared Generation (SSG), a normalized factorized weight generation method for nested elastic convolutional networks. The method uses CP factorization with layer-specific directions and a network-wide shared spectrum to generate convolutional weights. Through full-column normalization and fixed layer identities, SSG supports nested width and depth configurations while reducing persistent learnable parameter storage. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate substantial storage reduction with relatively small accuracy differences compared with dense elastic supernets.
Critical strengths
- The paper addresses a meaningful problem in elastic neural networks, namely reducing the storage cost of a single model that supports multiple network configurations. The motivation and overall framework are clearly presented, and the explanations of the proposed method are detailed. The authors also provide mathematical analyses to explain the structural properties of the proposed parameterization, which makes the method more understandable and convincing.
- The authors provide comparisons under different storage and computation budgets, together with detailed ablation studies to examine the contributions of different components. The supplementary material also includes analyses of different factorization strategies, rank settings, unseen configurations, and robustness under distribution shifts. These experiments provide useful evidence for the effectiveness and practical value of the proposed method.
Critical weaknesses
The authors report that SSG achieves 76.67% accuracy on CIFAR-100, outperforming their retrained OFA baseline by 1.14 points. However, Appendix M mentions some implementation differences between the main paper run and the controlled implementation. In the controlled implementation, SSG achieves 75.55% accuracy at the largest configuration, compared with 76.67% in the main paper run. Although the authors acknowledge these differences, it remains unclear how they affect the reported performance advantage over OFA. A clearer explanation would help make the experimental conclusions more convincing.
Questions for authors
- Could the authors further explain the implementation differences between the main paper run and the controlled implementation in Appendix M, and whether these differences affect the reported improvement over OFA?
- Are the best accuracies of the eight sub-networks in Table 1 obtained from the same shared checkpoint?
2 comments
Dear Reviewer Dtgx,
Thank you again for your careful reading. Your question about the experimental protocol led to an important correction, and we are grateful for it. Under a single code base with three seeds and the final checkpoint, SSG is within 0.79 mean points of OFA while using 5.4× less storage, and 6.35 points higher at equal storage.
As the discussion deadline approaches, we would be happy to clarify anything further. Thank you for your time.
Response to Reviewer Dtgx
We thank the reviewer for the positive assessment; both questions are important. All numbers are final-checkpoint, 3-seed means.
W1/Q1: Do the implementation differences in Appendix M affect the improvement over OFA? To answer this, we trained OFA in the same code base, recipe and configuration space as SSG (keeping progressive shrinking, channel reorganisation and shared BN with recalibration):
SSG comes within 0.79 mean points of OFA with 5.4× less storage, and at equal storage it is 6.35 points higher.
Regarding Appendix M: the 76.67 of SSG comes from an earlier version of our training code, while the controlled experiments use the current code. Both are the same model with the same main settings; they differ in some code details (e.g., the distillation temperature) and in reporting (the original table takes each sub-network's best epoch). Re-training with the current code under the earlier settings gives 75.66±0.09 at the largest configuration. The original OFA row was likewise a single run, from a different training code base whose recipe did not fully match ours. The reviewer's concern is therefore valid: in the unified setting, the "+1.14 over OFA" claim does not hold. The revision replaces Table 1 with the same-code, multi-seed results above and makes the equal-storage comparison the headline.
Q2: Are the eight best accuracies in Table 1 from the same checkpoint? No; the original table reports each sub-network's best value over 200 epochs. We agree that a single checkpoint is the right protocol for a model serving several configurations. Using epoch 200, the mean of every method in the main table drops by only 0.10–0.16 and no comparison changes. All numbers in this rebuttal use the final checkpoint, the revision will do the same, and we will remove the inaccurate "within 0.01" statement from the appendix.