- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper discussed the approach of using Shannon-mutual information as a criteria to expand width/depth of a neural network. This information has several limitations like being difficult to estimated in practice hence the authors proposed v-NAS, based on v-Information (Xu et al. 2020.) The main practical implementation is to limit the classifier each layer as linear functions, train them and get useful information from the CE loss of this classifier, to tell whether the width/depth is saturated. This also a general progressive architecture search method.
Experiments are conducted on several datasets on CIFAR and ImageNet.
Critical strengths
The usage of v-information into the arch search/parameter tuning works is interesting. To my best knowledge, this is prior NAS work that tries to approach the problem from such a perspective.
Critical weaknesses
It remains questionable that in 2026, doing NAS works under a fixed search dimension (e.g., width or depth) is still meaningful. In practice, this is the local optimal solution which likely cannot generalize to the global one, especially given the large model complexity people generally feel interested now. Even if we do not consider the general scaling law literature, there was also related works like "Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design (NeurIPS 2023)" that studies the joint scaling of width and depth of ViT under a compute budget. The authors seem not reporting any comparison towards it even not citing.
The small scale of the datasets used in the experiments is not enough to remove the above concern as well. A more comprehensive experiments would be helpful to answer a fundamental question: whether the v-information based approach can provide a better capacity allocation principle than prior works under a fixed computation budget.
Questions for authors
Check the weakness part