- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the information loss caused by global aggregation in zero-cost neural architecture search. It provides a general perspective on existing zero-cost proxies and investigates the importance of preserving layer-wise information during architecture evaluation. Based on this analysis, the authors propose a layer-aware aggregation method that uses logarithmic aggregation to construct architecture scores. Experiments on a parameter-matched subset of NAS-Bench-101 demonstrate improvements in architecture ranking correlation and Top-K screening.
Critical strengths
-
The motivation of this paper is relatively clear. Investigating the information loss caused by global aggregation is an interesting problem in zero-cost NAS.
-
The paper provides theoretical analysis of layer-wise aggregation and evaluates different aggregation functions, which helps explain the motivation behind the proposed score.
Critical weaknesses
-
The experimental validation does not sufficiently support one of the main motivations of this paper. The authors repeatedly argue that existing zero-cost proxies perform inconsistently across different tasks and search spaces. However, all experiments are conducted on only one parameter-matched subset of NAS-Bench-101 using CIFAR-10. It is unclear whether the proposed method can address this limitation across different tasks, datasets, or search spaces. I think more comprehensive experiments are necessary to support the generality of the proposed approach.
-
The experimental settings are not sufficiently described. Although Table 1 lists many compared proxies, the paper does not clearly introduce these baselines or provide enough details about their implementation and evaluation settings. Moreover, the computational cost of the proposed method is not compared with existing zero-cost proxies. These details are important for evaluating the effectiveness and efficiency of the proposed method.
-
The readability of this paper needs to be improved. Several symbols and mathematical expressions are not clearly explained, particularly in the Preliminaries and Section 4. For example, the construction and role of some intermediate representations and information extraction operators are difficult to understand. The authors should provide clearer definitions and explanations to make the proposed framework easier to follow.
-
The Related Work section is not well organized. Many existing studies are directly listed together without sufficient classification or discussion of their limitations. The authors should systematically review different types of zero-cost proxies and clearly explain the differences between these methods and the proposed approach, especially existing methods that already consider layer-wise information.
-
THe practical advantages of the proposed method are not sufficiently demonstrated. Although the proposed score achieves better ranking correlations, the improvements in Top-K screening are relatively small compared with some baselines. For example, Overlap@50 only increases from 0.140 for SNIP to 0.160 for the proposed method. It would be more convincing to evaluate whether these improvements can lead to better architectures in a complete NAS process.
Questions for authors
See weakness.1