- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LESS, a lightweight evolutionary neural architecture search method that combines fair supernet training, CMA-ES, and candidate-conditioned calibration. Specifically, LESS first performs a short supernet warm-up and then evaluates candidate architectures through a few balanced training updates. The best candidate determines the supernet state for the next search iteration, and Borda-based ranking is used to select the final architecture. Experiments on NAS-Bench-201 and the DARTS search space are conducted to evaluate its search accuracy and efficiency.
Critical strengths
1.The paper studies an interesting topic, aiming to reduce the computational cost of neural architecture search while maintaining reliable architecture evaluation. 2.The paper provides relatively detailed experiments, including multiple random seeds, component ablations, and evaluations across different datasets and search spaces.
Critical weaknesses
-
I think the motivation of this paper is not sufficiently clear. Although the authors discuss the trade-off between evaluation cost and accuracy in NAS, they do not clearly explain the specific limitations of existing training-free and weight-sharing NAS methods that motivate this work. In particular, why are the proposed calibration and search strategies necessary to address these limitations? The abstract and introduction mainly describe what the proposed method does, but the motivation for these designs is not sufficiently explained.
-
The novelty of the proposed method seems limited. The framework mainly combines several existing techniques, such as fair supernet training, CMA-ES, and Borda-based selection. However, the main idea behind this combination is not clearly presented. Moreover, some components do not show clear advantages in the ablation studies. For example, Borda selection does not clearly outperform the last-epoch selection, while using only one calibration update achieves comparable performance with lower computational cost than the selected six-update setting. These results make the necessity of some proposed strategies questionable.
-
The related work and experimental comparisons are not sufficiently up to date. Most baseline methods in Tables 1 and 2 were published between 2018 and 2021. Although the paper discusses some more recent methods published in 2024, these methods are not included in the experiments. Considering that this paper is submitted at 2026, comparisons with more recent efficient NAS methods are necessary to demonstrate its contribution over existing research.
-
The experimental results do not show sufficiently convincing advantages over existing methods. For example, in Table 1, LESS achieves 93.189% accuracy on CIFAR-10, which is lower than FairNAS (93.23%) and TE-NAS (93.90%). Similar issues can also be observed in the DARTS experiments, where several existing methods achieve better accuracy. Although LESS reports lower search costs than some baselines, these results are obtained using different hardware and implementations. Therefore, it is difficult to determine whether the proposed method achieves a better trade-off between search accuracy and computational cost. More comprehensive comparisons under the same computational budget are needed.
-
The effectiveness of the proposed calibration strategy across different datasets is not sufficiently demonstrated. For example, the improvement of the six-update calibration over the zero-update setting on CIFAR-100 is very small, and the confidence interval includes zero. The results on ImageNet16-120 also show relatively large selection regret. These observations raise questions about whether the proposed calibration strategy can consistently improve architecture selection across different tasks.
-
Figures 1 and 2 are not explicitly referenced or discussed in the main text. The authors should better explain how these figures illustrate the main ideas of the proposed method.
Questions for authors
See above weakness