acceptodds
Withdrawn from ICLR 2027

LESS: Lightweight Evolutionary Supernet Search in Minutes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-cost NAS must both explore high-performing architectures and identify them reliably, yet reducing evaluation cost often weakens the fidelity of candidate comparisons. Training-free methods reduce evaluation cost by replacing learned task feedback with proxy signals measured at initialization. We introduce LESS (Lightweight Evolutionary Supernet Search), a data-driven method that combines a brief fair hard-path warm-up with discrete search under a single CMA-ES distribution. Each proposal is evaluated as its decoded hard genotype after six candidate-conditioned supernet updates. On NAS-Bench-201, LESS achieves 93.189 ± 0.467% CIFAR-10 test accuracy in 409.1 seconds, coming within 0.04 percentage points of FairNAS using approximately 1/24 of its source-reported search time. Matched controls show that calibration improves selected validation accuracy by 0.577 percentage points while changing best-visited accuracy by only 0.054 points, indicating that its primary effect is to reduce selection regret. The frozen configuration transfers without tuning to CIFAR-100 and ImageNet16-120 with 69.615 ± 1.139% and 43.720 ± 1.697% accuracy. Applied without tuning to the larger DARTS space, LESS achieves 96.95 ± 0.14% on CIFAR-10 and 82.43 ± 0.80% on CIFAR-100, with each search completing in approximately 43.5 minutes on a single GPU. Together, these results show that short, balanced, data-dependent updates enable competitive neural architecture search across datasets and search spaces within minutes.

Primary area:
optimization
Keywords:
neural architecture search, weight-sharing NAS, evolutionary optimization, CMA-ES, supernet training, architecture selection, efficient deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iYvCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LESS, a lightweight evolutionary neural architecture search method that combines fair supernet training, CMA-ES, and candidate-conditioned calibration. Specifically, LESS first performs a short supernet warm-up and then evaluates candidate architectures through a few balanced training updates. The best candidate determines the supernet state for the next search iteration, and Borda-based ranking is used to select the final architecture. Experiments on NAS-Bench-201 and the DARTS search space are conducted to evaluate its search accuracy and efficiency.

Critical strengths

1.The paper studies an interesting topic, aiming to reduce the computational cost of neural architecture search while maintaining reliable architecture evaluation. 2.The paper provides relatively detailed experiments, including multiple random seeds, component ablations, and evaluations across different datasets and search spaces.

Critical weaknesses
  1. I think the motivation of this paper is not sufficiently clear. Although the authors discuss the trade-off between evaluation cost and accuracy in NAS, they do not clearly explain the specific limitations of existing training-free and weight-sharing NAS methods that motivate this work. In particular, why are the proposed calibration and search strategies necessary to address these limitations? The abstract and introduction mainly describe what the proposed method does, but the motivation for these designs is not sufficiently explained.

  2. The novelty of the proposed method seems limited. The framework mainly combines several existing techniques, such as fair supernet training, CMA-ES, and Borda-based selection. However, the main idea behind this combination is not clearly presented. Moreover, some components do not show clear advantages in the ablation studies. For example, Borda selection does not clearly outperform the last-epoch selection, while using only one calibration update achieves comparable performance with lower computational cost than the selected six-update setting. These results make the necessity of some proposed strategies questionable.

  3. The related work and experimental comparisons are not sufficiently up to date. Most baseline methods in Tables 1 and 2 were published between 2018 and 2021. Although the paper discusses some more recent methods published in 2024, these methods are not included in the experiments. Considering that this paper is submitted at 2026, comparisons with more recent efficient NAS methods are necessary to demonstrate its contribution over existing research.

  4. The experimental results do not show sufficiently convincing advantages over existing methods. For example, in Table 1, LESS achieves 93.189% accuracy on CIFAR-10, which is lower than FairNAS (93.23%) and TE-NAS (93.90%). Similar issues can also be observed in the DARTS experiments, where several existing methods achieve better accuracy. Although LESS reports lower search costs than some baselines, these results are obtained using different hardware and implementations. Therefore, it is difficult to determine whether the proposed method achieves a better trade-off between search accuracy and computational cost. More comprehensive comparisons under the same computational budget are needed.

  5. The effectiveness of the proposed calibration strategy across different datasets is not sufficiently demonstrated. For example, the improvement of the six-update calibration over the zero-update setting on CIFAR-100 is very small, and the confidence interval includes zero. The results on ImageNet16-120 also show relatively large selection regret. These observations raise questions about whether the proposed calibration strategy can consistently improve architecture selection across different tasks.

  6. Figures 1 and 2 are not explicitly referenced or discussed in the main text. The authors should better explain how these figures illustrate the main ideas of the proposed method.

Questions for authors

See above weakness

Reviewer_kRiYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, authors propose LESS, it addresses unreliable architecture selection in low-cost weight-sharing NAS. It combines fair supernet warm-up, CMA-ES proposals, candidate-conditioned balanced updates, BatchNorm recalibration, winner-state inheritance, and cumulative Borda selection. On NAS-Bench-201/CIFAR-10, it reports 93.189 ± 0.467% test accuracy in 409.1 seconds. Its main finding is that calibration improves the architecture returned primarily by reducing selection regret, rather than discovering better architectures. The frozen configuration is also evaluated on CIFAR-100, ImageNet16-120, and the DARTS space.

Critical strengths
  1. The paired CIFAR-10 experiment provides credible evidence that calibration improves the complete pipeline. The mean validation gain is 0.577 percentage points across 24 seeds, with a 95% confidence interval of [0.262, 0.891], all eight prospective replication pairs favor calibration.

  2. The experimental reporting is quite thorough. The appendix distinguishes development runs from the principal evaluation, provides seed-level outcomes and returned genotypes, and documents computational costs and inconclusive controls.

Critical weaknesses
  1. The comparative accuracy–cost claim needs stronger evaluation, specifically, there are roughly 24-fold time ratio against FairNAS uses different hardware and implementations, authors acknowledge this, but it cannot establish an algorithmic speedup. Tables 1–2 also omit CMANAS and AZ-NAS, despite their relevance to the paper’s positioning.

  2. The (K=6) versus (K=0) comparison changes both candidate evaluation and subsequent inherited weights. The conditioning-specific control is less decisive: its median gain is only 0.057 points, its mean-effect confidence interval spans zero, and two failures account for most of the improvement. This does not rule out a useful robustness benefit, but it leaves the necessity of candidate conditioning insufficiently established.

Questions for authors
  1. Can authors provide evidence that separates the benefit of candidate-conditioned updates from continued fair training of the shared parent and winner-state inheritance? e.g., a matched-update control would directly affect my assessment of the methodological contribution.

  2. Does CMA-ES still outperform uniform proposals when repeat evaluations per genotype are matched?

  3. Can the main accuracy–cost comparison include relevant baselines measured on the same hardware, with comparable architecture-selection and checkpoint-reporting protocols?

Reviewer_AyfwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LESS, a zero-cost NAS. It consists of 3 stages. First, a supernet is trained under a fair setting. Then, a subset is form by copying supernet weight and perform 6 SGD step over a fair group of paths anchored on that candidate. Its BatchNorm is refreshed, and it is scored by validation loss. The epoch winner’s weights become the next parent, and a Borda count over all 60 epochs picks the final architecture. Experiments on NAS-Bench-201 and DARTS search space are performed to validate the effectiveness of LESS.

Critical strengths
  1. Studying zero-cost NAS is an important topic, as it accelerates the search step.
  2. A simple method with reasonable acceleration.
Critical weaknesses

Major:

  1. The abstract is not well written, which makes it difficult for readers to understand the paper contribution. In the abstract, several terms are introduced without a brief explanation. What are "Matched controls", "frozen configuration"? Why are six candidates used? More than half of the abstract reports the results, while the proposed component is just 2 lines. I think the reader may be more interested in what the problem is and how this paper solves the problem than in reading these experimental results.
  2. The method is simple. The novelty is limited. The candidate-condition fair selection is introduced without a strong motivation. It is unclear why doing so helps.
  3. The experimental results on the large search space (DARTS) are weak. Most baselines for comparison were published 7 years ago (2019). It is required to compare with recent zero-cost NAS on the DARTS search space.
  4. The experiments are limited. Lack of experiments on large-scale datasets (ImageNet) and a stronger search space, such as the transformer search space [1].
  5. The related work section is limited. Most methods in this section are outdated. Most of them were introduced in 20~23.

Minor:

  1. Figure 1 and 2 are low quality. (design style/font size/etc.)

[1] AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search, Junghyup Lee et.al.

Questions for authors

See weaknesses. Addressing each weakness may help to strengthen the paper and improve the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.