acceptodds
Under review as a conference paper at ICLR 2027

Prediction-powered neural architecture search

Abstract

Evaluating candidate architectures in neural architecture search (NAS) faces an inherent trade-off: on the one hand, reliable performance labels are limited because training and evaluating architectures is costly; on the other hand, zero-cost proxies (ZCPs) are cheap to compute at large scale but can be noisy. Yet, how to effectively combine these two sources of supervision remains unclear. In this paper, we propose PPNAS, a novel prediction-powered inference (PPI) approach for NAS. PPNAS fuses (1) a small set of architectures with observed performance labels and (2) a large set of architectures with ZCP information. To combine these two sources of supervision, PPNAS exploits the ordinal information provided by ZCPs to construct additional pairwise ranking supervision, while PPI debiases systematic discrepancies between ZCP-based and true performance rankings. We evaluate PPNAS in end-to-end predictor-based NAS, where it achieves state-of-the art under limited evaluation budgets. To the best of our knowledge, PPNAS is the first prediction-powered approach for label-efficient NAS

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
neural architecture search, architecture search, ranking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yJBxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PP NAS, a method for combining scarce ground-truth architecture evaluations with a larger set of zero-cost proxy scores. Both sources are converted into pairwise ranking supervision. A prediction-powered objective then uses the labeled architectures to correct discrepancies between proxy and ground-truth rankings. The method estimates variability at the architecture level, selects the weight of proxy supervision, and uses the resulting parameter covariance for uncertainty-aware acquisition.

Experiments cover NAS-Bench-201, NAS-Bench-101, and TransNASBench, with additional analyses of proxy choice, initial labeling budget, unlabeled sample size, and architecture encoding. The results suggest that PP NAS often finds strong architectures earlier in the search, although its final accuracy is not consistently the highest.

Critical strengths
  1. The method addresses a practical problem with a coherent formulation.

    Reliable architecture evaluations are expensive, while zero-cost proxies provide inexpensive but imperfect information. Expressing both through pairwise preferences offers a natural way to combine signals with different numerical scales.

    I also appreciate the attention to dependence between pairwise comparisons. Comparisons sharing an architecture cannot be treated as independent observations. The architecture-level variance construction in Section 3.4 is a sensible response to this issue.

  2. The empirical evaluation covers several relevant settings.

    The paper evaluates multiple search spaces, proxies, initial budgets, and architecture encodings, with 30 repetitions per search configuration. In Table 1, PP NAS achieves the highest AUC on all three datasets. These results support its usefulness for finding good architectures early under a limited evaluation budget. The additional search-space experiments provide evidence that the benefit is not confined to a single benchmark.

Critical weaknesses
  1. The assumptions underlying the correction and covariance estimates do not directly hold under the proposed search procedure.

    Section 3.1 assumes independent labeled and unlabeled samples drawn from the same architecture–proxy distribution. In Section 3.5, however, architectures are selected using the current model and then added to the labeled set. As search progresses, this set becomes concentrated in regions favored by the acquisition rule.

    This matters for both parts of the method. The correction in Equation (7) relies on the two proxy-loss terms having matching expectations, while Equation (10) relies on the stated sampling structure. Accounting for shared architectures within pairwise comparisons does not, by itself, address adaptive selection across search iterations.

    The paper does not establish when these expressions remain applicable during search or evaluate the calibration of the resulting uncertainty estimates. I therefore find the statistical interpretation less convincing than the empirical search results.

  2. The experiments do not isolate the contribution of the proposed correction and adaptive weighting.

    PP NAS combines proxy ranking supervision, bias correction, adaptive weighting, and uncertainty-aware acquisition. The current comparisons mainly evaluate complete methods. They do not separate these components under a common predictor and search protocol. Showing that the selected weight changes over time does not establish that adaptive weighting improves on a fixed weight.

    This distinction matters because several absolute improvements are modest. In Table 1, the AUC gains over the strongest baseline on each dataset are approximately 0.08, 0.05, and 0.03 percentage points. Final accuracy is not consistently better. Aggregate ranks are useful, but they do not establish the magnitude or reliability of these improvements.

    Controlled ablations and confidence intervals for paired performance differences would help determine whether the proposed correction and weighting mechanisms explain the gains.

Questions for authors
  1. How is the distribution difference between the adaptively collected labeled set and the unlabeled set handled? Under what conditions do Equations (7) and (10) remain applicable during search? If they are used heuristically, can you provide an empirical assessment of uncertainty calibration?

  2. Under the same predictor, data, and acquisition rule, how do ground-truth-only training, direct mixing of proxy supervision, correction with a fixed λ, and correction with an adaptive λ compare? What changes when the uncertainty term in acquisition is removed?

  3. Equation (7) does not specify regularization or parameter constraints. What ensures stable optimization, particularly with only 5 or 10 initial labels? Please clarify the implementation choices and how they enter the Hessian and covariance calculations.

Reviewer_sAM9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PP NAS, a prediction-powered inference (PPI) framework for label-efficient neural architecture search. The method combines a small set of ground-truth architecture evaluations with a larger set of zero-cost proxy (ZCP) observations through a rectified pairwise ranking objective. To account for dependencies among architecture pairs, the authors employ architecture-level Hájek projections to estimate gradient variability and adaptively adjust the contribution of ZCP supervision. The resulting ranking model is integrated into an end-to-end NAS procedure using an uncertainty-aware upper confidence bound acquisition strategy. Experiments on NAS-Bench-201, NAS-Bench-101, and TransNAS-Bench demonstrate competitive search performance under limited ground-truth evaluation budgets. Additional analyses investigate robustness across different ZCPs, architecture encodings, search spaces, and evaluation budgets.

Critical strengths

The paper addresses an important problem in label-efficient NAS by combining scarce ground-truth evaluations with abundant but noisy zero-cost proxy (ZCP) information. The proposed pairwise ranking formulation provides a meaningful adaptation of prediction-powered inference (PPI) to NAS, while the use of architecture-level Hájek projections explicitly accounts for dependencies among pairwise comparisons. Furthermore, the experimental evaluation is relatively comprehensive, covering multiple NAS benchmarks, ZCPs, architecture encodings, and evaluation budgets. The results demonstrate competitive search performance, particularly in terms of search efficiency under limited ground-truth evaluation budgets.

Critical weaknesses
  1. Previous studies have already explored combining ZCP information with ground-truth evaluations. However, the paper lacks a controlled comparison with simple weighted fusion, e.g., (L_{\mathrm{naive}}=L_{\mathrm{GT}}^Y+\lambda L_{\mathrm{ZCP}}^Z), under the same predictor and search protocol, making it difficult to isolate the contribution of the proposed PPI rectification mechanism.

  2. The experiments are limited to tabular NAS benchmarks, without exploring more challenging search spaces such as DARTS or larger-scale datasets such as ImageNet-1K, where obtaining ground-truth architecture evaluations is particularly expensive. This limits the evidence for the practical effectiveness and scalability of PP NAS in realistic, label-scarce NAS scenarios.

  3. The authors are encouraged to discuss and compare against more recent relevant NAS studies, particularly methods that combine zero-cost proxies with ground-truth evaluations, to better establish the novelty and competitiveness of PP NAS.

  4. There are minor typographical errors: (P_{XZ}) is incorrectly formatted as (PXZ) in Section 3.1, and the main results in Section 5 are incorrectly referenced as being in the appendix.

Questions for authors

1。The PPI formulation assumes that labeled and unlabeled architectures are independently sampled from the same distribution. However, the labeled set becomes adaptively selected through UCB acquisition during NAS. Could the authors clarify whether the theoretical properties of the proposed rectification and variance estimation still hold under this adaptive sampling process? 2.Can the authors provide an ablation comparing the proposed PPI rectification with simple weighted fusion, (L_{\mathrm{GT}}^Y+\lambda L_{\mathrm{ZCP}}^Z), and fixed values of (\lambda), under the same experimental settings? 3.How well would PP NAS scale to non-tabular search spaces such as DARTS or larger-scale datasets such as ImageNet-1K, where ground-truth evaluations and uncertainty estimation may introduce substantial computational overhead? 4. I would be willing to reconsider my rating if the authors could provide the suggested ablation studies or additional evaluations on more challenging search spaces.

Reviewer_asPGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PP NAS, which adapts PPI to NAS to combine a few architectures with ground-truth performance and many with only ZCP scores. Both are used as pairwise ranking supervision, with a PPI-style rectifier correcting ZCP bias and an adaptively tuned weight λ. PP NAS achieves the best average ranks on NB201, NB101, and TransNASBench under limited budgets.

Critical strengths
  1. A new and well-motivated combination. This work introduces prediction-powered inference (PPI) into NAS. The motivation is clear.

  2. Broad and convincing evaluation. The experiments cover many ZCPs (Fig. 3), different initial ground-truth budgets, different architecture encodings (including arch2vec), and multiple search spaces and datasets. This provides broad empirical evidence of the method's effectiveness.

Critical weaknesses
  1. Missing ablations that would isolate the PPI correction. The paper explains its gains by comparing against baselines. Those baselines also differ in ranking loss, predictor, and acquisition strategy, so this comparison does not show where the improvement comes from. A controlled ablation within the same framework, using the same model, loss and UCB acquisition, would be much more convincing.

  2. The criterion for choosing λ may not match the NAS goal. λ is chosen by minimizing the trace of the estimated parameter covariance (Eq. 11). This is a statistical-efficiency criterion inherited from PPI++. NAS, however, cares about pairwise ranking quality and ultimately about search performance. It is not clear that lower parameter variance leads to better rankings or better architectures.

  3. The UCB acquisition is not analyzed. The exploration weight is fixed at β = 0.5 without justification. The paper does not examine how sensitive the results are to β, or whether the uncertainty term helps at all compared with a purely greedy acquisition (β = 0). Since the covariance estimate also feeds into acquisition, this analysis would help show whether PP NAS's uncertainty estimates are actually useful.

  4. Sloppy writing: the sentence at L309 ("At this fixed-λ estimate, we compute the") appears incomplete.

  5. (Minor) Typos: L50 "combine combine"; L83 "supervison"; L420 "Appenidx".

Questions for authors
  1. Naming in Fig. 12. What does "RankingPPI (ours)" in Fig. 12 refer to? Is it the same method as PP NAS (e.g., an earlier name), or a different variant?

  2. Ablating the rectifier. Could the authors report a controlled ablation that compares λ= 0 (ground truth only), naive fusion without the rectifier, and full PP NAS, using the same predictor and acquisition function?

  3. Choosing λ by ranking or search performance. Does minimizing the trace of the parameter covariance actually correlate with ranking quality (e.g., Kendall's τ on held-out architectures) or with search performance? Did the authors consider a ranking-aware criterion for choosing λ? How does adaptive λ compare with well-chosen fixed values?

  4. Sensitivity to β. How sensitive are the results to the UCB parameter β? How does PP NAS perform with β = 0 (no exploration term)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.