- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PP NAS, a method for combining scarce ground-truth architecture evaluations with a larger set of zero-cost proxy scores. Both sources are converted into pairwise ranking supervision. A prediction-powered objective then uses the labeled architectures to correct discrepancies between proxy and ground-truth rankings. The method estimates variability at the architecture level, selects the weight of proxy supervision, and uses the resulting parameter covariance for uncertainty-aware acquisition.
Experiments cover NAS-Bench-201, NAS-Bench-101, and TransNASBench, with additional analyses of proxy choice, initial labeling budget, unlabeled sample size, and architecture encoding. The results suggest that PP NAS often finds strong architectures earlier in the search, although its final accuracy is not consistently the highest.
Critical strengths
-
The method addresses a practical problem with a coherent formulation.
Reliable architecture evaluations are expensive, while zero-cost proxies provide inexpensive but imperfect information. Expressing both through pairwise preferences offers a natural way to combine signals with different numerical scales.
I also appreciate the attention to dependence between pairwise comparisons. Comparisons sharing an architecture cannot be treated as independent observations. The architecture-level variance construction in Section 3.4 is a sensible response to this issue.
-
The empirical evaluation covers several relevant settings.
The paper evaluates multiple search spaces, proxies, initial budgets, and architecture encodings, with 30 repetitions per search configuration. In Table 1, PP NAS achieves the highest AUC on all three datasets. These results support its usefulness for finding good architectures early under a limited evaluation budget. The additional search-space experiments provide evidence that the benefit is not confined to a single benchmark.
Critical weaknesses
-
The assumptions underlying the correction and covariance estimates do not directly hold under the proposed search procedure.
Section 3.1 assumes independent labeled and unlabeled samples drawn from the same architecture–proxy distribution. In Section 3.5, however, architectures are selected using the current model and then added to the labeled set. As search progresses, this set becomes concentrated in regions favored by the acquisition rule.
This matters for both parts of the method. The correction in Equation (7) relies on the two proxy-loss terms having matching expectations, while Equation (10) relies on the stated sampling structure. Accounting for shared architectures within pairwise comparisons does not, by itself, address adaptive selection across search iterations.
The paper does not establish when these expressions remain applicable during search or evaluate the calibration of the resulting uncertainty estimates. I therefore find the statistical interpretation less convincing than the empirical search results.
-
The experiments do not isolate the contribution of the proposed correction and adaptive weighting.
PP NAS combines proxy ranking supervision, bias correction, adaptive weighting, and uncertainty-aware acquisition. The current comparisons mainly evaluate complete methods. They do not separate these components under a common predictor and search protocol. Showing that the selected weight changes over time does not establish that adaptive weighting improves on a fixed weight.
This distinction matters because several absolute improvements are modest. In Table 1, the AUC gains over the strongest baseline on each dataset are approximately 0.08, 0.05, and 0.03 percentage points. Final accuracy is not consistently better. Aggregate ranks are useful, but they do not establish the magnitude or reliability of these improvements.
Controlled ablations and confidence intervals for paired performance differences would help determine whether the proposed correction and weighting mechanisms explain the gains.
Questions for authors
-
How is the distribution difference between the adaptively collected labeled set and the unlabeled set handled? Under what conditions do Equations (7) and (10) remain applicable during search? If they are used heuristically, can you provide an empirical assessment of uncertainty calibration?
-
Under the same predictor, data, and acquisition rule, how do ground-truth-only training, direct mixing of proxy supervision, correction with a fixed λ, and correction with an adaptive λ compare? What changes when the uncertainty term in acquisition is removed?
-
Equation (7) does not specify regularization or parameter constraints. What ensures stable optimization, particularly with only 5 or 10 initial labels? Please clarify the implementation choices and how they enter the Hessian and covariance calculations.