acceptodds
Under review as a conference paper at ICLR 2027

Auditing Size Dependence in Zero-Cost NAS Proxies: Confound-Controlled Discovery and Held-Out Evaluation

Abstract

Zero-cost proxy evaluation and automated discovery rely on score-accuracy correlation, but this correlation does not distinguish size-related association from association remaining after size control. In our audit, log parameter count has the largest absolute unadjusted Kendall correlation among 15 baselines in four of the original five cell-based search spaces from Network Design Spaces (NDS), using the finite support of each score. This strong size baseline makes it important to establish what predictive information a proxy retains after size control. Our audit of 24 scores shows that joint parameter-count and FLOPs controls change rankings and reveal obscured associations. We therefore use absolute residual association as the fitness objective for genetic programming (GP), yielding three candidates. The formulas, signs, and control analyses of these three candidates were frozen before NDS access. We designated the focal score and registered the 15-baseline comparison after opening. On architectures where all compared methods produce finite scores in these five spaces, exceeds the strongest eligible baseline in absolute association under residual and stratified controls. All ten margin confidence intervals remain positive after adjustment for the choice among the three candidates. Separately, a post-hoc comparison of 100 matched GP pairs finds higher mean absolute residual association on the five NDS spaces under controlled than unadjusted fitness. This association advantage alone does not establish better selection. Separate retrospective simulations using stored benchmark accuracies on these populations yield positive mean selected-accuracy gains over the strongest of the 15 baselines in 49 of 50 conditions. A post-hoc Bonferroni analysis retains 41 positive gain intervals. The evidence supports controlled discovery of stronger residual associations and, separately, selection gains within the evaluated scope.

Primary area:
optimization
Keywords:
Zero-Cost NAS Proxies, Held-out Evaluation, Controlled discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JW7QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that common zero-cost NAS evaluation conflate proxy quality with model size because parameter count and FLOPs are also predictive of accuracy. In the paper they audit many popular proxies and show that controlling for parameter count and FLOPS can change proxy rankings and reverse some associations. The authors remove the effects of model size and computation from both proxy scores and accuracy and then measure how strongly the remaining parts are related. They use this size-adjusted relationship to automatically search for new proxy formulas. The resulting proxy combines each layer’s activation diversity with its largest gradient value. On five cell-based NDS search spaces the new proxy outperforms the strongest baselines after controlling for model size and computation and it usually selects better architectures in simulations. The authors support these results with statistical tests, comparisons on shared architectures and checks across random seeds. The proxy does perform poorly relative to existing baselines on the tested ResNet and ResNeXt search spaces though.

Critical strengths

I think the paper identifies an important weakness in common zero-cost NAS evaluation namely that raw proxy–accuracy correlation can largely reflect parameter count and FLOPs. The authors show that controlling for these factors can reorder proxy rankings and reverse some associations.

The empirical analysis is done solid. The authors compare methods on shared architectures, reselect the strongest baseline within bootstrap samples, correct for multiple comparisons, test stability across random seeds and evaluate architecture selection separately from correlation.

Critical weaknesses

Based on the appendix the proposed proxy does not really generalize beyond the cell-based spaces used for the main evaluation. It is worse than existing baselines on the tested ResNet and ResNeXt families and it seems that its advantage is inconsistent under nonlinear controls. This limits the broader impact of the main result.

The proxy is undefined for many architectures and these architectures tend to have lower accuracy. The main correlation analysis excludes them while the selection experiments rank them as worst, so part of the reported benefit comes from informative missingness rather than the proxy’s continuous ranking I fear.

The paper does not discuss the following core related work:

White et al., 2021: https://proceedings.neurips.cc/paper_files/paper/2021/file/ef575e8837d065a1683c022d2077d342-Paper.pdf

Bhardwaj et al., 2023: https://openaccess.thecvf.com/content/ICCV2023W/RCV/papers/Bhardwaj_ZiCo-BC_A_Bias_Corrected_Zero-Shot_NAS_for_Vision_Tasks_ICCVW_2023_paper.pdf

Lai et al., 2026: https://arxiv.org/abs/2608.21927

Questions for authors

How broadly do the authors expect the proxy to work? It performs much worse than existing baselines on all four tested ResNet and ResNeXt families which suggests that it may be specific to cell-based search spaces?

Why do the authors evaluate the version of the proxy that can produce undefined scores rather than the numerically safe version used during discovery?

Why do the authors mainly use a linear method to remove the effects of parameters and FLOPs?

Reviewer_UrAKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new Zero-Cost Proxy to address the issue that the performance of existing Zero-Cost Proxies is often influenced by parameter count and FLOPs. The authors first systematically analyze existing Zero-Cost Proxies and then select a metric ϕfrz\phi_{frz} that combines activation entropy and maximum gradient magnitude to evaluate candidate network architectures. Extensive experiments on different benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper investigates a meaningful problem in zero-cost NAS proxy evaluation.
  • The experimental analysis is comprehensive.
  • The authors provide detailed statistical analyses.
Critical weaknesses
  • Limited generalization across architecture families.
  • The clarity and readability of the paper could be further improved.
Questions for authors
  • Why can the combination of activation entropy and gradient magnitude in ϕfrz\phi_{frz} more effectively reflect network architecture quality? The authors should provide further explanation.
  • Is controlling only for parameter count and FLOPs sufficient to demonstrate the superior ability of the proposed proxy? Should model depth, width, and the number of residual connections, also be considered?
  • The generalization ability of the proposed method appears to be limited. Comparing Tables 2 and 28, the proposed proxy seems to perform poorly on residual network families, raising concerns about its applicability across different model architectures.
  • The paper is difficult to follow, as some important details are mainly presented in the appendix. The authors should reorgnize the paper and provide clearer explanations of the key technical details in the main text.
Reviewer_2eEjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that several popular NAS proxies highly correlate with parameter count and flops, but fail to correlate with performance once these quantities are controlled for. It runs an evolutionary search that identifies a proxy, called ϕfrz\phi_{frz}, which evaluates a ratio of activation entropy to gradient magnitude and has predictive power orthogonal to parameter count and flops.

Critical strengths
  • Figure 2 provides a nice illustration of how different metrics used for architecture selection lose much of their predictive power after controlling for model size and flops
  • The strategy of training ϕfrz\phi_{frz} on one class of NAS benchmarks and then evaluating it on a different family of benchmarks gives more confidence in the results, especially as the candidates had to be selected prior to evaluation on the heldout set.
  • The paper conducts a detailed analysis into the best estimator found by the evolutionary search process, and giving insight into the mechanisms by which it worked (notably depending heavily on the number of layers)
Critical weaknesses
  • The main figure contains AI-generation artifacts in the held-out evaluation datapoint alignment (.369 is closer to 0.402 than 0.366).
  • One concern I have with the projection methodology is that many NAS proxies compute a sum of some quantity over parameters, and so have very high colinearity with parameter count. It seems hard to say that they are ‘only’ capturing parameter count as a useful predictive signal, but if the non-parameter-count signal they are capturing is hard to linearly disentangle from ∣θ∣|\theta| I could imagine the residual metric would underestimate their predictive power.
  • “Controlled-fitness champions have higher mean absolute residual association on the five cell-based NDS spaces than unadjusted-fitness champions, using the common-finite support for each pair” This is roughly what would be expected, no? Training one method to maximize size-adjusted score and another to maximize unadjusted score should result in the size-adjusted method being better per the size-adjusted score. How significant is the distribution shift between these
  • The paper contains another common artifact of AI-generated scientific writing, which is to list the number of experimental conditions in which the method outperforms some baseline, which contributes to text complexity without adding any insight (lines 083-084).
  • The description of the experiment protocol makes it sound like some automated research agent conducted the search over proxies, where the hash was intended to ensure that it did not accidentally access the test set beforehand. I cannot think of another reason to encrypt a publicly available dataset that the authors of this paper presumably had to have known and likely had trained or evaluated other models on beforehand. However, the AI use description is vague and doesn’t explain the exact role this automated researcher may have played or why else the dataset needed to be encrypted before running evals.
  • One take-away that could have been impactful but that I didn’t see explored in the paper is whether the correlation between NAS proxies and parameter count means that these proxies implicitly guide search towards higher-than-necessary parameter counts. This would have provided stronger support for the idea that these methods pick up on a spurious association between size and performance at the cost of more useful and parameter-efficient architectural features.
  • Conclusions from empirical results count the number of settings where one method outperforms another, but don’t provide concise, readable conclusions that give insight into what the numbers actually mean.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.