- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors argue that common zero-cost NAS evaluation conflate proxy quality with model size because parameter count and FLOPs are also predictive of accuracy. In the paper they audit many popular proxies and show that controlling for parameter count and FLOPS can change proxy rankings and reverse some associations. The authors remove the effects of model size and computation from both proxy scores and accuracy and then measure how strongly the remaining parts are related. They use this size-adjusted relationship to automatically search for new proxy formulas. The resulting proxy combines each layer’s activation diversity with its largest gradient value. On five cell-based NDS search spaces the new proxy outperforms the strongest baselines after controlling for model size and computation and it usually selects better architectures in simulations. The authors support these results with statistical tests, comparisons on shared architectures and checks across random seeds. The proxy does perform poorly relative to existing baselines on the tested ResNet and ResNeXt search spaces though.
Critical strengths
I think the paper identifies an important weakness in common zero-cost NAS evaluation namely that raw proxy–accuracy correlation can largely reflect parameter count and FLOPs. The authors show that controlling for these factors can reorder proxy rankings and reverse some associations.
The empirical analysis is done solid. The authors compare methods on shared architectures, reselect the strongest baseline within bootstrap samples, correct for multiple comparisons, test stability across random seeds and evaluate architecture selection separately from correlation.
Critical weaknesses
Based on the appendix the proposed proxy does not really generalize beyond the cell-based spaces used for the main evaluation. It is worse than existing baselines on the tested ResNet and ResNeXt families and it seems that its advantage is inconsistent under nonlinear controls. This limits the broader impact of the main result.
The proxy is undefined for many architectures and these architectures tend to have lower accuracy. The main correlation analysis excludes them while the selection experiments rank them as worst, so part of the reported benefit comes from informative missingness rather than the proxy’s continuous ranking I fear.
The paper does not discuss the following core related work:
White et al., 2021: https://proceedings.neurips.cc/paper_files/paper/2021/file/ef575e8837d065a1683c022d2077d342-Paper.pdf
Bhardwaj et al., 2023: https://openaccess.thecvf.com/content/ICCV2023W/RCV/papers/Bhardwaj_ZiCo-BC_A_Bias_Corrected_Zero-Shot_NAS_for_Vision_Tasks_ICCVW_2023_paper.pdf
Lai et al., 2026: https://arxiv.org/abs/2608.21927
Questions for authors
How broadly do the authors expect the proxy to work? It performs much worse than existing baselines on all four tested ResNet and ResNeXt families which suggests that it may be specific to cell-based search spaces?
Why do the authors evaluate the version of the proxy that can produce undefined scores rather than the numerically safe version used during discovery?
Why do the authors mainly use a linear method to remove the effects of parameters and FLOPs?