Auditing Size Dependence in Zero-Cost NAS Proxies: Confound-Controlled Discovery and Held-Out Evaluation
Abstract
Zero-cost proxy evaluation and automated discovery rely on score-accuracy correlation, but this correlation does not distinguish size-related association from association remaining after size control. In our audit, log parameter count has the largest absolute unadjusted Kendall correlation among 15 baselines in four of the original five cell-based search spaces from Network Design Spaces (NDS), using the finite support of each score. This strong size baseline makes it important to establish what predictive information a proxy retains after size control. Our audit of 24 scores shows that joint parameter-count and FLOPs controls change rankings and reveal obscured associations. We therefore use absolute residual association as the fitness objective for genetic programming (GP), yielding three candidates. The formulas, signs, and control analyses of these three candidates were frozen before NDS access. We designated the focal score and registered the 15-baseline comparison after opening. On architectures where all compared methods produce finite scores in these five spaces, exceeds the strongest eligible baseline in absolute association under residual and stratified controls. All ten margin confidence intervals remain positive after adjustment for the choice among the three candidates. Separately, a post-hoc comparison of 100 matched GP pairs finds higher mean absolute residual association on the five NDS spaces under controlled than unadjusted fitness. This association advantage alone does not establish better selection. Separate retrospective simulations using stored benchmark accuracies on these populations yield positive mean selected-accuracy gains over the strongest of the 15 baselines in 49 of 50 conditions. A post-hoc Bonferroni analysis retains 41 positive gain intervals. The evidence supports controlled discovery of stronger residual associations and, separately, selection gains within the evaluated scope.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.