- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ZAPS combines zero-cost architecture proxies with active surrogate-based search. Its pipeline selects a non-redundant proxy subset with ProxyFit, uses a hybrid K-means initialization, reselects proxies online with bootstrap votes, and fits an XGBoost ensemble to proxy ranks and architecture encodings for UCB acquisition. On NAS-Bench-201, it reports improved top-100 recovery under a 200-evaluation budget across CIFAR-10 and CIFAR-100. It also evaluates proxy transfer, component ablations, and NAS-Bench-101.
Critical strengths
- The empirical study is unusually thorough for a low-budget NAS method. It uses many random seeds, reports both recovery and regret, discusses practical effect sizes, and separates the contributions of the main search components.
- The analysis of proxy redundancy, initialization, and online proxy selection is useful beyond the specific method. The paper also acknowledges cases where the gains diminish or do not improve the strongest baseline, which makes the presentation more credible and appropriately cautious.
Critical weaknesses
- The novelty framing needs clarification. The paper presents combining proxy scores and architecture encodings in one surrogate as a distinguishing contribution. However, NAS-Bench-Suite-Zero already evaluates XGBoost predictors using proxy values together with architecture encodings. The authors should distinguish ZAPS from this prior setup and compare directly. The stronger novelty may lie in ProxyFit, initialization, and active selection.
- The NAS-Bench-101 comparison omits Local Search and BANANAS because of difficulties defining mutations or neighborhoods for variable-topology graphs. This leaves an incomplete comparison on the harder space. ProxyFit also uses ground-truth accuracies across the full space for offline calibration, so transferred-proxy results do not fully replace a target-label-free comparison. A baseline that concatenates all proxy features with the architecture encoding would help isolate whether ProxyFit and the active loop provide gains beyond straightforward feature aggregation.
Questions for authors
- How does ZAPS compare against an XGBoost surrogate that concatenates all proxy features with the architecture encoding, as in earlier proxy-augmented predictor work? Does ProxyFit and the active loop still provide a clear gain?
- Can the authors provide a principled NAS-Bench-101 comparison for BANANAS or Local Search, or additional evidence that the ordering is not driven by baseline and projection choices?
- How does ProxyFit's full-space ground-truth calibration affect the interpretation of the transferred-proxy and target-label-free experiments?