- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates deployment-aware neural architecture selection, where an architecture is evaluated based on the expected risk of an ensemble formed by independently training (M) instances and uniformly averaging their predictions, rather than on single-model performance alone. Under homogeneous regeneration and squared loss, Theorem 3.2 decomposes the expected deployment risk into terms involving single-model risk, predictive variance, and within-architecture correlation. Building on this result, the authors derive a population-level acceptance criterion and a finite-sample screening rule under specified validation-noise and proxy-stability bounds. The paper further examines heterogeneous augmentation, highlighting the importance of candidate-to-system error alignment. Experiments on Criteo-1M, Avazu-1M, and MovieLens-1M show that the proposed reduced objective achieves Spearman rank correlations of 0.974–0.997 with empirical deployment risk, compared with 0.719–0.872 for single-run validation. The confidence-adjusted screening rule reduces the observed false acceptance rate from 26.4% to 1.8% on Criteo-1M and from 14.8% to 0.4% on Avazu-1M, using (K=3) proxy training runs per candidate.
Critical strengths
- Well-defined deployment-aware objective and transparent computational accounting. The work identifies a meaningful mismatch between single-run NAS validation and M-instance ensemble deployment, and distinguishes O(NK) screening from O(NM) full per-candidate reconstruction and O(AM) post-acceptance rebuilding (Appendix K.11). This makes the proposed efficiency target precise.
- Coherent theory-to-experiment structure with explicit scope limits. Theorem 3.2 offers a compact population characterization, the screening formulation separates validation noise from proxy uncertainty, and the heterogeneous-augmentation analysis identifies a meaningful boundary. Ranking, FAR/recall, and proxy-count studies provide relevant empirical evidence; Appendix L appropriately acknowledges limitations.
Critical weaknesses
- The claimed contribution is not sufficiently distinguished from classical ensemble theory. Appendix B derives Theorem 3.2 using a classical squared-loss ambiguity decomposition, while Related Work characterizes the contribution as “one level above.” The paper needs to distinguish inherited identities from genuinely new architecture-selection and finite-sample results, with a focused comparison to ensemble-aware NAS and conservative screening. This matters to whether the work establishes a substantive new methodological contribution rather than primarily reframing a known identity (Sections 2 and 3.1; Appendix B).
- Theorem 3.5 provides a high-probability screening guarantee conditional on valid proxy-stability bounds. However, the paper does not sufficiently explain how these bounds are constructed or calibrated for (K=3) independently trained proxy models. Matching the proxy and deployment training protocols does not, by itself, establish that the resulting bounds adequately capture proxy-estimation uncertainty. Moreover, the reported FAR and recall results do not directly validate the nominal coverage of these bounds. Beyond screening reliability, the evidence for improved end-to-end architecture selection is also limited. Although the proposed objective achieves high rank correlation with empirical deployment risk, Table 4 reports only marginal improvements in final deployment risk over the Low-fidelity baseline (approximately (1×10-5) on each CTR dataset), which are small relative to the reported run-to-run variability. Without paired confidence intervals or significance tests, it remains unclear whether the ranking improvements translate into statistically reliable or practically meaningful end-to-end gains (Theorem 3.5; Appendix C/K.5; Tables 1 and 4; Appendix K.7).
Questions for authors
- Which statements in Theorem 3.2 and Corollary 3.3 go beyond taking expectations in the classical squared-loss ambiguity decomposition? Please identify the closest prior ensemble-aware NAS or safe-screening alternatives and the exact increment of the proposed framework.
- How are the proxy-stability radii b_E, b_σ, and b_ρ determined for the reported K=3 experiments? Can the authors provide independent proxy-seed diagnostics of statistic error and event-matched empirical coverage, together with ranking variability across seed sets and K? Empirical FAR should not be equated directly with the theorem's δ. Please also clarify how K is selected and whether any calibration cost is included.
- Using matched proposal streams, can the authors report paired per-seed differences and 95% confidence intervals for final deployment risk between Exact safe and the Low-fidelity baseline, which have the same screening budget? A paired uncertainty analysis of harmful-accept counts would also be helpful. Given that the observed absolute risk improvement is only approximately (10^{-5}) on both CTR datasets, please clarify its statistical reliability and practical significance.
- Appendix L.1/L.6 appropriately acknowledge that the experiments are limited to Criteo-1M, Avazu-1M, MovieLens-1M and relatively restricted architecture families. However, the population-level identity being architecture-independent under its mathematical assumptions does not establish that fixed-K statistical estimation and safe screening remain reliable for other model families, training variability, or larger NAS search spaces. Which practical workflows genuinely follow Assumption 3.1 (independently regenerated homogeneous models with uniform prediction averaging)? Shared-trajectory checkpoints and parameter-sharing ensembles do not automatically qualify. Please clarify the representative deployment use cases and, if feasible, evaluate a substantially different architecture family or training regime with matched proxy budgets; otherwise explicitly limit empirical generalization claims to the tested settings. Also delimit the sequential guarantee when adaptive proposals reuse the same validation data.
2 comments
Thanks for the detailed reply. My score keeps unchanged.
Thank you again for your comments on our submission. As the discussion period ends soon, we would be grateful if you could take a look and let us know whether it addresses your concerns. We are happy to answer any further questions.