acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Architecture Search Cost from Deployment Multiplicity

Abstract

Neural architecture search typically ranks candidates by the validation score of a single trained model, while some deployed systems use a multi-instantiation protocol in which an architecture is trained independently times and predictions are averaged. The risk of the deployed object is then governed jointly by single-model accuracy, predictive variance, and the correlation between replicas, and rebuilding the full -replica protocol for every proposal makes screening cost scale with . Under homogeneous regeneration and squared loss, we prove an exact reduction: deployment risk equals , so deployment-level ranking is determined by three architecture-level statistics that can be estimated from a fixed number of proxy instantiations, independently of the target . We then lift this population identity to finite validation data, deriving screening rules that control harmful acceptance with high probability when the stated validation-noise and proxy-stability bounds hold. We also delimit the result: under heterogeneous augmentation, whether a candidate helps depends on its error alignment with the deployed system and cannot be decided from candidate-only statistics. On Criteo-1M, Avazu-1M, and MovieLens-1M, the reduced objective attains Spearman – against measured deployment risk, versus – for single-run validation. Confidence-adjusted screening reduces the observed false acceptance rate from to on Criteo-1M and admits no harmful updates in the reported end-to-end runs, while requiring a fixed number of proxy trainings per candidate rather than full -replica reconstruction.

Primary area:
learning theory
Keywords:
cost-efficient NAS, risk decomposition, architecture ranking

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_n3R3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates deployment-aware neural architecture selection, where an architecture is evaluated based on the expected risk of an ensemble formed by independently training (M) instances and uniformly averaging their predictions, rather than on single-model performance alone. Under homogeneous regeneration and squared loss, Theorem 3.2 decomposes the expected deployment risk into terms involving single-model risk, predictive variance, and within-architecture correlation. Building on this result, the authors derive a population-level acceptance criterion and a finite-sample screening rule under specified validation-noise and proxy-stability bounds. The paper further examines heterogeneous augmentation, highlighting the importance of candidate-to-system error alignment. Experiments on Criteo-1M, Avazu-1M, and MovieLens-1M show that the proposed reduced objective achieves Spearman rank correlations of 0.974–0.997 with empirical deployment risk, compared with 0.719–0.872 for single-run validation. The confidence-adjusted screening rule reduces the observed false acceptance rate from 26.4% to 1.8% on Criteo-1M and from 14.8% to 0.4% on Avazu-1M, using (K=3) proxy training runs per candidate.

Critical strengths
  1. Well-defined deployment-aware objective and transparent computational accounting. The work identifies a meaningful mismatch between single-run NAS validation and M-instance ensemble deployment, and distinguishes O(NK) screening from O(NM) full per-candidate reconstruction and O(AM) post-acceptance rebuilding (Appendix K.11). This makes the proposed efficiency target precise.
  2. Coherent theory-to-experiment structure with explicit scope limits. Theorem 3.2 offers a compact population characterization, the screening formulation separates validation noise from proxy uncertainty, and the heterogeneous-augmentation analysis identifies a meaningful boundary. Ranking, FAR/recall, and proxy-count studies provide relevant empirical evidence; Appendix L appropriately acknowledges limitations.
Critical weaknesses
  1. The claimed contribution is not sufficiently distinguished from classical ensemble theory. Appendix B derives Theorem 3.2 using a classical squared-loss ambiguity decomposition, while Related Work characterizes the contribution as “one level above.” The paper needs to distinguish inherited identities from genuinely new architecture-selection and finite-sample results, with a focused comparison to ensemble-aware NAS and conservative screening. This matters to whether the work establishes a substantive new methodological contribution rather than primarily reframing a known identity (Sections 2 and 3.1; Appendix B).
  2. Theorem 3.5 provides a high-probability screening guarantee conditional on valid proxy-stability bounds. However, the paper does not sufficiently explain how these bounds are constructed or calibrated for (K=3) independently trained proxy models. Matching the proxy and deployment training protocols does not, by itself, establish that the resulting bounds adequately capture proxy-estimation uncertainty. Moreover, the reported FAR and recall results do not directly validate the nominal coverage of these bounds. Beyond screening reliability, the evidence for improved end-to-end architecture selection is also limited. Although the proposed objective achieves high rank correlation with empirical deployment risk, Table 4 reports only marginal improvements in final deployment risk over the Low-fidelity baseline (approximately (1×10-5) on each CTR dataset), which are small relative to the reported run-to-run variability. Without paired confidence intervals or significance tests, it remains unclear whether the ranking improvements translate into statistically reliable or practically meaningful end-to-end gains (Theorem 3.5; Appendix C/K.5; Tables 1 and 4; Appendix K.7).
Questions for authors
  1. Which statements in Theorem 3.2 and Corollary 3.3 go beyond taking expectations in the classical squared-loss ambiguity decomposition? Please identify the closest prior ensemble-aware NAS or safe-screening alternatives and the exact increment of the proposed framework.
  2. How are the proxy-stability radii b_E, b_σ, and b_ρ determined for the reported K=3 experiments? Can the authors provide independent proxy-seed diagnostics of statistic error and event-matched empirical coverage, together with ranking variability across seed sets and K? Empirical FAR should not be equated directly with the theorem's δ. Please also clarify how K is selected and whether any calibration cost is included.
  3. Using matched proposal streams, can the authors report paired per-seed differences and 95% confidence intervals for final deployment risk between Exact safe and the Low-fidelity baseline, which have the same screening budget? A paired uncertainty analysis of harmful-accept counts would also be helpful. Given that the observed absolute risk improvement is only approximately (10^{-5}) on both CTR datasets, please clarify its statistical reliability and practical significance.
  4. Appendix L.1/L.6 appropriately acknowledge that the experiments are limited to Criteo-1M, Avazu-1M, MovieLens-1M and relatively restricted architecture families. However, the population-level identity being architecture-independent under its mathematical assumptions does not establish that fixed-K statistical estimation and safe screening remain reliable for other model families, training variability, or larger NAS search spaces. Which practical workflows genuinely follow Assumption 3.1 (independently regenerated homogeneous models with uniform prediction averaging)? Shared-trajectory checkpoints and parameter-sharing ensembles do not automatically qualify. Please clarify the representative deployment use cases and, if feasible, evaluate a substantially different architecture family or training regime with matched proxy budgets; otherwise explicitly limit empirical generalization claims to the tested settings. Also delimit the sequential guarantee when adaptive proposals reuse the same validation data.
2 comments

Thank you again for your comments on our submission. As the discussion period ends soon, we would be grateful if you could take a look and let us know whether it addresses your concerns. We are happy to answer any further questions.

Thanks for the detailed reply. My score keeps unchanged.

Reviewer_r8siOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the neural architecture selection problem under ensemble deployment scenarios. The authors propose a simplified evaluation objective based on predictive variance and inter-model correlation, and design finite-sample safe screening rules to reduce architecture search costs for multi-model deployment. Experiments conducted on three datasets show that the proposed method achieves high architecture-ranking consistency. However, its substance belongs to model selection and ensemble evaluation rather than traditional NAS, showing a slight disconnect, and its theoretical boundaries and practical performance exhibit limitations under cross-entropy and other metrics.

Critical strengths
  1. The research problem is practically meaningful, focusing on the trade-off between architecture selection and deployment cost in multi-instantiation ensemble deployment.
  2. The theoretical analysis is clear, providing a deployment risk decomposition and a safe screening method that effectively reduces search evaluation costs.
  3. Empirical validations across multiple datasets demonstrate favorable architecture-ranking results.
Critical weaknesses
  1. The method relies heavily on the proxy-model stability assumption, and its applicability in real-world scenarios requires further validation.
  2. The experiments cover only a limited architecture space, and the method's performance in larger-scale searches remains unclear.
  3. The comparison of search costs against existing low-cost NAS methods is insufficiently thorough.
  4. The theory is primarily tailored for squared loss and i.i.d. model instances, showing limited applicability to other loss functions and heterogeneous ensembles.
  5. The evaluation relies solely on sequential screening over a pre-established candidate pool of approximately 100 architectures without involving large-scale topological search spaces or active search strategies; thus, it is substantially closer to model selection rather than traditional NAS.
Questions for authors
  1. How are proxy-stability bounds estimated and validated?
  2. How does the method compare with existing NAS methods under equal compute budgets?
  3. Can this method be scaled to larger architecture search spaces?
Reviewer_fX1nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a proxy scoring and accept/reject rule for architecture candidates in Neural architecture search.

The NAS objective considered is not only the accuracy of the optimized resulting architecture, but the "ensemble accuracy" of diverse candidates that are typically desired in production setups related to ads click through prediction where diversity is also important.

Given N candidates, and M trained versions of each candidate for a total of NxM candidates, they consider (K < M) proxy versions.

The evaluations are conducted on CTR datasets (Criteo-1M and AVAZU-1M) and MovieLens-1M

Critical strengths
  • The paper introduces an interesting problem formulation of optimizing M-seed averaged performance.
  • the proxy scoring rule's formulation looks sound.
Critical weaknesses
  • The paper doesn't compare against any existing NAS method on the considered datasets.
  • Conversely, I would also like to see if the proposed criterion can improve existing NAS performance on standard datasets.
Questions for authors
  • Can the authors pair the acceptance rule with an existing NAS algorithm on Criteo / Azavu instead of only evaluating 40 random permutations?

  • Does the criterion help improve performance for standard NAS benchmarks as well such as NAS-Bench-201?

1 comment

Thank you again for your comments on our submission. As the discussion period ends soon, we would be grateful if you could take a look and let us know whether it addresses your concerns. We are happy to answer any further questions.

Reviewer_nKbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The current paper addresses the problem of selecting an appropriate neural architecture for deployment where M different instances of a single architecture are independently trained. Assuming the same regeneration condition and the squared prediction loss, the deployment risk is split into three terms: single-model error, variance, and cross-instance correlation. Next, the problem of selecting the most suitable architecture is simplified using the estimation of the former two via K proxy trainings. The screening rule is derived under the assumptions of the validation noise and proxy stability. The theorem proving that heterogeneous ensemble improvement requires aligned errors between the candidate and system architectures is provided. The empirical study is conducted on three datasets (Criteo-1M, Avazu-1M, and MovieLens-1M) in order to evaluate ranking accuracy and false acceptance compared with different screening baselines.

Critical strengths
  1. This study looks into a particular example of system instantiation where it becomes clearly seen that validating only once may not be enough for finding out the architecture that is most suitable. The theoretical basis for the approach is provided by Theorem 3.2 and by the heterogeneous-augmentation criterion in Theorem 3.7, with the structural boundary stated in Proposition 3.8.

  2. The results regarding the evaluation of ranking seem positive. According to Table 1, the correlations between the value of reduced cost function and risk of deployment are in the range of 0.974-0.997 as opposed to 0.719-0.872 found in case of single validation run. As for Table 2, it shows that the results in terms of false acceptance rate are better than those produced by alternative approaches for both data sets.

Critical weaknesses
  1. First of all, the safety guarantee depends on some quantities, which were not defined in the text. Indeed, theorem 3.5 uses bounds on validation noise and proxy-training noise. However, Assumption C.10 (pp. 18–19) introduces the finite-K proxy constants b_E, b_sigma, and b_rho, but gives neither their values nor a calibration procedure. Moreover, even if we make training process totally similar to proxy and deployment process, the problem of the sampling variance still exists if K = 2-8. Thus, one should report the real radius for every experiment and show its coverage without information about real deployment risks. Moreover, Table 8 is non-monotonic in delta: moving from delta = 10^-1 to 10^-2 decreases recall (0.716 to 0.467) and acceptance rate (0.514 to 0.320), while moving from 10^-2 to 10^-3 increases both (0.467 to 0.545; 0.320 to 0.330). The caption does not state whether the three delta settings share the same candidate pairs, proxy realizations, and validation subsamples, nor whether the exact rule or the simplified heuristic is used. If these conditions are shared, a stricter threshold should yield a nested, no-larger acceptance set. The observed non-monotonicity therefore needs to be reconciled before the safety claim can be assessed.

  2. Finally, the performance comparison of the end-to-end process was done at different total costs of the selection procedure. The Table 4 shows 120 trained instances (instead of 40 in case of the single-run validation) when using the proposed screening rule, although differences in the risks are small. The reconstruction cost for M-instance incumbents is not included: the Table 4 caption explicitly states that post-acceptance deployment reconstruction is excluded, while Appendix K.11 accounts for AM reconstruction trainings. The reported cost comparison therefore understates the proposed rule's total cost relative to single-run selection. The authors should compare methods at the same total trained-instance or GPU-hour budget, including reconstruction, and report uncertainty over the ten search seeds. A separate test split and fresh deployment seeds would also help distinguish the theorem's expected-protocol risk from the risk of one realized M-member ensemble measured on validation data that also inform screening.

Questions for authors
  1. What were the exact figures or procedures adopted to derive b_E, b_sigma, b_rho, and the variance floor from the exact safe rule? What proportion of the proxy samples (out of the total of K) does the corresponding confidence interval contain the desired quantities?
  2. Do the three delta levels get examined using the same pairs of candidates, proxies, and validation set in Table 8? Why is there an increase in recall and acceptance rate with stricter confidence level?
  3. When comparing between the two strategies that have similar overall budget allocation to train the model (including the reconstruction cost of the incumbent), does the difference in deployment risk persist for the test data set and new deployment seeds?
  4. Which of these three features other than the decomposition of the squared loss – screening rule, the heterogeneous augmentation boundary, or empirical protocol – the authors see as their contribution, and how do the claims of the authors get affected if any one of them is taken away?
1 comment

Thank you again for your comments on our submission. As the discussion period ends soon, we would be grateful if you could take a look and let us know whether it addresses your concerns. We are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their comments. Reviewers found the problem practically meaningful (r8si, fX1n), the theory clear and coherent (r8si, n3R3), and the ranking results positive (nKbi). Reviewer r8si noted that some concerns have been resolved. Reviewers nKbi, fX1n and n3R3 have not yet replied. We summarize the main concerns and our responses below.

1. Radii and calibration (nKbi, r8si, n3R3). The submitted experiments used asymptotic radii with bq=0b_q=0, so those results were empirical, not certified. We now calibrate bqb_q on half of the architectures and evaluate on the other half. Coverage is 0.95–0.97 for K≥3K\ge3. Theorem 3.5 controls Pr⁡(accept∧harmful)\Pr(\text{accept}\wedge\text{harmful}), not FAR. We now report it separately, and it stays below δ\delta. Table 8 re-run on shared candidate pairs is nested and monotone in δ\delta.

2. Cost and significance (nKbi, r8si, n3R3). With reconstruction included, our method costs about 1.8× single-run, not 3×. Under an equal total budget, on a held-out test split with fresh deployment seeds. The absolute risk gain is small. The main benefit is fewer harmful updates: 0.02 per run on Criteo, versus 0.64 for single-run and 0.87 for low-fidelity.

3. NAS searchers and benchmarks (fX1n, r8si, n3R3). Our rule changes how a proposal is scored, so any searcher can use it. Paired with Random Search, REA, TPE and Successive Halving, it gives the lowest risk under every searcher. On NAS-Bench-201, the reduced objective reaches Spearman 0.968, versus 0.812 for single-run.

4. Revision. We will rewrite Appendix C with the calibrated radii and report Pr⁡(accept∧harmful)\Pr(\text{accept}\wedge\text{harmful}). We will also state the scope limits: squared loss, Assumption 3.1, and pre-specified proposal sequences.

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.