acceptodds
Under review as a conference paper at ICLR 2027

NICE-CV: Neural Inference- and Cost-Efficient architecture search for Computer Vision models with online proxy calibration

Abstract

Efficient neural architecture search (NAS) has two coupled objectives: the discovered network must maintain strong performance while achieving low inference latency, and the search must cost training time selectively. Zero-cost proxies reduce search cost, but their reliability varies across tasks and training fidelities, while fixed proxy combinations cannot adapt to new validation evidence. Therefore, we introduce NICE-CV, a multi-fidelity NAS method for inference- and search-efficient architecture discovery. To the best of our knowledge, NICE-CV is the first multi-fidelity NAS method to treat zero-cost proxy reliability as an online search state, continuously updating it from purchased validation evidence to guide architecture–fidelity decisions that jointly account for measured inference latency and wall-clock search cost. Concretely, NICE-CV converts heterogeneous proxy scores into comparable ranks, updates proxy reliability from pairwise validation comparisons, and combines the calibrated proxy evidence with cross-fidelity observations to guide subsequent search decisions. Each legal start or promotion is scored by posterior expected hypervolume improvement per predicted search second, while measured inference latency is incorporated directly into the search objective. The resulting search is invariant to monotone transformations of proxy scores, and its online reliability learner enjoys a no-regret guarantee relative to the best fixed proxy in hindsight. Across five complete tabular NAS benchmark families, NICE-CV achieves the highest mean normalized hypervolume ratio (HVR) among executable search methods once online feedback becomes available. On recorded NAS-Bench-201 fidelities, it achieves the highest HVR among the evaluated multi-fidelity methods at 0.9360 while using substantially less search time than full-fidelity baselines. Training-based searches across classification, segmentation, and detection further show that NICE-CV achieves the highest mean HVR on all four tasks; compared with the fastest baseline on each task, it requires only marginal additional search time while consistently recovering higher-quality architecture sets. These results show that continuously adapting inexpensive proxy guidance to task-specific feedback can improve both inference-efficient architecture discovery and wall-clock-efficient search, yielding stronger quality–latency tradeoffs under limited search budgets.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Neural Architecture Search, Hardware-Aware NAS, Efficiency AI

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Ndm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a new NAS method named NICE-CV, which introduces an online rank-calibration mechanism to adapt across tasks and fidelities and develop a unified multi-fidelity(validation performance, inference latency and search cost) architecture–fidelity acquisition rule to discovery high fidelity architecture. Some theoretical analyses are also involved to support the method. NICE-CV are evaluated across five complete tabular NAS benchmark families and four trainingbased computer-vision tasks. Experiment results show that NICE-CV achieves superior performance than other sota NAS methods.

Critical strengths
  1. NICE-CV proposes a new proxy metric, which considers performance, inference cost and search cost all in one. The hybrid metrics make NICE-CV search a lot of high fidelity architecture.
  2. Comprehensive experiments are conducted across various tasks, and search spaces, which support the generalization ability and effectiveness of NICE-CV.
Critical weaknesses
  1. Formulating NAS as a multi-objective search problem has been discussed by pervious work, such as MONAS、DPP-Net. NICE-CV further includes search costs for the search process. The novelty of the problem itself and of the solution is limited.
  2. As for evaluation of search results, I wonder to see more specific results, such as accuracy, inference latency and search cost rather than only the HVR metric.
Questions for authors

From a technical perspective, the multi-fidelity metric is the main contribution, but it does not seem to yield new findings, since multi-dimensional considerations have already been discussed in prior work. Moreover, the theoretical derivation is somewhat difficult to follow; it would be better to provide intuitive explanations and descriptions.

Reviewer_8edVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose NICE-CV, a multi-fidelity NAS method for inference- and search-efficient architecture discovery. Across five complete tabular NAS benchmark families, NICE-CV achieves the highest mean normalized hypervolume ratio (HVR) performance. The resulting search is invariant to monotone transformations of proxy scores.

Critical strengths
  1. A new method for multi-fidelity NAS with zero-shot proxies is proposed.
  2. Comprehensive empirical evaluation involves five tabular NAS benchmarks.
Critical weaknesses
  1. In introduction, authors write "As a result, accuracy-first search followed by latency filtering can spend substantial search budget on architectures that are ultimately unattractive for deployment". But alternatives exist, for example, optimizing linear combination of accuracy and latency, e.g. : Cai, H., Zhu, L., & Han, S. (2018). Proxylessnas: Direct neural architecture search on target task and hardware.
  2. Some notation is not defined, eg. Eq 2, 3.
  3. The paper is sometimes hard to read. For example, Section 6.2 is a mind-numbing listing of numbers. Please consider more high-level presentation of experimental results instead of diving into details.
  4. NAS-Bench-201 includes several methods like ENAS, DARTS, GDAS, RE, etc. why you don't compare with them? Standard NAS methods can be easily extended to accuracy-latency optimization by considering a linear combination of targets.
  5. The performance is mixed. The proposed method is not the best one in Tables 3, 4.
Questions for authors
  1. What is \Delta^{J-1}, L_{d,f}, E_{static} in Eq. 2 ? I am surprised to that Eq. 2 uses lots of undefined notations.
  2. The notation HV() is used in Eq. 3 before its formal definition.
  3. Why the HVR metrics is optimized instead of a standard accuracy, or a accuracy vs. latency trade-off (e.g. accuracy for a given latency constraint)?
  4. How your paper is related to:

Trofimov, I., Klyuchnikov, N., Salnikov, M., Filippov, A., & Burnaev, E. (2023). Multi-fidelity neural architecture search with knowledge distillation. IEEE Access, 11, 59217-59225.

  1. While Authors include Table 5 with a detailed list of evidence artifacts supporting the experimental results, the source code is not uploaded with the paper.
  2. You claim that "NAS method treats zero-cost proxy reliability as an online search state". Does it mean that you train a predictor for architecture performance with zero-cost proxies as a feature? The use of the term "online search state" is not clear.
  3. What is "static" proxy ranking? What does "online feedback" mean?
Reviewer_oTikOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates online proxy calibration based efficient neural architecture search (NAS) approaches for computer vision tasks.

Critical strengths

NAS approaches traditionally suffers from a notable disadvantages of performance loss due to disjointed stages of architecture candidates selection and their respectively parameters update, in which the correlation with the “ground truth” architecture candidates ranking order obtained using cross validation tends to be in practice weak.

Critical weaknesses

Very limited technical novelty: many aspects of the proposed techniques of this submission in relation to well established prior researches:, e.g., efficient proxylessNAS (Cai et al. ICLR2019), BayesNAS (Zhou et al. ICML2019), present limited novelty.

Lack of scientific controlled experimental design: Authors need to crystal clear in terms of what are the key performance metrics to be used when accessing performance efficiency trade-off tasks, e.g., the single, and vaguely defined HVR metric presents a thin and mixed gruel of what is desired. There is no performance and efficiency analysis against the ground truth, upper bound NAS system using explicit held-out data based selection. I would like also to see more detailed, separate analysis of performance, NAS training time, and the resulting systems’ inference time on test sets over different model complexity operating points.

Lack of presentation clarity: The paper is badly written. In particular, there is no discussion of how it is positioned against other related prior works. The use of math notations is inconsistent and messy, whilst not always being clearly defined before their first use. This not only degrades the readability of the paper overall, but more importantly obscures the technical contribution, if any, that it seeks to claim.

Questions for authors

Please address the above issues for possible future submissions.

Reviewer_gYZaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors are proposing a multi-fidelity neural architecture search method that trades off task quality against measured inference latency under a search budget. The core idea is to treat the reliability of training free proxies as a search state that is updated online from the validation evidence that we gain via the search itself. Each of the proxies is an expert whose weight is updated with exponential weights using a disagreement between the proxy's ranking and observed validation outcomes. Experiments cover tabular benchmarks, a multi-fidelity NAS-Bench-201 study against full-fidelity and successive-halving baselines, and training-based search over a MobileNet-like space on small datasets.

Critical strengths

The paper tackles a real problem: training-free proxies are unreliable, and how well they rank architectures varies across tasks and fidelities.

Critical weaknesses
  1. Novelty may be overstated. ProxyBO (Shen et al., AAAI 2023) already re-estimates each proxy's reliability during the search, from its rank agreement with the architectures evaluated so far, and weights the proxies accordingly. The paper does not cite it and should explain how its approach differs. The theory adds little. Proposition 1 follows immediately from the method using only ranks, and no later result relies on it. Theorem 1 and its proof (App. A.2) restate a standard result, the regret bound for exponential weights (e.g., Theorem 2.2 in Cesa-Bianchi & Lugosi, Prediction, Learning, and Games, Cambridge University Press, 2006), which the paper does not cite either. Corollary 1 follows by exponentiating the second equation in (10).

  2. The gains are not consistently significant. As Sec. 6.2 itself states, the confidence intervals for the gain over Early-SH include zero on every task except Tiny-ImageNet. The conclusion nevertheless claims that NICE-CV "consistently improves quality–latency frontier coverage".

  3. The experiments are small-scale. The tabular benchmarks have been studied extensively in prior work, and the training-based tasks are small: CIFAR-100, Tiny-ImageNet, Oxford-IIIT Pet and Penn-Fudan (170 images), with complete searches taking about 2 to 29 minutes (Table 4). A large-scale dataset such as ImageNet-1k should be part of the study, to show how the search cost scales. Appendix B gives the complexity of the search controller, but the dominant cost is training each candidate at each fidelity.

  4. The presentation is weak, particularly the mathematics in Section 3. ◦ Lines 150–155: the loss L_{d,f} in (2) is never defined, nor is the distribution over tasks and fidelities, so it is unclear how a fixed proxy combination affects the quality measured in (2). As written, (2) only defines a gap that is non-negative by construction; the paper does not show when, or by how much, it is positive. ◦ The terminal quality–cost efficient set P⋆ is introduced on line 160 but does not appear in (3), which as written only states that hypervolume is monotone under set inclusion. ◦ There is also a numerical slip: Sec. 6.2 says NICE-CV "leads at 0.7248" on Penn-Fudan, while Table 4 gives 0.7724.

Questions for authors
  1. What is the "separately calibrated prior"? On what data is it calibrated, is that cost charged to search time, and is it used in the tabular and NAS-Bench-201 experiments?
  2. How were AZ-NAS, MF-NAS and R-MF-NAS adapted to the latency objective, if at all? Appendix C.5 mentions no latency term for any of them.
  3. Appendix C.2 names the tabular proxies, but which proxies are the experts on the real tasks? If J = 2 there, reliability learning reduces to a single mixing weight.
  4. Sec. 5 lists individual proxies and an oracle best single proxy among the tabular comparisons, but Table 2 omits both. How does NICE-CV compare with the oracle?
  5. How does the method scale, at least in principle, to large models and search spaces such as ImageNet-1k, given that each candidate has to be trained at several fidelities?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.