- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose a new NAS method named NICE-CV, which introduces an online rank-calibration mechanism to adapt across tasks and fidelities and develop a unified multi-fidelity(validation performance, inference latency and search cost) architecture–fidelity acquisition rule to discovery high fidelity architecture. Some theoretical analyses are also involved to support the method. NICE-CV are evaluated across five complete tabular NAS benchmark families and four trainingbased computer-vision tasks. Experiment results show that NICE-CV achieves superior performance than other sota NAS methods.
Critical strengths
- NICE-CV proposes a new proxy metric, which considers performance, inference cost and search cost all in one. The hybrid metrics make NICE-CV search a lot of high fidelity architecture.
- Comprehensive experiments are conducted across various tasks, and search spaces, which support the generalization ability and effectiveness of NICE-CV.
Critical weaknesses
- Formulating NAS as a multi-objective search problem has been discussed by pervious work, such as MONAS、DPP-Net. NICE-CV further includes search costs for the search process. The novelty of the problem itself and of the solution is limited.
- As for evaluation of search results, I wonder to see more specific results, such as accuracy, inference latency and search cost rather than only the HVR metric.
Questions for authors
From a technical perspective, the multi-fidelity metric is the main contribution, but it does not seem to yield new findings, since multi-dimensional considerations have already been discussed in prior work. Moreover, the theoretical derivation is somewhat difficult to follow; it would be better to provide intuitive explanations and descriptions.