- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the trade-off between accuracy and computational cost in neural architecture search. Since Pareto dominance alone cannot rank architectures with different trade-offs, the authors propose CATOS, which constructs reference boundaries from observed architectures and fits curves to score each architecture by its relative distance to them. The score is used for parent and survival selection. The paper reports results on NATS-Bench and evaluates a version on HW-GPT-Bench.
Critical strengths
Many conventional multi-objective methods select solutions based on dominance relations and the distribution of the current solution set. CATOS instead tries to infer a structural relationship between the objectives from observed solutions and uses it to guide the search. I think this idea is meaningful.
Critical weaknesses
- The novelty of some geometric constructs is unclear. Local ideal points, for example, have been used in earlier studies. The paper could clarify which parts of CATOS are new.
- The paper does not sufficiently discuss the robustness of its power-law fit.
- Selecting by a single score may reduce population diversity. The use of additional coverage terms on HW-GPT-Bench raises the question of whether this is also a concern on NATS-Bench.
- The wall-clock analysis is limited to a comparison with NSGA-II.
Questions for authors
- CATOS uses a single score for parent and survival selection, which may concentrate the population in a narrow region and reduce diversity. On HW-GPT-Bench, the authors add coverage terms to Linear CATOS. Is this intended to address a diversity issue? Could the original CATOS face the same issue on NATS-Bench? Why is a coverage term not needed on NATS-Bench?
- The fitted curves assume a power-law cost–accuracy relationship. If this assumption does not hold, errors in the curves could change how architectures are ranked. How would this affect the final search results?
- Appendix B reports wall-clock time only for CATOS-EA and NSGA-II, with CATOS-EA taking longer. How does it compare with the other baselines? Apart from the implementation differences noted in the paper, what accounts for the additional runtime?
2 comments
Thank you for the response. I will maintain my original score.
Rebuttal
We thank the reviewer for finding CATOS’s corridor-based ranking meaningful for its structural relationship between the objectives.
[W1] The paper could clarify which parts of CATOS are new, since local ideal points have been used in earlier studies.
To the best of our knowledge, local ideal points, the ideal front, the negative Pareto front and the Pareto corridor are all novel in multiobjective optimization (lines 79-80). We assume the reviewer refers to local upper bounds [1]. Local ideal points share a similar construction but are the opposite point for consecutive Pareto points: For consecutive Pareto points yi,yi+1 with y1i<y1i+1 (minimization setting):
Local upper bound: ui:=(y1i+1,y2i)=max{yi,yi+1}
Local positive ideal point: ℓi+:=(y1i,y2i+1)=min{yi,yi+1}
(with max/min taken componentwise)
Since each ui is dominated by both yi and yi+1 and each ℓi+ dominates both, the local positive ideal points form a best-case envelope of the positive front.
If the reviewer meant something else, could a reference be shared?
[1] Klamroth et al. "On the representation of the search region in multi-objective optimization." European Journal of Operational Research 245.3 (2015): 767-778.
[Q3 and W4, part 1] How does CATOS compare in wall-clock time with other baselines except NSGA-II (Appendix B)?
We agree a computationally heavier baseline than NSGA-II would strengthen the comparison, and add NSGA-III with its heavier reference-point selection. We also optimized CATOS, vectorizing the scoring, and reran the wall-clock comparison against both baselines at a much larger scale (~1M architectures). The search runs in the OFA MobileNetV3 space, which we found has the smallest framework overhead, optimizing predicted ImageNet accuracy against Note10 latency (population 300, 80 children per generation), repeated 5 times. All algorithms share every component except parent selection and survival, isolating the optimizer itself.
CATOS-EA is only ~12% slower than NSGA-II and slightly faster than NSGA-III, with negligible overhead per generation. This replaces the original runtime experiment (unoptimized CATOS) in Appendix B of the revised manuscript.
~0.27 ms (new) vs. ~0.6 s per architecture (old), from batching, fast objective retrieval and vectorized scoring.
[Q3 and W4, part 2] Apart from the implementation differences noted in the paper, what accounts for the additional runtime?
The dominating costs in CATOS are curve-fitting and distance calculation (scoring), the latter especially depending on an efficient implementation.
[W3 and Q1] Selecting by a single score may reduce population diversity. For HW-GPT-Bench additional coverage terms are used. Why is a coverage term not needed on NATS-Bench?
We thank the reviewer for pointing this out! We didn't observe improvements in coverage-sensitive metrics (hypervolume regret and ε-indicator) for CATOS-EA on NATS-Bench, when augmenting it with coverage signals. If the scaling prior fits the Pareto front well, each model is scored similarly to its tradeoff, thus selection is not biased towards any region. On HW-GPT-Bench, the denser front and larger budget (Section 4.4) produce many candidates with near-identical scores, so explicit coverage signals are needed to maintain spread.
[Q2] If power-law cost–accuracy does not hold, how would this affect the final search results?
If the power law does not fit the fronts well, closeness to the front is no longer scored uniformly, which can concentrate the search on a subregion of the objective space or flatten scores across models and slow convergence. This is avoided by the linear interpolation fallback, which is assumption-free and adapts to the shape of the Pareto fronts. This is shown by the new result in Figure 2 of the revised document, where linear CATOS outperforms CATOS with scal. prior, both without coverage.
[W2] The paper does not sufficiently discuss the robustness of its power-law fit.
We fully agree robustness deserves a clear treatment, and note the submission provides evidence along three lines. First, in the low-sample regime the scaling prior outperforms general interpolators (RBF, linear) (Table 4). Second, a warm-start analysis (Tables 9 and 10; 7 and 8 in the original submission) tests sensitivity to poor initialization: runs with the worst initialization (low initial hypervolume) show the largest hypervolume gains, indicating robustness. Third, Section A.2 discusses curve-fitting rare failure cases that resolve as more models are sampled. If a particular aspect is not covered here, we would gladly investigate further.