- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OrdinalForge, a training-free framework for joint DNN and hardware multi-objective optimization. The method first evaluates every architecture in a finite DNN search space using a training-free NAS proxy, primarily RBFleX-NAS, sorts the architectures by proxy score, and assigns each architecture a scalar rank. During multi-objective Bayesian optimization, this scalar rank replaces the original nnn-dimensional architecture encoding, while the mmm-dimensional hardware configuration is retained. The surrogate input is therefore reduced from n+m dimensions to 1+m dimensions.
The main intended benefit is to make Gaussian-process surrogate modeling more tractable while avoiding repeated DNN training. The method is evaluated on image classification, semantic segmentation, recurrent language modeling, and GPT-S causal language modeling using public NAS benchmarks and hardware evaluators or analytical models. The paper reports improved cumulative hypervolume over the evaluated MOBO, random-search, and NSGA-II baselines, together with substantial search-time reductions relative to methods that repeatedly train candidate networks. An additional GPT-S experiment applies the rank representation using both RBFleX-NAS and NASWOT.
Critical strengths
-
The paper targets a genuine difficulty in surrogate-based DNN-hardware co-optimization: a Gaussian process must model a joint input containing a high-dimensional, discrete architecture encoding and hardware parameters. Replacing the complete architecture encoding with one ordered variable is simple, easy to integrate into an existing MOBO pipeline, and can materially reduce surrogate complexity.
-
The method is evaluated across image classification, semantic segmentation, recurrent language modeling, and GPT-S causal language modeling, rather than on a single NAS benchmark. The authors also demonstrate the rank construction with both RBFleX-NAS and NASWOT in the GPT-S setting.
Critical weaknesses
1. The evaluation does not isolate the contribution of scalar rank
The central missing baseline is MOBO using the raw or normalized proxy score itself as the one-dimensional architecture input. OrdinalForge constructs scalar rank by sorting the same proxy score used as the DNN objective, so the relationship between rank and that objective is monotonic by construction. However, the current baseline uses the proxy score as the objective while retaining the original high-dimensional architecture encoding. This comparison changes both the dimensionality and the representation at the same time. It therefore cannot establish whether the improvement comes specifically from ordinal ranking, from generic one-dimensional compression, or simply from presenting proxy-derived information to the surrogate in an easier form.
Necessary controls include using the raw or normalized proxy score as a scalar input, using the proxy-score percentile or empirical CDF, and using a randomly permuted scalar architecture index. Without these comparisons, the experiments demonstrate that a proxy-derived one-dimensional representation can help, but they do not establish that scalar rank is the appropriate or necessary representation.
2. Scalar rank preserves only the ordering induced by the DNN proxy
Scalar rank preserves only the ordering induced by the DNN proxy. It does not preserve proxy-score gaps, architecture similarity, or hardware-relevant characteristics. For example, adjacent ranks can correspond to architectures with substantially different operators, tensor shapes, memory traffic, available parallelism, and hardware utilization. Conversely, architectures that are far apart in rank may have similar execution characteristics.
This is particularly concerning because a Gaussian-process kernel generally assumes that nearby input values tend to produce similar outputs. The proxy objective is guaranteed to be monotonic with rank because rank is constructed from that objective. No equivalent guarantee exists for latency, cycle count, energy, or EDP. The paper does not demonstrate that these hardware metrics vary smoothly along the proxy-induced rank axis, nor does it report separate surrogate-prediction diagnostics for the DNN and hardware objectives.
Rank also discards the magnitude of proxy-score differences. Two adjacent ranks are always separated by one unit, even if one pair has nearly identical proxy scores while another pair has a large score difference. Therefore, the proposed representation may simplify the DNN proxy objective while imposing an arbitrary and potentially misleading geometry on the hardware objectives. This issue is central because the paper claims effectiveness for joint DNN-hardware optimization, not only for modeling a training-free NAS score.
3. The closed-space assumption and runtime accounting limit the scalability and efficiency claims
The paper assumes a closed, finite architecture space in which every candidate architecture is enumerated, evaluated using the training-free proxy, sorted, and stored before optimization begins. An architecture that is “unseen” by MOBO therefore already has a known proxy score and scalar rank in the offline table. The method does not directly support a truly new architecture outside this table, and it is unclear how it would handle very large, continuous, conditional, or open-ended architecture spaces for which exhaustive pre-scoring is impractical. The experiments demonstrate effectiveness on finite and enumerable benchmark spaces, but they do not establish scalability in the broader NAS setting.
This setup also raises a methodological question about the use of the Gaussian process. Once proxy scores have been computed for every architecture, the complete DNN proxy objective is already known over the architecture space. Nevertheless, OrdinalForge fits a GP to observations of this known deterministic function and uses its posterior predictions during acquisition. After a candidate is selected, the exact proxy value is retrieved from the ranking table. It is therefore unclear why the known proxy objective should be approximated by a GP rather than supplied directly to the acquisition procedure while modeling only the unknown or expensive hardware objectives. A comparison against such a formulation is needed to establish whether the proposed MOBO design is statistically and computationally appropriate.
Questions for authors
-
Please report a controlled baseline using the raw or normalized proxy score as the one-dimensional architecture input. Because rank is deterministically derived from the proxy score, this comparison is essential to establish whether ordinal ranking provides value beyond one-dimensional proxy-derived compression.
-
Please compare against a randomly permuted scalar architecture index and, if possible, another low-dimensional representation such as PCA, random projection, or a learned architecture embedding. This would distinguish the effect of meaningful ordering from dimensionality reduction alone.
-
Since the proxy score is precomputed for every architecture, why is it modeled with a GP? Please compare against an acquisition process that uses the exact proxy objective and models only the unknown hardware objective.
-
Please report local smoothness or correlation between rank distance and differences in latency, cycles, or EDP.