acceptodds
Withdrawn from ICLR 2027

OrdinalForge: A Score-Induced Rank Representation for Scalable Training-Free DNN–Hardware Co-Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As modern edge AI applications continue to grow in computational complexity, co-optimizing Deep Neural Networks (DNNs) and their underlying hardware becomes increasingly critical. Existing Neural Architecture Search (NAS) approaches face two major challenges: the high dimensionality of the joint DNN–hardware representation and the substantial costs of evaluating DNNs through training. Particularly, while training-free NAS can reduce evaluation costs, using the proxy score as the DNN objective alone does not reduce the dimensionality of the surrogate input. To address this limitation, we introduce OrdinalForge, a training-free DNN–hardware co-optimization framework using a score-induced rank formulation for surrogate-based optimization. By sorting the proxy scores and assigning each DNN candidate a scalar rank, OrdinalForge replaces the original multidimensional DNN candidates supplied to the surrogate with a one-dimensional ordered input, allowing the surrogate to model the proxy score as a monotonic function of rank. Experiments show that OrdinalForge improves hypervolume by up to 13.7× and reduces search time by up to 57.8× compared to training-based baselines in image classification and semantic segmentation workloads. It also achieves the highest mean final cumulative hypervolume and better Pareto-optimal solutions among the evaluated methods in image classification, semantic segmentation, and RNN-based language modeling tasks. In the GPT-2-based language processing workload, the score-induced rank formulation improves the mean final cumulative hypervolume compared to its corresponding original-encoding baselines, validating that the proposed approach is effective, generalizable, and scalable.

Primary area:
optimization
Keywords:
software-hardware co-optimization, neural architecture search, Bayesian optimization, AI accelerator

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hMgxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OrdinalForge, a training-free framework for joint DNN and hardware multi-objective optimization. The method first evaluates every architecture in a finite DNN search space using a training-free NAS proxy, primarily RBFleX-NAS, sorts the architectures by proxy score, and assigns each architecture a scalar rank. During multi-objective Bayesian optimization, this scalar rank replaces the original nnn-dimensional architecture encoding, while the mmm-dimensional hardware configuration is retained. The surrogate input is therefore reduced from n+m dimensions to 1+m dimensions.

The main intended benefit is to make Gaussian-process surrogate modeling more tractable while avoiding repeated DNN training. The method is evaluated on image classification, semantic segmentation, recurrent language modeling, and GPT-S causal language modeling using public NAS benchmarks and hardware evaluators or analytical models. The paper reports improved cumulative hypervolume over the evaluated MOBO, random-search, and NSGA-II baselines, together with substantial search-time reductions relative to methods that repeatedly train candidate networks. An additional GPT-S experiment applies the rank representation using both RBFleX-NAS and NASWOT.

Critical strengths
  • The paper targets a genuine difficulty in surrogate-based DNN-hardware co-optimization: a Gaussian process must model a joint input containing a high-dimensional, discrete architecture encoding and hardware parameters. Replacing the complete architecture encoding with one ordered variable is simple, easy to integrate into an existing MOBO pipeline, and can materially reduce surrogate complexity.

  • The method is evaluated across image classification, semantic segmentation, recurrent language modeling, and GPT-S causal language modeling, rather than on a single NAS benchmark. The authors also demonstrate the rank construction with both RBFleX-NAS and NASWOT in the GPT-S setting.

Critical weaknesses

1. The evaluation does not isolate the contribution of scalar rank

The central missing baseline is MOBO using the raw or normalized proxy score itself as the one-dimensional architecture input. OrdinalForge constructs scalar rank by sorting the same proxy score used as the DNN objective, so the relationship between rank and that objective is monotonic by construction. However, the current baseline uses the proxy score as the objective while retaining the original high-dimensional architecture encoding. This comparison changes both the dimensionality and the representation at the same time. It therefore cannot establish whether the improvement comes specifically from ordinal ranking, from generic one-dimensional compression, or simply from presenting proxy-derived information to the surrogate in an easier form.

Necessary controls include using the raw or normalized proxy score as a scalar input, using the proxy-score percentile or empirical CDF, and using a randomly permuted scalar architecture index. Without these comparisons, the experiments demonstrate that a proxy-derived one-dimensional representation can help, but they do not establish that scalar rank is the appropriate or necessary representation.

2. Scalar rank preserves only the ordering induced by the DNN proxy

Scalar rank preserves only the ordering induced by the DNN proxy. It does not preserve proxy-score gaps, architecture similarity, or hardware-relevant characteristics. For example, adjacent ranks can correspond to architectures with substantially different operators, tensor shapes, memory traffic, available parallelism, and hardware utilization. Conversely, architectures that are far apart in rank may have similar execution characteristics.

This is particularly concerning because a Gaussian-process kernel generally assumes that nearby input values tend to produce similar outputs. The proxy objective is guaranteed to be monotonic with rank because rank is constructed from that objective. No equivalent guarantee exists for latency, cycle count, energy, or EDP. The paper does not demonstrate that these hardware metrics vary smoothly along the proxy-induced rank axis, nor does it report separate surrogate-prediction diagnostics for the DNN and hardware objectives.

Rank also discards the magnitude of proxy-score differences. Two adjacent ranks are always separated by one unit, even if one pair has nearly identical proxy scores while another pair has a large score difference. Therefore, the proposed representation may simplify the DNN proxy objective while imposing an arbitrary and potentially misleading geometry on the hardware objectives. This issue is central because the paper claims effectiveness for joint DNN-hardware optimization, not only for modeling a training-free NAS score.

3. The closed-space assumption and runtime accounting limit the scalability and efficiency claims

The paper assumes a closed, finite architecture space in which every candidate architecture is enumerated, evaluated using the training-free proxy, sorted, and stored before optimization begins. An architecture that is “unseen” by MOBO therefore already has a known proxy score and scalar rank in the offline table. The method does not directly support a truly new architecture outside this table, and it is unclear how it would handle very large, continuous, conditional, or open-ended architecture spaces for which exhaustive pre-scoring is impractical. The experiments demonstrate effectiveness on finite and enumerable benchmark spaces, but they do not establish scalability in the broader NAS setting.

This setup also raises a methodological question about the use of the Gaussian process. Once proxy scores have been computed for every architecture, the complete DNN proxy objective is already known over the architecture space. Nevertheless, OrdinalForge fits a GP to observations of this known deterministic function and uses its posterior predictions during acquisition. After a candidate is selected, the exact proxy value is retrieved from the ranking table. It is therefore unclear why the known proxy objective should be approximated by a GP rather than supplied directly to the acquisition procedure while modeling only the unknown or expensive hardware objectives. A comparison against such a formulation is needed to establish whether the proposed MOBO design is statistically and computationally appropriate.

Questions for authors
  • Please report a controlled baseline using the raw or normalized proxy score as the one-dimensional architecture input. Because rank is deterministically derived from the proxy score, this comparison is essential to establish whether ordinal ranking provides value beyond one-dimensional proxy-derived compression.

  • Please compare against a randomly permuted scalar architecture index and, if possible, another low-dimensional representation such as PCA, random projection, or a learned architecture embedding. This would distinguish the effect of meaningful ordering from dimensionality reduction alone.

  • Since the proxy score is precomputed for every architecture, why is it modeled with a GP? Please compare against an acquisition process that uses the exact proxy objective and models only the unknown hardware objective.

  • Please report local smoothness or correlation between rank distance and differences in latency, cycles, or EDP.

Reviewer_aijyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OrdinalForge, a training-free DNN–hardware co-optimization framework using a score-induced rank formulation for surrogate-based optimization. By sorting the surrogate scores and assigning a scalar rank to each candidate DNN, the the dimensionality of the joint input representation is effectively reduced, thereby accelerating the search process. Experiments across tasks including image classification, semantic segmentation, recurrent language modeling, and GPT-style language modeling demonstrate that this approach outperforms baselines.

Critical strengths

1.This paper introduce OrdinalForge which addresses the practical challenge of jointly optimizing the performance and hardware efficiency of deep neural networks (DNNs). It uses a score-induced rank formulation that enables training-free DNN evaluation through precomputed proxy scores and reduces the dimensionality of the joint input representation. 2. The evaluation covers several distinct workloads and includes comparisons with training-based and training-free optimization baselines. 3. The paper provides an anonymized implementation link and detailed descriptions and make it clear to reproduce.

Critical weaknesses
  1. This paper lacks significant novelty. The methodology primarily employs existing approaches—RBFleX-NAS and MOBO—with new contributions limited to the representation of rankings.

  2. Minor comment: In some figures, the font size and legends are too small, which hinders readability. For Figure 8, it is necessary to adjust the starting point of the Y-axis or switch to a tabular format.

Questions for authors
  1. How do the construction time and memory requirements for the ranking table change as the number of candidates increases?
Reviewer_AjiJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose OrdinalForge, a NAS framework that mainly solves the expensive evaluation cost of high dimensional candidates. To speedup model evaluation, they define a so-called proxy-score rank which acts a surrogate to performance estimation through model training. The method is evaluated on a diverse set of benchmarks across multiple domains. Their results show an improvement in hypervolume and a significant reduction in search time in comparison to chosen baseline methods.

Critical strengths

[S1] The authors evaluate their methods on different benchmark tasks (image classification, semantic segmentation, causal language modelling, RNN-based language modelling) to demonstrate the technique's applicability across different domains.

Critical weaknesses

[W1] The authors do not provide a dedicated related work section, nor does the paper position or directly compare itself against existing hardware-aware NAS techniques, which makes it hard to assess the overall contribution of the work. The main selling point of using training-free proxy scores to accelerate architecture evaluations during neural architecture search has already been studied by a great number of papers. A section with related work and how OrdinalForge differentiates itself from existing literature would make it clearer to the reader what the work actually contributes.

[W2] The experimental setup is flawed and favoured towards OrdinalForge. 1) Most of the methods described do not use training-free proxies, so it seems logical that OrdinalForge wins when compared as a function of search cost. The experiments could be strengthened by comparing against other training-free NAS algorithms. 2) The setup mainly studies hypervolume. It is not clear whether the hypervolume is calculated with the performance objective used by each technique. If so, the comparison does not make any sense, because the objective space of each method differs in the quality metric used. 3) It would also make more sense if the authors compared each task’s performance for a given compute budget and hardware constraint instead of only relying on hypervolume to assess the performance. This would demonstrate that the proposed technique can find models with better performance for a given compute time and hardware constraints.

[W3] The overall motivation remains unclear: why using a training-free NAS proxy reduces the input dimensionality of the candidates and how dimensionality reduction really contributes to the faster evaluation rationale. Also, one of the core components, the proxy score ρ\rho, is nowhere explained in terms of how it is computed.

[W4] The presentation in the paper does not meet the quality bar for an ICLR submission. The computation of the proxy score ρ\rho is nowhere explained. Symbolic notations are not used consistently. For example, rr is first defined as a scalar (line 141), then used as a function r(xs)r(x_s) (line 198). The figures are not really clear, e.g., how does the zoom in Figure 3 relate to the overall view? The conclusion does not mention any headline numbers. The appendix consists of just one section instead of being divided into relevant subsections.

[W5] Only results are provided by the anonymous git repository. No research code, making it impossible to reproduce the authors' results.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.