- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates replacing value-based objectives with rank-based objectives in high dimensions, trying to reveale situations where order learning is preferable to value regression. Through comparisons using 20 synthetic instances and 4 real-world tasks, rank surrogates generally showed superior performance under the same evaluation budget, with this benefit being particularly marked in high-dimensional problems. Furthermore, the study found that a crossover between value and rank estimation, controlled by the complexity of the numerical representation, exists. For end-to-end optimization, Pairwise and ListMLE achieved the second-best average rank across all BO configurations, suggesting that direct order learning can be particularly advantageous when the complexity of the objective function's numerical representation introduces unnecessary complexity in candidate selection.
Critical strengths
-
Quantification of Conditions Favoring Rank Learning: The paper is theoretically valuable for explicitly identifying the conditions under which rank learning is superior to value regression. Specifically, demonstrating the criteria for the inversion of preference, defined by the difficulty of numerical representation due to monotonic transformation () and the dimension-to-sample ratio (), is a significant theoretical contribution. Furthermore, the proper separation of the covariance matrix inverse computation via one-step comparison is appropriate.
-
Alignment Between Theoretical Setup and Empirical Validation: The mapping between the theoretical problem formulation and the empirical validation is relatively clear. The structure, which systematically extends the methodology—by controlling the difficulty of numerical representation through monotonic transformations and validating it from limited samples to neural networks and actual BO—is highly rational. The comparison with Rank-NN is particularly strong, as it effectively distinguishes the proposed approach from simple rank transformation and ranking loss methods.
Critical weaknesses
-
The primary failing of this paper is its poor presentation. In the Theoretical Analysis of Section 3, the rationale behind introducing individual definitions and propositions is unclear, making it difficult to understand what the study is actually investigating. While it is possible the reviewer is simply unable to follow the paper's logic, the manuscript cannot be considered highly comprehensible.
-
A Gap Between Theoretical Formulation and the Proposed BO Algorithm. The theory focuses on affine estimators, initial gradient directions, and independent candidate sets within the Gaussian Single-Index Model. Conversely, the proposed method is a BO that iteratively learns an ensemble of neural networks. Therefore, the theoretical superiority demonstrated currently does not sufficiently explain the performance gains in the actual BO setting. Specifically, the correspondence between the theoretical condition and the high-dimensional settings explored in the experiments is ambiguous. Consequently, the loss incurred by relying only on ordinal information is not discussed adequately. Furthermore, in the regret decomposition of Proposition 3, the cost of ranking errors depends on the gap in objective function values. However, Pairwise and ListMLE appear to be designed to operate without utilizing that gap (please correct the reviewer if this is a misunderstanding). Thus, it is not necessarily true that learning the rank correctly is the most efficient way to minimize regret. The trade-off between the robustness gained by discarding numerical information and the information lost should be the core focus of this research.
Questions for authors
-
Robustness Against Regularized Value Regression: Does the advantage of rank learning still hold when compared against properly regularized value regression? The theory currently focuses mainly on comparison with standard least squares estimation, but instability near could potentially be mitigated using techniques such as ridge regression. It would be important to verify if superiority persists even when compared to such sophisticated baselines.
-
Validity of Using Rank Model Uncertainty for BO Exploration: Is it valid to utilize the uncertainty derived from ranking models in the BO exploration phase? While the variance across ensembles is employed for exploration, there is no guarantee that this variance accurately represents the uncertainty of the predicted rank. Specifically, can the study rigorously validate whether the rank-based improvement achieved in HeSBO successfully implements an appropriate exploration–exploitation trade-off?
-
Core Differentiation from Existing Ranking-Based BO: What is the key differentiator from existing ranking-based BO research? The mere introduction of ordinal learning into BO is not novel. While we believe the novelty resides in the theoretical characterization of the conditions under which rank learning excels, the paper must more clearly demonstrate what novel insights are revealed by this theory that previous studies failed to capture.