acceptodds
Under review as a conference paper at ICLR 2027

Within-Top Ranking Quality: A Masked Dimension in Regression Evaluation

Abstract

Regression models are widely used to score candidates and select the best for downstream action, from drug compounds to financial assets. In many such settings, model scores do more than determine which candidates are selected: they also determine how resources are allocated among them, often in proportion to predicted score. A reversed ranking within the top group can therefore direct larger resource commitments toward poorer-performing candidates within that group. Yet existing metrics either evaluate prediction accuracy across the entire candidate pool, or measure how many good candidates enter the model's predicted top group. Neither directly isolates whether the candidates with the highest realised outcomes are correctly ordered relative to one another. We formalise this missing evaluation dimension with , the rank correlation within the true top-outcome group rather than the model's predicted top group. To characterise when within-top reversal can occur, we prove a safety theorem: under independent additive noise with a log-concave density, the Bayes-optimal score cannot reverse; extending this guarantee to a fitted model requires a separate rank-alignment condition. Under feature-dependent noise structure, reversal is possible and standard metrics are largely insensitive to it. Across 17 regression domains, the dominant reversal pattern is domain-structured and persists across model classes. In the reversing case, standard model selection tends to favour models with poorer within-top ranking, and -aware constrained selection improves returns in chronological backtests. Beyond the reversing case, the degree to which provides information beyond standard metrics varies widely, and the two criteria can prefer different models even when all values are positive. This suggests that monitoring is warranted whenever regression models drive costly selection decisions.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Regression Evaluation, Ranking Metrics, Model Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ddv8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes W_T, a metric for whether a regression model gets the order right among the samples that actually turn out best. W_T is the Spearman correlation between scores and outcomes, computed only on samples whose realised outcomes fall in the top 20%. The authors argue that standard metrics barely register errors in this region: a complete reversal within the top group changes overall Spearman by only about 2q³ ≈ 0.016. On the theory side, they give sufficient conditions under which the Bayes-optimal score cannot reverse. One is additive, constant-scale noise with a log-concave density. The other is symmetric log-concave noise with non-decreasing scale. They also give counterexamples where these conditions fail. Across 17 domains and 684 models, W_T is positive everywhere except financial return prediction, where every standard-loss model reverses. The authors attribute this to a direction–magnitude structure in returns: features predict how large a move will be but not its direction. Finally, they argue that W_T is useful for decisions. It correlates with how many top outcomes a model would have captured in hindsight. Selecting models by W_T, subject to a minimum Spearman, also improves point-estimate returns in a chronological stock backtest and allocation gain in a traffic-forecasting task.

Critical strengths

1.Figure 1 is a useful diagnostic, where the core observation is real and clearly illustrated. The distinction between ranking within the predicted top group and ranking within the realised top group is an important one.

2.The theoretical appendix is carefully designed. Theorem 2’s reflection argument is, to my reading, the most original technical piece.

Critical weaknesses

Overall, the scope of this paper is limited, much of the theory is standard, and its link to the experiments is loose. The paper is also hard to read and follow. So I lean to rejection.

1.The theory is standard, and its link to the experiments is loose. Theorem 1 is the classical implication from stochastic monotonicity to association to non-negative rank covariance. Corollary 1 is the standard IFR residual-life ordering. Theorem 3 is the positive-semidefinite bound for a 3×3 correlation matrix.

2.W_T does not evaluate the set that the motivating decision acts on. The introduction motivates W_T with score-proportional allocation among selected candidates. In practice, allocation happens over the predicted top group, while W_T conditions on the realised top group.

3.The financial reversal is consistent with a known, bayes-consistent effect. Economically, the pattern in Figure 1(a,b) is the familiar volatility structure. Both tails of realised returns are populated by high-volatility stocks. W_T-aware selection may be selecting for risk.

4.Top-weighted and tail-conditional rank correlation has a substantial literature, none of which is cited. W_T conditions on only one variable, which differs from most of these, but the paper should compare against them both conceptually and empirically.

Questions for authors

1.Please compute W_T between the model scores and predicted or realised volatility (and beta/size). Please also compute it on volatility-scaled outcomes. 2.For the backtest, please report factor exposures and risk-adjusted returns (alpha) for the W_T-selected versus ρ_S-selected models.

Without these, I cannot tell whether the 6.7 pp gross-return gain over 2023–2026 comes from better ranking or just from higher risk during a period that included a strong rally.

3.Can you report equation (3) on the prediction-rank sets that are actually occupied, for the models in Figure 1(a,b)?

Reviewer_8ju7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ranking quality within the group of candidates with the highest realized outcomes. It evaluates this using WTW_T, the Spearman correlation restricted to the true top-outcome group. The paper characterizes when the metrics can mask within-top reversal, finds conditions that prevents reversal, and conducts experiments across multiple regression domains. It also discusses whether historical metric WTW_T can improve model selection.

Critical strengths
  1. The financial reversal is an interesting finding. It appears in 161 models and 14 architectures. The controlled objective comparisons outside finance domain also show that overall ranking and within-top ranking can favor different models even without reversal.
  2. The theoretical analysis is solid. It clearly explains when reversal is possible. The paper distinguishes these statements from guarantees about actual decision utility.
Critical weaknesses
  1. The novelty of the metric needs clearer positioning. WTW_T is a direct special case of conditional Spearman correlation, already defined in [1], but this literature is not discussed. The contribution should therefore be established through the regression-evaluation application, theoretical characterization, and empirical findings.
  2. The evidence for improved decision-making is limited. In Table T3, all financial return and Sharpe confidence intervals include zero. The Traffic result is significant for allocation gain, but not for the corresponding total utility, while results in other domains are mixed. These limitations are acknowledged, but they make it not convincing enough whether the metric can actually help model selection. Also, Appendix G.3 says random splits for non-financial models, but Appendix U does not clearly specify whether its candidate models were separately trained using only historical data.

[1] Schmid, Friedrich, and Rafael Schmidt. "Multivariate conditional versions of Spearman's rho and related measures of tail dependence." Journal of multivariate analysis 98.6 (2007): 1123-1140.

Questions for authors

Could the authors address the weaknesses above, particularly the relationship of the metric to existing literatures, when WTW_T-based selection provides gains in end-to-end utility over retrieval- or utility-based selection, and the data split in the financial experiment?

Reviewer_QN7sOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors study the problem that in regression models, the ranking of predicted scores within the top responses can have a reverse ordering. The proposed method is to compute Spearman’s correlation only among the observations whose responses are among the highest. The authors evaluated the masking effect and provide several theorems with sufficient conditions for the proposed statistic to be nonnegative.

Critical strengths
  1. The paper is clearly written with the proposal of a simple and inexpensive diagnostic.
  2. The theory is carefully developed.
Critical weaknesses
  1. The novelty of the proposed statistic WTW_T is limited, as it is essentially about a conditional Spearman correlation. The related paper on “Multivariate conditional versions of Spearman's rho and related measures of tail dependence” by Schmid and Schmidt should be carefully discussed and compared. The authors need to provide convincing theoretical and numerical evidence to show the advantages of using WTW_T.
  2. The current explanation of the reversal in financial data is still insufficient. What are the roles of mechanism in Section 4.4 in explaining the results from financial data? These are not clearly stated.
  3. The gain discussed in Section 5.2 and Section S.4 is not very useful in practice. The strong association does not guarantee a better future selection or allocation.
Questions for authors

Since WTW_T can be negative even when the conditional mean is correctly estimated, when should people in practice treat a negative WTW_T as a problem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.