- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes W_T, a metric for whether a regression model gets the order right among the samples that actually turn out best. W_T is the Spearman correlation between scores and outcomes, computed only on samples whose realised outcomes fall in the top 20%. The authors argue that standard metrics barely register errors in this region: a complete reversal within the top group changes overall Spearman by only about 2q³ ≈ 0.016. On the theory side, they give sufficient conditions under which the Bayes-optimal score cannot reverse. One is additive, constant-scale noise with a log-concave density. The other is symmetric log-concave noise with non-decreasing scale. They also give counterexamples where these conditions fail. Across 17 domains and 684 models, W_T is positive everywhere except financial return prediction, where every standard-loss model reverses. The authors attribute this to a direction–magnitude structure in returns: features predict how large a move will be but not its direction. Finally, they argue that W_T is useful for decisions. It correlates with how many top outcomes a model would have captured in hindsight. Selecting models by W_T, subject to a minimum Spearman, also improves point-estimate returns in a chronological stock backtest and allocation gain in a traffic-forecasting task.
Critical strengths
1.Figure 1 is a useful diagnostic, where the core observation is real and clearly illustrated. The distinction between ranking within the predicted top group and ranking within the realised top group is an important one.
2.The theoretical appendix is carefully designed. Theorem 2’s reflection argument is, to my reading, the most original technical piece.
Critical weaknesses
Overall, the scope of this paper is limited, much of the theory is standard, and its link to the experiments is loose. The paper is also hard to read and follow. So I lean to rejection.
1.The theory is standard, and its link to the experiments is loose. Theorem 1 is the classical implication from stochastic monotonicity to association to non-negative rank covariance. Corollary 1 is the standard IFR residual-life ordering. Theorem 3 is the positive-semidefinite bound for a 3×3 correlation matrix.
2.W_T does not evaluate the set that the motivating decision acts on. The introduction motivates W_T with score-proportional allocation among selected candidates. In practice, allocation happens over the predicted top group, while W_T conditions on the realised top group.
3.The financial reversal is consistent with a known, bayes-consistent effect. Economically, the pattern in Figure 1(a,b) is the familiar volatility structure. Both tails of realised returns are populated by high-volatility stocks. W_T-aware selection may be selecting for risk.
4.Top-weighted and tail-conditional rank correlation has a substantial literature, none of which is cited. W_T conditions on only one variable, which differs from most of these, but the paper should compare against them both conceptually and empirically.
Questions for authors
1.Please compute W_T between the model scores and predicted or realised volatility (and beta/size). Please also compute it on volatility-scaled outcomes. 2.For the backtest, please report factor exposures and risk-adjusted returns (alpha) for the W_T-selected versus ρ_S-selected models.
Without these, I cannot tell whether the 6.7 pp gross-return gain over 2023–2026 comes from better ranking or just from higher risk during a period that included a strong rally.
3.Can you report equation (3) on the prediction-rank sets that are actually occupied, for the models in Figure 1(a,b)?