Within-Top Ranking Quality: A Masked Dimension in Regression Evaluation

Regression models are widely used to score candidates and select the best for downstream action, from drug compounds to financial assets. In many such settings, model scores do more than determine which candidates are selected: they also determine how resources are allocated among them, often in proportion to predicted score. A reversed ranking within the top group can therefore direct larger resource commitments toward poorer-performing candidates within that group. Yet existing metrics either evaluate prediction accuracy across the entire candidate pool, or measure how many good candidates enter the model's predicted top group. Neither directly isolates whether the candidates with the highest realised outcomes are correctly ordered relative to one another. We formalise this missing evaluation dimension with $W_T$, the rank correlation within the true top-outcome group rather than the model's predicted top group. To characterise when within-top reversal can occur, we prove a safety theorem: under independent additive noise with a log-concave density, the Bayes-optimal score cannot reverse; extending this guarantee to a fitted model requires a separate rank-alignment condition. Under feature-dependent noise structure, reversal is possible and standard metrics are largely insensitive to it. Across 17 regression domains, the dominant reversal pattern is domain-structured and persists across model classes. In the reversing case, standard model selection tends to favour models with poorer within-top ranking, and $W_T$-aware constrained selection improves returns in chronological backtests. Beyond the reversing case, the degree to which $W_T$ provides information beyond standard metrics varies widely, and the two criteria can prefer different models even when all $W_T$ values are positive. This suggests that $W_T$ monitoring is warranted whenever regression models drive costly selection decisions.