acceptodds
Under review as a conference paper at ICLR 2027

Beyond Surrogates: A Quantitative Analysis for Ranking Metric Relationships

Abstract

The **Consistency** property between surrogate losses and evaluation metrics has been extensively studied to ensure that approaching the optimal surrogate risk leads to metric optimality. However, such guarantees do not ensure that different evaluation metrics improve together during optimization. This gap limits our understanding of the “Metric Mismatch” observed when improvements in one metric fail to translate into gains in another. To bridge this gap, this paper proposes a unified theoretical framework designed to quantify the relationships between ranking metrics. For fixed binary-relevance lists, we characterize the optimal linear **Regret Transfer** constants between ranking metrics and show that global regret transfer and local gain sensitivity can exhibit different scaling behavior. We further investigate **Dynamic Gain Transfer** through changes in cumulative relevance across ranking prefixes, deriving exact metric-gain identities and conditions for simultaneous improvement or conflict. To connect these relationships to optimization, we analyze score-margin dynamics and prove that equal-weight population binary cross-entropy gradient flow with independently optimized logits introduces no new relevance-order inversions. Through this framework, we provide a new perspective on how the location and direction of ranking changes govern cross-metric improvements, offering a theoretical basis for understanding metric alignment along optimization trajectories.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Learning-to-rank, Regret Transfer, Metric

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vcpkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies regret transfer and dynamic gain transfer among AUC, NDCG, Precision@k, Recall@k, and NDCG@k on fixed binary relevance ranking lists. Experiments verify the derived constants through exhaustive enumeration, illustrate the predicted asymptotic scaling, demonstrate the effects of parameter coupling on ranking dynamics, and observe metric conflicts during BCE and BPR training on MovieLens. These experiments validate the mathematical identities and demonstrate the existence of metric conflicts.

Critical strengths

The main contribution is a mathematical characterization of binary relevance ranking metrics. Specifically, the paper derives exact optimal linear regret transfer constants, conditions for finite regret transfer, asymptotic scaling with list size and relevance density, and identities characterizing metric gains through cumulative relevance changes. The results on the sharp regret bounds and their dependence on ranking structure are sound.

Critical weaknesses

See questions.

Questions for authors

The regret transfer bounds are derived for fixed binary relevance lists, using an oracle ranking determined by the realized relevance labels and optimized over all permutations. In practice, relevance labels are unknown at prediction time, and the rankings attainable by a scoring function are restricted by its hypothesis class. Could the authors comment on how these assumptions affect the interpretation and practical relevance of the derived bounds, particularly when the oracle ranking may not be attainable by the model? It would also be helpful to see whether the bounds are informative for the learned rankings in the existing experiments, for example, by examining their tightness or how often they provide nonvacuous guarantees.

Reviewer_uYATOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper aims to study the metric mismatch issues between different ranking metrics such as AUC and NDCG. it provides the Regret Transfer analysis between ranking metrics and show that global regret transfer and local gain sensitivity can exhibit different scaling behavior. It also analyzes Dynamic Gain Transfer through changes in cumulative relevance across ranking prefixes, deriving exact metric-gain identities and conditions for simultaneous improvement or conflict.

Critical strengths
  1. the metric mismatch between different ranking metrics is interesting.

  2. the theoretical transfer bounds between different metics are novel and interesting.

Critical weaknesses
  1. it is unclear how these transfer bounds inform the practice.

  2. there is little discussion on the insights from these transfer bounds.

  3. some existing works have studied the relationship between some metrics, e.g. AUROC, AUPRC.

  4. the NDCG the more interesting scenario is that the relevance score is not binary.

Questions for authors

see weakness.

Q: what about the relationship between AUC and partial AUC, AUROC and AUPRC?

Reviewer_KfGKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unified theoretical framework designed to quantify the relationships between ranking metrics. It gives answer to the question that how should one determine whether an improvement of a metric will result in gain or loss in another metric. It characterize structural relationships among ranking metrics. The paper also bounds regret-transfer between AUC and NDCG , as well as establishing dynamic gain-transfer identities.

Critical strengths

This paper has a strong theoretical development of regret-transfer quantities. It gives a thourough regret representation of different metrics, and also give optimal regret transfer constants between AUC, NDCG, Precision@k, Recall@k, and NDCG @k. It also characterizes the asymptotic scaling of AUC–NDCG transfer. Overall, I found this paper has a strong motivation, clear definition of problem, easy to follow, and has a complete theoretical development.

Critical weaknesses

My major concern lies in the experiments and the application.

  1. The experimental setting is not clear to me. I wonder what dataset is considered in the experiment study in Section 6. I would recommend a more detailed description of the experimental study.
  2. To further improve the motivation, I would expect a real-world application setting as an example.
Questions for authors
  1. What are other metrics in machine learning can one consider besides the metric the paper considers?
  2. What are some specific real-world examples dynamic metric transfer? (This question is also considered in the weakness section.) I would like to see an example in the introduction section to improve the motivation.
  3. Can one extend the problem setting to machine learning problems with multiple labels? (i.e., y∈{0,1,⋯ ,k}y\in\{0,1,\cdots, k\} )
  4. In the result of AUC–NDCG Transfer with fixed positive count, the AUC →\rightarrow NDCG is of oder nn, but the reverse is of order 1. For Fixed positive proportion, the two are of the same order. Could the author give an intuitive explanation of these?
  5. Can the authors give some application insights of the results?
  6. Generally, can one give a relationship for CA→B∗C^*_{A \rightarrow B} and CB→A∗C^*_{B \rightarrow A}? Can one bound each other?
  7. Section 5.3 is unclear to me. Can the author elaborate on how this is related to the ranking changes using previous notations? To further understand this, I have two additional questions. Does Eq. (30) only work for specific ηi\eta_i and ρ\rho chosen in line 402? In general, how do we see (20) is greater than or less than 0?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.