- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies regret transfer and dynamic gain transfer among AUC, NDCG, Precision@k, Recall@k, and NDCG@k on fixed binary relevance ranking lists. Experiments verify the derived constants through exhaustive enumeration, illustrate the predicted asymptotic scaling, demonstrate the effects of parameter coupling on ranking dynamics, and observe metric conflicts during BCE and BPR training on MovieLens. These experiments validate the mathematical identities and demonstrate the existence of metric conflicts.
Critical strengths
The main contribution is a mathematical characterization of binary relevance ranking metrics. Specifically, the paper derives exact optimal linear regret transfer constants, conditions for finite regret transfer, asymptotic scaling with list size and relevance density, and identities characterizing metric gains through cumulative relevance changes. The results on the sharp regret bounds and their dependence on ranking structure are sound.
Critical weaknesses
See questions.
Questions for authors
The regret transfer bounds are derived for fixed binary relevance lists, using an oracle ranking determined by the realized relevance labels and optimized over all permutations. In practice, relevance labels are unknown at prediction time, and the rankings attainable by a scoring function are restricted by its hypothesis class. Could the authors comment on how these assumptions affect the interpretation and practical relevance of the derived bounds, particularly when the oracle ranking may not be attainable by the model? It would also be helpful to see whether the bounds are informative for the learned rankings in the existing experiments, for example, by examining their tightness or how often they provide nonvacuous guarantees.