- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper develops a theoretical framework for finding optimal tradeoffs between different evaluation metrics in binary classification, such as accuracy, precision, recall, F-score, IoU, balanced accuracy, and Cohen’s kappa. It models these metrics as members of a unified family of ranking scores and studies how to construct rankings that optimally balance any pair of metrics
Critical strengths
- Rigorous mathematical development
- Generalization of previous results from Piérard et al.
- Clear conceptual motivation
Critical weaknesses
-
Limited novelty beyond prior work. The paper repeatedly builds upon the framework introduced in multiple earlier works by the same research group and explicitly generalizes the methodology developed for the precision-recall tradeoff in Piérard et al. (2026). The core conceptual ingredients (ranking scores, ranking manifolds, shortest paths, optimal tradeoffs) already appear to exist in earlier publications. The current contribution reads more like an small extension than a fundamentally new advance.
-
Empirical evidence is relatively weak Although the paper provides synthetic experiments and several case studies, the practical impact of the proposed tradeoffs remains unclear.
-
Limited relevance to the broader ICLR audience The work focuses on evaluation theory for binary classification metrics rather than on learning algorithms, representation learning, optimisation, or foundation models. While theoretically interesting, its applicability to mainstream machine learning research may be relatively narrow.
Questions for authors
The paper does not provide strong evidence of the theory's application. For instance, it would be more convincing if a certain challenge that relied on F1 had adopted their proposed score instead, would change the winner of the competition.
The format of references is almost always wrong (\citet vs \citep format)
Answers to reviewer zbsM
We thank the reviewer for his supportive review. We are pleased that our conceptual motivation is deemed clear and that our mathematical development is recognized rigorous.
Answer to weakness 1. The sole work in the literature that rigorously defines an optimal tradeoff between two scores is only about the tradeoff between precision and recall. It is true that we kept the same framework (i.e., core conceptual ingredients) since there was no reason to change it. However, generalizing the previously known theoretical result to any pair of ranking scores was not straightforward. In particular, we would like to underline that, to the best of our knowledge, it was not previously known that it is possible to obtain intermediate ranking scores by averaging ranking scores by f-means generated by carefully chosen homographic functions.
Answer to weakness 2. We have three empirical experiments. The first shows evidence that the current practice, that consists in establishing tradeoffs by averaging the values of two scores, is flawed. The second involves 21 cases corresponding to 7 pairs of common scores between which one needs to find a tradeoff multiplied by 3 large sets of performances that come from reference distributions. And the third involves 28 cases corresponding to the same 7 pairs of scores multiplied by 4 real world rankings taken from the literature or online benchmarks. If the reviewer still thinks that the empirical evidence is relatively weak, can (s)he explain precisely what is missing and the additional value it would bring to our paper?
Answer to weakness 3. Our paper is submitted in the “datasets and benchmarks” primary area. For us, the existence of this primary area shows that ICLR welcomes papers on evaluation. So, we do not understand why the reviewer considers it as a weakness.
Answer to the question. As a general rule, a classifier ranked number one for some importance I may rank very poorly for others. This is why it is essential not to overlook the choice of ranking criterion to use. Consider the example developed by Halin et al. (2024). In their comparison of 74 classifiers (semantic segmentation methods), the best methods (depending on the preferences considered) are the deep models “Mask2Former” that ranks between 1 and 32, “SETR” that ranks between 1 and 14, “DeepLabV3+” that ranks between 1 and 58, and ISANet, that ranks between 1 and 65. The significant variations in the ranks reflects the risk that one takes when not considering the right ranking objective.
Note on the ethics concern. Our paper is anonymous. As recognized by the reviewer himself, citing several works published by a same team of researchers is not a violation. From our point of view, all cited works are relevant for the topic of our paper. If you disagree, please feel free to let us know which of the cited works you believe is not related to this one and explain why you think so.