acceptodds
Under review as a conference paper at ICLR 2027

The Optimal Ranking Tradeoffs Between Two Ranking Scores in Two-Class Classification

Abstract

The accuracy, true negative rate (negative class recall), true positive rate (positive class recall), negative predictive value (negative class precision), positive predictive value (positive class precision), class-specific intersections over unions, class-specific F-scores, Cohen’s kappa, and balanced accuracy are all scores that one intuitively wants to maximize. However, they lead to different rankings. So, how can we establish a ranking that is an optimal tradeoff between any pair of these scores? It turns out that all these scores are special cases of ranking scores: a continuum of scores parameterized by the relative importance values assigned to the true negative, false positive, false negative, and true positive outcomes. Since all these scores are known to induce meaningful rankings, we seek the optimal tradeoffs among them. We consider the tradeoff between two scores to be optimal when it minimizes the Fréchet variance with respect to the Kendall distance (also known as the bubble-sort distance) between rankings. For any given set of classifiers to rank, the optimal tradeoff lies at the center of the shortest path (geodesic) on the manifold of all rankings induced by the ranking scores. However, in the parameter space of the ranking scores, it is rarely achieved by averaging the importance values corresponding to the initial scores. In this paper, we present an exact analytical solution for determining a ranking score that is an optimal tradeoff, for any given pair of initial scores, in the common case where the class priors are the same for all compared performances. We also experimentally investigate what happens when performance follows some distributions with fixed priors.

Primary area:
datasets and benchmarks
Keywords:
Two-Class Classification, Performance, Ranking, Accuracy, Cohen kappa, Balanced Accuracy, Precision, Positive Predictive Value, Recall, True Positive Rate, Sensitivity, F-Scores, Ranking Scores, Optimal Tradeoff, Rank Correlation, Kendall Distance, Bubble Sort Distance, Fréchet Variance, Karcher Means, Rank Aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_zbsMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper develops a theoretical framework for finding optimal tradeoffs between different evaluation metrics in binary classification, such as accuracy, precision, recall, F-score, IoU, balanced accuracy, and Cohen’s kappa. It models these metrics as members of a unified family of ranking scores and studies how to construct rankings that optimally balance any pair of metrics

Critical strengths
  1. Rigorous mathematical development
  2. Generalization of previous results from Piérard et al.
  3. Clear conceptual motivation
Critical weaknesses
  1. Limited novelty beyond prior work. The paper repeatedly builds upon the framework introduced in multiple earlier works by the same research group and explicitly generalizes the methodology developed for the precision-recall tradeoff in Piérard et al. (2026). The core conceptual ingredients (ranking scores, ranking manifolds, shortest paths, optimal tradeoffs) already appear to exist in earlier publications. The current contribution reads more like an small extension than a fundamentally new advance.

  2. Empirical evidence is relatively weak Although the paper provides synthetic experiments and several case studies, the practical impact of the proposed tradeoffs remains unclear.

  3. Limited relevance to the broader ICLR audience The work focuses on evaluation theory for binary classification metrics rather than on learning algorithms, representation learning, optimisation, or foundation models. While theoretically interesting, its applicability to mainstream machine learning research may be relatively narrow.

Questions for authors

The paper does not provide strong evidence of the theory's application. For instance, it would be more convincing if a certain challenge that relied on F1 had adopted their proposed score instead, would change the winner of the competition.

The format of references is almost always wrong (\citet vs \citep format)

1 comment
AuthorsOpenReview ↗

Answers to reviewer zbsM

We thank the reviewer for his supportive review. We are pleased that our conceptual motivation is deemed clear and that our mathematical development is recognized rigorous.

Answer to weakness 1. The sole work in the literature that rigorously defines an optimal tradeoff between two scores is only about the tradeoff between precision and recall. It is true that we kept the same framework (i.e., core conceptual ingredients) since there was no reason to change it. However, generalizing the previously known theoretical result to any pair of ranking scores was not straightforward. In particular, we would like to underline that, to the best of our knowledge, it was not previously known that it is possible to obtain intermediate ranking scores by averaging ranking scores by f-means generated by carefully chosen homographic functions.

Answer to weakness 2. We have three empirical experiments. The first shows evidence that the current practice, that consists in establishing tradeoffs by averaging the values of two scores, is flawed. The second involves 21 cases corresponding to 7 pairs of common scores between which one needs to find a tradeoff multiplied by 3 large sets of performances that come from reference distributions. And the third involves 28 cases corresponding to the same 7 pairs of scores multiplied by 4 real world rankings taken from the literature or online benchmarks. If the reviewer still thinks that the empirical evidence is relatively weak, can (s)he explain precisely what is missing and the additional value it would bring to our paper?

Answer to weakness 3. Our paper is submitted in the “datasets and benchmarks” primary area. For us, the existence of this primary area shows that ICLR welcomes papers on evaluation. So, we do not understand why the reviewer considers it as a weakness.

Answer to the question. As a general rule, a classifier ranked number one for some importance II may rank very poorly for others. This is why it is essential not to overlook the choice of ranking criterion to use. Consider the example developed by Halin et al. (2024). In their comparison of 74 classifiers (semantic segmentation methods), the best methods (depending on the preferences considered) are the deep models “Mask2Former” that ranks between 1 and 32, “SETR” that ranks between 1 and 14, “DeepLabV3+” that ranks between 1 and 58, and ISANet, that ranks between 1 and 65. The significant variations in the ranks reflects the risk that one takes when not considering the right ranking objective.

Note on the ethics concern. Our paper is anonymous. As recognized by the reviewer himself, citing several works published by a same team of researchers is not a violation. From our point of view, all cited works are relevant for the topic of our paper. If you disagree, please feel free to let us know which of the cited works you believe is not related to this one and explain why you think so.

Reviewer_HnX4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to construct an optimal ranking tradeoff between two evaluation scores in binary classification. The authors observe that commonly used metrics, including accuracy, precision, recall, F-scores, and IoU, may induce different rankings of classifiers, while conventional arithmetic, geometric, or harmonic means do not necessarily provide balanced compromises between these rankings.

Critical strengths
  1. A principled theoretical formulation of ranking tradeoffs.
  2. Generalization to arbitrary pairs of ranking scores.
Critical weaknesses
  1. The practical significance of the proposed optimality criterion is insufficiently established.
  2. The empirical validation is largely illustrative and does not sufficiently investigate the robustness of the resulting rankings.
Questions for authors
  1. How does the proposed ranking optimality relate to practical model-selection preferences?
  2. How stable is the optimal tradeoff when the set of classifiers changes?
  3. How robust is the proposed construction to uncertainty in estimated classifier performance?
1 comment
AuthorsOpenReview ↗

Answers to reviewer HnX4

We thank the reviewer for his supportive review. We are pleased that the theoretical formulation of ranking tradeoffs and our generalization to arbitrary pairs of ranking scores are appreciated.

Answer to weakness 1. With the approach we are following, we have 4 guarantees that are important in practice.

  1. As of the search space we consider, the have the guarantee that the chosen ranking corresponds to a score for which an increase (resp. decrease) in value can be safely interpreted as an improvement (resp. worsening).
  2. As the continuum of ranking scores is parameterized by importance values, it is possible to be fully transparent and tell which application-specific preferences correspond to the selected tradeoff (i.e., the relative importance given to the true positives with respect to the true negatives and the relative importance given to the false negatives with respect to the false positives).
  3. We give the guarantee that the chosen ranking is, from the point of view of rank correlations, between the two original rankings. This is what is called being on the shortest path.
  4. As we look for the midpoint on that path, we give the guarantee that the chosen ranking does not perfectly mimic one of the base rankings while totally ignoring the other.

Note that, with a simple average of values, the resulting tradeoff only comes with the third guarantee.

Answer to weakness 2. The robustness of rankings can be understood in several ways, depending on the type of disturbances under consideration: changing the learning set, retraining when the learning method is not deterministic, changing the test set, or even considering other classifiers to rank. Experimentally examining all of this is beyond the scope of this paper. However, in the presented theory, we already gave strong results. First, no matter what the disturbances are, the shortest path remains the same and the chosen score remains on this shortest path. Second, if both original scores agree on the order between two classifiers, then the recommended tradeoff score also agrees with them. So, for a pair of classifiers for which both original scores agree on the order, if the disturbance does not change this order, then the tradeoff ranking will remain stable.

Answer to question 1. What do you mean by practical model-selection preferences? If it is the relative importance given to the true positives with respect to the true negatives and the relative importance given to the false negatives with respect to the false positives, then it is possible to compute and report these values for the recommended tradeoff, which is done in Fig. 3 and Fig. 4.

Answer to question 2. (1) On the Tile, the optimal tradeoff score can only move along a curve (the shortest path) joining the two original performance orderings (with the parameterization that we suggest, this curve is a line segment). (2) If, in the pool of classifiers to rank, one is undoubtedly better or worse than all others, and if you remove it from the pool, then the recommended optimal tradeoff does not change because the set of values that appears in Eq. (10) does not change. (3) In practice, when the pool is large enough, the optimal tradeoff is stable. For example, the optimal tradeoffs shown in Fig. 3 do not change, to the naked eye, when I reexectue the code that generated this figures, which includes a random sampling of the performances. But for small pools, it is a proven fact that the optimal tradeoff should be adapted to the pool of classifiers.

Answer to question 3. Without knowing the source and extent of the uncertainty, it is impossible to provide a quantitative answer. However, we can say that an uncertainty about the performances leads to an uncertainty about the optimal tradeoff. Here is how to calculate it. When there is some uncertainty about a classifier’s performance, no matter its origin and its magnitude, we can replace each performance by a distribution of performances. We can then adopt a Monte Carlo approach: draw the performance of each classifier from the corresponding distribution and then apply the theory presented in our paper. The uncertainty regarding the optimal tradeoff is then given by the distribution of the optimal tradeoffs determined in this way. With the parameterization we recommend, on the Tuile, this is a 1D distribution of points along the line segment connecting the two initial scores.

Reviewer_vbPZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to construct a compromise between the classifier rankings induced by two evaluation scores in binary classification. Under fixed class priors, it extends an earlier precision–recall analysis to arbitrary pairs within a family of ranking scores. The proposed compromise minimizes the sum of squared Kendall distances to the two endpoint rankings. The paper introduces a parameterization adapted to class priors, derives paths between ranking scores, and proposes a median-based procedure for selecting a compromise. Experiments illustrate the construction using synthetic performance distributions and four real-world performance sets.

Critical strengths

1, the distinction between averaging score values and balancing the rankings induced by those scores is relevant to benchmark design. The paper addresses this distinction explicitly and provides a concrete optimization objective. 2, the generalization beyond precision and recall is potentially useful, and the proposed parameterization offers a convenient way to visualize how different evaluation scores order classifiers. 3, the real-world case studies include different class prevalences and candidate-set sizes. They help illustrate that the location of a compromise depends on the performances being compared.

Critical weaknesses

1, the practical motivation for the optimality criterion is insufficiently established. Equation (1) defines an optimal compromise through squared Kendall distances, but the paper does not sufficiently justify why this is the appropriate criterion for the intended evaluation decisions. Equal distance from two rankings is a particular interpretation of balance. It does not establish that the resulting ranking better reflects application preferences or supports better model selection. In particular, Kendall distance gives equal importance to all classifier pairs and does not distinguish small, uncertain performance differences from large, reliable differences. Although unequal endpoint weights are discussed, the connection between these weights and application requirements remains unclear. The claims about optimality should therefore be explicitly qualified by the chosen ranking objective. 2, the experiments mainly illustrate the construction rather than establish its practical advantage. Figure 2 evaluates the method using an optimality measure that directly rewards the balance the method is designed to achieve. This is a useful consistency check, but it provides limited independent evidence for preferring the resulting rankings over conventional alternatives. Section 4.3 does contain real-data analyses: IWDD learning and test performances, BDD100K segmentation performances, and CDnet background-subtraction performances. However, Figure 4 primarily displays ranking boundaries and compromise locations. The paper does not systematically show which consequential model-selection decisions change, whether those changes serve a specified evaluation purpose, or how reliable they are. Since this is an evaluation-methodology paper, training new classifiers is unnecessary; a focused quantitative analysis using the existing performance sets would be more informative. 3, dependence on the candidate set creates an unresolved issue for benchmark use. Equation (10) selects the compromise using pairwise comparisons among the available classifiers. Consequently, the selected score depends on the composition of that set. The paper does not establish its sensitivity to adding or removing models, including groups of very similar models. This matters because benchmark metrics are often fixed before submissions arrive. Section 4.2 acknowledges this issue and suggests choosing a score using a reference distribution, but does not validate how well that choice transfers to actual performances. A limited experiment that selects the score on one set, freezes it, and evaluates its balance on another would directly address this concern. 4, the theoretical contribution relative to the earlier precision–recall work needs sharper delineation. The paper explicitly adopts the methodology of Piérard et al. (2026), and Sections 3.3–3.4 repeatedly state that the previous reasoning applies or generalizes easily. The new parameterization and homographic construction may constitute a substantive extension, but the presentation should identify precisely which results are new and which technical obstacles they resolve. This distinction is especially important because the empirical contribution is primarily illustrative. 5, there is a concrete algebraic error in the theoretical development. Immediately after Equation (7), the transformation is claimed to be an involution, (f^{-1}=f). Consider the valid endpoint coordinates for TNR and PPV, giving (\Delta a^{(\pi)}=1) and (\Delta b^{(\pi)}=0). Equation (7) then gives f(x)=\frac{x+1}{1-x}, \qquad f(f(x))=-\frac{1}{x}, wherever the compositions are defined. Thus, this transformation is not an involution. This error does not by itself disprove the proposed ranking construction, but the authors should correct the statement and explain whether any subsequent argument or implementation relies on it.

Questions for authors

1, what benchmark decision or application preference makes the objective in Equation (1) preferable to other interpretations of compromise? Can one existing case study demonstrate that connection quantitatively? 2, how stable are the selected weights and resulting rankings when the candidate set changes? Does a score chosen from a reference distribution or development set retain its intended balance on held-out performances? 3, how should Equation (7) or the involution claim be corrected, and which later results depend on that claim? Please also clarify how the finite-set procedure handles ties and simultaneous ranking crossings.

1 comment
AuthorsOpenReview ↗

Answer to reviewer vbPZ

We thank the reviewer for his review and in particular for his/her careful reading of the theory section.

Answer to question 1. In our community, I never saw a paper starting by rigorously specifying the application-specific preferences (which, following previous work, we understand as the relative importance given to the true positives with respect to the true negatives and the relative importance given to the false negatives with respect to the false positives), then deriving a score tailored for these preferences, and finally producing a ranking based on it. Instead, researchers are used to think in terms of a trade-off, such as the one between TNR and TPR (shown in the ROC space), or the one between PPV and TPR (shown in the Precision-Recall space), and then they try to establish a unique ranking between the two, by averaging the two scores. While we totally agree with you that the first way of working would be a very good one when one has crystal-clear understanding of the target application and the related preferences, the objective of this paper was to present a better alternative for those who work in the second way. We chose to generalize a previous result of the literature (Piérard et al., 2026), with the same notion of optimal tradeoff. Theoretically studying alternative notions of ranking tradeoffs is beyond the scope of this paper as it may require to write many papers …

Answer to question 2. As explained in section “Case Studies: With Real-World Rankings”, the comparison between Fig. 4c and Fig. 4d with Fig. 3c shows that the optimal tradeoffs that we have found for the previously reported performances of pixel-based classifiers (MMSegmentation semantic segmentation models, Halin et al., 2024, and background subtraction methods, Wang et al., 2014) are close to those the optimal tradeoffs that are predicted by the reference distribution that is called “uniform distribution of performances, close to the oracle” in the paper. Thus, it seems that a realistic assumption for a reference distribution can be used to choose the ranking scores to use with real performances of classifiers. A benchmark designer who can roughly guess the kind of performances that will be submitted by the participants can most probably make a realistic assumption for a reference distribution and therefore make an educated choice for the score to use for ranking. As an alternative, the benchmark organizer could only make the choice for the ranking score at the end of the contest. But we would not recommend dynamically changing the score each time a new participant enters into the competition because the conclusion about which method is better than another would not be perennial.

Answer to question 3. (a) Thank you very much for pointing the mistake. Indeed, it is −f-f that is an involution. This has been corrected. Our mistake in the text had no impact on the rest of the paper. (b) There are different versions of Kendall’s tau that correspond to different arbitrary choices in how one wants to consider the ties. We don't have a strong opinion on this matter. I feel that the most important is that the solution should be 100% correct in the absence of ties. When there are ties, with some particular choice for the τ\tau to use, we can observe small differences between dτ(RIA;RIB)d_\tau(R_{I_A}; R_{I_B}) and dτ(RIA;RI∗)+dτ(RI∗;RIB)d_\tau(R_{I_A}; R_{I_*})+ d_\tau(R_{I_*}; R_{I_B}). Nevertheless, I would like to underline that ties are very unlikely, unless the performances are estimated from very small test sets, which is not really a good idea. The probability that two classifiers would have exactly the same performance in a realistic ranking is extremely small.

Reviewer_DQS7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to construct an optimal ranking tradeoff between any two evaluation scores for two-class classification, such as precision/recall, TPR/TNR, or NPV/PPV, rather than combining them through an arbitrarily chosen arithmetic, geometric, or harmonic mean.

Its main theoretical contribution is an analytical framework showing that, under fixed class priors, ranking scores admit universal shortest paths under Kendall distance and that an optimal tradeoff can be characterized as the midpoint/Karcher mean of this path, with an explicit solution based on pairwise ranking transitions.

Experiments on synthetic performance distributions and four real-world performance sets show that common score means are generally not optimal in ranking space, while the proposed construction attains the defined ranking optimum by design and can be applied to realistic classifier comparisons.

Critical strengths

1. A principled and fairly general theoretical treatment of metric tradeoffs. The paper moves beyond individual constructions such as F1 and asks a more fundamental question: given any two meaningful ranking scores, what score induces a ranking that is genuinely intermediate between them? Under fixed class priors, the authors derive universal shortest paths between ranking-score-induced orderings under Kendall distance and characterize the optimal tradeoff through Fréchet variance/Karcher means. This is consequential because it converts what is usually an ad-hoc choice of metric averaging into a mathematically defined ranking optimization problem.

2. The framework exposes a real weakness of conventional metric averaging. The analysis and experiments demonstrate that arithmetic, geometric, or harmonic averaging of two scores does not generally produce a ranking midway between the rankings induced by those scores, and may not even preserve the desired axioms of performance-based ranking. In contrast, the proposed construction explicitly operates in ranking space and provides an exact optimal solution according to the stated Kendall-distance criterion. This is consequential for benchmark design because apparently reasonable composite metrics can encode surprisingly unbalanced preferences between their constituent metrics.

Critical weaknesses

1. The notion of “optimal tradeoff” is mathematically well defined but not sufficiently justified as the practically desirable notion of compromise. The paper defines optimality as minimizing Fréchet variance under Kendall distance between the two induced rankings. This is elegant, but Kendall distance treats pairwise inversions uniformly, regardless of where they occur in the ranking or how large the underlying performance differences are. For many benchmark settings, swapping the top two methods may matter much more than swapping two methods near the bottom, and a tiny score difference may reasonably be treated differently from a large one. The paper convincingly solves the optimization problem it defines, but provides less evidence that this particular ranking geometry reflects the preferences benchmark users actually care about.

2. The strongest theoretical result relies on a restricted setting, and the empirical evaluation does not yet demonstrate substantial downstream benefit. The analytical construction assumes two-class classification and common fixed class priors across the compared performances. This is an important and practically relevant setting, but substantially narrower than modern multiclass, multilabel, and distribution-shifted evaluation scenarios. Moreover, the experiments primarily verify that arbitrary means deviate from the paper’s mathematically defined optimum; they do not clearly establish that using the proposed tradeoff produces better model selection or more useful benchmark conclusions according to an independent, application-level criterion. This limits my confidence in the broader practical significance of the contribution.

Questions for authors
  1. Clarification: The paper defines the optimal compromise as the midpoint/Karcher mean under Kendall distance, effectively treating all pairwise ranking inversions equally. Can the authors provide either theoretical justification or empirical evidence that this criterion corresponds to desirable benchmark decisions compared with alternatives that emphasize the top of the ranking or account for the magnitude of performance differences? My assessment would be strengthened if the proposed optimum could be shown to align with an independent notion of model-selection quality rather than only with the ranking-distance objective used to define it.

  2. Clarification: How essential is the shared fixed-prior assumption to the core result? In realistic evaluations, classifiers may be compared across datasets, subpopulations, folds, or deployment distributions with different effective class priors. Can the authors clarify which parts of the shortest-path and optimal-tradeoff construction fail when priors differ, and whether an exact or approximate extension remains possible? The practical scope of the theory would materially affect my assessment.

Suggestion for future work, not required for the present assessment: Extending the framework to multiclass classification and alternative ranking geometries, particularly top-weighted ranking distances, would substantially broaden its applicability.

1 comment
AuthorsOpenReview ↗

Answers to reviewer DQS7

We would like to thank the reviewer for his positive feedback and his very insightful suggestions regarding future works, which are fully in line with our long-term goals.

Answer to suggestion 1. It seems to me that what you are looking for is a weighted correlation. So, I suggest to replace Kendall’s tau by Kendall’s weighted tau (see the paper "A weighted Kendall's tau statistic" by G. S. Shieh) (which is already implemented in scipy) and Kendall’s distance by Kendall’s weighted distance (To be honest, I just discovered this paper: "Generalized Distances between Rankings" by R. Kumar and S. Vassilvitskii). Note that not every weighting scheme leads to a distance: this is an indicator that not every weighted Kendall correlation can be used. The good point is that the modification to the optimization problem seems straightforward: we replace a distance by another in the objective function's formula. That being said, I don't yet have enough perspective on this to know exactly how the solution I proposed in this paper (to solve the optimization problem) should be modified. This is a very good suggestion for a future research direction. Nevertheless, the solution (†\dagger) I propose for your second suggestion (see below) could be used.

Answer to suggestion 2. Let us make the distinction between how the optimal trade-off is defined (as an optimization problem) and the technique we present for determining it.

  • The definition of optimal tradeoffs as the ranking scores that minimize the Fréchet variance for Kendall's distance can be used without any constraint on performances. These constraints can be seen as some knowledge that we can use in the design of techniques to solve the optimization problem.
  • (†\dagger) I suggest you take another look at Figure 4, where we have plotted the boundaries between the rankings. In the general case where the priors are not fixed, we cannot use the parameterization (a(π),b(π))(a^{(\pi)}, b^{(\pi)}) tailored to the class priors. Instead, we can use the parameterization (a,b)(a, b) (these quantities are shown in Figure 1) with a=ItpItn+Itpa=\frac{I_{tp}}{ I_{tn}+ I_{tp}} and b=IfnIfp+Ifnb=\frac{I_{fn}}{ I_{fp}+ I_{fn}}. The boundaries are then no longer straight lines but curves with a quadratic equation. These curves partition the search space into cells in which the ranking is constant and thus where the value of the objective function (Fréchet variance) is also constant. In the space (a,b)(a,b), we must therefore minimize a piecewise constant function. Determining which optimization algorithm is best suited for this task is beyond the scope of this paper, but would be a very interesting topic for future research.
  • In the general case where no performance constraint is known, we can still, in certain cases, use the results we have presented in this paper. This is when, on the Tile, the two starting scores lie on the same horizontal or same vertical line. Indeed, in both of these cases, it is known since Piérard et al. (2025) that there exists an f-mean that yields the intermediate scores. This is, in fact, a fairly straightforward extension of the method to find the best tradeoff between precision and recall presented by Piérard et al. (2026). It is because that we have supposed fixed priors that we were able to present a general method in this paper.
  • If you suppose fixed prediction rates instead of fixed class priors, then all we have to do is to swap the ground-truth and predicted classes. That is a swap between the false negatives and false positives that should be done on the importance values and on the probability values. After that, everything that we have presented applies. At the end, the swap has indeed to be undone.

If you want to extent the framework to multiclass classification, then the second point given hereabove (†\dagger) could be a starting point. But then the values aa and bb have to be replaced by points in simplexes: the first is for the relative importance values given to the satisfying outcomes; the second is for the relative importance values given to the unsatisfying outcomes. Intuitively, I feel that the minimization problem becomes very hard. The dimension of the search space can be very high (e.g. with the 1000 classes of imagenet-1K, the search space has 999998 dimensions) and the number of cells in which the ranking is constant remains potentially high. So, my feeling is that we will need clever solutions in multiclass settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated PDF

Dear Reviewers,

We would like to thank all reviewers for their positive feedbacks.

We have noticed that, with the exception of Reviewer DQS7 who finds our presentation « clear and easy to follow », you have indicated that our paper « requires substantial effort in places ». Given the option to add a tenth page, we propose adding a section describing a detailed algorithm to determine a ranking score that is one of the optimal tradeoffs between two base scores (perfectly rank-correlated with some ranking scores), for a finite set of performances sharing the same class priors. The goal of this addition is to point, step by step, to the concepts and equations that are already presented in the Section 3 (Theory). We hope this will make it easier for some readers to understand.

Also, we warmly thank Reviewer vbPZ for his meticulous review of our theoretical section, during which he pointed out a minor error in the text. We previously wrote that the function ff is involutive, whereas it is in fact −f-f that is involutive. We have corrected this in the text. This error has no impact on the rest of the paper. In particular, the code used to generate the presented results is correct.

Have a good day!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.