- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes two families of (rejection) calibrated general Crammer-Singer losses for multi-class classification with abstention in the predictor-rejector setting, where the rejector decides which samples to abstain from (at a cost ). The main contribution of this work is to derive analytically tractable necessary-sufficient conditions for calibration, that can be used to set the parameters of the calibrated CS loss functions to guarantee calibration. This work further derives excess risk bounds for certain cases of the proposed loss families that have at most linear excess risk rate. A generalisation bound is also proved based on the Rademacher complexities of the predictor and rejector. Four variants of the proposed loss functions are evaluated on five datasets (yeast, Imagenet, vehicle, CIFAR-10, and satimage) with respect to the abstention loss on the test set, where they are shown to be competitive with baselines.
Critical strengths
- The results show how to modify any general Crammer-Singer loss for the predictor-rejector framework, and prove relatively interpretable necessary sufficient conditions for the modification to be calibrated.
- The paper is reasonably self-contained, and notation is well-defined, allowing the reader to understand the main results successfully.
Critical weaknesses
The main weakness of the manuscript is its clarity. Important details and discussions are omitted in several places, making it hard to understand the significance and motivation of the key results. Moreover there are some ambiguities in the key definitions and results.
- The definition of the point-wise risk (Lines 154-161) is ambiguous, and it is not clear if takes scalar and vector inputs (), or functions as input (). If is a function of and parametrised by , this should be made explicit by defining .
- The definition of rejection calibration (Lines 162-167) is ambiguous, since it does not mention for which the condition should hold (presumably for all ). Similarly, Proposition 5.1 does not make it clear for which the inequalities should hold (presumably for all ).
- The significance of Proposition 5.1 is unclear, beyond its use as a helper result to prove Theorems 5.3-5.5. Line 253 refers to Ni et al, which states that it is an "unnatural constraint" and that it is "hard to satisfy in the multi-class case". In what manner, is the constraint unnatural? The comment about it being hard to satisfy is confusing, since the condition is equivalent to rejection calibration, thus the comment seems to imply that rejection calibration is hard in the multi-class case. However, this work exactly proves the opposite.
- Similarly, Proposition 5.2 provides limited value to the reader since it is followed by no discussion beyond stating that it underpins the proofs of Theorems 5.3-5.5. In particular, it is hard to judge what or how much progress Proposition 5.2 makes towards the final goal of characterising calibration in this setting.
- Theorem 5.3-5.5 derive necessary and sufficient conditions for MCS and ACS losses. However, the central claim of the paper is that "general CS losses can be calibrated" (abstract, line 53). To meet this claim, additional discussion needs to be made about how to choose or construct the MCS and ACS losses (i.e. choose ) so that the conditions in Theorem 5.3-5.5 are satisfied, beyond simply asserting that this can be done, as in lines 308-310.
- Indeed, the examples presented in the empirical validation (ACS-Hinge, ACS-Logistic, MCS-Hinge, MCS-Logistic) are never explicitly stated in the main text. Presumably, the parameters of these examples are set to achieve calibration using Theorems 5.3-5.5 (this is briefly touched upon in Appendix E, lines 2682-2690). A complete worked-out example on how to do this would be beneficial.
- There are some inconsistencies in the claims. Line 320 states that the excess risk bounds are the main contribution. This is not consistent with the abstract (ignores the excess risk bounds), and introduction (positions it as one of 3 contributions). The claim in the abstract : "our calibration proof is interesting in its own right because it is applicable to broader losses composed of binary ones" is not addressed in the main text, i.e. "broader losses" are not addressed and no dedicated proof sketch is provided.
There are some ambiguities in the proofs of Theorem 5.3 and 5.4 that should be clarified. Specifically, they pertain to the use of Proposition 5.1. For example, in the proof of Theorem 5.3, the goal is to derive the equivalence between the condition and the inequalities in Proposition 5.1. This requires proving both directions. The proof is clear upto the step proving that (it follows from proving that the subdifferential is a singleton). However, the next step, i.e. Equation 35, is a series of equalities. It is not clear why those equalities should hold, or why the equalities imply the equivalence stated above.
Similarly, the proof of Theorem 5.4 attempts to prove the equivalence between the conditions in Theorem 5.4 and Proposition 5.1 at equation 38, which is again a series of equalities. Again, it is not obvious to me why equation 38 implies the equivalence of the two sets of conditions (or why it holds).
I have the following concern about the experimental setup and results. Line 427-428 states that "we choose around the best misclassification error with no abstention" and Lines 2678-2682 clarify that setting in this way encourages a "reasonable number of samples to be abstained". However, there is a big gap in the rejection ratios of the various predictor-rejector methods, for example, from 38% to 89% in Table 3. Indeed, in Table 5, ACS-Logistic rejects nearly all the samples (99.92% rejection ratio). This indicates that the hyperparameter has a disparate effect on different variants of the CS loss during training. Therefore, setting to one global value for one dataset may not be a fair comparison. This raises a number of questions :
- Should the abstention cost during training be detangled from the abstention cost during evaluation, and the former tuned for each method based on a validation loss?
- What threshold (on the maximum class score) is used for the confidence-based baselines? Should this threshold also be tuned for a given value of to ensure a fair comparison?
- Figure 4 shows that the abstention loss monotonically increases with on the CIFAR-10 dataset. Can you give an argument, or intuition, why this is the case?
- Continuing the discussion on the validity of evaluating the methods on one value of : can you compare all predictor-rejector methods with respect to the three metrics across values of , like Figure 4, at least on one dataset?
The following are minor issues and suggestions :
- The exact training procedure of the predictor and rejector pair is not detailed in the main text, or in the appendix.
- Figure 4 shows a non-monotonic trend in the rejection ratio with respect to on CIFAR-10. Why is it not decreasing, i.e. why does increasing not lead to fewer rejections? (This could potentially be explained by the generalization gap between training and test error).
- Figure 1 is confusing, since it mixes the comparison of the method and baseline on various datasets, and the trend of abstention loss on the test set vs the abstention cost (which also affects training for the proposed method). The latter trend appears to be independent of the dataset (see Figure 4). Consider disentangling the two points, and focusing only on one for this illustration. Also consider that the abstention cost has not been introduced at that point in the text, limiting the value of Figure 1.
- Consider making the relationship between and explicit in Section 5.
- Line 1665-1666 : The application of Proposition 5.2 to the given expression for is not immediate, consider making the transformation from to explicit.
- Theorems 5.3-5.5, 6.2, 6.3 are followed by short descriptions of the proofs. However, they provide limited value, for example, Line 281-282 says that the non-increasing property of is used to make the proof. It is not explained why this is interesting, or indeed, how to interpret the construction itself. Similarly, the comment following Theorem 5.4 saying that "a case-by-case analysis and Prop. 5.4 is needed" is of limited value without context about the cases that are considered. Consider omitting these comments altogether, and instead replacing them with a discussion of the result and its significance (e.g. like in line 353-357 interpreting Theorem 6.3), or expanding these comments into a dedicated proof sketch.
- The definition of excess risk is made at the end of Section 3, and is out of place with the rest of the section, which deals with calibration. Consider moving it to the end of Section 2.
- Line 474-477 : the recommendation of choosing logistic variants when is small is not justified well. Only a "speculation" is provided that logistic variants are underconfident in this setting. Why are the logistic variants underconfident, and how does this affect the final goal of calibration?
- Section 3 introduces and defines various notions of calibration which makes the overall presentation confusing. Consider beginning at the definition of the pointwise risk and only defining rejection calibration, since this work only pertains to that. Other definitions of calibration can be discussed in a remark.
- The authors state (line 151-152) that rejection-calibration is necessary for calibration of (r,g) (defined in line 141-143), while that of is not. While a reference is provided, consider justifying both statements, i.e. (a) (r,g) calibration implies r-calibration, and (b) (r,g) calibration does not imply g-calibration, directly in the text.
- A similar notion of calibration in the predictor-rejector setting is the analogue of (r,g) calibration, where is calibrated iff for every minimizer of , where and are the Bayes optimal pair for , as defined in lines 136-139. How does this differ from rejection calibration, and why does this work focus on the latter?
- Can you include a toy experiment, where the optimal risks and/or Bayes predictor rejector pair can be analytically computed? This would give a direct estimate of the calibration error.
- Line 256-257 is confusing, it states that "We will reveal that our losses satisfy this constraint" (are the authors referring to the inequalities in Prop. 5.1?) , "that is, only depends on " (why is it important that only depends on ?)
- Proposition 5.1 should explicitly state that the subdifferential of with respect to at fixed is considered.
- Typo : line 178 : "predictor-rejector".
- Theorem 7.2 : should be independent of .
- Line 239 : The authors probably mean "non-differentiable" instead of "indifferentiable".
- Line 422 : Typo : "Thus we also (MISSING) with".
Questions for authors
See weaknesses.
5 comments
Responses to Reviewer wsLF
- Ambiguities in the proofs of Theorems 5.3 and 5.4: Indeed, in the proof of Theorem 5.3, the goal is to derive the equivalence between the condition and the inequalities in Proposition 5.1. In the proof, we are directly proving the equivalence rather than proving two directions separately. The equalities in (35) come from an equivalent transformation of the inequalities in the specific context. This equivalent transformation holds because, in the context where is non-increasing and specific forms of and are given, the inequalities in Proposition 5.1 can hold only when equality is achieved at the point such that , and the value for which the equality is achieved is surely . Then, under the same context, the equalities in (35) can be further simplified to . In the proof of Theorem 5.4, the equivalent transformation from inequalities in Proposition 5.1 to equalities in (38), and then to , is in a similar way.
- [Q1] Should the abstention cost be detangled during training from that during evaluation: Since our calibration guarantee requires or , to satisfy some equalities about abstention which is used in evaluation (as shown in Theorem 5.3-5.5), fixing the abstention cost in training to be the same as that value is reasonable. Otherwise, tuning the abstention cost heuristically during training may lead to the calibration guarantee not holding.
- [Q2] What threshold is used for the confidence-based baselines: Theoretically grounded are considered in the submitted version. The updated results with tuned can be found in Tables 1-2 in our reply to reviewer 5hds. For both [Ramaswamy+2018] (CS-hinge) and [Ramaswamy+2018] (BEP), the tuned threshold is 0.7 on the vehicle dataset and 0.8 on Imagenette.
- [Q3] Why does the abstention loss monotonically increase with in Figure 4: A larger means that it is harder for the learner to avoid misclassification on low-confidence samples via abstention, thus tending to have a larger abstention loss.
- [Q4] Can the methods be compared across values of on one dataset: is not a hyperparameter, but an environment setting. A reasonable value of is around the best misclassification error with no abstention. We will add comparison experiments on different in a future version.
- Training procedure of the predictor and rejector: The formulation of is given in Section 4.2, and details like the optimization method are given in Appendix E as mentioned in line 429. We believe that sufficient information is given in Appendix E, but please let us know if some points should be detailed further.
- Why the trend in Figure 4 is not monotonic: Although the trend in Figure 4 is not absolutely monotonic, the rejection ratio generally tends to decrease with the increasing abstention cost . Note that the rejection ratio is affected not only by the rejection ratio, but also by the training condition of the predictor (note that the predictor and the rejector are trained jointly). Thus, when the abstention cost increases, if the predictor is not trained sufficiently due to an insufficient number of samples, the rejection ratio can also increase within a small range.
- Figure 1: For a given dataset, a reasonable value of on this dataset is around the best misclassification error with no abstention on this dataset. Thus, when displaying Figure 1, we actually implicitly assumed that is naturally determined from the dataset and meant to focus on comparing different methods on various datasets. The abstention cost indeed has not been introduced formally at that point in the text, and we will move the introduction of to an earlier position in a future version.
- and in Section 5: and are strongly related but not the same quantity. is a more general quantity with as input, and does not have as input. See our reply below for the quantitative relationship between and .
Responses to Reviewer wsLF
- Application of Proposition 5.2 to the given expression for : By Proposition 5.2, can be obtained by the following transformation: \begin{aligned} W_{\mathrm{MCS}}(r, \boldsymbol{g}_{\boldsymbol{\eta}}^{\dagger}; \boldsymbol{\eta}) &= h_{\eta^*}(r)\\ &= \min_{\Delta\geqslant 0}\left\{\eta^*\Phi(\Delta, r) + (1 - \eta^*)\Phi(-\Delta, r)\right\}\\ &= \min_{\Delta\geqslant 0}\left\{\eta^*\phi(\Delta)\psi_1(r) + (1 - \eta^*)\phi(-\Delta)\psi_1(r) + \psi_2(r)\right\}\\ &= \min_{\Delta\geqslant 0}\left\{\eta^*\phi(\Delta) + (1 - \eta^*)\phi(-\Delta)\right\}\psi_1(r)+ \psi_2(r)\\ &= H_{\eta^*}\psi_1(r) + \psi_2(r). \end{aligned} We would like to add a related statement in a future version to make the transform from to explicit.
- Descriptions for the proofs of Theorems 5.3-5.5, 6.2, 6.3: During the proof of 5.3, The non-increasing of is important because, with this property, we can reduce the equalities in Proposition 5.1 to a calibration condition in equality. During the proof of Theorem 6.2, the different cases of and are needed because the infimum of , , and depends on them. Besides, in Remarks 6.2 and 6.3, we state the significance of Theorems 6.2 and 6.3. We would like to expand the short descriptions of the proofs of Theorems 5.3-5.5, 6.2, and 6.3 in the submitted version into a dedicated proof sketch like the one above in a future version if space permits.
- Definition of excess risk bound: The definition of the excess risk bound is placed in Section 3 because it has a strong relationship with calibration (as mentioned in our reply to [W7]). We would like to add the statement about the relationship between them at the end of Section 3 to make the connection more fluent in a future version.
- Reason for lower confidence of logistic variants: As mentioned in lines 2722-2733, the logistic variants are speculated to be more conservative in accepting low-confidence samples. The detailed cause analysis may be an interesting future direction. All of our losses in experiments are calibrated, and the lower confidence of logistic variants does not affect the final goal of calibration because the calibration is under the assumption of infinite samples. However, in a practical scenario with finite samples, there might exist a lower confidence phenomenon in logistic variants.
- Various notations of calibration: we also think the reviewer's suggestion is reasonable, while the notations of calibration of and calibration of and separately can serve as a foundation to introduce the notation of rejection-calibration of . The motivation to introduce the rejection-calibration comes from the fact that, for the calibration of , the calibration of is necessary while that of is not. Besides, the definition of rejection-calibration of actually can be interpreted as the rejection-calibration of (where is the minimizer of the corresponding pointwise risk).
- Justification for the necessary condition for calibration of : Related justification can be found at the end of [Ramaswamy+2018, Section 1.2], where they take SVM as an example: SVM is consistent for standard binary classification, but it avoids consistent estimation of the conditional probability (calibration for the predictor). Since it is a generally accepted conclusion, we will add discussions in the appendix and add a reference to [Ramaswamy+2018, Section 1.2] considering the space limitation.
- Calibration condition for : Note that, the calibration for does not necessarily need the calibration of , i.e., , while the calibration of is enough (where are minimizers for , in reviewer's notation). The difference is that rejection calibration is a more streamlined necessary condition. Besides, consider that pointwise risk is easier to analyze than ; in this work we focus on rejection calibration for (where is a minimizer for pointwise risk ).
Responses to Reviewer wsLF
We thank the reviewer for the detailed review and suggestions. Regarding the issues pointed out, we would like to add the following information: