- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors are motivated by the plain fact that two samples can receive the same confidence score while having different probabilities of being correct since one has lower-quality data. They propose Quality-Aware Uncertainty Calibration (QAC), a second post-hoc calibration stage that adjusts a classifier’s probabilities using both the output of an existing calibrator and a per-example quality score Q. QAC fits a separate gradient-boosted regressor for each class, constrained to be monotone in that class’s calibrated probability but free to learn the relationship with Q. The experiments are conducted on seven datasets, nine base calibrators, and ten seeds. The empirical results show QAC consistent improveson a synthetic sensor task and three datasets with multi-annotator information, while does not improve results on the two datasets whose quality scores are computed from the model input. On other metrics, the performance is mixed.
Critical strengths
- The paper is easy to follow and the motivation is clear.
- The definition of quality score Q is novel, which adds information beyond the model output.
- The evaluation is soundness for a calibration paper.
Critical weaknesses
- The empirical results achieve the best performance on ChaosNLI, CIFAR-10H, and CIFAR-10N, while do not improve on AudioMUSAN and Camelyon17, and on eICU, it shows no significant gain.
- Some quality signals are closely tied to the labels being evaluated. For example, on ChaosNLI and the CIFAR annotation datasets, Q is derived from annotator distributions that also determine or relate closely to the evaluation labels.
- The method fits a class-wise regressor to estimate each class probability and then normalizes the outputs. However, according to the performance on ECE and NLL, it seems the method could not guarantee calibrated multiclass probabilities after normalization.
- The experiments should clarify which comparisons are covered by the correction and how uncertainty over datasets or deployment settings is represented.
Questions for authors
- For the annotator-based dataset, like CIFAR-10 series, how is the evaluation label constructed, and how much information does Q share with that label by construction?
- Why is the performance of ECE and NLL so poor, and are put in the appendix?
2 comments
Reply to Reviewer rTAL (Part 2 of 2)
Weakness 4. The experiments should clarify which comparisons are covered by the correction and how uncertainty over datasets or deployment settings is represented.
We acknowledge that this was unclear. Sec. 4.1 and App. D now state it.
- Main comparison: QAC with the real q against the same second stage with a constant q. It isolates the effect of q.
- Its interval. Our ten seeds split one pool of data, so they are not independent, and a standard interval is too narrow. We widen the 95% intervals with the Nadeau and Bengio correction, by about 3.3 times when the calibration and test sets have the same size. Unlike the first version's Bonferroni correction, it addresses dependence, not multiple comparisons. It covers the main comparison, QAC against the best plain calibrator, the effect of q per base calibrator, the leftover overconfidence, and QAC against competitors (listed in App. D); not Camelyon17, which has one fixed split.
- "Informative" dataset: the widened interval of the main comparison excludes zero in favor of the real q. This holds for E_toy, ChaosNLI, CIFAR-10H and CIFAR-10N. The label describes a result; it is not a test made beforehand.
- Other comparisons carry no claim of significance; Fig. 3 and Table 24 say so.
- Across datasets: we report each dataset separately. The one claim across datasets, the ranking of gains by leftover overconfidence, has a bootstrap interval over pairs (App. E.6).
- Deployment: we added two tests. In the labor survey (CPS), everything is fitted on two months and applied unchanged to 31 other months (Sec. 4.4). On five splits where calibration and test samples come from different periods or sites, recalibrating on samples from the target lowers the Brier score 2 to 13 times more than any quality score we tested (Sec. 4.7, App. E.9.6). QAC does not correct a shift. Recalibrating does, but it needs labeled samples from the target, and QAC does not.
Question 1. For the annotator-based datasets such as the CIFAR-10 series, how is the evaluation label constructed, and how much information does q share with it by construction?
For CIFAR-10H and CIFAR-10N, the label is the original CIFAR-10 test label. It is not built from the CIFAR-10H votes, so the label and q come from different annotators. q is computed from the CIFAR-10H votes: the share of annotators on the most voted class (CIFAR-10H) or the negative entropy of the votes (CIFAR-10N). q measures how much the annotators agree, not which class they chose. The two are related through the image: the most voted class equals the label on 99.2% of images (Sec. 5). We did not measure the mutual information between q and the label; q predicts the model's errors a little beyond its output (gain in of +0.010 and +0.028, Table 1). For ChaosNLI, where they share annotators, disjoint annotators give the same effect of q (-0.0141 against -0.0138; App. E.7). Table 1 now states who produced each q.
Question 2. Why is the performance on ECE and NLL so poor, and why are they in the appendix?
We agree that they belong in the main text. The Brier score was our main metric because it is a proper score that needs no binning (unlike ECE) and is not dominated by a few probabilities near zero (unlike NLL). ECE and NLL are now in Table 3, and Sec. 4.3 states where QAC is worse.
Table R2 (from Table 3). Mean of 10 seeds; lower is better.
| Metric | Method | E_toy | ChaosNLI | CIFAR-10H | CIFAR-10N | eICU |
|---|---|---|---|---|---|---|
| ECE | Dirichlet calibration | .0117 | .0321 | .0087 | .0127 | .0066 |
| ECE | QAC (ours) | .0153 | .0289 | .0080 | .0117 | .0079 |
| NLL | Dirichlet calibration | .3385 | .6484 | .1809 | .3064 | .2716 |
| NLL | QAC (ours) | .3103 | .6522 | .2132 | .3582 | .2722 |
- ECE: QAC is lower than Dirichlet calibration on ChaosNLI and the two CIFAR datasets, and higher on E_toy and eICU. No difference is significant (intervals computed for this reply), so they neither show nor rule out an ECE penalty.
- NLL: QAC is lower on E_toy and higher on the other four, by 17-18% on the two CIFAR datasets.
- Cause of the NLL penalty (App. E.4, Table 18): QAC gives the true class a probability below 0.001 on 0.30% (CIFAR-10H) and 0.51% (CIFAR-10N) of test samples. NLL penalizes these few cases heavily: they account for about a quarter of QAC's NLL. With a lower bound on the probabilities, chosen on the calibration set for each method, the difference to Dirichlet calibration falls from +0.0323 to +0.0067 (CIFAR-10H) and from +0.0518 to +0.0034 (CIFAR-10N). The paper reports the standard NLL, without this bound.
- On Audio MUSAN and Camelyon17, where q is not informative, QAC is worse than Dirichlet calibration on both metrics (Tables 11 and 12). QAC should not be used there; Table 1 shows, before fitting, that q does not predict errors beyond the model's output.
We thank the reviewer for the careful review. We hope these answers and the new experiments address the reviewer's concerns.
Reply to Reviewer rTAL (Part 1 of 2)
We thank the reviewer for the careful review. We appreciate the reviewer acknowledging that the manuscript is easy to follow, that its motivation is clear, that the definition of the quality score q is new, as it adds information beyond the model output, and that the evaluation is sound. The review led to three new experiments and three new analyses: a check of where QAC can gain, labels and q built from disjoint annotators, a dataset whose quality score comes from the data provider, not from the label, intervals that account for our data splits, splits with a shift between calibration and test data, and an analysis of the NLL.
Weakness 1. QAC performs best on ChaosNLI, CIFAR-10H and CIFAR-10N, does not improve on Audio MUSAN and Camelyon17, and shows no significant gain on eICU.
We agree. The manuscript now explains most of this pattern with a check that can be run in advance (Sec. 4.6, Table 5). In our data, QAC helped only where the calibrated model was still more overconfident on low-quality cases than on the others. We call this gap leftover overconfidence: confidence minus accuracy on low-quality samples, minus the same on the others.
Table R1 (from Tables 5 and 15). Leftover overconfidence after Dirichlet calibration on test samples (95% interval), and the effect of q: Brier score of QAC with the real q minus with a constant q (negative: q helps). Low-quality: the worst fifth of q (E_toy: sensors disagree).
During the rebuttal, some numbers changed as we refitted three plain calibrators that were too weak (App. C.4); a plain calibrator is a post-hoc calibrator that does not use the quality score q. The main result is unchanged: the effect of q, measured against the same second stage with a constant q, is significant on the same four datasets. QAC still has the lowest Brier score on these four (Table 2), but its margin over the best plain calibrator is now significant only on E_toy and ChaosNLI. On the two CIFAR datasets, q helps significantly, but QAC is not significantly better than the best plain calibrator.
Weakness 2. Some quality signals are closely tied to the evaluation labels: on ChaosNLI and the CIFAR annotation datasets, q is derived from annotator distributions that also determine or relate to the labels.
We agree for ChaosNLI, where the label is the majority vote, and q is the entropy of the same votes.
Weakness 3. The method fits one regressor per class and normalises; judging from ECE and NLL, it does not seem to guarantee calibrated multiclass probabilities after normalisation.
We agree: it does not. Sec. 3 now states that with more than two classes, the output has no calibration guarantee and that the second stage can change the predicted class (0.4% to 8.1% of test samples; App. C.4). In ECE, QAC is lower than Dirichlet calibration on ChaosNLI and both CIFAR datasets, and higher on E_toy and eICU (Table 3).