acceptodds
Under review as a conference paper at ICLR 2027

QAC: Data Quality-Aware Uncertainty Calibration for Trustworthy AI

Abstract

A classifier's predicted probability is its statement of uncertainty about the label, and calibration makes this statement match observed frequencies. Post-hoc calibrators use only the model's output, so two samples with the same output receive the same probability. Consider two patients whose laboratory values are almost the same, but one patient's value is wrong because the blood sample was stored too long before analysis: the model gives both the same prediction and the same confidence, although the second prediction is more likely to be wrong. Laboratories and other data pipelines often record such problems. We call a per-sample value that describes how reliable a data point's input or label is its data quality. When it is not available at training time, it cannot be added to the model's input, but it can be given to the calibrator. A standard result implies that, for the best possible predictor, a data-quality value can improve the prediction only if it carries information about the label that the input does not. We propose Quality-Aware Uncertainty Calibration (QAC), a second calibration stage that can follow any existing calibrator. For each class, it estimates on the calibration set how often the class is correct given the first-stage probability and the quality score; the direction of the effect of quality is learned, not assumed. On seven datasets, nine base calibrators and ten seeds, and compared with the same stage given a constant quality score, QAC lowers the Brier score by - on the four datasets where quality is informative (a simulation and three datasets where quality is annotator agreement). It does not help on the two datasets whose quality score is computed from the input, and gives no significant gain on a clinical dataset. The gain is mainly in separating correct from wrong predictions (error-detection AUROC to ), while the separation between classes improves about a quarter as much or less.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty calibration, post-hoc calibration, data quality, aleatoric uncertainty, selective prediction, error detection, proper scoring rules, trustworthy machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rTALOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors are motivated by the plain fact that two samples can receive the same confidence score while having different probabilities of being correct since one has lower-quality data. They propose Quality-Aware Uncertainty Calibration (QAC), a second post-hoc calibration stage that adjusts a classifier’s probabilities using both the output of an existing calibrator and a per-example quality score Q. QAC fits a separate gradient-boosted regressor for each class, constrained to be monotone in that class’s calibrated probability but free to learn the relationship with Q. The experiments are conducted on seven datasets, nine base calibrators, and ten seeds. The empirical results show QAC consistent improveson a synthetic sensor task and three datasets with multi-annotator information, while does not improve results on the two datasets whose quality scores are computed from the model input. On other metrics, the performance is mixed.

Critical strengths
  1. The paper is easy to follow and the motivation is clear.
  2. The definition of quality score Q is novel, which adds information beyond the model output.
  3. The evaluation is soundness for a calibration paper.
Critical weaknesses
  1. The empirical results achieve the best performance on ChaosNLI, CIFAR-10H, and CIFAR-10N, while do not improve on AudioMUSAN and Camelyon17, and on eICU, it shows no significant gain.
  2. Some quality signals are closely tied to the labels being evaluated. For example, on ChaosNLI and the CIFAR annotation datasets, Q is derived from annotator distributions that also determine or relate closely to the evaluation labels.
  3. The method fits a class-wise regressor to estimate each class probability and then normalizes the outputs. However, according to the performance on ECE and NLL, it seems the method could not guarantee calibrated multiclass probabilities after normalization.
  4. The experiments should clarify which comparisons are covered by the correction and how uncertainty over datasets or deployment settings is represented.
Questions for authors
  1. For the annotator-based dataset, like CIFAR-10 series, how is the evaluation label constructed, and how much information does Q share with that label by construction?
  2. Why is the performance of ECE and NLL so poor, and are put in the appendix?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer rTAL (Part 1 of 2)

We thank the reviewer for the careful review. We appreciate the reviewer acknowledging that the manuscript is easy to follow, that its motivation is clear, that the definition of the quality score q is new, as it adds information beyond the model output, and that the evaluation is sound. The review led to three new experiments and three new analyses: a check of where QAC can gain, labels and q built from disjoint annotators, a dataset whose quality score comes from the data provider, not from the label, intervals that account for our data splits, splits with a shift between calibration and test data, and an analysis of the NLL.

Weakness 1. QAC performs best on ChaosNLI, CIFAR-10H and CIFAR-10N, does not improve on Audio MUSAN and Camelyon17, and shows no significant gain on eICU.

We agree. The manuscript now explains most of this pattern with a check that can be run in advance (Sec. 4.6, Table 5). In our data, QAC helped only where the calibrated model was still more overconfident on low-quality cases than on the others. We call this gap leftover overconfidence: confidence minus accuracy on low-quality samples, minus the same on the others.

Table R1 (from Tables 5 and 15). Leftover overconfidence after Dirichlet calibration on test samples (95% interval), and the effect of q: Brier score of QAC with the real q minus with a constant q (negative: q helps). Low-quality: the worst fifth of q (E_toy: sensors disagree).

DatasetLeftover overconfidenceEffect of q on the Brier score
E_toy+0.161 [+0.096, +0.225]-0.0108
ChaosNLI+0.114 [+0.083, +0.146]-0.0155
CIFAR-10H+0.062 [+0.038, +0.086]-0.0011
CIFAR-10N+0.104 [+0.068, +0.140]-0.0039
eICU-0.008 [-0.023, +0.007]+0.0001
Audio MUSAN+0.011 [-0.013, +0.035]+0.0008
Camelyon17+0.038 [+0.034, +0.042]+0.0029
  • Where QAC gains, the gap is large. On eICU, there is no gap and no gain.
  • Over 27 pairs of a dataset and a quality score with calibration and test cases from one population, a squared form of the gap ranks the gains of one-stage QAC (Spearman 0.84 [0.59, 0.95]; App. E.6; 10 of the 27 gaps on test cases). Under a shift, it does not (0.09 over 9 pairs).
  • On Audio MUSAN and Camelyon17, q makes QAC slightly worse; there, q does not predict errors beyond the model’s output (Table 1), and Camelyon17 also has a hospital shift (Sec. 4.7).
  • The check is not sufficient: on Camelyon17 a gap remains without a gain (Sec. 4.6).

During the rebuttal, some numbers changed as we refitted three plain calibrators that were too weak (App. C.4); a plain calibrator is a post-hoc calibrator that does not use the quality score q. The main result is unchanged: the effect of q, measured against the same second stage with a constant q, is significant on the same four datasets. QAC still has the lowest Brier score on these four (Table 2), but its margin over the best plain calibrator is now significant only on E_toy and ChaosNLI. On the two CIFAR datasets, q helps significantly, but QAC is not significantly better than the best plain calibrator.

Weakness 2. Some quality signals are closely tied to the evaluation labels: on ChaosNLI and the CIFAR annotation datasets, q is derived from annotator distributions that also determine or relate to the labels.

We agree for ChaosNLI, where the label is the majority vote, and q is the entropy of the same votes.

  • Disjoint annotators (App. E.7, Table 21). We built the label from votes 1-50 and q from votes 51-100. The effect of q on the Brier score is -0.0141, compared with -0.0138 with shared annotators.
  • A quality score not derived from annotators (Sec. 4.4, Table 4). In a US labor survey, the publisher flags each answer it edited. QAC with the age edit flag lowers the Brier score of cases with an edited age from 0.639 to 0.589 in 31 held-out months. Given the same flag, three calibrators fitted per flag value do better (0.565 to 0.574, lower ECE) and change 9-18% of these predictions (QAC 2%). That bears on the choice of method, not on this point: a score not derived from the label can help.
  • CIFAR datasets: the label is the original CIFAR-10 test label, which is not built from these annotators' votes.

Weakness 3. The method fits one regressor per class and normalises; judging from ECE and NLL, it does not seem to guarantee calibrated multiclass probabilities after normalisation.

We agree: it does not. Sec. 3 now states that with more than two classes, the output has no calibration guarantee and that the second stage can change the predicted class (0.4% to 8.1% of test samples; App. C.4). In ECE, QAC is lower than Dirichlet calibration on ChaosNLI and both CIFAR datasets, and higher on E_toy and eICU (Table 3).

AuthorsOpenReview ↗

Reply to Reviewer rTAL (Part 2 of 2)

Weakness 4. The experiments should clarify which comparisons are covered by the correction and how uncertainty over datasets or deployment settings is represented.

We acknowledge that this was unclear. Sec. 4.1 and App. D now state it.

  • Main comparison: QAC with the real q against the same second stage with a constant q. It isolates the effect of q.
  • Its interval. Our ten seeds split one pool of data, so they are not independent, and a standard interval is too narrow. We widen the 95% intervals with the Nadeau and Bengio correction, by about 3.3 times when the calibration and test sets have the same size. Unlike the first version's Bonferroni correction, it addresses dependence, not multiple comparisons. It covers the main comparison, QAC against the best plain calibrator, the effect of q per base calibrator, the leftover overconfidence, and QAC against competitors (listed in App. D); not Camelyon17, which has one fixed split.
  • "Informative" dataset: the widened interval of the main comparison excludes zero in favor of the real q. This holds for E_toy, ChaosNLI, CIFAR-10H and CIFAR-10N. The label describes a result; it is not a test made beforehand.
  • Other comparisons carry no claim of significance; Fig. 3 and Table 24 say so.
  • Across datasets: we report each dataset separately. The one claim across datasets, the ranking of gains by leftover overconfidence, has a bootstrap interval over pairs (App. E.6).
  • Deployment: we added two tests. In the labor survey (CPS), everything is fitted on two months and applied unchanged to 31 other months (Sec. 4.4). On five splits where calibration and test samples come from different periods or sites, recalibrating on samples from the target lowers the Brier score 2 to 13 times more than any quality score we tested (Sec. 4.7, App. E.9.6). QAC does not correct a shift. Recalibrating does, but it needs labeled samples from the target, and QAC does not.

Question 1. For the annotator-based datasets such as the CIFAR-10 series, how is the evaluation label constructed, and how much information does q share with it by construction?

For CIFAR-10H and CIFAR-10N, the label is the original CIFAR-10 test label. It is not built from the CIFAR-10H votes, so the label and q come from different annotators. q is computed from the CIFAR-10H votes: the share of annotators on the most voted class (CIFAR-10H) or the negative entropy of the votes (CIFAR-10N). q measures how much the annotators agree, not which class they chose. The two are related through the image: the most voted class equals the label on 99.2% of images (Sec. 5). We did not measure the mutual information between q and the label; q predicts the model's errors a little beyond its output (gain in R2R^2 of +0.010 and +0.028, Table 1). For ChaosNLI, where they share annotators, disjoint annotators give the same effect of q (-0.0141 against -0.0138; App. E.7). Table 1 now states who produced each q.

Question 2. Why is the performance on ECE and NLL so poor, and why are they in the appendix?

We agree that they belong in the main text. The Brier score was our main metric because it is a proper score that needs no binning (unlike ECE) and is not dominated by a few probabilities near zero (unlike NLL). ECE and NLL are now in Table 3, and Sec. 4.3 states where QAC is worse.

Table R2 (from Table 3). Mean of 10 seeds; lower is better.

MetricMethodE_toyChaosNLICIFAR-10HCIFAR-10NeICU
ECEDirichlet calibration.0117.0321.0087.0127.0066
ECEQAC (ours).0153.0289.0080.0117.0079
NLLDirichlet calibration.3385.6484.1809.3064.2716
NLLQAC (ours).3103.6522.2132.3582.2722
  • ECE: QAC is lower than Dirichlet calibration on ChaosNLI and the two CIFAR datasets, and higher on E_toy and eICU. No difference is significant (intervals computed for this reply), so they neither show nor rule out an ECE penalty.
  • NLL: QAC is lower on E_toy and higher on the other four, by 17-18% on the two CIFAR datasets.
  • Cause of the NLL penalty (App. E.4, Table 18): QAC gives the true class a probability below 0.001 on 0.30% (CIFAR-10H) and 0.51% (CIFAR-10N) of test samples. NLL penalizes these few cases heavily: they account for about a quarter of QAC's NLL. With a lower bound on the probabilities, chosen on the calibration set for each method, the difference to Dirichlet calibration falls from +0.0323 to +0.0067 (CIFAR-10H) and from +0.0518 to +0.0034 (CIFAR-10N). The paper reports the standard NLL, without this bound.
  • On Audio MUSAN and Camelyon17, where q is not informative, QAC is worse than Dirichlet calibration on both metrics (Tables 11 and 12). QAC should not be used there; Table 1 shows, before fitting, that q does not predict errors beyond the model's output.

We thank the reviewer for the careful review. We hope these answers and the new experiments address the reviewer's concerns.

Reviewer_sBn1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Quality-Aware Uncertainty Calibration, a second calibration stage that combines existing predicted probabilities with sample-level quality information. It fits one boosted-tree regressor per class, constrained to increase with the original probability while learning the effect of quality. Information-theoretic results are acquired to explain when additional information can enhance prediction. Experiments across several datasets and calibration methods show improvements in probability estimates and error detection in some settings.

Critical strengths

The experiments evaluates whether the quality scores actually account for the gains. The authors keep the second calibration stage but ablate the quality scores with constant or shuffled values. This helps indicate whether it benefits from quality, beyond an extra calibration step.

Critical weaknesses

The practical benefit is not yet well demonstrated. Those real-data results which are positive, all require multiple human annotations to construct the quality score. Such information can be expensive for new examples. In contrast, the experiments using measurement or record-quality information show no clear benefit. The results therefore show limited support for the paper's motivation.

Questions for authors
  1. When human annotations are already available to compute the quality score, what practical benefit does QAC provide beyond using those annotations directly?

  2. Are Tables 2 and 5 directly comparable? If so, why is Dirichlet excluded from the comparison in Table 2?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer sBn1 (Part 1 of 2)

We thank the reviewer for the clear and comprehensive review. We appreciate that the reviewer found useful the experiments that keep the second calibration stage and replace the quality score with a constant or shuffled value for testing whether the gain comes from quality. This review led to six new experiments.

Weakness. The practical benefit is not yet well demonstrated: the positive real-data results all need several human annotations to build the quality score, which can be expensive for new examples; the experiments with measurement or record-quality information show no clear benefit; so the motivation has limited support.

We agree with both observations and addressed each with a new experiment.

(a) A quality score that exists for every new case, at no annotation cost (new Sec. 4.4, Table 4). The US Current Population Survey (CPS) is a monthly labor survey. Its publisher flags each answer it edited. We fitted the model (boosted trees, 7 classes) with a post-hoc calibrator that does not use the quality score q, which we call the "plain calibrator" in the manuscript (in this case, Dirichlet calibration), and QAC on two months of 2024, then applied them unchanged to 31 other months.

Result: after plain calibration, the model is overconfident by 0.175 on age-edited cases and close to calibrated on the others. QAC with the age-edit flag as q lowers the Brier score for these cases from 0.639 to 0.589 and their overconfidence to 0.074. It changes the other cases by at most 0.0003. We identified the age-edited cases on the 2024 data and then tested them on 31 other months, where the result held.

Table R1 (from Tables 4 and 19 in the manuscript; -: denotes results not reported). Dataset: CPS, 10 seeds. Overconfidence: confidence minus accuracy after plain calibration.

CasesnOverconfidenceBrier, plainBrier, QACBrier, Dirichlet refitted per flag value
Age edited45,917+0.1750.6390.5890.565
Other answers edited only109,025+0.00050.38630.3866-
Not edited1,160,911-0.0050.3530.353-

In the manuscript, we state two limits of this result.

  • Given the same flag, three calibrators fitted separately for flagged and unflagged cases do better (0.565 to 0.574, lower ECE; last column) and change 9-18% of these predictions (QAC 2%). So for a single yes/no flag, per-group calibration is the better tool. Our implementations of Pan et al., MBCT, and Kelly & Smyth with its published leaf size stay at 0.636-0.640; with the any-edit flag, Kelly & Smyth also beats QAC (0.596 vs 0.609; Table 19). When the quality score takes many values, an equivalent approach is to group cases into quality tiers and fit one calibrator per tier. QAC has a lower mean Brier score than this method on the four datasets where q helps, significantly on both CIFAR datasets (Table 17).
  • On a second survey (American Community Survey), no method that keeps the model frozen lowers the flagged cases’ Brier score by more than 0.0024, so a provider’s edit flag does not always help.

Where the overconfidence sits: for 5.7% of the age-edited cases, the unedited age is known from an adjacent month. The overconfidence is 0.194 when the two ages differ by more than a year, and 0.034 otherwise (Sec. 4.4; App. E.5).

(b) Why the real measurement and record-quality datasets show no benefit (new Sec. 4.6, Table 5). In our data, QAC helped only where the calibrated model was still more overconfident on low-quality cases. We now measure this leftover overconfidence on seven datasets. It can be computed on the calibration set.

  • On the four datasets where QAC helps, it is 0.06 to 0.16.
  • On the clinical dataset eICU (record-quality scores), it is -0.008 with an interval that includes zero: the plain calibrator has left little for q to correct. QAC then changes the Brier score by +0.0001 against the constant-q control (Table 15).
  • Over 27 pairs of a dataset and a quality score with calibration and test cases from one population, a squared form of this gap ranks the gains of one-stage QAC (Spearman 0.84 [0.59, 0.95]; App. E.6, Table 20; 10 of the 27 gaps on test cases); under a shift it does not (0.09, 9 pairs).
  • On Audio MUSAN and Camelyon17, q makes QAC slightly worse; there, q does not predict errors beyond the model’s output (Table 1), and Camelyon17 also has a hospital shift (Sec. 4.7). However, the check is not sufficient: on Camelyon17 a gap remains without a gain (Sec. 4.6).

So the motivation now has support beyond annotations: with a provider's edit flag, available for every new case, QAC lowers the flagged cases' Brier score in 31 months not used for fitting, and a screen for a gain can be measured beforehand.

AuthorsOpenReview ↗

Reply to Reviewer sBn1 (Part 2 of 2)

Question 1. When human annotations are already available to compute the quality score, what practical benefit does QAC provide beyond using those annotations directly?

We agree: when the votes on a new sample are available, they should be used directly. We tested it (App. E.7, Table 22, Sec. 4.7) by combining the votes with the model's probabilities in a fitted log-linear pool. This pool has a lower Brier score than QAC for every number of votes.

Table R2 (from Table 22 of the manuscript). Brier score of QAC (q computed from m votes) against a pool of the model and the same m votes. We split the votes on ChaosNLI as follows: label from votes 1-50, methods see votes 51-100. This run uses the first version's untuned Dirichlet calibration, so QAC's values differ slightly from Table 2.

Datasetm = 3m = 10all votes
ChaosNLI.3965 vs .2872.3886 vs .2202.3841 vs .1614
CIFAR-10H.0879 vs .0204.0874 vs .0120.0870 vs .0094
CIFAR-10N.1433 vs .0263.1419 vs .0142.1406 vs .0105

Votes are opinions about the label itself, so pooling them estimates the label directly; QAC uses only how much the annotators agree. The annotation datasets are a controlled test of the mechanism: with the same second stage, the gain comes from q (Table R3), also when q and the label come from disjoint annotators (ChaosNLI, Table 21). QAC is meant for a quality score that does not reveal the label, such as the provider’s edit flag of the US Current Population Survey (CPS). The paper states the need for annotators (Limitations).

Question 2. Are Tables 2 and 5 directly comparable? If so, why is Dirichlet excluded from the comparison in Table 2?

Yes. Both used Dirichlet calibration as the first stage, and the QAC values in Table 5 match those in Table 2, so Dirichlet calibration should have been included in Table 2. We thank the reviewer for pointing this out. Table 2 now lists all nine plain calibrators, including Dirichlet calibration, with QAC on top. It also has two second stages without q, to test whether the gain comes from q or from calibrating twice.

Table R3 (from Table 2 of the paper). Brier score, mean of 10 seeds; bold: lowest per column. q is informative (the real q is significantly better than a constant q) in the first four columns.

MethodE_toy (simulated sensors)ChaosNLICIFAR-10HCIFAR-10NeICU
Dirichlet calibration (best plain calibrator).2049.3932.0859.1426.1529
Second stage with a constant q (control).2064.3942.0863.1437.1532
Best plain calibrator as second stage.2049.3929.0858.1426.1529
QAC (ours).1956.3787.0852.1398.1532
  • On Dirichlet calibration, a second stage alone does not help: with a constant q it is slightly worse, and no plain calibrator used as a second stage improves on it by more than 0.0004.
  • The quality score does: QAC with the real q is below the control by 0.0108, 0.0155, 0.0011 and 0.0039 on the four informative datasets. Each difference is significant.
  • Against Dirichlet calibration itself, QAC is lower by 4.5%, 3.7%, 0.8% and 2.0%. This difference is significant on E_toy and ChaosNLI only. On the two CIFAR datasets, q helps significantly, but QAC is not significantly better than the best plain calibrator.
  • On eICU, q is not informative, and QAC brings nothing, consistent with the conditional claim.

Additionally, we made two revisions in the manuscript based on the review:

  • We found that three plain calibrators were weaker in our experiments than they should be: temperature scaling stopped before convergence, Dirichlet calibration was not tuned, and top-versus-all isotonic regression spread the remaining probability evenly. We refitted them (App. C.4): temperature scaling now runs to convergence and Dirichlet calibration uses its published regulariser, tuned by cross-validation (untuned on the survey); the new isotonic rule is ours. On ChaosNLI, QAC's margin over Dirichlet calibration is 3.7% both before the refit (0.3800 against 0.3945, old Table 5) and after it; the first version's 7.6% was against isotonic regression, because its Table 2 lacked Dirichlet calibration.
  • The old Table 5 (alternative designs of the second stage) is now Table 8 in App. E.1.3. It still uses the untuned Dirichlet calibration, so it is not directly comparable with Table 2; its caption says so and gives the Table 2 values.

ECE and NLL are now in the main text (Table 3). QAC's NLL is 17-18% worse than Dirichlet calibration on the two CIFAR datasets because it assigns a few probabilities close to zero. A lower bound on the probabilities removes most of this gap (App. E.4, Table 18: 0.032 to 0.007 and 0.052 to 0.003), but QAC stays slightly worse in NLL.

We thank the reviewer for the clear and comprehensive review. We hope these answers and the new experiments address the reviewer's concerns.

Reviewer_bmVqOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a post-hoc calibration method that uses a per-sample data-quality score Q together with the probabilities produced by an existing calibrator. A second stage fits one gradient-boosted regressor per class, monotone in the class probability and unconstrained in Q, and normalises the outputs. Standard results on proper scoring rules motivate when additional quality information can improve predictions. Experiments on seven datasets, nine base calibrators and ten seeds show improvements in Brier score and error detection on four datasets, relative to a second calibration stage with constant Q.

Critical strengths

S1. The method is simple and can be applied to a fixed classifier without retraining. Learning the direction of the quality effect, rather than imposing a universal direction, is a reasonable design choice.

S2. The constant-Q and shuffled-Q controls address an important confound: a second calibration stage can improve predictions even without additional information. The paired experiments across nine base calibrators and ten seeds provide useful evidence that Q contributes on the positive datasets.

S3. The paper reports negative results and acknowledges the small absolute effects, the shared annotation source of the two CIFAR experiments, and the inference-time requirements of annotation-based quality scores. The distinction between error detection and class discrimination is informative.

Critical weaknesses

W1. Practical relevance is not convincingly demonstrated. Three positive datasets use multiple annotations of each test item to construct Q, and the two CIFAR experiments share this signal. Such information must also be available at inference. Among the measurement-quality datasets, only synthetic E_toy clearly benefits; eICU shows no significant Brier gain, while Audio MUSAN and Camelyon17 deteriorate against constant Q.

W2. The contribution beyond existing feature-aware calibration is unclear. Pan et al. (WWW 2020) combine score and feature-dependent corrections, and MBCT (Huang et al., WWW 2022) uses boosted trees. Theorem 1 is acknowledged to be standard. Appendix E.3 shows similar Brier scores for alternative models receiving Q, although QAC generally has better ECE. A comparison with existing methods given the same Q would clarify the contribution.

W3. Appendix I incorrectly treats Q computed from X as uninformative for calibration. Corollary 3 requires Q to be determined by C(X), not merely X. The diagnostic Delta R^2 measures additional error predictability beyond model outputs, so it cannot establish I(Q;Y|X)>0. This distinction matters for the proposed deployment advice.

W4. Appendix B.1 reverses the Bregman-divergence arguments: under its definition, excess loss is d_phi(q,p), not d_phi(p,q). For log loss, it must be KL(q||p). The gap expressions in Theorem 1 and Corollary 7 therefore need correction, although the underlying information-monotonicity result remains valid.

Feiyang Pan et al. Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions. WWW 2020 Siguang Huang et al. MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration. WWW 2022

Questions for authors
  1. For each annotation-based dataset, what is the exact construction of the evaluation label Y and Q? Using the existing annotations, how do the gains change when Q and the evaluation label are constructed from disjoint groups of annotators?

  2. How much of the advantage remains against an existing feature-aware calibrator supplied with the same Q, in settings where it is applicable? This would clarify the contribution of the proposed construction beyond the additional information.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer bmVq (Part 1 of 2)

We thank the reviewer for the thorough review. We appreciate the strengths the reviewer highlighted: a simple method that applies to a fixed classifier and learns the direction of the quality effect; the constant-q and shuffled-q controls across nine base calibrators and ten seeds; and the reporting of negative results, small effects, shared CIFAR annotations and inference-time needs, with error detection separated from class discrimination. The review led to six new experiments and two corrections of the theory.

This is how we define two terms used below: A plain calibrator is a post-hoc calibrator that does not use the quality score q, and we consider a dataset informative when QAC with the real q is significantly better than with a constant q.

Weakness 1. Practical relevance is not convincingly demonstrated: three positive datasets build q from several annotations of each test item, which must be available at inference, and the two CIFAR datasets share them; among the measurement-quality datasets only the synthetic E_toy benefits.

We agree (the paper states that the two CIFAR datasets are not independent evidence, Sec. 4.1) and added three experiments.

(a) A dataset where the provider supplies q for every new case (Sec. 4.4, Table 4). The US Current Population Survey (CPS) is a monthly labor survey. Its publisher flags each answer it edited. We fitted the model, a plain calibrator (Dirichlet calibration) and QAC on two months, then applied them unchanged to 31 others. After plain calibration, the model is overconfident by 0.175 on cases whose age was edited, and QAC, with the age edit flag as q, lowers their Brier score from 0.639 to 0.589.

Table R1 (from Tables 4 and 19 of the manuscript; -: not computed). Dataset: CPS over 10 seeds. Overconfidence: confidence minus accuracy after plain calibration.

CasesnOverconfidenceBrier, plainBrier, QACBrier, Dirichlet refitted per flag value
Age edited45,917+0.1750.6390.5890.565
Other answers edited only109,025+0.00050.38630.3866-
Not edited1,160,911-0.0050.3530.353-

So a provider's score, with no annotation, finds overconfident cases, and QAC corrects part of this in new months. Given the same flag, three calibrators fitted separately for flagged and unflagged cases do better (0.565 to 0.574, lower ECE; last column) and change 9-18% of these predictions (QAC 2%). So for a single yes/no flag, per-group calibration is the better tool; the general response states this scope. On a second survey of the same publisher (American Community Survey), no method that keeps the model frozen lowers the flagged cases’ Brier score by more than 0.0024, so the flags helped on one of two surveys.

(b) A check of where QAC can help (Sec. 4.6, Table 5). In our data, QAC helped only where the calibrated model was still more overconfident on low-quality cases than on the others. This leftover overconfidence is 0.06 to 0.16 on the four datasets where QAC helps. On eICU it is -0.008 (interval includes zero), matching the absence of a gain. On Audio MUSAN and Camelyon17, q makes QAC slightly worse; there, q does not predict errors beyond the model’s output (Table 1), and Camelyon17 also has a hospital shift (Sec. 4.7). The check misses Camelyon17 (a gap, no gain; Sec. 4.6).

Weakness 2. The contribution beyond existing feature-aware calibration is unclear (Pan et al.; MBCT); Theorem 1 is standard; Appendix E.3 shows similar Brier scores for alternative models receiving q.

We agree that Theorem 1 is standard and that QAC is a feature-aware calibrator. A new related-work paragraph says both and cites Pan et al., MBCT, and Kelly & Smyth (Sec. 1). Our contribution: (1) A calibrator for a quality score, monotone in the probability and free in q, that works on top of any first-stage calibrator. (2) When a post-hoc stage can gain from q: q must carry information beyond the calibrator's output (Corollary 8), with the check described above. (3) Controls for the effect of a second stage alone. (4) Given the same graded q, QAC has a lower mean Brier score (not NLL) than Pan et al., MBCT (two forms), Kelly & Smyth, and per-tier isotonic regression in all 20 comparisons on the four informative datasets, significantly in 8. We also agree on Table 8 (App. E.1.3): alternative second stages given q reach similar Brier scores; the monotone constraint mainly lowers the ECE. This does not affect the claim about q. For a single yes/no flag, calibrators fitted per flag value do better than QAC.

AuthorsOpenReview ↗

Reply to Reviewer bmVq (Part 2 of 2)

Weakness 3. Appendix I treats q computed from X as uninformative for calibration, but Corollary 3 requires q to be determined by C(X), not by X; ΔR2\Delta R^2 cannot establish I(Q;Y∣X)>0I(Q;Y|X)>0.

We agree, and thank the reviewer for pointing this out. A q computed from X adds nothing to the best predictor from X but can help a real calibrator C that has discarded that information. Sec. 4.2 and App. E.10 now say so: to help a calibrator, q must carry information about the label beyond the model's output. App. E.10 also says that ΔR2\Delta R^2 measures how much q predicts errors beyond the model's output and cannot establish I(Q;Y∣X)>0I(Q;Y|X)>0.

This point led to a new control experiment (App. E.2, Table 10). On ChaosNLI, we replaced q with a predictor of the model's errors built from its logits and sentence embedding (no votes). In one-stage QAC, it lowers the Brier score by 0.0305 against a constant score, more than q (0.0148). So part of the ChaosNLI gain is available from the model. The two signals are complementary: adding q to the predictor lowers the Brier score by a further 0.0132, about as much as q alone.

Weakness 4. Appendix B.1 reverses the Bregman-divergence arguments: the excess loss is dϕ(q,p)d_\phi(q,p), and KL(q∥p)\mathrm{KL}(q\|p) for the log loss.

We agree, and thank the reviewer for the careful review. We corrected the gap expression in Theorem 1 and, in Appendix B, the setup, the proof of Theorem 1, and Corollaries 7 and 8. As the reviewer notes, the information-monotonicity result is unchanged.

Question 1. For each annotation-based dataset, how are Y and q constructed? How do the gains change when they come from disjoint groups of annotators?

The gain does not change with disjoint annotators (App. E.7, Table 21). On ChaosNLI we built Y from votes 1-50 and q from votes 51-100. The effect of q on the Brier score (real q minus constant q) is -0.0141, against -0.0138 when Y and q share annotators.

Construction. ChaosNLI: Y is the majority label, and q is the entropy of the 100-or-so votes, so they share annotators. CIFAR-10H and CIFAR-10N: Y is the original CIFAR-10 test label, which is not built from the CIFAR-10H votes; q is the share of CIFAR-10H annotators on the plurality label (CIFAR-10H) or the negative entropy of the same votes (CIFAR-10N). Table 1 now states who produced each q.

Question 2. How much of the advantage remains against an existing feature-aware calibrator supplied with the same q?

On the four informative datasets, QAC has the lowest mean Brier score in all 20 comparisons, but not the lowest NLL (App. E.4, Table 17). All methods receive the same q and first stage (Dirichlet calibration), and we reimplemented the competing methods (App. C.4).

Table R2 (from Table 17 of the paper). Brier score, mean of 10 seeds; bold: lowest per column. q is informative in the first four columns.

MethodE_toyChaosNLICIFAR-10HCIFAR-10NeICU
Dirichlet calibration (no q).2049.3932.0859.1426.1529
QAC (ours).1956.3787.0852.1398.1532
Per-tier isotonic regression.1966.3792.0878.1422.1537
Kelly and Smyth.1960.3831.0936.1456.1529
Pan et al. (field only).2052.3923.0858.1408.1529
MBCT (q only).2040.3931.0857.1417.1529
MBCT (q and probability).2025.3939.0855.1408.1529
  • QAC's lower Brier score is significant in 8 of the 20 comparisons: against Pan et al. on E_toy and ChaosNLI; MBCT (q only) on E_toy and ChaosNLI; MBCT (q and probability) on ChaosNLI; per-tier isotonic regression on both CIFAR datasets; Kelly & Smyth on CIFAR-10H. In the other 12, it is lower but not significantly.
  • Where q is not informative (eICU, Audio MUSAN, Camelyon17), QAC is 0.0003 to 0.0057 worse than MBCT. There, q adds almost nothing beyond the model’s output, which Table 1 shows before use (ΔR2≤+0.003\Delta R^2 \le +0.003), so QAC should not be applied.
  • On CPS, given the yes/no age edit flag, QAC (Brier 0.589 on age-edited cases) beats our implementations of Pan et al. and MBCT, and Kelly & Smyth with its published leaf size, which cannot split this 3.5% group (0.636 to 0.640). Three calibrators fitted per flag value, including Kelly & Smyth with leaves of at least 50 cases, perform better (0.565 to 0.574, lower ECE) and change 9-18% of these predictions (QAC 2%). With the any-edit flag, Kelly & Smyth also beats QAC (0.596 vs 0.609). So for a single yes/no flag, per-group calibration is the better choice (Table 19).
  • The result holds for the Brier score, not for NLL or ECE: Kelly & Smyth has a lower NLL than QAC on all four informative datasets. On ECE, QAC is not the lowest on three of the four: Pan et al. and MBCT (q only) are lower on E_toy, Kelly & Smyth on ChaosNLI, and MBCT (q and probability) on CIFAR-10N (App. E.4).

We thank the reviewer for the thorough review and hope we addressed the reviewer's concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment (Part 1 of 2)

We thank the three reviewers for their careful and constructive reviews. We appreciate the strengths they highlighted: a simple method that applies to a fixed classifier and learns the direction of the quality effect from data; a clear motivation and a manuscript that is easy to follow; a definition of the quality score that adds information beyond the model's output; a sound evaluation, with constant-score and shuffled-score controls across nine base calibrators and ten seeds; and the reporting of negative results, small effects and inference-time needs. The reviews led to ten new experiments and several corrections.

In our work, we studied when per-sample data-quality information improves calibration after training; QAC is the method we use to test it. The quality score q is a value recorded with each sample that describes how reliable its input or label is. A plain calibrator is a post-hoc calibrator that does not use q. The revision supports three findings:

  1. A data provider's edit flag. A US labor survey flags each answer its publisher edited. After calibration, the model remains overconfident by 0.175 on cases with an edited age. Given the age edit flag, QAC lowers their Brier score from 0.639 to 0.589 in 31 months not used for fitting.
  2. Graded quality scores. On the four datasets where the real score is significantly better than a constant score (the informative datasets), QAC has the lowest mean Brier score among calibrators given the same score in all 20 comparisons (significant in 8, none against the closest competitor), but not the lowest NLL.
  3. A screening method. In our data, QAC helped only where the calibrated model stayed more overconfident on low-quality cases. This can be measured before fitting.

Main concerns of the reviewers:

1. Practical relevance: the positive real datasets need several annotations per test item (all three reviewers). We agree, and added a US labor survey whose provider supplies q for every case, with no annotation (Sec. 4.4, Table 4). There, the provider's edit flag finds the cases on which a calibrated model is still overconfident, and QAC corrects part of this error in new months. Given the same flag, QAC also does better than our implementations of Pan et al. and MBCT (0.636 to 0.640; Table 19). So the benefit of a quality score does not depend on annotations. Two limits apply. Calibrators fitted per flag value do better than QAC (0.565 to 0.574, lower ECE): a real limitation of QAC for a single yes/no flag. And on a second survey from this provider, the flags do not help: no method that keeps the model frozen gains more than 0.0024 (App. E.5). Averaging the model's output over plausible values of the flagged answer (an earlier method, which needs the model) detects errors best (AUROC 0.729 vs 0.631; 0.730 with QAC's second stage; Table 19 (iii)).

2. Why QAC helps on some datasets and not on others (all three reviewers). We measure leftover overconfidence: confidence minus accuracy on low-quality cases, minus the same on the others (Sec. 4.6, Table 5). It is 0.06 to 0.16 where QAC helps, and about zero on eICU, where it does not. Over 27 dataset-score pairs from one population, a squared form of it ranks the gains of one-stage QAC (Spearman 0.84 [0.59, 0.95]; App. E.6). Under a shift, it does not (0.09, 9 pairs). On Audio MUSAN and Camelyon17, q does not predict errors beyond the model’s output (Table 1), and QAC is slightly worse there; Camelyon17 also has a hospital shift (Sec. 4.7). On Camelyon17, a gap remains without a gain: the screen is not sufficient.

3. Contribution beyond existing feature-aware calibrators (Reviewer bmVq). We compared QAC with our implementations of Pan et al., MBCT (two forms), Kelly & Smyth and per-tier isotonic regression, each given the same q and first stage (App. E.4, Table 17). This comparison gives the second finding above.

4. Is the gain due to q or to a second stage? ECE and NLL (Reviewers sBn1, rTAL). Table 2 now includes Dirichlet calibration; two second stages without q improve on it by no more than 0.0004. On the four informative datasets, giving q to the same stage lowers the Brier score by 0.001 to 0.016. QAC's NLL is 17-18% worse than Dirichlet calibration on the two CIFAR datasets (Table 3). This is a real limitation where the NLL matters; a few probabilities near zero cause it, and a lower bound on the probabilities removes most of it (App. E.4, Table 18).

5. Quality scores tied to the evaluation labels (Reviewers bmVq, rTAL). On ChaosNLI, we built the label and q from two disjoint groups of annotators. The effect of q on the Brier score is unchanged: -0.0141, against -0.0138 with shared annotators (App. E.7, Table 21).

6. Theory (Reviewer bmVq). We agree with both points, and thank the reviewer for the careful check. We corrected the order of the Bregman-divergence arguments (Theorem 1; App. B) and the statements about a q computed from the input (Sec. 1, 4.2; App. E.10).

AuthorsOpenReview ↗

General Comment (Part 2 of 2)

In the revised manuscript, new or changed text is blue.

New experiments.

  1. A labor survey whose provider supplies a quality flag with each answer, tested on 31 months not used for fitting (Sec. 4.4, Table 4; App. E.5).
  2. A second survey of the same provider, where the flags do not help (Sec. 4.4; App. E.5).
  3. The leftover-overconfidence screen on seven datasets, and its rank correlation with the gain over 27 pairs of a dataset and a quality score (Sec. 4.6, Table 5; App. E.6, Table 20).
  4. A comparison with four feature-aware calibrators given the same q (App. E.4, Table 17).
  5. Three plain calibrators refitted (App. C.4); all nine reported, with ECE and NLL (Tables 2, 3, 11 and 12).
  6. A second control without q: each plain calibrator used as a second stage, beside the constant-score control (Table 2).
  7. The label and q built from disjoint groups of annotators (App. E.7, Table 21).
  8. Pooling the annotators' votes directly with the model, compared with QAC (App. E.7, Table 22).
  9. A control in which q is replaced by a predictor of errors built from the model itself (App. E.2, Table 10).
  10. Three ways of using a quality flag without retraining the model, and retraining with the flags as inputs (App. E.5, Table 19).

We also analyzed why QAC's NLL is worse on the CIFAR datasets (App. E.4, Table 18) and how QAC behaves when calibration and test data come from different periods or sites (App. E.9.6).

A change that affects the main numbers: stronger baselines. While preparing this response, we found that three plain calibrators were weaker in our experiments than they should be. Temperature scaling stopped before convergence, while Dirichlet calibration was not tuned, and the top-versus-all isotonic regression spread the remaining probability evenly over the other classes. We refitted the three calibrators (App. C.4). The main result is unchanged: the effect of q is still significant on E_toy, ChaosNLI, CIFAR-10H and CIFAR-10N. QAC with the real q, compared with the same stage with a constant q, changes the Brier score by -0.0108 (E_toy), -0.0155 (ChaosNLI), -0.0011 (CIFAR-10H), and -0.0039 (CIFAR-10N). Against the best plain calibrator, which is now stronger, QAC's Brier score is lower by 4.5%, 3.7%, 0.8%, and 2.0%. This second difference is significant on E_toy and ChaosNLI only. On the two CIFAR datasets, this is a limitation of the evidence: q helps significantly there, but QAC is not significantly better than the best plain calibrator.

Statistics. The ten seeds are ten splits of one pool of data, so they are not independent. We now widen the intervals of the main comparisons for this, with the Nadeau and Bengio correction (Sec. 4.1; App. D). They become about 3.3 times wider when the calibration and test sets have the same size. We call a dataset informative when the widened 95% interval of the main comparison shows a lower Brier score with the real q than with a constant q.

Other changes to the paper.

  • Abstract, Sec. 4.4 and Sec. 4.6: the survey result and the screen.
  • Sec. 1: a related-work paragraph on feature-aware calibration, the grouping loss, and unreliable or missing inputs.
  • Sec. 3: QAC has no multiclass calibration guarantee after normalization, and it can change the predicted class (0.4% to 8.1% of test cases), which lowers the accuracy by at most 0.012. It is meant for a model that cannot be retrained.
  • Table 1: who produced each q, whether it needs new annotations for a new case, and whether it is computed from the model input.
  • Theorem 1, App. B, App. E.10 and one sentence each in Sec. 1 and 4.2: the two theory corrections.
  • The section order of Sec. 4 changed: the survey is Sec. 4.4, and the check is Sec. 4.6.
  • Removed: two appendix sections of the first version, on routing samples by q (previous App. J) and on quality scores given by users (previous App. E.4).

Scope of QAC. QAC is for a trained model that cannot be retrained, and for a quality score that describes how reliable a sample is without revealing its label. The paper now states what to use outside this scope: retraining with q as an input when retraining is possible (Sec. 3); the annotators' votes themselves when they exist for a new sample, since pooling them with the model beats QAC (Sec. 4.7); recalibration on target data under a shift between calibration and test data (Sec. 4.7); and, when q is a single yes/no flag, calibrators fitted per flag value, which did better than QAC on the survey (Brier 0.565 to 0.574 against 0.589; Sec. 4.4; App. E.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.