acceptodds
Under review as a conference paper at ICLR 2027

Epistemic Learning from Imprecise Annotation

Abstract

Imprecise annotations may support several plausible labelling distributions, yet learning methods often discard this ambiguity into a single predictive distribution. This can obscure what the annotation evidence leaves unresolved. We introduce *epistemic learning from credal supervision*, a framework that uses convex sets of plausible labelling distributions, called credal sets, as supervision and learns sets of compatible predictive distributions. We instantiate the framework with a *pessimistic–optimistic credal classifier* (POCC), which combines a shared backbone with two classification heads trained to minimise worst-case and best-case losses over the available supervisions. Their outputs define a predictive credal set whose spread provides an uncertainty score. We also show how credal labels can practically be obtained through a simple relaxation of existing probabilistic labels, reducing commitment to their precise probability assignments. This construction admits closed-form inner optimisation under cross-entropy loss, enabling efficient training. Assuming the supervision sets contain the true conditional label distributions, and other regularity assumptions, we establish a finite-sample generalisation bound for the averaged predictor with an explicit penalty for supervision imprecision. We evaluate POCC using human annotator disagreement and teacher predictions, alongside label smoothing as a controlled proxy for annotation imprecision. Across these settings, POCC achieves a favourable balance of predictive accuracy, calibration, and uncertainty-based selective classification against competitive baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
epistemic learning, imprecise annotation, credal sets, epistemic uncertainty

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qS6QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies epistemic learning from credal supervision and proposes a Pessimistic–Optimistic Credal Classifier (POCC). The method constructs credal labels from imprecise supervision, trains two prediction heads using worst-case and best-case cross-entropy objectives, and represents predictive uncertainty using the total variation distance between their outputs. The paper also establishes a finite-sample risk bound under conditional coverage assumptions and evaluates the method on annotator disagreement, teacher predictions, and label smoothing. The problem is relevant, the framework is clearly presented, and the experimental evaluation is relatively extensive.

Critical strengths

1、The paper addresses an important problem, retaining unresolved annotation information rather than collapsing all supervision into a single probability distribution. 2、The proposed two-head architecture admits efficient optimization under cross-entropy loss. 3、The theoretical analysis explicitly separates model complexity, finite-sample error, optimization error, and supervision imprecision. 4、The experimental evaluation covers multiple datasets and three different sources of imprecise supervision.

Critical weaknesses

1、1. The proposed uncertainty score is not shown to identify annotation-induced epistemic uncertainty. The total variation distance between the pessimistic and optimistic heads measures their predictive disagreement, but this disagreement may also arise from differences in optimization objectives, parameterization, and training dynamics. The method does not establish that the score isolates uncertainty induced by imprecise supervision. In particular, when the credal label degenerates to a singleton, the two heads have the same nominal objective, but the architecture does not guarantee identical predictions. Controlled experiments with known annotation ambiguity and singleton supervision are necessary. 2、The semantic validity of the predictive credal set remains unestablished. The line segment between two learned predictions is a valid convex set of probability distributions, but the paper does not demonstrate that it represents the full set of plausible predictions induced by the original credal supervision. The theoretical guarantee concerns the risk of the averaged predictor, not coverage or calibration of the predictive set. The authors should either establish a meaningful set-coverage result or narrow their claims about the interpretation of the predictive credal set. 3、The theoretical analysis requires the true conditional label distribution to belong to the supplied credal set. However, the proposed construction derives its probability lower bound from the maximum probability of a reference soft label. This does not guarantee coverage for finite annotator samples or potentially overconfident teacher predictions. The authors should evaluate sensitivity to coverage violations and clarify which experimental settings plausibly satisfy the assumptions. 4、The uncertainty evaluation does not directly validate supervision-induced ambiguity. AUARC and the supplementary uncertainty–error correlation assess whether the uncertainty score ranks prediction errors. They do not directly establish that the score captures human annotation disagreement or uncertainty in the underlying label distribution. The paper would benefit from direct comparisons with human disagreement, teacher disagreement, and controlled ground-truth ambiguity.

Questions for authors

1、Under what conditions does the distance between the two heads monotonically reflect the true degree of annotation imprecision? 2、Can the authors demonstrate coverage or calibration of the predicted credal set on synthetic data where the true conditional distribution is known? 3、How does POCC compare against direct label-relaxation and credal-supervision baselines using the same information? 4、How robust are the theoretical and empirical results when the conditional coverage assumption is violated? 5、Would the conclusions remain unchanged if BQS were replaced by raw metrics and the statistical analysis accounted for multiple comparisons?

5 comments

Dear Reviewer,

We thank the reviewer for your review effort and the thoughtful comments. Below, we address your concerns.

Responses to W1 and W4: whether MMI captures supervision-induced imprecision.

Thank you for this constructive feedback. Both points concern whether MMI captures supervision-induced imprecision, so we would like to answer them together. In short, POCC's MMI vanishes without supervision imprecision, grows with it under control, and tracks human and teacher ambiguity more closely than the baselines' EU.

Singleton supervision (W1). The two heads' objectives differ only through the credal label and coincide when it is a singleton (Proposition 3). We trained POCC with one-hot labels: MMI is zero up to floating-point precision on CIFAR-10 and CIFAR-100, versus 5.8% and 14.3% at ϵ=0.05\epsilon = 0.05. Although initialised independently, the two linear heads minimise the same objective and converge to the same predictions, so parameterisation and training dynamics alone create no disagreement.

POCC's EU captures supervision-induced imprecision (W1, W4). We agree that AUARC evaluates how well the EU ranks errors, not whether it reflects the imprecision of the supervision. Following your suggestion, we test the latter directly. Label smoothing injects a known imprecision d=ϵ(K−1)/Kd = \epsilon(K-1)/K while the images stay fixed (Here, KK denotes the number of classes). As ϵ\epsilon grows, POCC's mean test MMI increases in every seed, by about the injected amount:

POCC's mean test MMI (%) and the injected imprecision dd (%):

ϵ\epsilondd, CIFAR-10MMI, CIFAR-10dd, CIFAR-100MMI, CIFAR-100
0.054.55.8 ± 0.14.9514.3 ± 0.1
0.19.010.5 ± 0.19.919.0 ± 0.1
0.1513.515.3 ± 0.114.8522.7 ± 0.2
0.218.020.0 ± 0.119.826.4 ± 0.2

Comparison with the baselines' EU (W4). We further compare MMI with the baselines' EU, first under controlled imprecision and then under human and teacher disagreement.

Controlled imprecision. Since the baselines' EU scores have different units, we compare them by the ratio of each method's mean test EU at ϵ=0.2\epsilon = 0.2 to that at ϵ=0.05\epsilon = 0.05. Only DAPPr's EU also increases, and less than MMI; the others stay flat or decrease:

MethodCIFAR-10CIFAR-100
EDL×1.0×1.0
LbBnn×0.8×0.5
Decali×0.8×0.5
DAPPr×1.8×1.1
POCC×3.5×1.8
MCDO×0.7×1.0
DE×0.7×0.9

Human and teacher disagreement. We correlate each method's EU on test images with the imprecision 1−max⁡kq(k∣x)1 - \max_k q(k \mid x) of the held-out human or teacher distribution (Spearman $\times$100, mean ± std over 10 seeds; best in bold). For human disagreement, we use MiceBone and TreeVersity, whose test splits are annotated; CIFAR-10H is not included because our protocol (Appendix C.3.1) uses all its annotated images for training and evaluates on CIFAR-10 images with a single label each.

Human disagreement:

MethodMiceBoneTreeVersity
EDL62.8 ± 2.536.8 ± 1.2
Decali51.6 ± 1.545.9 ± 1.1
DAPPr64.3 ± 1.544.5 ± 1.4
POCC70.4 ± 1.358.0 ± 0.7
DE50.0 ± 2.121.9 ± 2.4

Teacher predictions (T=2.5T = 2.5), CIFAR-10:

MethodResNet-20ShuffleNetV2-0.5$\times$MobileNetV2-x0.5
EDL53.6 ± 0.754.0 ± 0.747.8 ± 0.5
LbBnn79.5 ± 0.373.5 ± 0.370.9 ± 0.6
Decali66.0 ± 0.558.1 ± 0.557.8 ± 0.5
DAPPr69.9 ± 0.871.2 ± 0.965.2 ± 0.7
POCC80.8 ± 0.280.1 ± 0.273.9 ± 0.3
MCDO68.7 ± 0.669.5 ± 0.559.3 ± 0.6
DE70.3 ± 0.371.1 ± 0.360.4 ± 0.3

Teacher predictions (T=2.5T = 2.5), CIFAR-100:

MethodResNet-20ShuffleNetV2-0.5$\times$MobileNetV2-x0.5
EDL32.9 ± 4.932.0 ± 4.830.6 ± 1.7
LbBnn−40.0 ± 1.2−43.3 ± 0.8−54.0 ± 0.7
Decali9.1 ± 0.30.7 ± 0.514.3 ± 0.9
DAPPr41.6 ± 0.547.1 ± 0.456.5 ± 0.4
POCC64.0 ± 0.660.5 ± 0.960.6 ± 0.5
MCDO54.3 ± 0.752.6 ± 0.655.9 ± 0.5
DE59.8 ± 0.660.4 ± 0.462.2 ± 0.4

POCC is best in 7 of the 8 settings, and one-sided Wilcoxon signed-rank tests over 10 seeds (Holm-corrected) show that it is significantly better than every baseline (p=0.008p = 0.008 for human disagreement, p=0.012p = 0.012 for teachers), with one exception: on CIFAR-100, it beats the deep ensemble for ResNet-20 but ties for ShuffleNetV2 and trails for MobileNetV2. This ensemble, however, trains and evaluates five networks, about five times the cost of POCC's single backbone.

Responses to W2, W3, Q1–Q3.

We sincerely thank you for these points, which help us sharpen the scope of our claims. We first clarify the scope.

  • Predictive credal set. The segment in Eq. (6) is the convex hull of the predictions of the pessimistic and optimistic criteria, and its width is our predictive epistemic uncertainty score. We do not claim that it covers the true conditional distribution or contains every prediction compatible with the credal labels, as already stated in Section 4 and Appendix J. We agree that some wording in Sections 1 and 3.1 (e.g., the set "represents the consequences of unresolved annotation evidence") suggests a stronger reading, and will revise it accordingly, for instance by stating that the set reflects how much the supervision leaves unresolved.

  • Theory. Theorem 5 is a first guarantee for valid but imprecise supervision: it bounds the risk of the averaged predictor under coverage, which separates the effect of imprecision from that of invalid constraints. As acknowledged in Section 6, coverage need not hold for finite annotations or imperfect teachers; a guarantee under probabilistic coverage remains future work, but below we show how the current one degrades when coverage fails.

Within this scope, and following your suggestions, we add the analyses below.

How plausible coverage is in our settings, and sensitivity to violations (W3).

With deterministic labels, as for CIFAR, the true conditional is one-hot (Section 4), so QαQ_\alpha covers it if and only if the reference's top class is correct, whatever the value of α\alpha. Overconfidence alone therefore never violates coverage; it matters only when the top class is wrong, where the TV distance from p0(x)p_0(x) to Qα(x)Q_\alpha(x) is ρ(x)=α(x)\rho(x) = \alpha(x), the confidence placed on the wrong class.

In our experiments:

SettingViolation rateMean violation E[ρ]\mathbb{E}[\rho]
Label smoothing (all ϵ\epsilon)0%0
CIFAR-10H (human majority vs. CIFAR-10 label)0.79%0.0046
CIFAR-10, ResNet-20 teacher0.05%0.0003 (0.0004)
CIFAR-10, ShuffleNetV2-0.5× teacher0.92%0.0040 (0.0060)
CIFAR-10, MobileNetV2-x0.5 teacher0.13%0.0006 (0.0008)
CIFAR-100, ResNet-20 teacher6.59%0.0147 (0.0344)
CIFAR-100, ShuffleNetV2-0.5× teacher3.21%0.0074 (0.0171)
CIFAR-100, MobileNetV2-x0.5 teacher0.96%0.0026 (0.0053)

Summary: For the teachers, E[ρ]\mathbb{E}[\rho] is reported at T=2.5T = 2.5, as in our experiments, with T=1T = 1 in brackets: temperature scaling does not change which teacher predictions are wrong, but lowers their confidence and hence the violation by more than half on CIFAR-100. POCC's results are stable across these settings: it dominates every baseline in all six teacher configurations, including CIFAR-100 with ResNet-20, which has the most violations (see our response to W1 and W4 on multiple comparisons). Coverage is least plausible with few annotators on genuinely ambiguous inputs (MiceBone, TreeVersity), where the top frequency tends to overestimate p0p_0.

We also would like to note that annotator frequencies are nonetheless the standard estimate of the ground-truth conditional distribution [R1, R2, R3], as are teacher probabilities in distillation [R4]. Under this approximation, coverage holds by construction, since qL∈Qαq_L \in Q_\alpha (Eq. (8)); our assumption is weaker, requiring only that p0p_0 lies somewhere in QαQ_\alpha. Since these datasets provide no ground-truth conditional distribution, we study the few-annotator case on synthetic data below.

Regarding the degradation and monotonicity (W3, Q1, Q3).

  • Violated coverage. Cross-entropy is linear in the target, so ∣ℓ(f,p)−ℓ(f,q)∣≤L⋅TV(p,q)|\ell(f,p) - \ell(f,q)| \le L\cdot\mathrm{TV}(p,q) under the boundedness condition of Theorem 5. Without coverage, R+≤R0≤R−R_+ \le R_0 \le R_- still holds up to L E[ρ(X)]L\thinspace\mathbb{E}[\rho(X)], and Theorem 5 holds with an additional term proportional to L⋅E[ρ(X)]L\cdot\mathbb{E}[\rho(X)]: the guarantee degrades linearly in the violation, which is small in our settings.

  • Monotonicity. For a training input, the optimistic loss is minimised by eje_j and the pessimistic loss by sαs_\alpha, which puts α\alpha on jj and spreads 1−α1-\alpha evenly over the other classes. These are the minimum- and maximum-entropy members of QαQ_\alpha, and their TV distance is 1−α=dQ(x)1-\alpha = d_Q(x). When the heads attain these per-sample optima, the predictive segment therefore spans the full TV diameter of the credal label, and MMI equals the imprecision, linearly and hence monotonically. On test inputs, this further requires the imprecision to be learnable from xx.

Synthetic study with known p0p_0 (Q1–Q3).

We train POCC on a synthetic three-class problem in which the true conditional p0p_0 is known for every input. For each training input, NN simulated annotators draw labels from p0(x)p_0(x), and QαQ_\alpha is built from their empirical distribution as in Section 3.2 (oracle: qL=p0q_L = p_0); the detailed configuration is given at the end of this response.

Metrics. (a) Validity of the credal labels on the training inputs: the fraction with p0(x)∈Qα(x)p_0(x) \in Q_\alpha(x) (label coverage) and the mean TV distance E[ρ]\mathbb{E}[\rho] from p0p_0 to QαQ_\alpha.

(b) Ranking: the Spearman correlation between MMI and the true ambiguity 1−max⁡kp0(k∣x)1-\max_k p_0(k \mid x) on the test inputs.

(c) Magnitude: since Spearman only checks the ranking, we group the test inputs by their true ambiguity and compare, within each group, the mean MMI with the mean true ambiguity, in the spirit of a reliability diagram. Both are in TV units, and for the oracle the true ambiguity equals the diameter of QαQ_\alpha, which MMI matches at the per-sample optimum.

(d) The expected accuracy of the averaged prediction under p0p_0. Table S1 reports (a), (b), (d) and the mean MMI; Table S2 reports (c).

Results. (i) With valid labels (oracle), the width of the predictive set is calibrated to the true ambiguity: in every bin of Table S2, the mean MMI matches the mean true ambiguity, e.g., inputs whose true top-class probability is about 0.6 receive a width of about 0.4, and the ranking is near-perfect (Spearman 99.7).

(ii) With finite annotators, coverage is violated often but mildly (Table S1), since the top frequency frequently exceeds p0p_0 by a small amount. Accuracy nevertheless stays within 0.1 points of the Bayes accuracy and the ranking is preserved (Spearman ≥ 98); the violation mainly shrinks MMI in the most ambiguous bin (42.6/34.8/27.0% with 50/10/3 annotators, against 51.0%), so performance degrades gradually with E[ρ]\mathbb{E}[\rho]. We will add this study to the appendix.

Table S1. Summary over 10 seeds (mean true ambiguity 16.2%, Bayes accuracy 83.8%).

SupervisionLabel coverageE[ρ]\mathbb{E}[\rho] (%)Spearman (MMI, true ambiguity) ×100Mean MMI (%)Accuracy (%)
Oracle (qL=p0q_L = p_0)100%0.099.716.383.8
50 annotators41%1.899.215.883.8
10 annotators30%4.798.414.783.7
3 annotators17%10.498.011.983.7

Table S2. Mean MMI by bin of true ambiguity (%).

True ambiguity bin[0, 0.05)[0.05, 0.15)[0.15, 0.3)[0.3, 0.45)[0.45, 2/3]
Mean true ambiguity1.89.321.937.451.0
Mean MMI, oracle1.89.121.738.450.9
Mean MMI, 50 annotators1.89.122.239.742.6
Mean MMI, 10 annotators1.79.323.035.534.8
Mean MMI, 3 annotators1.78.619.026.227.0

Detailed configuration. Labels are uniform over three classes and inputs are drawn as x∣y∼N(μy,I2)x \mid y \sim \mathcal{N}(\mu_y, I_2), with the means μk\mu_k at the vertices of an equilateral triangle of radius 1.5. The true conditional p0(x)=softmaxk(μk⊤x)p_0(x) = \mathrm{softmax}_k(\mu_k^\top x) is then known for every input: it is close to uniform near the centre and increasingly confident away from it. For each of 3,000 training inputs, NN simulated annotators draw labels from p0(x)p_0(x). POCC is an MLP (2–64–64, ReLU) with two linear heads, trained with Eq. (5) and the closed-form targets of Proposition 3. We evaluate on 10,000 test inputs over 10 seeds.

Reviewer_dAtcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers the problem of classification, in which the supervision data is given as a pair of an object and a set of label distributions, i.e., a credal set, and the output is also a credal set that is able to represent the epistemic uncertainty. This imprecision in the supervision is put against the case where each object is associated with a single label distribution that might not be the true label distribution (e.g., due to annotation noise) or its degenerate form of having a single label. The proposed learning approach is to consider two predictors minimizing the worst-case and the best-case risk with respect to a set of plausible data generating distributions. The output credal set is simply the convex hull of the predicted label distribution from the two predictors. A specific kind of credal set supervision is considered, namely, the one that turns a single label distribution into a credal set by considering the probability of the argmax class as the lower probability of this class, and assigning the remaining mass arbitrarily among the classes. Under this credal set supervision, the optimization process in the learning procedure can be done efficiently. Furthermore, for the case of valid credal set supervision, the cross-entropy excess risk of the proposed method has an analytical upper bound. The proposed method is tested against deliberately designed competitors and turns out to perform really well empirically.

Critical strengths

I think the most important strength of this paper is that it is very well-written. One can simply follow the paper step by step from beginning to end and appreciate how carefully it is written. The authors considered a very interesting problem, provided an appropriate solution, and elaborated on their solution both analytically and empirically.

Critical weaknesses
  • When I read that the authors are criticizing the datasets that aggregate the disagreement of the different annotators in the form of a single label distribution, I thought that they were going to avoid this simple aggregation and propose a very systematic approach to derive a credal set from it. However, later it turns out that the only method they considered was to simply construct a credal set from that aggregation. Moreover, this approach has a clear disadvantage, and that is that the credal set constructed this way always lies near a simplex corner, which can limit its applicability, especially when the label distribution lies near the center of the simplex. Last but not least, this construction, when the original dataset consists of the standard object-label pairs, collapses into a singleton credal set of the degenerate distribution at that class.
  • line 235 and 236: Common supervision signals instead often provide a single reference distribution ==> this is not true. Indeed, the supervision is often given as a single class, i.e., degenerate probabilities.
Questions for authors
  • The representation of the epistemic uncertainty in the output is quite limited. The convex hull of the two predicted label distributions might not be the best choice. Have the authors considered other ways of doing so?
  • Why don't the authors discuss the aleatoric uncertainty at all? Is there a reason for that?
4 comments

Dear Reviewer,

We sincerely thank you for your review effort and for recognising our work. We respond to your comments below.

Responses to W1

Thank you for this thoughtful comment; we would like to clarify the points about the construction.

Our main contribution is on the learning side: given credal labels, POCC learns a predictive credal set, and its objective (Eq. 5) applies to any credal labels whose inner extrema can be evaluated. The construction in Section 3.2 is one practical instantiation for the most common case, where supervision comes as a single distribution, covering annotator frequencies, teacher predictions, and smoothed labels; we chose it so that POCC applies to the supervision most widely available in practice. Rather than using this distribution as a precise target, the relaxation keeps it as one admissible member among others.

As the reviewer notes, this relaxation [R1] keeps only a lower bound on the most supported class, so the set is anchored at that class, grows as the label distribution approaches the centre of the simplex, and reduces to a singleton for a hard label. The latter is by design: POCC targets imprecise supervision, and a hard label carries no supervision-induced imprecision to represent.

POCC is not tied to this construction. When raw annotations are available, one possibility is the imprecise Dirichlet model [R2], which gives a set centred on the empirical annotator distribution (an ϵ\epsilon-contamination with ϵ\epsilon = s/(N + s) for N annotations and prior strength s); it shrinks as annotations accumulate and stays non-degenerate for a single label. Since our relaxation is the special case centred at the one-hot distribution with ϵ\epsilon = 1 − α\alpha, Proposition 3 carries over with the same O(K)O(K) cost. Such constructions, however, need annotation counts, which teacher predictions and smoothed labels do not provide and few existing benchmarks release; we will discuss this extension in the revision and explore it in future work.

Responses to W2

Thank you for catching this. We meant imprecise supervision signals; standard supervision is indeed the degenerate case. We will revise lines 235–236 accordingly.

Responses to Q1

Thanks for raising this point, and we agree: the convex hull of two distributions is only a line segment in the simplex, and we will state this limitation explicitly in the revision. The two vertices correspond to the two endpoints of the credal risk profile (Section 3.1): the pessimistic and optimistic heads minimise the upper and lower credal risks, and the segment is the smallest convex set that retains both learned views (Eq. 6).

We considered two extensions:

  • More heads, each capturing different supervision information in a richer credal label. Each head would need its own principled objective, and the limiting case of one predictor per admissible labelling law is an infinite-task problem (Section 3.1). We leave this to future work.

  • An ensemble of M POCC models (POCC-Ens), whose 2M outputs span the convex hull of M segments. Besides the supervision-induced width of each segment, this hull reflects disagreement between independently trained POCC models, a complementary source of epistemic uncertainty. The price is M times the training and inference cost. Appendix J already shows that POCC-Ens \emph{substantially improves OOD detection over POCC and outperforms DE}.

Prompted by your question, we also evaluated POCC-Ens (M = 5, as for DE) on CIFAR-10H and on CIFAR-10 with distillation (ResNet-20 teacher, T = 2.5) or label smoothing (ϵ\epsilon = 0.05):

SettingMethodACC ↑ECE ↓AUARC ↑
CIFAR-10HPOCC87.0 ± 0.42.9 ± 0.397.1 ± 0.2
POCC-Ens89.9 ± 0.21.5 ± 0.297.9 ± 0.1
DE87.0 ± 0.34.3 ± 0.296.9 ± 0.1
MCDO85.6 ± 0.70.9 ± 0.196.6 ± 0.3
CIFAR-10, distillation (T = 2.5)POCC95.5 ± 0.13.0 ± 0.199.5 ± 0.0
POCC-Ens96.1 ± 0.14.4 ± 0.199.6 ± 0.0
DE94.2 ± 0.19.5 ± 0.199.2 ± 0.0
MCDO93.5 ± 0.28.5 ± 0.299.0 ± 0.1
CIFAR-10, label smoothing (ϵ\epsilon = 0.05)POCC95.5 ± 0.11.5 ± 0.199.2 ± 0.1
POCC-Ens96.3 ± 0.12.0 ± 0.199.6 ± 0.0
DE95.2 ± 0.15.5 ± 0.199.4 ± 0.0
MCDO93.7 ± 0.23.5 ± 0.199.1 ± 0.1

Results in %, mean ± std; best per setting in bold.

POCC-Ens improves accuracy and the AUARC over POCC in all three settings and outperforms the equally sized DE on all three criteria, while POCC itself remains the best-calibrated method under distillation and label smoothing. POCC is thus the efficient single-model choice, and POCC-Ens a richer alternative when the budget allows.

Responses to Q2

Thank you for raising this. We focus on epistemic uncertainty (EU) on purpose, but we agree we should have said more explicitly (we will emphasize more in Section 1). The EU we care about comes from the supervision itself: when annotations are imprecise, it is unclear which label distribution an input should have. Aleatoric uncertainty (AU), the randomness of the label once this distribution is known, is captured by a single predicted distribution [R4], so it is not the gap we address.

POCC nevertheless quantifies AU through the standard decomposition for credal sets [R3, R4]: total uncertainty is the upper entropy of the predictive credal set, AU its lower entropy, and EU their difference, i.e., the entropy difference of Appendix I.2.

We report AU and EU (entropy difference and MMI) on our controlled label-smoothing experiments, where the injected imprecision changes the supervision but not the images:

Datasetϵ\epsilonAU: lower entropyEU: entropy differenceEU: MMI (%)
CIFAR-100.050.079 ± 0.0030.456 ± 0.0025.77 ± 0.05
0.10.079 ± 0.0030.774 ± 0.00410.54 ± 0.10
0.150.079 ± 0.0021.057 ± 0.00615.30 ± 0.14
0.20.075 ± 0.0031.312 ± 0.00420.02 ± 0.10
CIFAR-1000.050.881 ± 0.0151.115 ± 0.00914.31 ± 0.13
0.10.838 ± 0.0111.509 ± 0.01118.96 ± 0.14
0.150.789 ± 0.0161.836 ± 0.01422.74 ± 0.23
0.20.736 ± 0.0172.150 ± 0.01626.37 ± 0.21

Entropies in bits; mean ± std over 10 seeds.

As ϵ\epsilon grows, AU does not increase (it stays flat on CIFAR-10 and decreases on CIFAR-100), whereas both EU measures rise monotonically. The injected supervision imprecision is thus reflected in EU rather than AU. We will add this discussion and the table to the revision.

We hope these responses adequately address your concerns.

[R1] Julian Lienen and Eyke Hüllermeier. Credal self-supervised learning. NeurIPS 2021.

[R2] Walley, P. Statistical Reasoning with Imprecise Probabilities. 1991.

[R3] J. Abellán, G. J. Klir, and S. Moral. Disaggregated total uncertainty measure for credal sets. International Journal of General Systems, 2006.

[R4] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 2021.

Thank you for elaborating on my comments.

Regarding the response to my Q1, as mentioned by the authors, the epistemic uncertainty represented by the proposed learner comes solely from the supervision, and mixing it with model/approximation uncertainty coming through ensembling is not a great idea, as it is not even clear how one can separate the two sources, and it definitely does not address my question. Therefore, I would advise the authors not to bring the ensemble experiments into the paper.

Nevertheless, I will maintain my score.

Dear Reviewer,

Thank you for the discussion and for keeping your positive assessment. You are right that the spread of POCC-Ens mixes supervision-induced uncertainty with the variation between independently trained models, and that the two cannot be told apart in the resulting hull. The hull is richer in the prediction space, but its EU cannot be attributed to the supervision alone.

We will take your advice and not bring the new ensemble experiments during rebuttal in the final version. We will also state clearly that the segment is a limited representation of supervision-induced EU, and mention richer representations, for example, more heads with their own principled training objectives, as future work.

Thanks again for your helpful feedback.

The authors

Reviewer_4wcVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes learning from credal supervision, where each training input is labeled with a set of plausible label distributions and the model learns a set of predictive distributions. Credal labels are built from existing soft labels by keeping the probability of the most likely class as a lower bound and leaving the remaining probability mass unassigned. The proposed method pessimistic-optimistic credal classifier (POCC) uses a shared backbone with two heads. One head minimizes the worst case cross-entropy over the credal label and the other minimizes the best case cross-entropy, and both have closed-form targets. The prediction is the average of the two heads, and the total variation distance between them is used as an uncertainty score. Under the assumption that the true conditional label distribution lies inside the credal label, the authors give a generalization bound for the averaged predictor with a term that grows with the width of the credal label. Experiments show that POCC achieves the best balanced score, which combines accuracy, expected calibration error (ECE) and the area under the accuracy-rejection curve.

Critical strengths

The paper addresses the loss of information when annotators disagree. Most methods merge their labels into one target, so the model cannot show that the label was uncertain. This paper trains on a set of possible labels and outputs a set of possible predictions, so the model can show when the labels were ambiguous. The training is also cheap, since it only adds a second head and the targets for both heads are computed directly without sampling.

Overall, the paper is well written and addresses an important problem, and I think it is worth publishing.

Critical weaknesses

I think the method is close to a simpler model. The optimistic head is trained on the most favorable label in the credal set, and since the one-hot label is always in the set, this head is pushed toward the hard label. The pessimistic head is trained on the least favorable label, and its best output is α\alpha on the top class with the rest spread evenly over the other classes, which is a smoothed hard label. In the label smoothing experiments, this is the same target the baselines are trained on. POCC is then close to a shared backbone with one head trained on hard labels and one trained on smoothed labels, where the uncertainty score is the gap between them.

The paper does not compare with this two-head model or with a model trained on q_L that uses the maximum softmax probability as its uncertainty score

Questions for authors

The best target for the optimistic head is always the one-hot label, and the best target for the pessimistic head is a smoothed hard label. Can you report results for each head alone and for a two-head model trained directly on these two targets? This would show whether the gains come from the credal formulation or from combining hard and smoothed targets.

2 comments

Dear Reviewer,

We are grateful for your valuable feedback and for acknowledging our work. Below, we address your comments.

Responses to W1 and Q2: POCC vs. a hard-label head plus a smoothed-label head.

We thank the reviewer for this insightful observation, which is correct at the per-sample optimum: for Qα(x)Q_\alpha(x), the optimistic and pessimistic objectives are minimised by the one-hot label eje_j and by the maximum-entropy member sαs_\alpha (α\alpha on jj, the rest spread evenly), respectively. We will add this analysis to the revision.

The suggested model can thus be seen as another instantiation of our framework, with both targets fixed before training. POCC instead minimises the two endpoints of the credal risk profile, so its extremal labels depend on the current predictions (Proposition 3). When α(x)<1/2\alpha(x) < 1/2, the optimistic objective also has local minimisers αej+(1−α)ek\alpha e_j + (1-\alpha) e_k (k≠jk \neq j), so the optimistic head can retain an alternative class supported by similar inputs, which a head trained on eje_j cannot. When α(x)>1/2\alpha(x) > 1/2, as under label smoothing, the per-sample optima coincide, and the two models differ mainly in their training dynamics.

Empirically, we implemented the suggested model (POCC-ab: same backbone and training protocol, one head trained on eje_j, one on sαs_\alpha, and their TV distance as the uncertainty score) and compared it with POCC on CIFAR-100 over 10 seeds.

In short, POCC-ab matches POCC's epistemic uncertainty only under nearly precise supervision (T=1T = 1, ϵ≤0.2\epsilon \le 0.2), and POCC is significantly better calibrated in five of the six settings. These gains come from optimising over the credal set during training, not from combining hard and smoothed targets. We will include this comparison in the revision.

Knowledge distillation (ResNet-20 teacher):

TTTeacher imprecision (%)MethodACC (%)ECE (%)AUARC (%)Spearman ×100
120.8POCC77.9 ± 0.23.9 ± 0.492.4 ± 0.165.5 ± 0.3
POCC-ab77.9 ± 0.34.1 ± 0.392.8 ± 0.167.1 ± 0.4
2.555.0POCC77.2 ± 0.210.6 ± 0.490.6 ± 0.264.0 ± 0.6
POCC-ab77.9 ± 0.316.8 ± 0.387.8 ± 0.253.9 ± 0.6
478.0POCC71.3 ± 0.418.0 ± 0.390.5 ± 0.269.9 ± 0.8
POCC-ab77.4 ± 0.125.8 ± 0.277.1 ± 0.422.8 ± 1.4

Label smoothing:

ϵ\epsilonMethodACC (%)ECE (%)AUARC (%)
0.1POCC79.4 ± 0.21.5 ± 0.293.3 ± 0.1
POCC-ab79.3 ± 0.32.5 ± 0.393.1 ± 0.2
0.2POCC79.3 ± 0.32.7 ± 0.392.8 ± 0.2
POCC-ab79.5 ± 0.36.2 ± 0.392.8 ± 0.2
0.5POCC78.4 ± 0.210.5 ± 0.389.1 ± 0.5
POCC-ab79.4 ± 0.117.4 ± 0.388.3 ± 0.3

Teacher imprecision: mean 1−max⁡kq(k∣x)1-\max_k q(k \mid x) over the test images. AUARC uses MMI for both models; Spearman: rank correlation between MMI and the teacher imprecision of each test image. Bold: significantly better (one-sided Wilcoxon signed-rank test over seeds, Holm over both directions, p<0.05p < 0.05).

  • Distillation. At T=1T = 1, where the teacher is confident, the two models are close, with POCC-ab slightly ahead in AUARC and Spearman correlation. As TT grows and α(x)\alpha(x) falls below 1/2 for more inputs, they diverge: POCC is significantly better calibrated (ECE lower by 6.2 and 7.8 points), and its MMI keeps tracking the teacher's imprecision (Spearman 64.0 and 69.9), whereas POCC-ab's loses this information (53.9 and 22.8). At T=4T = 4, the two models have nearly the same mean MMI (68.3% and 68.7%), but POCC-ab's no longer helps selective classification (AUARC 77.1 vs. ACC 77.4), whereas POCC's raises an accuracy of 71.3% to an AUARC of 90.5%. The difference thus lies in which inputs receive a large MMI, not in its overall size. POCC-ab is more accurate at high TT, as POCC's optimistic head can follow an alternative class when the teacher is nearly flat.

  • Label smoothing. Here α=1−ϵ(K−1)/K>1/2\alpha = 1-\epsilon(K-1)/K > 1/2 for all three values of ϵ\epsilon (K=100K = 100), so only the training dynamics differ.

Responses to W2 and Q1: Comparison with a model trained on qLq_L using MSP.

Following your constructive suggestions, we further add single networks trained on qLq_L and on sαs_\alpha (the smoothed head alone requested in Q2), with the same backbone and training protocol, on CIFAR-100 over 10 seeds; under label smoothing, sα=qLs_\alpha = q_L, so the two coincide. A head trained on eje_j alone is the same model for every TT and ϵ\epsilon, since eje_j does not depend on them, so it cannot reflect the imprecision.

Because single networks with single probabilistic predictions cannot express EU [R1], AUARC uses the maximum softmax probability (MSP) of each model's (mean) prediction:

  • Single networks. Without an EU, a single network can express the imprecision only by lowering its confidence, so both single-head models become increasingly underconfident: at T=2.5T = 2.5, the smoothed head is 77.1% accurate but only 41.2% confident on average (ECE 35.9%). The exception is T=1T = 1, where the unscaled teacher is nearly calibrated and so is the qLq_L model (ECE 2.4%), at 3.6 points lower accuracy than POCC.

  • Two-head models. Both are much better calibrated than the single networks under substantial imprecision (ECE 10.6% and 16.8% vs. 27.8% and 35.9% at T=2.5T = 2.5), and POCC is the best calibrated model in five of the six settings. With MSP, POCC-ab ranks errors better than POCC at high imprecision (by 2.1 points at T=4T = 4), but its EU loses track of the imprecision there (Spearman 22.8 vs. 69.9; see W1 and Q2).

Thus, single networks trained on qLq_L or sαs_\alpha can only fold the supervision imprecision into their probabilities, whereas POCC represents it as the width of its predictive set while keeping its mean prediction the best calibrated in five of the six settings. We will add these baselines to the revision.

Knowledge distillation (ResNet-20 teacher):

TTMethodACC (%)ECE (%)AUARC, MSP (%)
1POCC77.9 ± 0.23.9 ± 0.494.2 ± 0.1
POCC-ab77.9 ± 0.34.1 ± 0.394.2 ± 0.1
Smoothed head78.4 ± 0.212.3 ± 0.394.4 ± 0.1
qLq_L74.3 ± 0.42.4 ± 0.392.5 ± 0.1
2.5POCC77.2 ± 0.210.6 ± 0.493.5 ± 0.1
POCC-ab77.9 ± 0.316.8 ± 0.394.0 ± 0.1
Smoothed head77.1 ± 0.235.9 ± 0.393.7 ± 0.1
qLq_L71.0 ± 0.227.8 ± 0.290.2 ± 0.1
4POCC71.3 ± 0.418.0 ± 0.391.5 ± 0.2
POCC-ab77.4 ± 0.125.8 ± 0.293.6 ± 0.1
Smoothed head76.1 ± 0.356.2 ± 0.492.9 ± 0.1
qLq_L69.0 ± 0.447.8 ± 0.388.2 ± 0.2

Label smoothing:

ϵ\epsilonMethodACC (%)ECE (%)AUARC, MSP (%)
0.1POCC79.4 ± 0.21.5 ± 0.294.3 ± 0.1
POCC-ab79.3 ± 0.32.5 ± 0.394.3 ± 0.1
Smoothed head (=qL= q_L)75.9 ± 0.55.0 ± 0.392.4 ± 0.2
0.2POCC79.3 ± 0.32.7 ± 0.393.9 ± 0.1
POCC-ab79.5 ± 0.36.2 ± 0.394.2 ± 0.2
Smoothed head (=qL= q_L)75.4 ± 0.313.5 ± 0.391.6 ± 0.3
0.5POCC78.4 ± 0.210.5 ± 0.392.1 ± 0.2
POCC-ab79.4 ± 0.117.4 ± 0.394.1 ± 0.1
Smoothed head (=qL= q_L)74.9 ± 0.339.1 ± 0.490.2 ± 0.3

Bold: significantly better of POCC and POCC-ab (one-sided Wilcoxon signed-rank test over seeds, Holm over both directions, p<0.05p < 0.05).

We hope to have thoroughly addressed your concerns and trust that this provides stronger support for our paper.

[R1] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.