- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies epistemic learning from credal supervision and proposes a Pessimistic–Optimistic Credal Classifier (POCC). The method constructs credal labels from imprecise supervision, trains two prediction heads using worst-case and best-case cross-entropy objectives, and represents predictive uncertainty using the total variation distance between their outputs. The paper also establishes a finite-sample risk bound under conditional coverage assumptions and evaluates the method on annotator disagreement, teacher predictions, and label smoothing. The problem is relevant, the framework is clearly presented, and the experimental evaluation is relatively extensive.
Critical strengths
1、The paper addresses an important problem, retaining unresolved annotation information rather than collapsing all supervision into a single probability distribution. 2、The proposed two-head architecture admits efficient optimization under cross-entropy loss. 3、The theoretical analysis explicitly separates model complexity, finite-sample error, optimization error, and supervision imprecision. 4、The experimental evaluation covers multiple datasets and three different sources of imprecise supervision.
Critical weaknesses
1、1. The proposed uncertainty score is not shown to identify annotation-induced epistemic uncertainty. The total variation distance between the pessimistic and optimistic heads measures their predictive disagreement, but this disagreement may also arise from differences in optimization objectives, parameterization, and training dynamics. The method does not establish that the score isolates uncertainty induced by imprecise supervision. In particular, when the credal label degenerates to a singleton, the two heads have the same nominal objective, but the architecture does not guarantee identical predictions. Controlled experiments with known annotation ambiguity and singleton supervision are necessary. 2、The semantic validity of the predictive credal set remains unestablished. The line segment between two learned predictions is a valid convex set of probability distributions, but the paper does not demonstrate that it represents the full set of plausible predictions induced by the original credal supervision. The theoretical guarantee concerns the risk of the averaged predictor, not coverage or calibration of the predictive set. The authors should either establish a meaningful set-coverage result or narrow their claims about the interpretation of the predictive credal set. 3、The theoretical analysis requires the true conditional label distribution to belong to the supplied credal set. However, the proposed construction derives its probability lower bound from the maximum probability of a reference soft label. This does not guarantee coverage for finite annotator samples or potentially overconfident teacher predictions. The authors should evaluate sensitivity to coverage violations and clarify which experimental settings plausibly satisfy the assumptions. 4、The uncertainty evaluation does not directly validate supervision-induced ambiguity. AUARC and the supplementary uncertainty–error correlation assess whether the uncertainty score ranks prediction errors. They do not directly establish that the score captures human annotation disagreement or uncertainty in the underlying label distribution. The paper would benefit from direct comparisons with human disagreement, teacher disagreement, and controlled ground-truth ambiguity.
Questions for authors
1、Under what conditions does the distance between the two heads monotonically reflect the true degree of annotation imprecision? 2、Can the authors demonstrate coverage or calibration of the predicted credal set on synthetic data where the true conditional distribution is known? 3、How does POCC compare against direct label-relaxation and credal-supervision baselines using the same information? 4、How robust are the theoretical and empirical results when the conditional coverage assumption is violated? 5、Would the conclusions remain unchanged if BQS were replaced by raw metrics and the statistical analysis accounted for multiple comparisons?
5 comments
Responses to W2, W3, Q1–Q3.
We sincerely thank you for these points, which help us sharpen the scope of our claims. We first clarify the scope.
-
Predictive credal set. The segment in Eq. (6) is the convex hull of the predictions of the pessimistic and optimistic criteria, and its width is our predictive epistemic uncertainty score. We do not claim that it covers the true conditional distribution or contains every prediction compatible with the credal labels, as already stated in Section 4 and Appendix J. We agree that some wording in Sections 1 and 3.1 (e.g., the set "represents the consequences of unresolved annotation evidence") suggests a stronger reading, and will revise it accordingly, for instance by stating that the set reflects how much the supervision leaves unresolved.
-
Theory. Theorem 5 is a first guarantee for valid but imprecise supervision: it bounds the risk of the averaged predictor under coverage, which separates the effect of imprecision from that of invalid constraints. As acknowledged in Section 6, coverage need not hold for finite annotations or imperfect teachers; a guarantee under probabilistic coverage remains future work, but below we show how the current one degrades when coverage fails.
Within this scope, and following your suggestions, we add the analyses below.
How plausible coverage is in our settings, and sensitivity to violations (W3).
With deterministic labels, as for CIFAR, the true conditional is one-hot (Section 4), so covers it if and only if the reference's top class is correct, whatever the value of . Overconfidence alone therefore never violates coverage; it matters only when the top class is wrong, where the TV distance from to is , the confidence placed on the wrong class.
In our experiments:
| Setting | Violation rate | Mean violation |
|---|---|---|
| Label smoothing (all ) | 0% | 0 |
| CIFAR-10H (human majority vs. CIFAR-10 label) | 0.79% | 0.0046 |
| CIFAR-10, ResNet-20 teacher | 0.05% | 0.0003 (0.0004) |
| CIFAR-10, ShuffleNetV2-0.5× teacher | 0.92% | 0.0040 (0.0060) |
| CIFAR-10, MobileNetV2-x0.5 teacher | 0.13% | 0.0006 (0.0008) |
| CIFAR-100, ResNet-20 teacher | 6.59% | 0.0147 (0.0344) |
| CIFAR-100, ShuffleNetV2-0.5× teacher | 3.21% | 0.0074 (0.0171) |
| CIFAR-100, MobileNetV2-x0.5 teacher | 0.96% | 0.0026 (0.0053) |
Summary: For the teachers, is reported at , as in our experiments, with in brackets: temperature scaling does not change which teacher predictions are wrong, but lowers their confidence and hence the violation by more than half on CIFAR-100. POCC's results are stable across these settings: it dominates every baseline in all six teacher configurations, including CIFAR-100 with ResNet-20, which has the most violations (see our response to W1 and W4 on multiple comparisons). Coverage is least plausible with few annotators on genuinely ambiguous inputs (MiceBone, TreeVersity), where the top frequency tends to overestimate .
We also would like to note that annotator frequencies are nonetheless the standard estimate of the ground-truth conditional distribution [R1, R2, R3], as are teacher probabilities in distillation [R4]. Under this approximation, coverage holds by construction, since (Eq. (8)); our assumption is weaker, requiring only that lies somewhere in . Since these datasets provide no ground-truth conditional distribution, we study the few-annotator case on synthetic data below.
Regarding the degradation and monotonicity (W3, Q1, Q3).
-
Violated coverage. Cross-entropy is linear in the target, so under the boundedness condition of Theorem 5. Without coverage, still holds up to , and Theorem 5 holds with an additional term proportional to : the guarantee degrades linearly in the violation, which is small in our settings.
-
Monotonicity. For a training input, the optimistic loss is minimised by and the pessimistic loss by , which puts on and spreads evenly over the other classes. These are the minimum- and maximum-entropy members of , and their TV distance is . When the heads attain these per-sample optima, the predictive segment therefore spans the full TV diameter of the credal label, and MMI equals the imprecision, linearly and hence monotonically. On test inputs, this further requires the imprecision to be learnable from .
Synthetic study with known (Q1–Q3).
We train POCC on a synthetic three-class problem in which the true conditional is known for every input. For each training input, simulated annotators draw labels from , and is built from their empirical distribution as in Section 3.2 (oracle: ); the detailed configuration is given at the end of this response.
Metrics. (a) Validity of the credal labels on the training inputs: the fraction with (label coverage) and the mean TV distance from to .
(b) Ranking: the Spearman correlation between MMI and the true ambiguity on the test inputs.
(c) Magnitude: since Spearman only checks the ranking, we group the test inputs by their true ambiguity and compare, within each group, the mean MMI with the mean true ambiguity, in the spirit of a reliability diagram. Both are in TV units, and for the oracle the true ambiguity equals the diameter of , which MMI matches at the per-sample optimum.
(d) The expected accuracy of the averaged prediction under . Table S1 reports (a), (b), (d) and the mean MMI; Table S2 reports (c).
Results. (i) With valid labels (oracle), the width of the predictive set is calibrated to the true ambiguity: in every bin of Table S2, the mean MMI matches the mean true ambiguity, e.g., inputs whose true top-class probability is about 0.6 receive a width of about 0.4, and the ranking is near-perfect (Spearman 99.7).
(ii) With finite annotators, coverage is violated often but mildly (Table S1), since the top frequency frequently exceeds by a small amount. Accuracy nevertheless stays within 0.1 points of the Bayes accuracy and the ranking is preserved (Spearman ≥ 98); the violation mainly shrinks MMI in the most ambiguous bin (42.6/34.8/27.0% with 50/10/3 annotators, against 51.0%), so performance degrades gradually with . We will add this study to the appendix.
Table S1. Summary over 10 seeds (mean true ambiguity 16.2%, Bayes accuracy 83.8%).
| Supervision | Label coverage | (%) | Spearman (MMI, true ambiguity) ×100 | Mean MMI (%) | Accuracy (%) |
|---|---|---|---|---|---|
| Oracle () | 100% | 0.0 | 99.7 | 16.3 | 83.8 |
| 50 annotators | 41% | 1.8 | 99.2 | 15.8 | 83.8 |
| 10 annotators | 30% | 4.7 | 98.4 | 14.7 | 83.7 |
| 3 annotators | 17% | 10.4 | 98.0 | 11.9 | 83.7 |
Table S2. Mean MMI by bin of true ambiguity (%).
| True ambiguity bin | [0, 0.05) | [0.05, 0.15) | [0.15, 0.3) | [0.3, 0.45) | [0.45, 2/3] |
|---|---|---|---|---|---|
| Mean true ambiguity | 1.8 | 9.3 | 21.9 | 37.4 | 51.0 |
| Mean MMI, oracle | 1.8 | 9.1 | 21.7 | 38.4 | 50.9 |
| Mean MMI, 50 annotators | 1.8 | 9.1 | 22.2 | 39.7 | 42.6 |
| Mean MMI, 10 annotators | 1.7 | 9.3 | 23.0 | 35.5 | 34.8 |
| Mean MMI, 3 annotators | 1.7 | 8.6 | 19.0 | 26.2 | 27.0 |
Detailed configuration. Labels are uniform over three classes and inputs are drawn as , with the means at the vertices of an equilateral triangle of radius 1.5. The true conditional is then known for every input: it is close to uniform near the centre and increasingly confident away from it. For each of 3,000 training inputs, simulated annotators draw labels from . POCC is an MLP (2–64–64, ReLU) with two linear heads, trained with Eq. (5) and the closed-form targets of Proposition 3. We evaluate on 10,000 test inputs over 10 seeds.
Dear Reviewer,
We thank the reviewer for your review effort and the thoughtful comments. Below, we address your concerns.
Responses to W1 and W4: whether MMI captures supervision-induced imprecision.
Thank you for this constructive feedback. Both points concern whether MMI captures supervision-induced imprecision, so we would like to answer them together. In short, POCC's MMI vanishes without supervision imprecision, grows with it under control, and tracks human and teacher ambiguity more closely than the baselines' EU.
Singleton supervision (W1). The two heads' objectives differ only through the credal label and coincide when it is a singleton (Proposition 3). We trained POCC with one-hot labels: MMI is zero up to floating-point precision on CIFAR-10 and CIFAR-100, versus 5.8% and 14.3% at ϵ=0.05. Although initialised independently, the two linear heads minimise the same objective and converge to the same predictions, so parameterisation and training dynamics alone create no disagreement.
POCC's EU captures supervision-induced imprecision (W1, W4). We agree that AUARC evaluates how well the EU ranks errors, not whether it reflects the imprecision of the supervision. Following your suggestion, we test the latter directly. Label smoothing injects a known imprecision d=ϵ(K−1)/K while the images stay fixed (Here, K denotes the number of classes). As ϵ grows, POCC's mean test MMI increases in every seed, by about the injected amount:
POCC's mean test MMI (%) and the injected imprecision d (%):
Comparison with the baselines' EU (W4). We further compare MMI with the baselines' EU, first under controlled imprecision and then under human and teacher disagreement.
Controlled imprecision. Since the baselines' EU scores have different units, we compare them by the ratio of each method's mean test EU at ϵ=0.2 to that at ϵ=0.05. Only DAPPr's EU also increases, and less than MMI; the others stay flat or decrease:
Human and teacher disagreement. We correlate each method's EU on test images with the imprecision 1−maxkq(k∣x) of the held-out human or teacher distribution (Spearman $\times$100, mean ± std over 10 seeds; best in bold). For human disagreement, we use MiceBone and TreeVersity, whose test splits are annotated; CIFAR-10H is not included because our protocol (Appendix C.3.1) uses all its annotated images for training and evaluates on CIFAR-10 images with a single label each.
Human disagreement:
Teacher predictions (T=2.5), CIFAR-10:
Teacher predictions (T=2.5), CIFAR-100:
POCC is best in 7 of the 8 settings, and one-sided Wilcoxon signed-rank tests over 10 seeds (Holm-corrected) show that it is significantly better than every baseline (p=0.008 for human disagreement, p=0.012 for teachers), with one exception: on CIFAR-100, it beats the deep ensemble for ResNet-20 but ties for ShuffleNetV2 and trails for MobileNetV2. This ensemble, however, trains and evaluates five networks, about five times the cost of POCC's single backbone.