- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that conventional evidential deep learning (EDL) provides no explicit training target for uncertainty, so evidence can grow on all inputs as training proceeds (Eq. 3, Fig. 1). The proposed CSEDL splits the output layer into a direction head π, trained with cross-entropy, and a scalar uncertainty head mapped to u ∈ (0.05, 0.95). The two are recombined into Dirichlet parameters α_k = 1 + (K/u − K)π_k, so the predictive distribution is p_k = u/K + (1 − u)π_k. The uncertainty head is regressed onto a "conformal rank": the rank of each training sample's nonconformity score, 1 − p_y, among the scores of recent training batches held in a first-in-first-out cache, smoothed with a sigmoid. The standard EDL KL regularizer on non-target evidence is kept. Experiments cover OOD detection with CIFAR-10 as the in-distribution set and six OOD sets, tumor referral on PatchCamelyon, trajectory-risk forecasting on Argoverse 1, multi-view classification with missing views, and calibration of a fine-tuned Qwen2.5-3B on four multiple-choice QA sets. CSEDL reports the best average OOD AUROC (91.43, against 89.46 for a five-model deep ensemble) and the best results on most other metrics. The supplement provides code for the CIFAR-10 OOD experiment.
Critical strengths
S1. A simple, low-cost mechanism with strong OOD results. CSEDL adds one scalar head and a cache of recent nonconformity scores. The supplementary script implements it compactly and consistently with Eqs. 4–16 and the stated hyperparameters. Training with the cache takes 218.1 s, against about 340 s when the scores are recomputed over the training set once per epoch (Fig. 4b). With a single model, CSEDL gives the best average OOD AUROC in Table 1 (91.43 vs. 89.46 for deep ensembles), with large margins on SVHN (97.01 vs. 93.59) and LFWPeople (99.44 vs. 93.10). This is consequential because a single-model plug-in that outperforms a five-model ensemble at OOD detection would be useful in practice, provided the gains survive the concerns below.
S2. Breadth of evaluation. The method is tested on five task families: image OOD detection, medical referral, driving risk, multi-view fusion and LLM calibration. It is compared with up to seven EDL variants (three, plus IB-EDL, for the LLM task) and, for OOD detection, with deep ensembles, PostN and NatPN, with means and standard deviations reported. CSEDL is best or close to best on most metrics. This matters because it suggests the effect is not tied to one setting, which is also why the attribution problems in W2 matter.
Critical weaknesses
W1. The uncertainty target is a uniform, in-sample rank. On CIFAR-10 this appears to leave the predictive distribution strongly underconfident, and the training signal is partly self-referential.
- Uniform by construction. The target is the empirical CDF of the nonconformity score (Eqs. 11–13), which the paper describes as "a smooth and uniform measure of uncertainty" (l.245). Over the training data it is therefore approximately uniform on [0, 1], however accurate the model is. On CIFAR-10 the learned uncertainty follows suit. In Fig. 5 (epoch 100), CIFAR-10 samples are spread over all five uncertainty bins (about 15–28% each), with about 28% in [0.8, 1], the bin that holds about 93% of SVHN samples.
- Consequence for calibration on CIFAR-10.
- Because p_k = u/K + (1 − u)π_k (Eq. 8), each prediction's confidence is at most 1 − (1 − 1/K)u.
- Fig. 1 reports u_ID = 0.3956. The four u values in Fig. 1 equal K/(K + mean evidence), so they are at most the arithmetic mean of u, and the bound below is conservative.
- If Fig. 1 shows CIFAR-10 test samples for the reported model, average confidence is at most about 0.64, against 92.5% accuracy (Table 1). Since ECE is at least the gap between accuracy and mean confidence, CIFAR-10 ECE would be at least about 28 points. Fig. 5's histogram implies a similar or larger gap.
- Whether the figures describe the Table 1 model needs confirming. The released script trains for 20 epochs by default, while Figs. 1 and 5 run to epoch 100.
- The paper reports no in-distribution calibration for CIFAR-10 or PCam, although the script already collects p on the test set. Calibration appears only for the LLM and multi-view tasks, where the reported ECE is low.
- By the same bound, the LLM results require much smaller u. With K = 4, an ECE of 2.54% at 94.07% accuracy (SciQ, Table 4) allows a mean u of at most about 0.11. The paper does not explain why the same uniform target yields small u there but not on CIFAR-10. (Fused multi-view predictions escape the bound because Dempster–Shafer fusion multiplies view-level uncertainties.)
- Self-referential target. Eq. 10 computes the nonconformity score from p, which contains u, and the released code does the same (
nonconformityin csedl.py usesprob). This gives Z_i = 1 − π_y + u(π_y − 1/K). For training samples the classifier already fits (π_y ≈ 1), Z_i ≈ (1 − 1/K)u. The target then reduces to the rank of the sample's current uncertainty among the uncertainty values the model produced for earlier batches. The stop-gradient prevents a direct collapse, but it is unclear what information the target carries once training accuracy saturates. The paper reports neither the training accuracy nor any analysis of this. - No conformal guarantees. The scores are computed in-sample, on training data, against a cache of the last 1,024 training scores (four batches of 256 in the released code). The cache is filled under earlier parameters, and there is no held-out calibration set and no exchangeability. None of conformal prediction's guarantees therefore apply. The claim that the rank "guarantees a better uncertainty quantification performance" (l.245–246) is supported only empirically, by the Entropy and Raw-score variants in Table 1 on one benchmark.
This is consequential because the paper's central claim is better uncertainty quantification, and these issues stem from the design of the target rather than from a particular experiment.
W2. The evaluation does not isolate what drives the gains.
-
Ablation.
- Without the KL term, conformal-rank supervision gives OOD AUROC of 76.74 / 71.25 / 80.22 on SVHN / CIFAR-100 / GTSRB (Table 5). That is below vanilla EDL on SVHN and CIFAR-100 (79.84 / 79.20, Table 1).
- The full model's 97.01 / 87.59 / 90.68 appears only when both terms are combined. "Complementary roles" (l.477–479) does not explain jumps of 10–20 points.
- A likely reason, not discussed in the paper: because α's non-target entries scale with K/u − K, the KL term also trains u directly. By my calculation, consider a training sample whose π puts at least 10% of its mass on wrong classes. Unless u is already high, the KL gradient on the uncertainty logit exceeds the largest possible rank-loss gradient (0.02 per sample with λ_unc = 0.01) by roughly 10× to over 1,000×. For confidently fitted samples it is near zero. The KL term may therefore be the main uncertainty signal for hard samples, with the rank loss mainly spreading the easy ones.
- The variant without either term sits at chance (50.00 / 49.98 / 50.01). The uncertainty head then receives no gradient (in csedl.py, L_dir uses only the direction head's logits), so this is not an informative ablation.
- Table 5 reports no standard deviations.
-
Missing baselines. Classification is decided entirely by the direction head (argmax p = argmax π under Eq. 8), which is trained with cross-entropy rather than an EDL loss. CSEDL's accuracy (92.50%) exceeds every EDL baseline (86.23–90.40%) and the deep ensemble (92.33%). Yet its own cross-entropy-only variant reaches 87.31% (Table 5), so the source of the extra accuracy is unclear. The paper also reports no standard single-model baseline (a cross-entropy model scored by maximum softmax probability, entropy or energy) and no auxiliary-confidence baseline such as ConfidNet. Without them, it is unclear whether CSEDL improves on standard practice outside the EDL family.
-
Selection on test data. App. F chooses M_max = 1024 "because it achieves the highest AUROC among the tested cache sizes" (l.874–876). The paper describes no validation split, and the released script trains on the full CIFAR-10 training set and evaluates directly on the test sets. The AUROC at 1024 in Fig. 4a equals Table 1's SVHN result (97.01), so the cache size appears to have been chosen on test OOD data. The choice matters: at M_max = 256, the SVHN margin over deep ensembles shrinks from 3.42 to 0.11 points. The paper does not say how λ_unc = 0.01 and T = 0.05 were chosen.
-
Reproducibility. The supplement contains a single script, for CSEDL on the CIFAR-10 OOD benchmark. There is no code for the baselines, the Entropy and Raw-score variants, Figs. 1, 4 and 5, or the other four tasks. The script's argument checks also reject λ_unc = 0 and λ_kl = 0, so Table 5 cannot be run as released. Most reported results therefore cannot be reproduced from the supplement, although the paper says the code is provided "to facilitate the reproduction of our results" (l.500–501).
-
Positioning and scope. The paper claims to be the first to supervise uncertainty explicitly in EDL (l.49–50). Sec. 2.3 also states that CP-based methods typically use conformal scores post hoc rather than as a training signal (l.153–155). Both overlook closely related work:
- auxiliary confidence heads, such as ConfidNet, which regresses the true-class probability (Corbière et al., NeurIPS 2019), and DeVries & Taylor (2018);
- conformal training (Stutz et al., ICLR 2022; Einbinder et al., NeurIPS 2022). Einbinder et al. penalize the deviation of hold-out conformity scores from uniformity using a smoothed empirical CDF, which is close to Eq. 13;
- analyses showing that EDL objectives do not yield faithful epistemic uncertainty, and that the resulting measures are relative rather than absolute (Bengs et al., NeurIPS 2022; Jürgens et al., ICML 2024; Shen et al., NeurIPS 2024).
Beyond OOD detection, comparisons are limited to EDL variants (Tables 2, 3, 6), and Table 4 uses a single LLM. Several of Table 4's differences are within one standard deviation, although the text says it "proves the generalization ability" (l.462–463).
This is consequential because the evidence that the conformal-rank target drives the gains comes from within-method ablations on one benchmark. That evidence also depends on the KL term, uses a cache size apparently chosen on test data, lacks standard single-model baselines, and mostly cannot be reproduced from the supplement.
Questions for authors
- Additional result requested (could change my assessment; W1).
- What are the ECE and NLL of the final prediction p (Eq. 8) on the CIFAR-10 and PCam test sets? The released script already collects p for the test set.
- Do Fig. 1 and Fig. 5 show test samples of the model in Table 1? The script trains for 20 epochs by default, while the figures run to epoch 100.
- The u values in Fig. 1 equal K/(K + mean evidence), and u_ID = 0.3956 caps mean confidence at about 0.64, against 92.5% accuracy. Why, by contrast, does Table 4 report ECE of 2.5–6.3%? Under Eq. 8 that requires a mean u of at most about 0.11–0.33.
- Clarification (W1). The released code computes Z_i from p, as Eq. 10 states. What is the training accuracy at the end of training? Once most training samples have π_y ≈ 1, so that Z_i ≈ (1 − 1/K)u, what does the rank target capture?
- Clarification and additional result (W2). Which data were used to choose M_max, λ_unc and T? Could you report Table 1 with these chosen on held-out in-distribution data only?
- Additional result requested (W2).
- How does CSEDL compare with a single cross-entropy model scored by maximum softmax probability, entropy or energy, and with a ConfidNet-style confidence head on the same backbone?
- Why does conformal-rank supervision without the KL term (Table 5) fall below vanilla EDL (Table 1) on SVHN and CIFAR-100? How much of u's training signal comes from the KL term, which acts on u through α?
- What are the standard deviations in Table 5?
- Clarification. The supplement gives the CIFAR-10 setup: VGG16-BN with global average pooling, AdamW, cosine schedule, batch size 256, 20 epochs and seed 42 by default.
- Did Table 1 use 20 epochs, or the 100 shown in Figs. 1 and 5?
- How many runs are behind the ± values?
- What setups were used for PCam, Argoverse, the multi-view task and the Qwen2.5-3B fine-tuning, and which quantity is the uncertainty score for each?
- Will code for the baselines, ablations and other tasks be released?
Dear Reviewer ePzp,
Thanks again for your valuable comments. We provided a rebuttal to your review comments four days ago. Would you please kindly check whether your concerns are well addressed by those responses? Thanks.
Regards from the authors.