- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce a simulation protocol that grades epistemic-uncertainty (EU) measures against a computed oracle. Their results show that credal sets beat Bayesian model averaging (BMA) for very small ensembles (fewer than five members) in sparse, low-class geological classification.
Critical strengths
- The paper is clearly written, and the oracle-graded approach to EU measurement is well motivated.
Critical weaknesses
- The spread oracle target mirrors what BMA does. It naturally agrees as increases.
- The headline claims are not supported by the evidence. The assumed scenario involves few samples that are very expensive to collect. Given that premise, computation is a negligible cost, leaving no reason not to avoid the small range where credal sets help.
Questions for authors
- Does the gap between credal sets and BMA survive comparisons with other dispersion measures such as standard deviation, bias-corrected MI, and pairwise divergences between members?
- Can the authors clarify the credal framing in the title when the results from the paper show that it doesn't matter? I might be missing something.
- Assuming computational cost matters after all, can the authors provide empirical evidence from a larger, more expensive experiment? I am aware that this might be infeasible within the relatively short rebuttal period, but it is necessary for me to see the value in the paper, unless it were rewritten in a "Credal wrappers don't matter for..." manner.
2 comments
Thank you for your rebuttal.
- The authors acknowledge the problems with the credal framing. Yet, several sections of the submission still heavily discuss its advantages (e.g., Contribution 2 and Sections 6.2 and 6.3).
- This is not in my review, but in their answer to ebtL, the authors claim to have checked their 43 entries... yet the new version is a mess. See, e.g., Wang et al. 2024a/b, 2025a/b, etc. I didn't bother to check further. Additionally, the authors sneakily more than doubled the number of references by adding a lot of papers by Cuzzolin, most of them single-author papers. (And by a lot, I mean a lot! There are more than 40 references with him as a co-author.) And yes, while the term "sneakily" is rather loaded for a review discussion, I mean it. I consider this a serious ethics violation.
- The new Section L was generated via AI, as revealed in the updated AI statements, without this being acknowledged in the rebuttal where only an "independent implementation" is mentioned.
- Despite acknowledging W2, the introduction still motivates the work via the expense of an ensemble.
- In the new related work section, where does this come from: "it predicts over four answer options rather than vocabulary tokens"? It has no relation to this paper, as far as I can tell, and I strongly expect to find that it is a copy and paste from another Cuzzolin paper if I were to search for it.
I lower my score.
Response to Reviewer EQJo
We thank the reviewer. Both weaknesses are correct, the experiment in Q1 confirms the reviewer's reading, and we have reframed the paper accordingly.
W1, the spread oracle mirrors BMA. We agree, and we tested it. Against a variance-form spread oracle and against squared-loss reducible error, which do not share BMA's functional, the reversal at large M disappears: the gap between width and BMA at M = 40 is +0.011 (P = 0.35) and +0.019 (P = 0.17), against -0.022 on the original oracle. The small-M lead survives on both spread targets, under a random design, and on squared-loss reducible error: +0.038 to +0.076 at M ≤ 5, all P < 0.001. Against log-loss reducible error it is +0.01 to +0.02 and mostly not significant. So the reversal in the submission was largely produced by the choice of target, and we no longer report it as a finding about credal sets.
Q1, other dispersion measures. The gap does not survive as a credal effect. At M = 2 the interval width is exactly twice the standard deviation of the member probabilities, so the two cannot differ in ranking. At M = 3 and 5 the paired gap between them is +0.000 and -0.003 (P = 0.61 and 0.47), and from M = 20 the standard deviation is ahead of the width (P < 0.01). Jackknife-corrected mutual information and mean pairwise KL match plug-in mutual information to within 0.001 at every M. The gap is therefore between probability-scale and information-scale dispersion. Wrapping the ensemble as a credal set adds nothing over the standard deviation in this regime.
W2, small M is avoidable when data are few. We agree for the setting in the paper. With a few hundred tabular samples, ten or more members are cheap. Our practical conclusion is now: use a probability-scale dispersion measure such as the standard deviation, which is at least as faithful as mutual information at every M we tested on the variance and squared-loss reducible-error targets; do not use the entropy interval for two-class problems. Small M is forced only when a member is expensive, such as a large fine-tuned network. We have not tested that case and do not claim our numbers transfer to it.
Q2, the credal framing in the title. The reviewer has not missed anything. We have changed the title to "Oracle-Referenced Evaluation of Epistemic Uncertainty Measures in Sparse Geological Data" and rewritten the abstract around three results: the oracle protocol; the finding that the scale of the dispersion measure matters and the credal wrapping does not, in sparse low-class classification; and the failure of the entropy interval in the binary case, which confirms the analysis of Hüllermeier, Destercke and Shaker (2022) against a computed reference.
Q3, a larger and more expensive experiment. We could not complete one in the rebuttal period. With the reframing above the paper no longer depends on small M being unavoidable, and we list the expensive-member case as the main open test. We also report a limit the new experiments exposed: all disagreement-based measures track reducible error weakly in our worlds (rank correlation 0.14 to 0.34 at n = 150), because squared bias is about 83% of it.