acceptodds
Desk rejected at ICLR 2027

Credal Wrapper for Uncertainty Modelling in Sparse Geological Data

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine learning models for mineral exploration are trained on a few hundred sparse, expensive samples and used to decide where to drill next, so their epistemic uncertainty (EU), the part that more data could remove, matters as much as their predictions. Yet EU cannot be measured on real data, because the counterfactual that defines it is never observed, so estimators are graded by proxies such as out-of-distribution detection and calibration. We contribute a general simulation protocol that makes EU directly computable for a given model and can grade any EU estimator. It separates two estimands the literature conflates, reducible error and estimator spread, builds an oracle reference for each in a synthetic prospectivity field sampled by clustered drilling surveys, and compares uncertainty measures of an identical ensemble within each simulated world. We use it to ask when credal (imprecise-probability) measures are more faithful than Bayesian model averaging (BMA) in the sparse, low-class regime of geological classification. Three findings follow. First, the credal interval width recovers the epistemic ordering better than BMA when members are few ( to in rank correlation for , ), confirming the small-ensemble advantage reported for credal wrappers; the advantage vanishes by and reverses at . Second, the choice of credal functional matters more than the choice between credal and BMA: in binary (deposit versus barren) classification the entropy interval carries no epistemic signal () while the interval width of the same credal set does. Third, the credal advantage does not widen as training data are thinned, so it compensates for few posterior samples rather than few data. On real drill-hole geochemistry from South Australia, a resampling oracle reproduces the first finding. The protocol applies to any EU estimator whose data-generating process can be simulated.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Quantification, Bayesian modelling, Credal sets

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EQJoOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce a simulation protocol that grades epistemic-uncertainty (EU) measures against a computed oracle. Their results show that credal sets beat Bayesian model averaging (BMA) for very small ensembles (fewer than five members) in sparse, low-class geological classification.

Critical strengths
  • The paper is clearly written, and the oracle-graded approach to EU measurement is well motivated.
Critical weaknesses
  • The spread oracle target mirrors what BMA does. It naturally agrees as MM increases.
  • The headline claims are not supported by the evidence. The assumed scenario involves few samples that are very expensive to collect. Given that premise, computation is a negligible cost, leaving no reason not to avoid the small range where credal sets help.
Questions for authors
  1. Does the M≤5M\leq 5 gap between credal sets and BMA survive comparisons with other dispersion measures such as standard deviation, bias-corrected MI, and pairwise divergences between members?
  2. Can the authors clarify the credal framing in the title when the results from the paper show that it doesn't matter? I might be missing something.
  3. Assuming computational cost matters after all, can the authors provide empirical evidence from a larger, more expensive experiment? I am aware that this might be infeasible within the relatively short rebuttal period, but it is necessary for me to see the value in the paper, unless it were rewritten in a "Credal wrappers don't matter for..." manner.
2 comments
AuthorsOpenReview ↗

Response to Reviewer EQJo

We thank the reviewer. Both weaknesses are correct, the experiment in Q1 confirms the reviewer's reading, and we have reframed the paper accordingly.

W1, the spread oracle mirrors BMA. We agree, and we tested it. Against a variance-form spread oracle and against squared-loss reducible error, which do not share BMA's functional, the reversal at large M disappears: the gap between width and BMA at M = 40 is +0.011 (P = 0.35) and +0.019 (P = 0.17), against -0.022 on the original oracle. The small-M lead survives on both spread targets, under a random design, and on squared-loss reducible error: +0.038 to +0.076 at M ≤ 5, all P < 0.001. Against log-loss reducible error it is +0.01 to +0.02 and mostly not significant. So the reversal in the submission was largely produced by the choice of target, and we no longer report it as a finding about credal sets.

Q1, other dispersion measures. The gap does not survive as a credal effect. At M = 2 the interval width is exactly twice the standard deviation of the member probabilities, so the two cannot differ in ranking. At M = 3 and 5 the paired gap between them is +0.000 and -0.003 (P = 0.61 and 0.47), and from M = 20 the standard deviation is ahead of the width (P < 0.01). Jackknife-corrected mutual information and mean pairwise KL match plug-in mutual information to within 0.001 at every M. The gap is therefore between probability-scale and information-scale dispersion. Wrapping the ensemble as a credal set adds nothing over the standard deviation in this regime.

W2, small M is avoidable when data are few. We agree for the setting in the paper. With a few hundred tabular samples, ten or more members are cheap. Our practical conclusion is now: use a probability-scale dispersion measure such as the standard deviation, which is at least as faithful as mutual information at every M we tested on the variance and squared-loss reducible-error targets; do not use the entropy interval for two-class problems. Small M is forced only when a member is expensive, such as a large fine-tuned network. We have not tested that case and do not claim our numbers transfer to it.

Q2, the credal framing in the title. The reviewer has not missed anything. We have changed the title to "Oracle-Referenced Evaluation of Epistemic Uncertainty Measures in Sparse Geological Data" and rewritten the abstract around three results: the oracle protocol; the finding that the scale of the dispersion measure matters and the credal wrapping does not, in sparse low-class classification; and the failure of the entropy interval in the binary case, which confirms the analysis of Hüllermeier, Destercke and Shaker (2022) against a computed reference.

Q3, a larger and more expensive experiment. We could not complete one in the rebuttal period. With the reframing above the paper no longer depends on small M being unavoidable, and we list the expensive-member case as the main open test. We also report a limit the new experiments exposed: all disagreement-based measures track reducible error weakly in our worlds (rank correlation 0.14 to 0.34 at n = 150), because squared bias is about 83% of it.

Thank you for your rebuttal.

  • The authors acknowledge the problems with the credal framing. Yet, several sections of the submission still heavily discuss its advantages (e.g., Contribution 2 and Sections 6.2 and 6.3).
  • This is not in my review, but in their answer to ebtL, the authors claim to have checked their 43 entries... yet the new version is a mess. See, e.g., Wang et al. 2024a/b, 2025a/b, etc. I didn't bother to check further. Additionally, the authors sneakily more than doubled the number of references by adding a lot of papers by Cuzzolin, most of them single-author papers. (And by a lot, I mean a lot! There are more than 40 references with him as a co-author.) And yes, while the term "sneakily" is rather loaded for a review discussion, I mean it. I consider this a serious ethics violation.
  • The new Section L was generated via AI, as revealed in the updated AI statements, without this being acknowledged in the rebuttal where only an "independent implementation" is mentioned.
  • Despite acknowledging W2, the introduction still motivates the work via the expense of an ensemble.
  • In the new related work section, where does this come from: "it predicts over four answer options rather than vocabulary tokens"? It has no relation to this paper, as far as I can tell, and I strongly expect to find that it is a copy and paste from another Cuzzolin paper if I were to search for it.

I lower my score.

Reviewer_ebtLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an oracle-referenced evaluation protocol for epistemic uncertainty in sparse geological data, distinguishing reducible error from estimator spread. It compares Bayesian model averaging, credal interval width, and the entropy interval from the same ensemble against a disagreement oracle. The main simulations report a small-ensemble advantage for interval width and near-zero fidelity of the entropy interval in binary classification. Additional validation uses bootstrap MLPs and a resampling oracle on real drill-hole geochemistry.

Critical strengths

The paper contributes an oracle-referenced evaluation protocol for EU, comparing uncertainty measures of an identical ensemble within each simulated world. The experiments in Sections 6.1–6.2 (Figure 3) show that the credal interval width recovers the epistemic ordering better than BMA when members are few, while the entropy interval has near-zero fidelity to the disagreement oracle in the binary classification experiments.

Critical weaknesses
  1. Wang et al. (2025a) already introduced the post-hoc credal wrapper and studied its performance with limited numbers of predictive samples. The main addition here is an oracle-referenced empirical comparison, but its conclusions are constrained by one family of synthetic worlds and a limited set of member models. Although the authors discuss a location effect and suggest an explanation involving the plug-in mutual-information estimate, these explanations are not sufficiently developed or validated. Further analysis is needed to explain how the credal advantage changes with the number of members.

  2. Mathematical issues in relating the two EU estimands. The claim in Section 3.2 that reducible error and estimator spread agree wherever bias is small does not hold in general under the stated definitions. The specific concern is detailed in Questions for Authors below.

Questions for authors
  1. What is the main contribution beyond Wang et al. (2025a), and do the findings generalize to other data-generating processes and member models? A standalone paragraph addressing these points would be helpful.

  2. Could the authors provide a clearer heuristic explanation, supported by analysis or experiments, for how the credal advantage changes with the number of members?

  3. Could the authors clarify what is meant by “agree” in Section 3.2? If this refers to agreement in rankings, small bias alone is insufficient. Consider two binary classification query points. At the first point, refit predictions are equally likely to be (0.45, 0.55) or (0.55, 0.45); at the second, they are equally likely to be (0.01, 0.99) or (0.09, 0.91). Set the true probability vectors to their respective means, (0.50, 0.50) and (0.05, 0.95), so both points have exactly zero bias. The rankings can be shown to be reversed despite zero bias. What additional conditions would support the claimed agreement?

  4. Could the authors discuss scalability to more complex member models and possible ways to accelerate the computation? No code repository is linked in the manuscript, and the supplementary code mentioned in the reproducibility statement could not be located. Could the authors provide anonymized code and instructions for reproducing the experiments during the rebuttal period?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ebtL

We thank the reviewer for the detailed report, and in particular for the counterexample and the reference checks. Both identified real errors, which we have corrected.

Ethics flag, references. Both entries were wrong. The first combined the author list of one paper with the title of another. The title belongs to Caté, Schetselaar, Mercier-Langevin and Ross (2018), Journal of Geochemical Exploration 188, 216-228. The authors we listed wrote "Machine learning predictive models for mineral prospectivity" (Ore Geology Reviews 71, 2015). We now cite each correctly, where it supports the sentence. In the second entry the first author is Dazheng Huang. We then checked all 43 entries against publisher or index records. We also reworded two sentences whose citations did not support them as written.

Q3 and W2, Section 3.2. The reviewer is right. The claim is withdrawn, together with the sentence saying the difference in scale does not matter under ranking. Definition 1 is expected squared distance in probability space; Definition 2 is mutual information. With zero bias, reducible error equals the variance of the refits exactly, but mutual information is a different function of the same refits. In the reviewer's example the reducible errors are 0.0050 and 0.0032 and the spreads are 0.0072 and 0.0278 bits, so the rankings reverse. The revised section states:

  1. Reducible error equals squared bias plus the variance-form spread (expected squared distance of a refit from the mean refit), exactly and pointwise. The two rank locations identically at zero bias.
  2. For two classes, mutual-information spread is, to second order, the variance-form spread divided by 4ln⁡2 qˉ(1−qˉ)4 \ln 2 \, \bar q (1 - \bar q), where qˉ\bar q is the mean refit probability. Its ranking matches squared-loss reducible error only if qˉ(1−qˉ)\bar q (1 - \bar q) is also similar across locations. The reviewer's example violates this (0.25 against 0.0475).
  3. Mutual-information spread is matched instead, to second order and at zero bias, with reducible error under log loss (0.0072 and 0.0381 bits in the reviewer's example).

Bias is also not small in our worlds. Across 40 worlds squared bias is about 83% of reducible error, and variance-form spread and reducible error have mean rank correlation 0.39 at n = 150 and 0.54 at n = 300. The 0.761 in the submission came from one world. We now report all measures against both forms of spread and both forms of reducible error. The two forms of spread rank locations almost identically (0.96), so the small-M results are unchanged by this choice, while the large-M comparison is not (see Q2).

Q1 and W1, contribution and generality. Wang et al. (2025a) introduce the wrapper and evaluate it by OOD detection and calibration on image benchmarks. We do not propose a method. We add (i) references computed by refitting, against which any uncertainty measure from a finite ensemble can be graded, (ii) the conditions under which the reported small-ensemble advantage holds, and (iii) a measured account of how two readings of the credal set behave as the number of classes changes. The small-M gap holds against four targets and for label sharpness β = 2.5, 5 and 10, but not at β = 1, where labels are noisiest (+0.012, P = 0.23 at M = 2). With MLP members the gap has the same sign at every M, but among M ≤ 5 it is significant only at M = 2, and it grows at large M (Table 7). We have not tested deep image models and say so.

Q2, why the advantage changes with M. To second order the plug-in mutual information is the variance of the member probabilities divided by a location term pˉ(1−pˉ)\bar p (1 - \bar p). With two to five members both parts are noisy, and the ratio is unstable where the mean prediction is near 0 or 1. The width has no location term. Three results support this. The standard deviation of the member probabilities, which also has no location term, matches the width at M ≤ 5. Other information-scale measures (mean pairwise KL, jackknife-corrected mutual information) match BMA, not the width. Against a variance-form oracle the reversal at large M disappears (+0.011 at M = 40, P = 0.35). The reversal in the submission arose because the oracle shared BMA's functional, and because the range uses only the two most extreme members.

Q4, scalability and code. Each measure is a function of the M member predictions and costs nothing beyond the ensemble (Table 10). The cost is in the oracle, which needs K refits per world and is used only in simulation: 51,200 refits in under 30 CPU-minutes for the full programme. For larger members it could be reduced by warm-starting or refitting only the last layer, which we have not tested. We cannot release anonymised code during review, and have corrected the reproducibility statement, which wrongly promised it. Algorithm 1 and Appendix C give every step and setting, and Appendix L was reproduced from them alone.

Reviewer_9MQ9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper aims to separate reducible prediction error from variation across refitted models. It includes a comparison between BMA mutual information, credal interval width, and credal entropy intervals using the same ensemble. The latter is shown to behave poorly on binary classification problems while credal interval width appear to perform well for small ensembles. The consider application is based on geological data, where the sparsity of the information is often pronounced.

Critical strengths
  • The empirical findings in the paper are convincing since the comparison appears to be fair (paired statistics) and suitably exhaustive (ensembles, label sets, and training sets of varying size).
  • The emphasis on the separation between reducible prediction error and variation across refitted models is useful and insightful
Critical weaknesses
  • The suggested novelty against the literature needs to be better justified: [R1] and [R2] below already include aspects in this paper: [R1] discusses some of the limitations with the credal entropy intervals in the binary classification case and [R2] uses repeated training-data sampling to assess uncertainty estimation in simulations.
  • The oracle and estimator do not use the same sampling scheme: the ensemble bootstraps input-label pairs, whereas the disagreement oracle redraws labels at fixed inputs.

[R1] Hüllermeier, E., Destercke, S., & Shaker, M. H. (2022, August). Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison. In Uncertainty in Artificial Intelligence (pp. 548-557). PMLR. [R2] Juergens, M., Meinert, N., Bengs, V., Hüllermeier, E., & Waegeman, W. (2024). Is epistemic uncertainty faithfully represented by evidential deep learning methods?. arXiv preprint arXiv:2402.09056.

Questions for authors
  1. Can you evaluate all measures directly against the reducible-error oracle across multiple worlds?
  2. How does interval width compare with plain ensemble variance? They should give similar results for binary classification.
  3. Is it possible to evaluate whether the observed gains help with active learning / sequential experimental design?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9MQ9

We thank the reviewer for a careful reading and for the two references, which we should have discussed. We ran the requested simulation experiments with a second, independent implementation of the protocol (new Appendix L). The results confirm the small-ensemble finding and change how we interpret it.

W1, novelty against [R1] and [R2]. We agree that both overlap with our paper, and we have revised the related work and the contribution list. [R1] shows analytically that upper minus lower entropy on a binary interval depends on where the interval sits, and uses the interval width as the epistemic part. Our binary result is consistent with theirs and we no longer present it as new. What we add is a measurement against a computed reference, where [R1] relies on accuracy-rejection curves and notes that no reference uncertainty is available, and the trend with the number of classes. [R2] builds a reference by retraining on resampled training sets and grades evidential methods against it. Our oracle follows the same logic and we now say so. We differ in the object of study: measures computed from one finite ensemble, the number of members, and two targets (reducible error and estimator spread) kept apart.

W2, the oracle and the ensemble use different sampling schemes. Correct. We added a random-design oracle that redraws locations as well as labels, which matches the pairs bootstrap. The finding is unchanged: the paired gap between width and BMA is +0.051, +0.047 and +0.038 at M = 2, 3, 5 (all P < 0.001) against +0.053, +0.055 and +0.045 under the fixed design, and the two oracles have rank correlation 0.84.

Q1, all measures against the reducible-error oracle across worlds. Done over 40 worlds, with reducible error computed at the same design as the ensemble. Two results. First, the width still leads BMA: +0.041, +0.045, +0.051, +0.041 and +0.036 at M = 2, 3, 5, 10, 20 (all P < 0.01), and +0.019 at M = 40 (P = 0.17), so there is no reversal against this target. Second, all disagreement-based measures track reducible error much less well than they track spread: at M = 5 the rank correlations are 0.21 for BMA and 0.26 for the width, against 0.48 and 0.52 for spread. The reason is that squared bias makes up about 83% of reducible error in these worlds, and no measure built from member disagreement can see bias. Across worlds the spread and reducible-error oracles have mean rank correlation 0.39 at n = 150 and 0.54 at n = 300. The value of 0.761 in the submission came from a single world and we have replaced it with these.

Q2, interval width against ensemble variance. The reviewer is right. At M = 2 the width is exactly twice the standard deviation of the member probabilities, so the two rank locations identically. At M = 3 and 5 the paired gap between them is +0.000 and -0.003 (P = 0.61 and 0.47). From M = 20 the standard deviation is ahead of the width (width minus standard deviation of -0.017 at M = 20 and -0.027 at M = 40, P < 0.01), and at the default setting it is never significantly behind BMA. Mean pairwise KL and jackknife-corrected mutual information behave like BMA to within 0.001. We therefore describe the result as an advantage of probability-scale dispersion over mutual information at small M. The credal interval width is one such measure, and it has no advantage over the standard deviation. We have revised the abstract and title to say this.

Q3, active learning. We have not tested this and make no claim about it. A gap of about 0.05 in rank correlation may or may not change which location is selected, and the low fidelity to reducible error reported under Q1 is a reason for caution. We list it as future work.

I would like to thank the authors for taking my comments into account seriously. The application to active learning was more an open-ended question and it is fine to keep it as future work. For the other points, the clarifications are welcomed but at the same time, the novelty is somewhat reduced as a consequence. Therefore I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.