acceptodds
Under review as a conference paper at ICLR 2027

JUCAL: Jointly Calibrating Aleatoric and Epistemic Uncertainty in Classification Tasks

Abstract

We study post-calibration uncertainty for trained ensembles of classifiers. Specifically, we consider both aleatoric (label noise) and epistemic (model) uncertainty. Among the most popular and widely used calibration methods in classification are temperature scaling (i.e., pool-then-calibrate) and conformal methods. However, the main shortcoming of these calibration methods is that they do not balance the proportion of aleatoric and epistemic uncertainty. Not balancing these uncertainties can severely misrepresent predictive uncertainty, leading to overconfident predictions in some input regions while being underconfident in others. To address this shortcoming, we present a simple but powerful calibration algorithm Joint Uncertainty Calibration (JUCAL) that jointly calibrates aleatoric and epistemic uncertainty. JUCAL jointly calibrates two constants to weight and scale epistemic and aleatoric uncertainties by optimizing the negative log-likelihood (NLL) on the validation/calibration dataset. JUCAL can be applied to any trained ensemble of classifiers (e.g., transformers, CNNs, or tree-based methods), with minimal computational overhead, without requiring access to the models' internal parameters. We experimentally evaluate JUCAL on various text classification tasks, for ensembles of varying sizes and with different ensembling strategies. Our experiments show that JUCAL significantly outperforms SOTA calibration methods across the majority of considered classification tasks, reducing NLL and predictive set size by up to 15% and 20%, respectively. Interestingly, even applying JUCAL to an ensemble of size 5 can outperform temperature-scaled ensembles of size up to 50 in terms of NLL and predictive set size, resulting in up to 6 times smaller inference costs. Thus, we propose JUCAL as a new go-to method for calibrating ensembles in classification.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Quantification, Epistemic Uncertainty, Aleatoric Uncertainty, Calibration, Classification, Deep Ensembles, predictive distribution, predictive sets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w9pJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes JUCAL, a post-calibration uncertainty quantification method for classifier ensembles that uses two scalar parameters to separately rescale aleatoric and epistemic uncertainty. The method reports better NLL, uncertainty ranking, and predictive set size across multiple benchmarks, while reducing the number of ensemble members needed for comparable performance.

Critical strengths
  • The paper addresses an important research problem, as existing temperature scaling methods do not balance the proportion of aleatoric and epistemic uncertainty.
  • The method is generally easy to follow and implement, and is computationally lightweight, as it only requires fitting two scaler parameters.
  • The experiments cover multiple tasks, datasets, ensemble sizes, pretrained networks, etc.
  • The method performs better than the selected baselines.
Critical weaknesses
  • Although the paper describes c1 and c2 as separately calibrating aleatoric and epistemic uncertainty, the two operations are still dependent to each other, as changing either one affects both. Is that assuming the total uncertainty is fixed, and therefore tuning one uncertainty can affect the other? If the total uncertainty is fixed, only one variable should be needed to tune aleatoric and epistemic uncertainty. The claimed disentanglement seems questionable.
  • Comparison to more uncertainty quantification methods are desirable. More quantitative results as tables should be shown in the main text.
  • More direct evidence for improved aleatoric/epistemic separation should be shown. Quantitative evaluations of uncertainty calibration, OOD detection, and statistical significance across multiple trainings would all strengthen the claims.
Questions for authors

Please refer to the aforementioned weaknesses.

3 comments
AuthorsOpenReview ↗

Thank you for your review!

Thank you for your review and for acknowledging the importance of this research problem!

@W1. We don’t fix the total uncertainty nor do we assume it is fixed; instead, we actively want to be able to rescale it. Increasing both c_1 and c_2 can increase the overall uncertainty. If both the aleatoric and the epistemic part are overconfident it can be desirable to increase both of them to become less overconfident. Additionally we can also change the ratio of the two types of uncertainty. Figure 14 in the appendix visualizes how c_1 and c_2 together can both adjust the aleatoric, the epistemic and the total uncertainty. (It’s a bit like a shower where you have one knob for cold water c_1 and another for the hot water c_2, which allows you to control both the total amount of water pressure and the ratio of hot and cold water.)

The following addresses W1 and part of W3: We do not claim to have strong evidence for providing better estimates of the epistemic uncertainty (EU) and the aleatoric uncertainty (AU). Our goal is to obtain better calibrated total predictive distributions that are both well calibrated in regions where epistemic uncertainty dominates and in regions where aleatoric uncertainty dominates to finally obtain better NLL on the unseen test dataset. The idea is that increasing c_2 increases the epistemic uncertainty and hence mainly increases the total uncertainty in regions where epistemic uncertainty dominates without increasing it in regions where epistemic uncertainty is zero. Figure 7 hints that our AU-EU decomposition could make sense: with more training data, estimated EU decreased on all six text datasets for JUCAL Greedy-50, whereas estimated AU showed no consistent decrease. However, we agree that this is not sufficient evidence to reliably claim that we improve estimations of epistemic uncertainty. We have now explicitly added the sentence “While we have strong empirical evidence that JUCAL improves the total predictive uncertainty for sufficiently large calibration datasets, further work is needed to establish whether JUCAL improves the estimation of epistemic and aleatoric uncertainty. “ in to the limitations section in the conclusion of the paper.

@W2: We’ll think about how to better present the results in the camera-ready version to please both readers who prefer plots and those who prefer tables.

As a first step, we have added a reference to Tables 2-9 in the caption to Figure 4.

Across six text datasets, two training-set sizes, and four metrics, a JUCAL variant achieved the best mean score in 46 of 48 settings and significantly outperformed all four non-JUCAL baselines in 29 of 48 settings. Since all methods operate on model outputs and retraining the language models would be prohibitively expensive, we kept the trained models fixed across five random validation/test splits. On each split, we fitted the calibrators on the validation set and evaluated all methods on the same test set. Tables 2–9 shade methods whose scores did not differ significantly from the best, based on paired tests of the five within-split score differences (=0.05).

The PlantNet tables below provide additional quantitative results and a comparison with a new competitor: calibrate–pool–calibrate.

AuthorsOpenReview ↗

Thank you for your review! part 2

@W3: Our answer regarding EU/AU is in @W1. Our main objective is to improve the total predictive uncertainty by selectively increasing or decreasing it, either in regions where AU dominates or in regions where EU dominates. If we are underconfident in regions where epistemic uncertainty dominates, we want to compensate for this by increasing the epistemic uncertainty globally to increase the total predictive uncertainty in these regions. It is not our goal to change the relative ordering of where EU should be larger and where it should be smaller. For OOD detection only this relative ranking of EU matters, not the overall average global magnitude of EU.

Instead of OOD detection experiments we’ve added experiments to evaluate the total predictive uncertainty on OOD data after distribution shifts as suggested by Reviewer iZoE. While our original submission has provided strong empirical evidence for improved total predictive uncertainty on out-of-sample OOS data, we had only synthetic examples for improvements on OOD data. Therefore, we have run additional OOD experiments on real-world data, which just finished. We evaluated JUCAL on PlantNet v2 under one geographic shift and two temporal shifts. For the geographic split, we used 75% of the French observations for training, 25% for validation, and all non-French observations for testing. For the first temporal split, we ordered observations by date and used the earliest 60% for training, the next 20% for validation, and the latest 20% for testing. For the second, we split the pre-2020 observations chronologically into training and validation sets and tested on observations from January 1, 2020 onward. In each case, we fitted the calibration parameters on the validation set before evaluating on the shifted test set. On all three shifts, JUCAL achieved the lowest test NLL, Brier score, and AORAC among the five ensemble methods, although the AORAC gains were small and top-1 ECE was higher than for the uncalibrated ensemble. Table 1. PlantNet v2 test NLL / top-1 ECE (%). Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.

MethodFrance → elsewhere60/20/20 chronologicalTest from Jan. 1, 2020
Uncalibrated1.970 / 3.391.579 / 3.041.729 / 2.61
Calibrate then pool1.977 / 5.191.575 / 6.001.714 / 5.08
Pool then calibrate1.966 / 2.021.574 / 3.381.731 / 2.65
Calibrate–pool–calibrate1.926 / 3.951.547 / 2.861.693 / 6.34
JUCAL1.925 / 3.651.525 / 4.121.690 / 5.40

Table 2. PlantNet v2 test Brier score / AORAC. AORAC is one minus the area under the rejection–accuracy curve. Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.

MethodFrance → elsewhere60/20/20 chronologicalTest from Jan. 1, 2020
Uncalibrated0.5652 / 0.18580.4605 / 0.12490.4877 / 0.1456
Calibrate then pool0.5684 / 0.18690.4657 / 0.12640.4908 / 0.1469
Pool then calibrate0.5633 / 0.18540.4612 / 0.12530.4877 / 0.1455
Calibrate–pool–calibrate0.5591 / 0.18360.4596 / 0.12540.4853 / 0.1434
JUCAL0.5587 / 0.18330.4534 / 0.12210.4837 / 0.1432

Thank all the authors for their rebuttal. My previous concerns are mostly addressed. I will change my score to 3.

Reviewer_LHy1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JUCAL, a post-hoc calibration method for classifier ensembles that aims to account for both aleatoric and epistemic uncertainty. The method combines shared temperature scaling of individual ensemble members with a second parameter that rescales their logits around the ensemble mean. Both parameters are jointly fitted by minimizing the negative log-likelihood of the averaged predictive distribution on a calibration set, without retraining the underlying models. Experiments on text, image, and tabular classification tasks compare JUCAL with uncalibrated ensembles and temperature-scaling baselines. The authors report improvements in predictive NLL, uncertainty ranking, and prediction-set efficiency, with smaller calibrated ensembles sometimes matching or outperforming larger temperature-scaled ensembles. Additional experiments examine how the estimated uncertainty components respond to changes in training sample size and available features.

Critical strengths

(1) Clear and intuitive motivation for exploiting ensemble structure. Figure 2 provides an accessible geometric explanation of the distinction between individually uncertain predictions and confident but disagreeing ensemble members. Together with the illustrative examples, it explains why averaged predictions can conceal useful information about uncertainty. This provides a compelling motivation for jointly adjusting member confidence and ensemble diversity.

(2) A lightweight method with practical empirical benefits. JUCAL fits only two global parameters to existing ensemble predictions, without retraining the underlying classifiers. Its simplicity and compatibility with different architectures make it straightforward to implement. Experiments spanning text, image, and tabular classification report improvements in predictive NLL, uncertainty ranking, and prediction-set efficiency. Smaller calibrated ensembles can sometimes match or outperform larger temperature-scaled ensembles, supporting the method’s practical usefulness.

Critical weaknesses

(1) The evidence does not directly establish improved calibration of both uncertainty components. The NLL objective evaluates the averaged predictive distribution, whereas the central interpretation concerns the aleatoric and epistemic components individually. Different distributions of ensemble predictions can have identical means and predictive NLL while yielding substantially different uncertainty decompositions. The sample-size and feature-count experiments provide useful qualitative consistency checks, but do not directly demonstrate more accurate estimation of each component after calibration. A clearer operational definition and more direct validation would strengthen this interpretation.

(2) The comparisons do not sufficiently isolate the benefit of the proposed structure. The calibration baselines primarily consist of temperature-scaling variants. A matched two-parameter baseline applying temperature scaling before and after pooling would help determine the distinctive benefit of rescaling ensemble disagreement. Comparison with a representative richer calibrator, such as Dirichlet calibration or Parameterized Temperature Scaling, would further contextualize the predictive improvements. These missing comparisons limit support for the broad state-of-the-art claims and leave uncertainty about the specific contribution of the proposed parameterization.

Questions for authors

(1) Definition and evidence for component-wise calibration. What precisely constitutes improved calibration of the aleatoric and epistemic components individually? Since predictive NLL depends only on the averaged predictive distribution, what assumptions or component-specific evidence support interpreting its improvement as better calibration of both components? Do the existing experiments demonstrate improved estimation after calibration, beyond qualitative responses to sample size and feature availability?

(2) Controlled comparisons supporting the proposed structure. Can you provide a controlled comparison with a two-parameter calibrate–pool–calibrate baseline, using a shared temperature before pooling and another afterward, under the same calibration-data budget? Comparison with one representative richer calibrator would also help assess the distinctive benefit of explicitly rescaling ensemble disagreement. These comparisons would materially affect my assessment of the methodological contribution and the broad performance claims.

(3) Effects on classification accuracy. Appendix J.1 states that predictive accuracy was evaluated, with results provided in the supplementary material. Could you point to these results and summarize accuracy changes before and after calibration across the main benchmarks, particularly whether improved NLL is accompanied by any systematic accuracy reductions? This would help assess the practical trade-off when applying JUCAL to an existing classifier.

2 comments
AuthorsOpenReview ↗

Thank you for your review!

Thank you for your detailed review! @W1: We fully agree that two different ensembles can have 2 very different AU-EU decompositions can lead to the same predictive distribution and hence the same NLL. We do not claim to have strong evidence for providing better estimates of the epistemic uncertainty (EU) and the aleatoric uncertainty (AU). Our main goal is to obtain better calibrated total predictive distributions that are both well calibrated in regions where epistemic uncertainty dominates and in regions where aleatoric uncertainty dominates to finally obtain a better test NLL. Figure 7 hints that our AU-EU decomposition could make sense: with more training data, estimated EU decreased on all six text datasets for JUCAL Greedy-50, whereas estimated AU showed no consistent decrease. However, we agree that this is not sufficient evidence to reliably claim that we improve estimations of EU. We have now explicitly added the sentence “While we have strong empirical evidence that JUCAL improves the total predictive uncertainty for sufficiently large calibration datasets, further work is needed to establish whether JUCAL improves the estimation of epistemic and aleatoric uncertainty. “ in to the limitations section in the conclusion of the paper.

W2: We very much appreciate your very exciting novel idea to introduce a new two-parameter calibrate–pool–calibrate baseline. We think that this idea has not been explored in the literature so far. We think that this idea of calibrate–pool–calibrate is a bit similar to JUCAL as the first calibration constant only adjusts the aleatoric uncertainty while the 2nd one adjusts the total uncertainty. [Sidenote: JUCAL is a bit like a shower where you have one knob (c_1) to adjust the cold water and another one (c_2) to adjust the hot water, while calibrate–pool–calibrate is bit like a shower where you can move a leverage in one axis to control the total amount of water pressure and another axis to adjust the amount of cold water. In both chases you can both adjust the total amount and the ratio of the two components but via a different coordinate system. For the showers this would be completely equivalent, but for uncertainty there are some subtle differences: e.g., JUCAL always outputs a valid ensemble which can be decomposed into EU and AU, and there are further subtle differences.]

In the new experiments on distribution shifts suggested by Reviewer iZoE, we have also implemented your proposed new calibrate–pool–calibrate method. We can see that calibrate–pool–calibrate using the same calibration data is usually the 2nd best method closely after JUCAL. JUCAL is not ranked best on top-1 ECE on PlantNet v2, but ECE is not a proper scoring rule and only assesses top-class confidence (see Appendix F.2), whereas JUCAL achieves the best NLL and Brier score, both strictly proper, on all three splits:

Table 1. PlantNet v2 test NLL / top-1 ECE (%). Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.

MethodFrance → elsewhere60/20/20 chronologicalTest from Jan. 1, 2020
Uncalibrated1.970 / 3.391.579 / 3.041.729 / 2.61
Calibrate then pool1.977 / 5.191.575 / 6.001.714 / 5.08
Pool then calibrate1.966 / 2.021.574 / 3.381.731 / 2.65
Calibrate–pool–calibrate1.926 / 3.951.547 / 2.861.693 / 6.34
JUCAL1.925 / 3.651.525 / 4.121.690 / 5.40

Table 2. PlantNet v2 test Brier score / AORAC. AORAC is one minus the area under the rejection–accuracy curve. Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.

MethodFrance → elsewhere60/20/20 chronologicalTest from Jan. 1, 2020
Uncalibrated0.5652 / 0.18580.4605 / 0.12490.4877 / 0.1456
Calibrate then pool0.5684 / 0.18690.4657 / 0.12640.4908 / 0.1469
Pool then calibrate0.5633 / 0.18540.4612 / 0.12530.4877 / 0.1455
Calibrate–pool–calibrate0.5591 / 0.18360.4596 / 0.12540.4853 / 0.1434
JUCAL0.5587 / 0.18330.4534 / 0.12210.4837 / 0.1432

Richer calibrators such as Dirichlet calibration (K² + K parameters for K classes, plus regularisation hyperparameters) or PTS (a neural network predicting a per-sample temperature, with its own architecture and training hyperparameters) fit far more parameters than JUCAL’s two, which increases the risk of overfitting the calibration set and requires additional tuning. JUCAL instead targets the setting in which temperature scaling remains the default in practice: a drop-in method that fits two scalars and requires no tuning. We therefore compare against temperature-scaling-based ensemble calibration, have restricted our SOTA claim accordingly in the revised paper, and leave a comparison with richer calibrators to future work.

@Q1: See our answer to W1.

@Q2: See our answer to W2.

AuthorsOpenReview ↗

Thank you for your review! part 2

@Q3: JUCAL improved mean test NLL in all 30 Greedy-ensemble settings, with no systematic loss in top-1 accuracy. Across the 24 text settings, accuracy improved in 17 and declined in 7, averaging 91.76% before calibration and 91.79% after. The largest decline was 0.04 percentage points (Tweet, full dataset, Greedy-50). Accuracy improved in all six image settings, from 88.34% to 88.49% on average. Per-setting misclassification rates are available in the supplementary text results and image results.

For the tabular experiments in Appendix J, JUCAL always improved accuracy over the uncalibrated ensembles when trained on all features and samples, and matched or improved on both temperature-scaling baselines (Appendix J.2). The per-dataset accuracy plots for Appendix J are not yet in the linked repository; we will add them together with the links above.

JUCAL also improved both accuracy and NLL on all three PlantNet v2 distribution shifts, with accuracy gains of 0.63–0.86 percentage points (table below).

PlantNet v2 test splitUncalibrated accuracy (%)JUCAL accuracy (%)Change (pp)Uncalibrated NLLJUCAL NLL
France → elsewhere56.7557.37+0.631.9701.925
60/20/20 chronological66.8567.71+0.861.5791.525
Test from Jan. 1, 202064.5165.13+0.631.7291.690

Accuracy changes are calculated before rounding.

Reviewer_iZoEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper's methodology is motivated by the calibration of both aleatoric uncertainty (AU) and epistemic uncertainty (EU) through two different operations with different effects on the ensemble logits that it uses as an input. The proposed method is reasonable and appears to be new. The provided intuition makes sense and the method performs better in the provided experiments under the considered metrics.

Critical strengths
  • Separately calibrating AU and EU in classification problems is appealing
  • The provided experimental results are good
  • The paper is accessible and well-written enough.
Critical weaknesses
  1. The proposed method is based on the NLL of the averaged prediction, but this does not guarantee that both AU and EU are calibrated (e.g. two ensembles with logit differences cc and −c-c will give a constant pˉ1J\bar{p}^J_1 when the entropy-based AU-EU decomposition is non-constant).
  2. The method is motivated and illustrated through OOD inputs, but the proposed NLL optimisation is in-distribution and this aspect is not evaluated in the experiments.
  3. The PCS framework seems to be largely over-emphasised.
  4. It seems important to better explain in the main text how the proposed method is different from CLEAR Azizi et al. (2025).
  5. The comparison with credal and conformal methods does not seem to be entirely fair: the fact that they might yield larger predictive sets is not necessarily a weakness when these sets have coverage guarantees (which do not follow from improving the NLL).

The fact that there is guarantee of calibration for AU and EU (W.1) means that the merit of the method relies entirely on experimental evidence, which is not currently aligned with the claims (W.2). I would increase my score if such evidence could be provided.

Questions for authors

What is the performance of the proposed method for OOD samples?

4 comments
AuthorsOpenReview ↗

Thank you for your review!

Thank you very much for your thoughtful review and the very constructive suggestions!

@1. We fully agree with your example: We fully agree that two different ensembles can have 2 very different AU-EU decompositions can lead to the same predictive distribution and hence the same NLL. We do not claim to provide better estimates of the EU and AU. Our goal is to obtain better calibrated predictive distributions that are both well calibrated in regions where epistemic uncertainty dominates and in regions where aleatoric uncertainty dominates. Figure 7 hints that our AU-EU decomposition could make sense: with more training data, estimated EU decreased on all six text datasets for JUCAL Greedy-50, whereas estimated AU showed no consistent decrease. However, we agree that this is not sufficient evidence to reliably claim that we improve estimations of epistemic uncertainty. We have now explicitly added the sentence “While we have strong empirical evidence that JUCAL improves the total predictive uncertainty for sufficiently large calibration datasets, further work is needed to establish whether JUCAL improves the estimation of epistemic and aleatoric uncertainty. “ in to the limitations section in the conclusion of the paper.

@2. Our method is as much motivated by out-of-sample OOS data as it is by out-of-distribution OOD data. We optimize the NLL on OOS data. In high dimensions almost all OOS datapoints are outside the convex hull of the training data. Some of them will be more similar to many training data points while others will be further away from the training data and its convex hull. Intuitively this should encourage JUCAL to neither be miscalibrated for those close data points nor for the further away OOS datapoints. If epistemic uncertainty is systematically underestimated, the NLL of further away OOS datapoints (and further away OOD datapoints) would suffer and vice versa for aleatoric uncertainty. We agree that while our original submission has provided strong empirical evidence for improved OOS performance, we had only synthetic examples for OOD improvements. Thank you for noticing this! Therefore, we have run additional OOD experiments on real-world data, which just finished.We evaluated JUCAL on PlantNet v2 under one geographic shift and two temporal shifts. For the geographic split, we used 75% of the French observations for training, 25% for validation, and all non-French observations for testing. For the first temporal split, we ordered observations by date and used the earliest 60% for training, the next 20% for validation, and the latest 20% for testing. For the second, we split the pre-2020 observations chronologically into training and validation sets and tested on observations from January 1, 2020 onward. In each case, we fitted the calibration parameters on the validation set before evaluating on the shifted test set. On all three shifts, JUCAL achieved the lowest test NLL, Brier score, and AORAC among the five ensemble methods, although the AORAC gains were small and top-1 ECE was higher than for the uncalibrated ensemble.

OOD Results

Table 1. PlantNet v2 test NLL / top-1 ECE (%). Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.

MethodFrance → elsewhere60/20/20 chronologicalTest from Jan. 1, 2020
Uncalibrated1.970 / 3.391.579 / 3.041.729 / 2.61
Calibrate then pool1.977 / 5.191.575 / 6.001.714 / 5.08
Pool then calibrate1.966 / 2.021.574 / 3.381.731 / 2.65
Calibrate–pool–calibrate1.926 / 3.951.547 / 2.861.693 / 6.34
JUCAL1.925 / 3.651.525 / 4.121.690 / 5.40

Table 2. PlantNet v2 test Brier score / AORAC. AORAC is one minus the area under the rejection–accuracy curve. Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.

MethodFrance → elsewhere60/20/20 chronologicalTest from Jan. 1, 2020
Uncalibrated0.5652 / 0.18580.4605 / 0.12490.4877 / 0.1456
Calibrate then pool0.5684 / 0.18690.4657 / 0.12640.4908 / 0.1469
Pool then calibrate0.5633 / 0.18540.4612 / 0.12530.4877 / 0.1455
Calibrate–pool–calibrate0.5591 / 0.18360.4596 / 0.12540.4853 / 0.1434
JUCAL0.5587 / 0.18330.4534 / 0.12210.4837 / 0.1432

@3. Many theoretical treatments of uncertainty take preprocessing and modeling choices as given. In practice, reasonable alternatives can yield substantially different predictions. PCS explicitly treats these judgment calls throughout the data science life cycle as sources of uncertainty by examining how results vary across those alternatives. For example, Tang et al. (2025) used PCS to reduce a prostate cancer test from 18 genes to 7, with a loss in predictive performance comparable to the variation across preprocessing choices. This perspective matters for JUCAL because ensemble disagreement depends on which choices vary across members. We therefore favor keeping this motivation brief in the main text, with the fuller discussion in Appendix C.1.

@4. CLEAR is only for regression, not for classification. CLEAR and JUCAL share the idea of fitting two parameters. CLEAR combines estimates of aleatoric and epistemic uncertainty to construct regression intervals and fits its parameters using pinball loss subject to a coverage constraint on the calibration data. JUCAL operates on classification logits: one parameter temperature-scales each member’s logits, and the other rescales their deviations from the ensemble mean. We then average the resulting class probabilities and fit both parameters by NLL to obtain a predictive distribution over classes. We are bringing this distinction from Appendix C.2 into the main text.

@5. We agree that set size should be interpreted alongside the coverage guarantee. When we compare against conformal methods we compare conformalized JUCAL Greedy-50 against conformalized Greedy-50, for example. Both of these pipelines have the same conformal guarantees for marginal coverage. Then conformalized JUCAL results in smaller predictive sets with the same conformal guarantees and basically the same test coverage (see Appendix F.3). Without JUCAL, conformal methods are usually overconfident in some regions while being under-confident in other regions which can result in unnecessarily large predictive sets on average. Therefore we consider this comparison fair. Or can you think about any even fairer comparison? We will acknowledge the more ambitious goal of credal sets to provide conditional coverage guarantees in the appendix. To clarify: Greedy-50 trains on some training dataset and then selects the ensemble based on a validation dataset. JUCAL then reuses the same validation dataset to adjust optimize c_1 and c_2. Then both conformalized JUCAL Greedy-50 and Conformalized Greedy-50 use a separate conformal calibration dataset. Then we evaluate them on a separate test dataset. We will link to Appendix F.3 more prominently in the main paper.

@Question regarding OOD: We care about OOS as much as we care about OOD. And we have strong empirical evidence for OOS. Conformalized JUCAL has the same theoretical conformal marginal coverage guarantee for OOS as other conformal methods. As other conformal methods, JUCAL has no theoretical guarantees on OOD data. As you suggested we have run experiments on OOD. Thank you for this suggestion! We think these additional experiments and the other clarifications based on your suggestions have further improved the paper.

Thank you for the detailed response and the additional PlantNet experiments.

I have two follow-up points before updating my score:

  1. Framing: You clarify that you do not claim improved estimates of aleatoric or epistemic uncertainty, which is reasonable. However, the title, abstract, and conclusion still state that JUCAL “jointly calibrates aleatoric and epistemic uncertainty.” Could you revise this framing to focus on calibrating the total predictive distribution?
  2. PlantNet: Only "France →\to elsewhere" keeps the validation data in-distribution. On this split, the NLL difference between JUCAL and calibrate–pool–calibrate is just 0.001. Could you also report variability across random seeds or provide confidence intervals, and briefly describe the ensemble setup?

I would be happy to raise my score if these points are satisfactorily addressed.

Thank you for your response!

@1. We will formulate this more precisely. The mechanism of our algorithm is still to tune two calibration constants for epistemic and aleatoric uncertainty, so we think our title is still justified. However, we will formulate more clearly that our ultimate goal is to improve the total predictive constant via these two constants. There are many well-established evaluation metrics to evaluate the total predictive uncertainty, which provide us with clear empirical evidence that our algorithm can indeed improve the total predictive uncertainty via these two constants. One possible explanation for this is that we are actually improving the calibration of EU and AU (underestimating EU would lead to overconfidence in different regions (far OOS) than underestimating AU would and adjusting the global magnitudes of both components could help to have both the right level of total uncertainty (TU) in the center of the distribution and far OOS). However, with our current experiments, we cannot rule out other explanations, as we don't have sufficient evidence to directly evaluate EU or AU. Therefore, we think future work is needed to evaluate whether the individual components are improved by JUCAL or not. (Our current recommendation for downstream applications where the distinction between AU and EU matters would be to compute both the original AU and EU and the versions adjusted by JUCAL. If they both lead to the same downstream decision, I would have more trust in this decision if they heavily disagree, I would feel some 3rd order uncertainty about this disentanglement, but this has to be studied more systematically in future work.)

@2. Thank you for this suggestion. We will add confidence bounds for the camera-ready version. Retraining these large models for multiple splits is quite expensive and not possible within the rebuttal period, unfortunately. We also want to note that calibrate–pool–calibrate is a novel method similar to JUCAL (which can to some also adjust both the magnitude of AU and the magnitude of total uncertainty separately, which indirectly also allows to calibrate EU), which was not explored in previous literature. JUCAL's improvements over the popular SOTA methods from the literature are much larger than JUCAL's improvement over calibrate–pool–calibrate. Also note that, for the temporal distribution shifts, the test data has a different distribution from the calibration data too (all the photos from the calibration set were taken before the photos from the test dataset). These temporal distribution shifts are the most common ones in practice, and is a very common to use temporal trian/val/test splits in these settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.