- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes JUCAL, a post-calibration uncertainty quantification method for classifier ensembles that uses two scalar parameters to separately rescale aleatoric and epistemic uncertainty. The method reports better NLL, uncertainty ranking, and predictive set size across multiple benchmarks, while reducing the number of ensemble members needed for comparable performance.
Critical strengths
- The paper addresses an important research problem, as existing temperature scaling methods do not balance the proportion of aleatoric and epistemic uncertainty.
- The method is generally easy to follow and implement, and is computationally lightweight, as it only requires fitting two scaler parameters.
- The experiments cover multiple tasks, datasets, ensemble sizes, pretrained networks, etc.
- The method performs better than the selected baselines.
Critical weaknesses
- Although the paper describes c1 and c2 as separately calibrating aleatoric and epistemic uncertainty, the two operations are still dependent to each other, as changing either one affects both. Is that assuming the total uncertainty is fixed, and therefore tuning one uncertainty can affect the other? If the total uncertainty is fixed, only one variable should be needed to tune aleatoric and epistemic uncertainty. The claimed disentanglement seems questionable.
- Comparison to more uncertainty quantification methods are desirable. More quantitative results as tables should be shown in the main text.
- More direct evidence for improved aleatoric/epistemic separation should be shown. Quantitative evaluations of uncertainty calibration, OOD detection, and statistical significance across multiple trainings would all strengthen the claims.
Questions for authors
Please refer to the aforementioned weaknesses.
3 comments
Thank you for your review! part 2
@W3: Our answer regarding EU/AU is in @W1. Our main objective is to improve the total predictive uncertainty by selectively increasing or decreasing it, either in regions where AU dominates or in regions where EU dominates. If we are underconfident in regions where epistemic uncertainty dominates, we want to compensate for this by increasing the epistemic uncertainty globally to increase the total predictive uncertainty in these regions. It is not our goal to change the relative ordering of where EU should be larger and where it should be smaller. For OOD detection only this relative ranking of EU matters, not the overall average global magnitude of EU.
Instead of OOD detection experiments we’ve added experiments to evaluate the total predictive uncertainty on OOD data after distribution shifts as suggested by Reviewer iZoE. While our original submission has provided strong empirical evidence for improved total predictive uncertainty on out-of-sample OOS data, we had only synthetic examples for improvements on OOD data. Therefore, we have run additional OOD experiments on real-world data, which just finished. We evaluated JUCAL on PlantNet v2 under one geographic shift and two temporal shifts. For the geographic split, we used 75% of the French observations for training, 25% for validation, and all non-French observations for testing. For the first temporal split, we ordered observations by date and used the earliest 60% for training, the next 20% for validation, and the latest 20% for testing. For the second, we split the pre-2020 observations chronologically into training and validation sets and tested on observations from January 1, 2020 onward. In each case, we fitted the calibration parameters on the validation set before evaluating on the shifted test set. On all three shifts, JUCAL achieved the lowest test NLL, Brier score, and AORAC among the five ensemble methods, although the AORAC gains were small and top-1 ECE was higher than for the uncalibrated ensemble. Table 1. PlantNet v2 test NLL / top-1 ECE (%). Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.
| Method | France → elsewhere | 60/20/20 chronological | Test from Jan. 1, 2020 |
|---|---|---|---|
| Uncalibrated | 1.970 / 3.39 | 1.579 / 3.04 | 1.729 / 2.61 |
| Calibrate then pool | 1.977 / 5.19 | 1.575 / 6.00 | 1.714 / 5.08 |
| Pool then calibrate | 1.966 / 2.02 | 1.574 / 3.38 | 1.731 / 2.65 |
| Calibrate–pool–calibrate | 1.926 / 3.95 | 1.547 / 2.86 | 1.693 / 6.34 |
| JUCAL | 1.925 / 3.65 | 1.525 / 4.12 | 1.690 / 5.40 |
Table 2. PlantNet v2 test Brier score / AORAC. AORAC is one minus the area under the rejection–accuracy curve. Lower is better for both metrics. Bold indicates the lowest value for each metric on each split.
| Method | France → elsewhere | 60/20/20 chronological | Test from Jan. 1, 2020 |
|---|---|---|---|
| Uncalibrated | 0.5652 / 0.1858 | 0.4605 / 0.1249 | 0.4877 / 0.1456 |
| Calibrate then pool | 0.5684 / 0.1869 | 0.4657 / 0.1264 | 0.4908 / 0.1469 |
| Pool then calibrate | 0.5633 / 0.1854 | 0.4612 / 0.1253 | 0.4877 / 0.1455 |
| Calibrate–pool–calibrate | 0.5591 / 0.1836 | 0.4596 / 0.1254 | 0.4853 / 0.1434 |
| JUCAL | 0.5587 / 0.1833 | 0.4534 / 0.1221 | 0.4837 / 0.1432 |
Thank all the authors for their rebuttal. My previous concerns are mostly addressed. I will change my score to 3.
Thank you for your review!
Thank you for your review and for acknowledging the importance of this research problem!
@W1. We don’t fix the total uncertainty nor do we assume it is fixed; instead, we actively want to be able to rescale it. Increasing both c_1 and c_2 can increase the overall uncertainty. If both the aleatoric and the epistemic part are overconfident it can be desirable to increase both of them to become less overconfident. Additionally we can also change the ratio of the two types of uncertainty. Figure 14 in the appendix visualizes how c_1 and c_2 together can both adjust the aleatoric, the epistemic and the total uncertainty. (It’s a bit like a shower where you have one knob for cold water c_1 and another for the hot water c_2, which allows you to control both the total amount of water pressure and the ratio of hot and cold water.)
The following addresses W1 and part of W3: We do not claim to have strong evidence for providing better estimates of the epistemic uncertainty (EU) and the aleatoric uncertainty (AU). Our goal is to obtain better calibrated total predictive distributions that are both well calibrated in regions where epistemic uncertainty dominates and in regions where aleatoric uncertainty dominates to finally obtain better NLL on the unseen test dataset. The idea is that increasing c_2 increases the epistemic uncertainty and hence mainly increases the total uncertainty in regions where epistemic uncertainty dominates without increasing it in regions where epistemic uncertainty is zero. Figure 7 hints that our AU-EU decomposition could make sense: with more training data, estimated EU decreased on all six text datasets for JUCAL Greedy-50, whereas estimated AU showed no consistent decrease. However, we agree that this is not sufficient evidence to reliably claim that we improve estimations of epistemic uncertainty. We have now explicitly added the sentence “While we have strong empirical evidence that JUCAL improves the total predictive uncertainty for sufficiently large calibration datasets, further work is needed to establish whether JUCAL improves the estimation of epistemic and aleatoric uncertainty. “ in to the limitations section in the conclusion of the paper.
@W2: We’ll think about how to better present the results in the camera-ready version to please both readers who prefer plots and those who prefer tables.
As a first step, we have added a reference to Tables 2-9 in the caption to Figure 4.
Across six text datasets, two training-set sizes, and four metrics, a JUCAL variant achieved the best mean score in 46 of 48 settings and significantly outperformed all four non-JUCAL baselines in 29 of 48 settings. Since all methods operate on model outputs and retraining the language models would be prohibitively expensive, we kept the trained models fixed across five random validation/test splits. On each split, we fitted the calibrators on the validation set and evaluated all methods on the same test set. Tables 2–9 shade methods whose scores did not differ significantly from the best, based on paired tests of the five within-split score differences (=0.05).
The PlantNet tables below provide additional quantitative results and a comparison with a new competitor: calibrate–pool–calibrate.