- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether high predictive accuracy implies trustworthy confidence in selective prediction systems. The authors introduce the positive likelihood ratio (LR+), adapted from diagnostic testing, to measure the relative retention of correct versus incorrect predictions at a fixed coverage level, arguing that conventional uncertainty metrics do not adequately capture reliability in the high-confidence region. The study evaluates 18 frozen vision and video foundation models from six families across five datasets, spanning image classification, video classification, action anticipation, and semantic segmentation. LR+ is compared against AUROC, FPR@95, ECE, and signed ECE. The main findings are: (1)on ImageNet-1K, DINOv3 ViT-H+ and SigLIP2 g-opt are accuracy-matched (88.4 vs 88.1) but differ ~2× in LR+ at 20% retention (24.0 vs 49.7); (2) AUROC and FPR@95 vary little across the 18 models and rank them differently from LR+; (3) the accuracy–LR+ relationship and the top-LR+ model change across tasks; (4) MC-dropout improves ECE but leaves median LR+ essentially unchanged. Overall, the paper argues that predictive accuracy alone is insufficient for assessing confidence reliability and advocates evaluating confidence at the operating point relevant to selective prediction.
Critical strengths
- Relevant and well-motivated research question. The deployment question is well motivated and clearly stated: in an abstaining system, the only predictions that matter are those above the action threshold, and a metric evaluated at that operating point is a reasonable and sensible goal.
- Broad empirical evaluation across modern foundation models. The empirical scope is broad for an uncertainty study of frozen backbones: six families spanning three pretraining paradigms, three scales, five datasets, four task types, with a uniform probe protocol and a shared hyperparameter search per task.
- Emphasis on operating-point-specific evaluation. The use of LR+ at fixed retention fractions provides an interpretable perspective on how confidence separates correct and incorrect predictions within the selected set. Unlike metrics aggregated across all thresholds, this analysis focuses on an operating region relevant to abstention. The retention curves further demonstrate that model comparisons may change depending on the selected coverage.
- Writing is clear. Limitations are stated honestly (single seed, probe-size confound, Diving48 size, label-noise sensitivity).
Critical weaknesses
-
LR+ at fixed coverage is a reparameterization of selective risk, not an independent quantity. The main methodological contribution is the adoption of LR+ at fixed coverage. However, LR+ can be expressed directly in terms of overall accuracy (a) and selective risk (r(q)): LR+(q) = [(1 − r(q)) / r(q)] · [(1 − a) / a]. Thus, at fixed coverage, LR+ provides no independent information beyond these two quantities. Importantly, a model may achieve lower selective risk yet have a lower LR+ due to differences in overall accuracy. While the authors discuss AURC, E-AURC, NAURC, and SAC (Sec. 2), they do not directly compare LR+ with selective risk at matched coverage or empirically evaluate its added value over existing selective prediction metrics. Consequently, it remains unclear what additional actionable insight LR+ provides, particularly for deployment decisions where the absolute error rate among accepted predictions is critical.
-
Insufficient evidence that conventional uncertainty metrics conceal meaningful differences. Comparing the range of AUROC (0.864–0.885) with that of LR+ (24.0–53.1) is not meaningful across metrics with different scales; a bounded metric near its ceiling is necessarily compressed, and a ratio is necessarily expanded. Table 6 shows that AUROC, FPR@95, and ECE all favor SigLIP2 with CIs excluding zero, so the conventional metrics agree in direction. The rank-disagreement example (DINOv3 ViT-L: AUROC 0.879 "among the highest", LR+ 24.0 "last") is given without AUROC CIs; with all 18 models spanning only 0.021 in AUROC, it is unclear how many of these ranks are statistically distinguishable. While LR+ may emphasize differences at a specific operating point, its additional discriminative value is not sufficiently demonstrated. A quantitative analysis of model-ranking disagreements, supported by paired statistical comparisons, would strengthen this claim.
-
Limited evaluation under deployment-relevant conditions. As acknowledged in Section 5, all results are on in-distribution data. Given the safety-critical motivation, evaluating the stability of the confidence threshold and of selective risk under at least one representative distribution shift would strengthen the practical claims.
-
Limited robustness analysis of probe training. Each probe configuration is trained once; bootstrap intervals cover evaluation-set variability only, not probe initialization or training stochasticity (Sections 3.5, D.3). Running 3–5 seeds for at least DINOv3 ViT-H+ and SigLIP2 g-opt on ImageNet-1K would be inexpensive and would establish whether the central finding is robust.
-
Figure 4 is unreadable at print size.
Questions for authors
- Added value of LR+. Since LR+ at fixed coverage can be expressed directly in terms of overall accuracy and selective risk, what additional information or practical benefit does it provide? Could the authors include direct comparisons with selective risk at matched coverage and with established metrics such as E-AURC, NAURC, or SAC?
- Agreement with conventional metrics. Table 6 shows that AUROC, FPR@95, and ECE all favor SigLIP2 g-opt over DINOv3 ViT-H+, consistent with LR+. How often does LR+ change model rankings relative to the conventional metrics, and are those disagreements supported by paired confidence intervals?
- Robustness to probe training. How sensitive are the reported LR+ values and rankings to probe initialization and training randomness? Could the authors provide multi-seed results for representative models, in particular the DINOv3 ViT-H+ vs. SigLIP2 g-opt comparison and the within-family scaling trends?
- Deployment-relevant thresholds. Thresholds are currently set from evaluation-set quantiles (Algorithm 1). How would the conclusions change if thresholds were selected on a separate calibration set and held fixed during evaluation? Do LR+ and the associated selective risk remain stable under distribution shift?
Thank you for the detailed response and additional experimental results. The robustness analyses and metric comparisons answered most of my questions. However, the practical benefits of LR+ over current metrics remain limited. Given the improvements, I will raise my score accordingly.