- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes post-hoc confidence estimation for frozen foundation models by adaptively combining decision-specific and context-conditioned probability estimates. The safety implementation uses a two-estimator mixture, while the non-safety experiments combine three experts. Across seven safety datasets, the method improves Brier score in all 14 primary comparisons and Macro balanced accuracy in 12. Additional evaluations cover other model families and prediction-correctness estimation on three non-safety benchmarks.
Critical strengths
-
Broad evidence of improved probability quality: Adaptive calibration reduces Brier score in all 14 primary safety comparisons, with 12 significant improvements after Holm correction. Additional results across Qwen3-VL, InternVL3.5, and three non-safety tasks strengthen the evidence that the approach is useful beyond one backbone or application.
-
Controls that meaningfully test the proposed explanation: The capacity-matched score-diagnostic, shuffled-context, and random-feature controls help distinguish useful context from additional estimator flexibility. Real visual context outperforms shuffled and random features in all six comparisons, with five significant improvements against each control. The authors also explicitly acknowledge weaker text results, making the scope of the evidence clearer.
Critical weaknesses
-
The incremental value of adaptive mixing needs clearer evidence. The framework builds on established contextual calibration and learned estimator combinations. Constant candidates are included, and non-safety experiments compare constant and adaptive gates, but the reported results do not clearly quantify the benefit of example-dependent mixing over the contextual estimator alone or a tuned constant mixture. Explicit comparisons would help isolate the contribution of the gate.
-
The added value of semantic text context is limited under the stronger control. In Table 20, real-context calibration beats the score-diagnostics-only gate in only 2/8 text comparisons, with no significant wins and a mean relative Brier reduction of −0.14%. Visual context has stronger support against shuffled and random features. Although the authors acknowledge this disparity, it limits how broadly the results support semantic conditioning as the source of improvement.
-
The evidence for improved within-context calibration remains incomplete. Brier score measures both calibration and resolution, so its improvement alone does not establish reduced subgroup miscalibration. Figure 2 already provides aggregate reliability curves against a calibrated baseline. However, Tables 22–23 evaluate a separate ECE-targeted attribute method against raw scores. Subgroup comparisons of the main adaptive estimator against calibrated decision-specific probabilities would more directly test the conditional-calibration claim.
Questions for authors
-
Could you report the selected contextual estimator alone, a tuned constant mixture, and the adaptive gate side by side? For the safety experiments, does constant-mixture selection include interior weights beyond (w=0,1)?
-
What accuracy–cost tradeoff does model-assisted text attribute extraction offer relative to rule-based attributes and the score-diagnostics-only gate?
-
How does the main Brier-selected adaptive estimator compare with calibrated per-category probabilities on subgroup calibration, particularly for low-support groups?
-
Grounding-based confidence calibration (Padhi et al., 2025) [1] is relevant prior work on using cross-modal evidence to improve multimodal confidence estimates. Please clarify how the proposed context-conditioned calibration differs from grounding-based approaches like these.
[1] Padhi, Trilok, Ramneet Kaur, Adam D. Cobb, Manoj Acharya, Anirban Roy, Colin Samplawski, Brian Matejek, Alexander M. Berenbeim, Nathaniel D. Bastian, and Susmit Jha. "Calibrating uncertainty quantification of multi-modal llms using grounding." arXiv preprint arXiv:2505.03788 (2025).