- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors argue that existing confidence estimators in MLLMs, which rely on output distributions, repeated sampling or hidden representations, may inherit language biases from the models, resulting in high-confidence predictions that are not adequately supported by multimodal evidence. To address this issue, they propose EviConf, a training-free confidence estimation method. Specifically, EviConf computes internal confidence using normalised predictive entropy while fixing the answer selected from the multimodal input. It then measures content evidence support by calculating the difference in probability between the full input and a question-only input, and temporal evidence support by determining the proportion of K temporal shuffles that decrease the probability of the selected answer. The product of these two signals is mapped to the range [0, 1] to obtain an evidence score, which is then combined with the internal confidence score using a geometric mean. Experiments across four benchmarks and eight models demonstrate that EviConf achieves the highest AUROC for correctness among the estimators compared in all evaluated settings. The authors also report reductions in confident error rates of 55–92% for video QA and 19–65% for audio-visual QA.
Critical strengths
1.The paper addresses the important issue of false confidence in multimodal models and provides an intuitive motivation for incorporating multimodal evidence into confidence estimation.
2.The method is evaluated across four benchmarks and eight models, consistently achieving the highest AUROC among the compared baselines. The ablation studies further investigate the contributions of different components.
Critical weaknesses
1.The proposed temporal support measures how frequently shuffling video frames decreases the probability of the selected answer. However, such a decrease does not necessarily indicate that the original temporal order provides correct evidence. For instance, an incorrect prediction may also become less confident after temporal shuffling. Therefore, the proposed metric may primarily capture sensitivity to temporal order rather than the reliability of temporal reasoning.
2.Equation 3 combines content and temporal support multiplicatively, which may be inappropriate for questions that do not require temporal reasoning. For example, when answering a static question such as "What color is the man's shirt?", temporal shuffling may not decrease the answer probability, resulting in zero temporal support. Consequently, the positive contribution of content evidence is completely eliminated, regardless of how strongly the visual input supports the answer. The paper does not adequately justify this design choice.
Questions for authors
1.Why was the number of temporal shuffles fixed at K=4? Could the authors provide a sensitivity analysis with different values of K?
2.Figure 1 appears inconsistent with Equation 3. For example, the content lift (+0.50) and temporal contribution (+0.27) are added to obtain +0.77, whereas Equation 3 uses multiplicative fusion. Additionally, the internal confidence for the Violin example differs between the left (0.91) and right (0.92) panels.
3.The content evidence score is defined as the probability difference between the full input and the question-only input. However, if the blind-input distribution is nearly uniform or exhibits systematic answer-position biases, this score may largely reflect the full-input prediction probability rather than independent evidence support.
- The Related Work section lacks a comprehensive discussion of multimodal uncertainty estimation methods based on input perturbation and sampling. For example, FESTA (Findings of EMNLP 2025) explores uncertainty estimation through functionally equivalent and complementary multimodal input samples, which appears closely related to EviConf's input intervention strategy.
5.The paper does not sufficiently explain why the proposed Full-vs-Blind Contrast and Temporal Shuffle strategies should be more effective than existing multimodal input-sampling approaches.