- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In order to quantify the performance of a VLM on a given data set without the use of ground truth labels, one can use either generation consistency, or the output token probabilities of the model as a proxy for its uncertainty. The latter option is more token efficient, but the output logits often exhibit overconfidence, leading to biased readings. The authors hypothesize that overconfidence is significantly associated with over-reliance of either the text or image modality, and this can therefore be used as a heuristic to detect poorly calibrated confidence. They present Visual Contrast for Reliable Uncertainty (VCRU), a technique that uses this information source dependence with the aim of obtaining more reliable label-free VLM rankings. The authors present a brief mathematical formalism to motivate their approach. They empirically evaluate the performance of their method on various benchmarks other label-free baseline methods, and ablate different design elements to quantify their effect. The authors qualitatively analyze the benchmark performance, and offer interpretations of how the different conflict types relate to different failure modes.
Critical strengths
The method performs well in a variety of benchmarks against other logit-based baselines, and the authors do well to ablate the different design elements.
Critical weaknesses
-
I do not find the mathematical treatment of the different modality dependencies to be robust or convincing. The 'contribution' terms and , as well as the formula assumes that the contributions of the text and visual input modality to the output logits are in some way separable, or at least that the influence of the visual input can be modulated in a controlled fashion. I do not think that it is reasonable to assume this, and the authors do not provide any explanation or justification for this assumption. All the analysis that follows relies on this premise.
-
Adding Gaussian noise to visual input does not simply 'remove visual information'. It affects high frequency features first, while lower frequency signals may still be resolved even at significant noise levels, so how much noise obfuscates the visual contribution to classification depends on a wide variety of factors, such as contrast of the image, the scale of the salient features, etc.
Questions for authors
Have you tried other methods of obfuscating the visual information, such as crops, distortions, or masking? How do you think that the type of corruption might affect the influence on the output token probabilities?
6 comments
Response to Reviewer iFFS
1. Revising the Theoretical Formulation and Providing Empirical Evidence [W1-1]
Does the NLL response relate to overconfidence? In Fig. 3(c), we group MMStar predictions from 36 VLMs by the NLL change of the original first token at Gaussian noise step . We measure both first-token confidence and answer accuracy on the original images. Among predictions in each model's confidence interval, mean confidence exceeds accuracy by 29.9 percentage points in the negative tail () and 30.6 points in the positive tail (), compared with 19.7 points near zero (). Thus, among high-confidence predictions on MMStar, substantial NLL shifts in either direction are associated with a larger confidence--accuracy gap, with positive shifts indicating stronger relative support from visual evidence and negative shifts indicating stronger relative support from linguistic priors.
| NLL change → | < −0.10 | [−0.10, −0.01) | [−0.01, +0.01] | (+0.01, +0.10] | > +0.10 |
|---|---|---|---|---|---|
| Original-image first-token confidence ↓ | |||||
| High | −29.9 | −34.0 | −19.7 | −27.8 | −30.6 |
| Mid | −21.6 | −24.5 | −21.2 | −20.4 | −23.1 |
| Low | −9.6 | −8.9 | −10.1 | −9.0 | −9.0 |
Response to Reviewer iFFS
2. Measuring Relative Answer Support Under Cross-Modal Interaction [W1-2]
Thank you for pointing out that our original formulation assumed an additive decomposition of visual and linguistic logits. We have revised the theorem. To examine whether perturbing the image while holding the question fixed can reveal changes in the model’s support for an answer, even when visual and textual information interact, we conducted two related experiments.
1. With the question held fixed, which answer gains support as the relevant visual evidence is perturbed? In Fig. 3(b), we selected questions for which the model gave different answers with and without the image. We fixed both answer tokens before applying Gaussian noise to the question-relevant object. At each noise level, we measured the probabilities of both tokens on the same image. The last column reports how much the ratio of the text-only answer’s probability to the original-image answer’s probability changed relative to the original image.
At , the probability ratio favoring that answer was 2.08 times its original value. So we can identify whether an answer gains relative support when the relevant visual evidence is perturbed while holding the question fixed.
| Noise step | Original-image answer token: mean ΔNLL | Text-only answer token: mean ΔNLL | Multiplier on the text-only / original-image answer probability ratio |
|---|---|---|---|
| 0 (original image) | 0.0000 | 0.0000 | 1.00× |
| 50 | −0.0009 | +0.0201 | 0.98× |
| 100 | +0.0019 | +0.0369 | 0.97× |
| 250 | +0.0224 | −0.1366 | 1.17× |
| 500 | +0.1554 | −0.5776 | 2.08× |
| 750 | +0.4674 | −1.2896 | 5.80× |
| 999 | +0.8772 | −1.6128 | 12.06× |
2. Why does this fixed-question comparison remain interpretable when the modalities interact? We evaluated question pairs from Visual Answerability benchmark Cha et al. (2026), When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes. Each pair asks about the same attribute of two objects, A and B, with different correct answers. We masked object A under both questions. For this experiment, we tracked the same margin in every cell: the logit of A’s answer minus the logit of B’s answer. Positive margins favor A’s answer, and negative margins favor B’s answer.
| Question held fixed | Mean margin with A and B visible | Mean margin with A masked | Mean decrease after masking A |
|---|---|---|---|
| Ask about object A | +3.662 | +1.172 | 2.489 |
| Ask about object B | −3.504 | −3.538 | 0.034 |
The first row shows what the perturbation reveals for a given question: when asked about A, the model still favored A’s answer on average after masking, but its advantage fell by 2.489 logit units. The second row shows why the question matters: when asked about B, the same mask changed the mean margin by only 0.034. The difference between these effects was 2.455 logit units .
Together, with the question held fixed, the direction of the shift tells us whether the model moves toward or away from the answer. In Fig. 3(b), perturbing the relevant object shifted preference toward that text-only answer on average at the stronger noise levels. The paired-question experiment shows why this interpretation remains valid under interaction: the same mask produced a larger shift when the question concerned the masked object, and each within-question comparison measured how much seeing that object favored one answer over the other. This gives us an observable measure of relative support under cross-modal Interaction.
Response to Reviewer iFFS
1. Revising the Theoretical Formulation and Providing Empirical Evidence [W1-1]
Thank you for pointing out that our original formulation assumed that visual and linguistic contributions to a model’s logits were separable. We have revised the theorems and narrowed our claims accordingly. To examine how the theoretical analysis relates to the behavior of VLMs, we added three empirical analyses in Fig. 3., We have also added three empirical analyses in Fig. 3 to examine how visual perturbation changes support for an answer and how those changes relate to overconfidence in VLMs.
Revised theoretical analysis Theorem 1 gives a lower bound on this confidence using the token’s logit margin over competing tokens within the same model. Theorem 2 shows that a group is overconfident on average when this lower bound exceeds the group’s answer accuracy. Theorem 3 characterizes how visual perturbation changes support for the same token. Its NLL change, Δℓm,iF, equals the logarithm of its probability on the original input divided by its probability on the perturbed input. The derivations use softmax probabilities and logit differences, avoiding strong assumptions about the separability of visual and linguistic contributions in modern VLMs.
Empirical analysis
What supports a visual or linguistic interpretation of the NLL change? In Fig. 3(a), we tested whether the response depends on which visual evidence is disrupted. on Visual Answerability benchmark Cha et al. (2026), When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes, whose annotations identify the object region needed to answer each question.We applied Gaussian noise at the same strength separately to the object needed to answer the question and to the other object in the same image. As shown in Table, As noise increases, the mean NLL of the original answer token rises more sharply when the key object is perturbed. This suggests that NLL shifts can reflect changes in the model’s support for an answer under visual perturbation.
In Fig. 3 (b), we tests whether weakening visual evidence shifts support toward the model’s text-only answer. We selected questions for which the model gave different answers with and without the image. As shown in table, as the noise strength increases, the NLL of the original-image answer token, which is supported by visual evidence, generally increases, whereas that of the text-only answer token, which is favored by linguistic priors, decreases. This opposite-direction pattern suggests that the direction of an answer token's NLL change under visual perturbation can indicate whether the answer is relatively supported by visual evidence or linguistic priors.