acceptodds
Under review as a conference paper at ICLR 2027

Visual Contrast for Reliable Uncertainty in Label-Free VLM Ranking

Abstract

Selecting a vision-language model (VLM) for a downstream task is challenging when ground-truth labels are unavailable. Existing uncertainty-based ranking methods use signals such as negative log-likelihood (NLL) and entropy, but overconfident predictions can make these signals unreliable. We observe that excessive reliance on either linguistic priors or visual information is associated with greater overconfidence than reliance on both sources. Motivated by this observation, we propose Visual Contrast for Reliable Uncertainty (VCRU), a label-free method for ranking VLMs on multiple-choice question-answering tasks. VCRU compares the NLL of the original first-token prediction and the output entropy between original and visually degraded inputs. It identifies confidence conflicts, where NLL and entropy shift in opposite directions, as diagnostic signals of potentially unreliable confidence. A drift gate then emphasizes predictions with high original confidence and substantial changes in relative confidence, yielding a model-level penalty that adjusts the baseline ranking score. Experiments with 36 heterogeneous VLMs across four benchmarks show that VCRU consistently improves upon its underlying uncertainty baselines and achieves the strongest ranking correlations on three benchmarks. On MMStar, it improves Spearman's correlation from 0.8273 to 0.9437 over first-token NLL. These results demonstrate the value of visual contrast for assessing the reliability of confidence in label-free VLM ranking.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Vision-language models, Unsupervised model ranking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iFFSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In order to quantify the performance of a VLM on a given data set without the use of ground truth labels, one can use either generation consistency, or the output token probabilities of the model as a proxy for its uncertainty. The latter option is more token efficient, but the output logits often exhibit overconfidence, leading to biased readings. The authors hypothesize that overconfidence is significantly associated with over-reliance of either the text or image modality, and this can therefore be used as a heuristic to detect poorly calibrated confidence. They present Visual Contrast for Reliable Uncertainty (VCRU), a technique that uses this information source dependence with the aim of obtaining more reliable label-free VLM rankings. The authors present a brief mathematical formalism to motivate their approach. They empirically evaluate the performance of their method on various benchmarks other label-free baseline methods, and ablate different design elements to quantify their effect. The authors qualitatively analyze the benchmark performance, and offer interpretations of how the different conflict types relate to different failure modes.

Critical strengths

The method performs well in a variety of benchmarks against other logit-based baselines, and the authors do well to ablate the different design elements.

Critical weaknesses
  1. I do not find the mathematical treatment of the different modality dependencies to be robust or convincing. The 'contribution' terms aka_k and vkv_k, as well as the formula sk(ρ)=ak+ρvks_k(\rho) = a_k + \rho v_k assumes that the contributions of the text and visual input modality to the output logits are in some way separable, or at least that the influence of the visual input can be modulated in a controlled fashion. I do not think that it is reasonable to assume this, and the authors do not provide any explanation or justification for this assumption. All the analysis that follows relies on this premise.

  2. Adding Gaussian noise to visual input does not simply 'remove visual information'. It affects high frequency features first, while lower frequency signals may still be resolved even at significant noise levels, so how much noise obfuscates the visual contribution to classification depends on a wide variety of factors, such as contrast of the image, the scale of the salient features, etc.

Questions for authors

Have you tried other methods of obfuscating the visual information, such as crops, distortions, or masking? How do you think that the type of corruption might affect the influence on the output token probabilities?

6 comments
AuthorsOpenReview ↗

Response to Reviewer iFFS

1. Revising the Theoretical Formulation and Providing Empirical Evidence [W1-1]

Thank you for pointing out that our original formulation assumed that visual and linguistic contributions to a model’s logits were separable. We have revised the theorems and narrowed our claims accordingly. To examine how the theoretical analysis relates to the behavior of VLMs, we added three empirical analyses in Fig. 3., We have also added three empirical analyses in Fig. 3 to examine how visual perturbation changes support for an answer and how those changes relate to overconfidence in VLMs.

Revised theoretical analysis Theorem 1 gives a lower bound on this confidence using the token’s logit margin over competing tokens within the same model. Theorem 2 shows that a group is overconfident on average when this lower bound exceeds the group’s answer accuracy. Theorem 3 characterizes how visual perturbation changes support for the same token. Its NLL change, Δℓm,iF\Delta\ell_{m,i}^{F}, equals the logarithm of its probability on the original input divided by its probability on the perturbed input. The derivations use softmax probabilities and logit differences, avoiding strong assumptions about the separability of visual and linguistic contributions in modern VLMs.

Empirical analysis

  1. Fig. 3(a) tests whether the original answer token’s NLL increases more when we perturb the object needed to answer the question than when we perturb another object in the same image.
  2. Fig. 3(b) tests whether weakening visual evidence shifts the model's preference toward the answer favored by its language prior under text-only conditions.
  3. Fig. 3(c) shows that when first-token confidence on the original image is high, the confidence--accuracy gap is larger for predictions whose original token's NLL changes substantially in either direction under visual perturbation.

What supports a visual or linguistic interpretation of the NLL change? In Fig. 3(a), we tested whether the response depends on which visual evidence is disrupted. on Visual Answerability benchmark Cha et al. (2026), When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes, whose annotations identify the object region needed to answer each question.We applied Gaussian noise at the same strength separately to the object needed to answer the question and to the other object in the same image. As shown in Table, As noise increases, the mean NLL of the original answer token rises more sharply when the key object is perturbed. This suggests that NLL shifts can reflect changes in the model’s support for an answer under visual perturbation.

Perturbed regionT = 0T = 50T = 100T = 250T = 500T = 750T = 999
Question-relevant object0+0.0002+0.0030+0.0185+0.1566+0.4734+0.7636
Control object0+0.0010+0.0011−0.0009+0.0110+0.0262+0.0617
Difference (relevant − control)0−0.0008+0.0019+0.0194+0.1457+0.4472+0.7020

In Fig. 3 (b), we tests whether weakening visual evidence shifts support toward the model’s text-only answer. We selected questions for which the model gave different answers with and without the image. As shown in table, as the noise strength increases, the NLL of the original-image answer token, which is supported by visual evidence, generally increases, whereas that of the text-only answer token, which is favored by linguistic priors, decreases. This opposite-direction pattern suggests that the direction of an answer token's NLL change under visual perturbation can indicate whether the answer is relatively supported by visual evidence or linguistic priors.

Answer token trackedT = 0T = 50T = 100T = 250T = 500T = 750T = 999
Text-only answer token0+0.0201+0.0369−0.1366−0.5776−1.2896−1.6128
Original-image answer token0−0.0009+0.0019+0.0224+0.1554+0.4674+0.8772
AuthorsOpenReview ↗

Response to Reviewer iFFS

1. Revising the Theoretical Formulation and Providing Empirical Evidence [W1-1]

Does the NLL response relate to overconfidence? In Fig. 3(c), we group MMStar predictions from 36 VLMs by the NLL change Δℓm,iF\Delta\ell_{m,i}^{F} of the original first token zm,i,1z_{m,i,1} at Gaussian noise step T=500T=500. We measure both first-token confidence Cm,iC_{m,i} and answer accuracy on the original images. Among predictions in each model's confidence interval, mean confidence exceeds accuracy by 29.9 percentage points in the negative tail (Δℓm,iF<−0.10\Delta\ell_{m,i}^{F}<-0.10) and 30.6 points in the positive tail (Δℓm,iF>0.10\Delta\ell_{m,i}^{F}>0.10), compared with 19.7 points near zero (∣Δℓm,iF∣≤0.01|\Delta\ell_{m,i}^{F}|\leq0.01). Thus, among high-confidence predictions on MMStar, substantial NLL shifts in either direction are associated with a larger confidence--accuracy gap, with positive shifts indicating stronger relative support from visual evidence and negative shifts indicating stronger relative support from linguistic priors.

NLL change Δℓm,iF\Delta\ell_{m,i}^{F} →< −0.10[−0.10, −0.01)[−0.01, +0.01](+0.01, +0.10]> +0.10
Original-image first-token confidence ↓
High−29.9−34.0−19.7−27.8−30.6
Mid−21.6−24.5−21.2−20.4−23.1
Low−9.6−8.9−10.1−9.0−9.0
AuthorsOpenReview ↗

Response to Reviewer iFFS

2. Measuring Relative Answer Support Under Cross-Modal Interaction [W1-2]

Thank you for pointing out that our original formulation assumed an additive decomposition of visual and linguistic logits. We have revised the theorem. To examine whether perturbing the image while holding the question fixed can reveal changes in the model’s support for an answer, even when visual and textual information interact, we conducted two related experiments.

1. With the question held fixed, which answer gains support as the relevant visual evidence is perturbed? In Fig. 3(b), we selected questions for which the model gave different answers with and without the image. We fixed both answer tokens before applying Gaussian noise to the question-relevant object. At each noise level, we measured the probabilities of both tokens on the same image. The last column reports how much the ratio of the text-only answer’s probability to the original-image answer’s probability changed relative to the original image.

At T=500T=500, the probability ratio favoring that answer was 2.08 times its original value. So we can identify whether an answer gains relative support when the relevant visual evidence is perturbed while holding the question fixed.

Noise step TTOriginal-image answer token: mean ΔNLLText-only answer token: mean ΔNLLMultiplier on the text-only / original-image answer probability ratio
0 (original image)0.00000.00001.00×
50−0.0009+0.02010.98×
100+0.0019+0.03690.97×
250+0.0224−0.13661.17×
500+0.1554−0.57762.08×
750+0.4674−1.28965.80×
999+0.8772−1.612812.06×

2. Why does this fixed-question comparison remain interpretable when the modalities interact? We evaluated question pairs from Visual Answerability benchmark Cha et al. (2026), When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes. Each pair asks about the same attribute of two objects, A and B, with different correct answers. We masked object A under both questions. For this experiment, we tracked the same margin in every cell: the logit of A’s answer minus the logit of B’s answer. Positive margins favor A’s answer, and negative margins favor B’s answer.

Question held fixedMean margin with A and B visibleMean margin with A maskedMean decrease after masking A
Ask about object A+3.662+1.1722.489
Ask about object B−3.504−3.5380.034

The first row shows what the perturbation reveals for a given question: when asked about A, the model still favored A’s answer on average after masking, but its advantage fell by 2.489 logit units. The second row shows why the question matters: when asked about B, the same mask changed the mean margin by only 0.034. The difference between these effects was 2.455 logit units .

Together, with the question held fixed, the direction of the shift tells us whether the model moves toward or away from the answer. In Fig. 3(b), perturbing the relevant object shifted preference toward that text-only answer on average at the stronger noise levels. The paired-question experiment shows why this interpretation remains valid under interaction: the same mask produced a larger shift when the question concerned the masked object, and each within-question comparison measured how much seeing that object favored one answer over the other. This gives us an observable measure of relative support under cross-modal Interaction.

Reviewer_aKJJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes VCRU, a label-free method designed to rank VLMs on multiple-choice question-answering tasks without ground-truth labels. The core motivation stems from the observation that VLM confidence can become unreliable due to "single-source reliance", over-relying excessively on either visual evidence or linguistic priors. To address this, VCRU introduces "confidence conflicts" by measuring changes in first-token negative log-likelihood and output entropy between original and visually degraded inputs, combined with a drift gate to penalize potentially overconfident predictions. Experiments across models and benchmarks are presented to show ranking improvements over existing logit-based baselines.

Critical strengths
  1. Framing VLM confidence failures through the lens of information-source dependence (visual vs. linguistic) provides a reasonable angle for exploring why standard uncertainty measures sometimes fail in multimodal settings.

  2. The evaluation covers a large number of candidate models and multiple downstream benchmarks, providing a broad empirical comparison against various logit- and consistency-based ranking proxies.

Critical weaknesses
  1. A major flaw of the paper is the lack of precise, formal definitions for "uncertainty" and "confidence". The text jumps straight into mathematical derivations without clearly establishing how confidence is quantified or mapped from raw logits and probabilities, leaving readers to guess the foundational probabilistic framework underlying the formulation.

  2. The theoretical section features several theorems that feel artificially forced into the paper to justify the heuristic method. Crucially, Theorem 1 and subsequent discussions introduce the concept of "confidence" directly, even though no prior definition or mathematical formalization of confidence was provided in the problem setup. Upon close inspection, these theorems rely on sweeping assumptions that do not hold for modern, autoregressive autoregulated VLMs, rendering the theoretical backing unconvincing.

  3. The evaluation pool relies heavily on outdated model architectures (such as older variants of LLaVA-1.5, LLaVA-NeXT, and InstructBLIP). The performance and behavior of these legacy models can no longer compete with or adequately represent state-of-the-art modern VLMs such as Qwen3.5VL. Consequently, optimizing a ranking proxy for obsolete architectures diminishes the practical utility and timeliness of the study for current research standards.

Questions for authors

Given that confidence is utilized in Theorem 1 without a formal preliminary definition or quantification method in the setup, how do the authors mathematically define confidence independent of first-token NLL? Furthermore, how can the authors justify that the theoretical bounds and assumptions hold true for modern, high-performing VLM families rather than legacy architectures?

5 comments
AuthorsOpenReview ↗

Response to Reviewer aKJJ

1. Clarification of Confidence and Uncertainty [W1,Q1]

Thank you for allowing us to clarify the uncertainty and confidence definition. We have added definitions to Section 2.1.

Confidence. For an original question–image input xix_i, let sm,i,ks_{m,i,k} be the raw logit that model fmf_m assigns to token kk at the first decoding step. We apply softmax over the model’s full vocabulary Vm\mathcal V_m, zm,i,1z_{m,i,1} is the first token the model generates, its confidence is the probability assigned to that token:

=\frac{\exp(s_{m,i,k})}{\sum_{v\in\mathcal V_m}\exp(s_{m,i,v})}, \qquad C_{m,i}=p_m(z_{m,i,1}\mid x_i).$$ **Uncertainty.** We use two measures derived from the same first-step distribution. Here, the $F$ indicates that NLL and entropy are computed using the original input $x_i$, the first-token negative log-likelihood and normalized predictive entropy are $$\ell_{m,i}^{F}=-\log C_{m,i}, \qquad h_{m,i}^{F} =-\frac{1}{\log|\mathcal V_m|} \sum_{k\in\mathcal V_m}p_m(k\mid x_i)\log p_m(k\mid x_i).$$ A higher NLL $\ell_{m,i}^{F}$ means lower confidence in the particular token the model generated. A higher entropy $h_{m,i}^{F}$ means that probability is spread more broadly across the vocabulary, indicating greater uncertainty about which token to generate first. **Changes under visual perturbation.** We evaluate the same question with a distorted image, denoted by $x_i'$. We keep the original first token $z_{m,i,1}$ fixed when computing its NLL and recompute entropy over the full next-token distribution under $x_i'$. Here, the $F'$ indicates that NLL and entropy are computed using the distorted input $x_i'$,, we define both changes relative to the original input: $$\Delta\ell_{m,i}^{F} =\ell_{m,i}^{F'}-\ell_{m,i}^{F} =\log\frac{p_m(z_{m,i,1}\mid x_i)} {p_m(z_{m,i,1}\mid x_i')}, \qquad \Delta h_{m,i}^{F}=h_{m,i}^{F'}-h_{m,i}^{F}.$$ A positive $\Delta\ell_{m,i}^{F}$ means that the model assigns less probability to its original first token after perturbation. A positive $\Delta h_{m,i}^{F}$ means that the first-step distribution becomes more uncertain.
AuthorsOpenReview ↗

Response to Reviewer aKJJ

2. Theoretical Interpretation and Empirical Evidence [W2-1,Q3]

Thank you for asking us to clarify the assumptions behind the theorems and how their conclusions relate to the behavior of VLMs. We revised the theoretical section and added three empirical analyses on VLMs directly in Fig. 3.

Theorem 1 gives a lower bound on this confidence using the token’s logit margin over competing tokens within the same model. Theorem 2 shows that a group is overconfident on average when this lower bound exceeds the group’s answer accuracy. Theorem 3 characterizes how visual perturbation changes support for the same token. Its NLL change, Δℓm,iF\Delta\ell_{m,i}^{F}, equals the logarithm of its probability on the original input divided by its probability on the perturbed input. The derivations use softmax probabilities and logit differences, avoiding strong assumptions about the separability of visual and linguistic contributions in modern VLMs. To examine how the theoretical analysis relates to the behavior of VLMs, we added three empirical analyses in Fig. 3.

  1. Fig. 3(a) tests whether the original answer token’s NLL increases more when we perturb the object needed to answer the question than when we perturb another object in the same image.
  2. Fig. 3(b) tests whether weakening visual evidence shifts the model's preference toward the answer favored by its language prior under text-only conditions.
  3. Fig. 3(c) shows that when first-token confidence on the original image is high, the confidence--accuracy gap is larger for predictions whose original token's NLL changes substantially in either direction under visual perturbation.

What supports a visual or linguistic interpretation of the NLL change? In Fig. 3(a), we tested whether the response depends on which visual evidence is disrupted. on Visual Answerability benchmark Cha et al. (2026), When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes, whose annotations identify the object region needed to answer each question. We applied Gaussian noise at the same strength separately to the object needed to answer the question and to the other object in the same image. As shown in Table, As noise increases, the mean NLL of the original answer token rises more sharply when the key object is perturbed. This suggests that NLL shifts can reflect changes in the model’s support for an answer under visual perturbation.

Perturbed regionT = 0T = 50T = 100T = 250T = 500T = 750T = 999
Question-relevant object0+0.0002+0.0030+0.0185+0.1566+0.4734+0.7636
Control object0+0.0010+0.0011−0.0009+0.0110+0.0262+0.0617
Difference (relevant − control)0−0.0008+0.0019+0.0194+0.1457+0.4472+0.7020

In Fig. 3 (b), we tests whether weakening visual evidence shifts support toward the model’s text-only answer. We selected questions for which the model gave different answers with and without the image. As shown in table, as the noise strength increases, the NLL of the original-image answer token, which is supported by visual evidence, generally increases, whereas that of the text-only answer token, which is favored by linguistic priors, decreases. This opposite-direction pattern suggests that the direction of an answer token's NLL change under visual perturbation can indicate whether the answer is relatively supported by visual evidence or linguistic priors.

Answer token trackedT = 0T = 50T = 100T = 250T = 500T = 750T = 999
Text-only answer token0+0.0201+0.0369−0.1366−0.5776−1.2896−1.6128
Original-image answer token0−0.0009+0.0019+0.0224+0.1554+0.4674+0.8772
AuthorsOpenReview ↗

Response to Reviewer aKJJ

2. Theoretical Interpretation and Empirical Evidence [W2-1,Q3]

Does the NLL response relate to overconfidence? In Fig. 3(c), we group MMStar predictions from 36 VLMs by the NLL change Δℓm,iF\Delta\ell_{m,i}^{F} of the original first token zm,i,1z_{m,i,1} at Gaussian noise step T=500T=500. We measure both first-token confidence Cm,iC_{m,i} and answer accuracy on the original images. Among predictions in each model's confidence interval, mean confidence exceeds accuracy by 29.9 percentage points in the negative tail (Δℓm,iF<−0.10\Delta\ell_{m,i}^{F}<-0.10) and 30.6 points in the positive tail (Δℓm,iF>0.10\Delta\ell_{m,i}^{F}>0.10), compared with 19.7 points near zero (∣Δℓm,iF∣≤0.01|\Delta\ell_{m,i}^{F}|\leq0.01). Thus, among high-confidence predictions on MMStar, substantial NLL shifts in either direction are associated with a larger confidence--accuracy gap, with positive shifts indicating stronger relative support from visual evidence and negative shifts indicating stronger relative support from linguistic priors.

NLL change Δℓm,iF\Delta\ell_{m,i}^{F} →< −0.10[−0.10, −0.01)[−0.01, +0.01](+0.01, +0.10]> +0.10
Original-image first-token confidence ↓
High−29.9−34.0−19.7−27.8−30.6
Mid−21.6−24.5−21.2−20.4−23.1
Low−9.6−8.9−10.1−9.0−9.0
Reviewer_9XohOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Visual Contrast for Reliable Uncertainty (VCRU), a method for ranking vision-language models without ground truth labels. It compares first-token negative log-likelihood and output entropy between original and visually degraded inputs. Samples where these signals change in opposite directions are treated as potentially unreliable confidence estimates. A drift gate emphasizes confident predictions that involve substantial changes, thereby producing a correction to the baseline ranking score. Experiments on 36 VLMs show improved ranking correlations over four MCQA benchmarks: MMStar, MMBench, AI2D, and ScienceQA.

Critical strengths
  1. Interesting and practical setup: selecting VLMs without annotating a new target dataset is useful. The approach requires no additional model training and can complement existing uncertainty scores.

  2. Strong evaluation across heterogeneous models: VCRU is tested on 36 VLMs from multiple families. On MMStar, it improves Spearman correlation from 0.8273 to 0.9437 over the first-token NLL baseline, with improvements on the other main benchmarks.

  3. Useful evidence of generalization: the appendix shows improvements across five datasets using a fixed correction coefficient. Additional experiments consider limited evaluation samples, non-MCQ tasks, and a newly added model.

  4. The component ablations support both conflict selection and the drift gate. The reported runtime overhead is also modest: approximately 22% over the NLL baseline in the tested configuration.

Critical weaknesses
  1. The assumption that degraded visual input should reduce the reliance on visual cues (lines 258- 261) may not be universal and may depend on the question. For example, if the question is "what's the color of the sky?", blurring may be useful as it will partially reduce the distractors in the image that otherwise get attention. More generally, the dependence on visual or textual cues should be question-specific. So over-reliance on one modality alone may not be indicative of the uncertainty. It will be good to see a study based on question types.

  2. First-token scoring and MCQ questions: The first-token scoring may be suitable for MCQ but generative models may generate sentence-like answers by choosing the options. The approach may be limited in those case. Also, how does the approch work for common use cases where answers are beyond a single word or phrase?

  3. Diversity of perturbation: The approach primarily uses Gaussian noise with a fixed strength, while the category analysis shows that different tasks favor different perturbation levels. How about the other types of degradation such as blur, masking, or other image degradations? For example, masking image regions may have a drastic impact on the visual reliance, as it may partially remove the evidence or confounders.

  4. The theoretical analysis does not fully establish the proposed explanation. The theorems show that a large logit margin produces high confidence and that overconfidence follows when accuracy is sufficiently low. They do not establish that dependence on one modality generally causes poorer calibration. The additive decomposition of visual and linguistic logits is also a simplifying assumption that is not validated for the evaluated models. Some questions legitimately require mostly visual recognition or mostly background knowledge. How does VCRU distinguish these cases from unreliable confidence?

Questions for authors

Please address the comments in the weaknesses section. I am happy to reconsider the final score based on the rebuttal.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9Xoh

1. Question-Specific Effects of Visual Evidence [W1-1]

Thank you for pointing out that visual degradation can affect predictions differently depending on question types.

(a) Correction of initially wrong answers under Gaussian noise. We first tested the possibility raised in your example: a model that answers incorrectly on the original image may answer correctly after degradation. We define the error correction rate as the percentage of predictions that were wrong on the original image and became correct after perturbation.

As shown in table, coarse perception had the highest correction rate at both noise strengths, reaching 19.08% at step 500 and 23.76% at step 750.

MMStar question typeOriginal accuracyAccuracy at step 500Accuracy at step 750Error correction at step 500Error correction at step 750
Coarse perception63.94%59.47%48.40%19.08%23.76%
Fine-grained perception39.72%35.37%29.53%13.38%16.76%
Instance reasoning54.84%51.09%40.67%11.15%13.90%
Logical reasoning41.07%37.73%31.32%10.29%12.10%
Math37.90%36.96%35.03%8.12%11.67%
Science & technology33.90%32.10%28.50%7.80%11.33%

(b) Effect of adding the original image after a text-only error. To examine how much the image helps with different questions, we started from cases in which the model chose a wrong option from text alone. After adding the original image, we recorded the ratio of:

  1. switched to the correct option
  2. kept the same wrong option
  3. switched to a different wrong option

We also measured changes in preference that could occur even when the selected option stayed the same. As shown in table, adding the image corrected 60.3% of text-only errors in coarse perception, compared with 26.1% in math. The probability comparison showed the same pattern: the correct option gained the most support in coarse perception (+4.24) and the least in math (+1.04).

MMStar question typeSwitched to the correct optionKept the same wrong optionSwitched to a different wrong optionMean shift toward correct option (log-odds units)
Coarse perception60.3%21.0%18.7%+4.24
Fine-grained perception36.5%30.6%32.9%+2.13
Instance reasoning48.6%30.1%21.3%+3.28
Logical reasoning34.0%38.3%27.8%+1.82
Math26.1%47.5%26.4%+1.04
Science & technology26.2%46.6%27.3%+1.11

(c) First-token uncertainty on initially correct answers. We also tracked the original first token under perturbation and measured the change in its negative log-likelihood (ΔNLL). A negative value means that the token became more probable after perturbation.

MMStar question typeCorrect predictions on original imagesΔNLL > 0ΔNLL < 0Mean ΔNLLMean when positiveMean when negative
Coarse perception5,50370.09%29.55%+0.287+0.450−0.098
Fine-grained perception3,43371.89%28.11%+0.423+0.641−0.136
Instance reasoning4,73966.93%32.69%+0.163+0.282−0.079
Logical reasoning3,40168.27%31.64%+0.234+0.383−0.087
Math3,25660.44%39.56%+0.078+0.173−0.067
Science & technology2,94164.37%35.50%+0.103+0.207−0.084

Math type had the highest proportion of negative ΔNLL values (39.56%) and the smallest mean change (+0.078). Fine-grained perception had the lowest proportion of negative values (28.11%) and the largest mean change (+0.423). These results support the point that reliance on visual evidence varies by question type.

Question-Specific Effects of VCRU [W1-2]

Question typeNLL-F baselineFull VCRUConflict-V onlyConflict-L only
Coarse perception0.57010.64160.62460.6195
Fine-grained perception0.68920.75000.74130.7139
Instance reasoning0.78170.85400.86460.7930
Logical reasoning0.83770.87100.87060.8659
Math0.64210.68240.68640.6879
Science & technology0.80000.83050.82700.8251

We split VCRU into two variants, using Conflict-V and Conflict-L separately. As shown in the table, for question types that rely more heavily on background knowledge (i.e., language priors), such as Math and Science & Technology, using Conflict-L alone to detect unstable confidence already achieves performance close to that of the full VCRU, and even slightly outperforms the full VCRU on Math.

AuthorsOpenReview ↗

VCRU Response to Reviewer 9Xoh

2. Analysis of Perturbation Type and Strength [W3]

We extended the Gaussian noise analysis to nine additional image degradations, each tested at five strengths,following the evaluation design of Qiu et al. (2025).

(a) Error correction and first-token probability. We report two quantities averaged over the five tested strengths: how often a perturbation corrected an initially wrong answer and how much it reduced the probability of the original first token.

PerturbationOverall error correction rateError correction rate on coarse perceptionMean drop in original first-token probability
JPEG compression6.44%11.31%3.56 pp
Coarse dropout8.67%12.18%6.58 pp
Cutout8.91%12.39%7.16 pp
Gaussian blur9.84%14.65%8.59 pp
Motion blur9.93%14.34%8.85 pp
Defocus blur10.64%15.41%9.78 pp
Glass blur11.36%17.87%10.96 pp
Random patch masking11.62%17.67%13.05 pp
Zoom blur12.50%18.98%14.04 pp

JPEG compression had the lowest mean error correction rate overall (6.44%) and on coarse perception (11.31%). It also produced the smallest mean first-token probability drop among the nine additional degradations for each of the 36 VLMs.

(b) Variation across question types and masking patterns. Across the 45 combinations of degradation and strength, fine-grained perception had the largest mean probability drop in 44 settings, while math had the smallest in 39.

MMStar question typeMean first-token probability drop across 45 settings
Fine-grained perception12.32 pp
Coarse perception10.66 pp
Logical reasoning9.09 pp
Instance reasoning8.59 pp
Science & technology7.73 pp
Math6.64 pp

Moreover, cutout and coarse dropout also let us compare different masking patterns at the same target coverage. At 5% and 10% coverage, masking 16 dispersed blocks produced the larger mean drop. The ordering reversed at 20%, 30%, and 40%. At 40% coverage, one contiguous block produced a 2.39 pp larger mean drop, and 33 of 36 models showed this pattern.

Target area maskedMean drop with one blockMean drop with 16 blocksModels with a larger drop under one block
5%2.32 pp2.98 pp3/36
10%3.81 pp4.31 pp9/36
20%6.83 pp6.49 pp23/36
30%9.86 pp8.55 pp32/36
40%12.99 pp10.60 pp33/36

(c) VCRU ranking performance across model pools. We measured Spearman’s ρ between the VCRU ranking and the ranking by MMStar accuracy. On the original 36 VLMs, JPEG compression at Q7 gave the highest correlation among the ten settings shown (ρ = 0.9452), followed by glass blur at level 2 (0.9402) and Gaussian noise at step 500 (0.9387). In general, moderate-to-strong perturbations often yielded better rankings. It can be attribute to several reasons: 1. If perturbation is too weak, the probability of the original answer token may change only slightly, making it difficult to detect the model’s reliance on visual evidence. 2. If perturbation is too strong, it may obscure key visual details. Predictions that rely on those details to different degrees may then show similar probability drops, making those differences harder to distinguish.

PerturbationLevel 1Level 2Level 3Level 4Level 5
Random patch masking0.9111 (+0.152)0.9204 (+0.258)0.9210 (+0.374)0.9316 (+0.509)0.9091 (+0.665)
Cutout0.8469 (+0.067)0.8466 (+0.117)0.8692 (+0.226)0.8654 (+0.343)0.8887 (+0.467)
Coarse dropout0.8536 (+0.081)0.8769 (+0.120)0.8873 (+0.188)0.8966 (+0.254)0.9061 (+0.321)
JPEG compression0.8657 (+0.051)0.9025 (+0.066)0.9111 (+0.085)0.9306 (+0.131)0.9452 (+0.192)
Defocus blur0.9004 (+0.125)0.9107 (+0.189)0.9071 (+0.306)0.9004 (+0.395)0.9254 (+0.463)
Glass blur0.9092 (+0.132)0.9402 (+0.216)0.9142 (+0.397)0.9347 (+0.440)0.9312 (+0.505)
Zoom blur0.9365 (+0.340)0.9187 (+0.425)0.9237 (+0.463)0.9273 (+0.523)0.9150 (+0.550)
Motion blur0.9017 (+0.083)0.9115 (+0.156)0.9313 (+0.262)0.9356 (+0.387)0.9273 (+0.459)
Gaussian blur0.8867 (+0.053)0.9096 (+0.153)0.9109 (+0.263)0.9024 (+0.354)0.9268 (+0.482)
AuthorsOpenReview ↗

Response to Reviewer 9Xoh

2. Analysis of Perturbation Type and Strength [W3]

comparison on 36 additional latest VLMs

We then do a comparison on 36 additional latest VLMs. Gaussian noise gave the higher correlation on both the newly added models and the combined pool.

PerturbationStrengthOriginal 36 VLMs ρNewly added 36 VLMs ρAll 72 VLMs ρ
JPEG compressionQ70.94520.67920.9029
Gaussian noiseStep 5000.93870.72190.9242

Experimental settings. We used five severity levels for each additional degradation, following the evaluation design of Qiu et al. (2025).

DegradationStrength parameterFive levels, weakest to strongest
Random patch maskingFraction of 16 × 16-pixel patches masked15%, 30%, 45%, 60%, 75%
CutoutTarget image area masked by one rectangle5%, 10%, 20%, 30%, 40%
Coarse dropoutTarget image area masked by 16 rectangles5%, 10%, 20%, 30%, 40%
Defocus blurDisk radius and edge-smoothing σ(3, 0.1), (4, 0.5), (6, 0.5), (8, 0.5), (10, 0.5)
Glass blurSmoothing σ, displacement, and shuffle passes(0.7, 1, 2), (0.9, 2, 1), (1.0, 2, 3), (1.1, 3, 2), (1.5, 4, 2)
Zoom blurZoom-factor stop and increment, starting at 1.00×(1.11, 0.01), (1.16, 0.01), (1.21, 0.02), (1.26, 0.02), (1.31, 0.03)
Motion blurKernel radius and weighting σ(10, 3), (15, 5), (15, 8), (15, 12), (20, 15)
Gaussian blurSpatial σ in pixels1, 2, 3, 4, 6
JPEG compressionJPEG quality25, 18, 15, 10, 7

36 additional latest VLMs

SeriesModel
Qwen3.5Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-35B-A3B
Qwen3.6 / Qwen3.8Qwen3.6-27B, Qwen3.8-27B
Qwen3-VLQwen3-VL-2B-Instruct, Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct
Qwen2.5-VLQwen2.5-VL-32B-Instruct
Gemma 4gemma-4-E2B-it, gemma-4-E4B-it, gemma-4-26B-A4B-it, gemma-4-31B-it
Gemma 3 / Gemma 3ngemma-3-4b-it, gemma-3-12b-it, gemma-3-27b-it, gemma-3n-E2B-it
InternVL3.5InternVL3_5-2B-HF, InternVL3_5-4B-HF, InternVL3_5-8B-HF, InternVL3_5-14B-HF
InternVL3InternVL3-8B, InternVL3-14B
Intern-SIntern-S1-mini, Intern-S2-Preview
OvisOvis2-8B, Ovis2.5-9B
Ministral 3Ministral-3-14B-Instruct-2512
MiniCPM-VMiniCPM-V-4.7-1B
Kimi-VLKimi-VL-A3B-Instruct
MiMo-VLMiMo-VL-7B-RL-2508
Phi-4 MultimodalPhi-4-multimodal-instruct
IsaacIsaac-0.1

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely appreciate the reviewers' thoughtful feedback and encouraging comments on our work. We are pleased that reviewers recognize the practical value of label-free VLM ranking (9Xoh) and the promising perspective of understanding confidence reliability through information-source dependence (aKJJ). Methodologically, the proposed VCRU provides a complementary approach to existing uncertainty-based ranking methods, requiring no additional model training and offering a practical way to improve existing uncertainty scores (9Xoh). Experimentally, the reviewers acknowledge the broad evaluation across heterogeneous VLM families and multiple benchmarks (9Xoh, aKJJ), as well as the consistent improvements over existing logit-based ranking baselines (9Xoh, iFFS). The substantial improvement on MMStar, together with evaluations on limited samples, non-MCQ tasks, and additional models, provides useful empirical evidence of generalization (9Xoh). and the presentation is praised for being clear and easy to follow (9Xoh).

The reviews allow us to strengthen our manuscript, and the changes are summarized below:

  1. Added analysis of Question Type in Appendix D.3. [9Xoh]
  2. Added analysis of Perturbation Type and Strength in Appendix D.2. [9Xoh,iFFS]
  3. Added the prompt used for first-token scoring (Figure 12). [9Xoh]
  4. Extended evaluation to two additional free-form benchmarks in Table 19. [9Xoh]
  5. Revised Theoretical analysis in Line 181-217. [9Xoh,aKJJ,iFFS]
  6. Added three empirical findings for Theoretical interpretation in Figure 3. [9Xoh,aKJJ,iFFS]
  7. Revised the problem formulation to define first-token confidence and the two uncertainty measures in Line 129-137. [aKJJ]
  8. Added an evaluation of VCRU on 36 newer VLMs in Appendix D.1. [aKJJ]

For clarity, we highlight the revised part of the manuscript in blue color.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.