- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates whether VLM errors arise because the required visual evidence is not encoded (representation failure) or is encoded but not used (utilization failure), and formalizes the share of utilization failures as an estimand with probe-based bounds. It proposes VELens, a logit-lens, rank-based probe that detects object and attribute evidence across LM layers. Evaluated on color, material, and shape questions from the VAW dataset with LLaVA-1.6-7B and Qwen2.5-VL-7B, it estimates that utilization failures account for 84.4% and 70.0% of errors, respectively.
Critical strengths
- The question is well posed and practically relevant: distinguishing missing from unused evidence indicates where improvements should come from.
- The paper proposes an interesting and intuitive approach to diagnosing the source of VLM errors.
- The rank-based probe is lightweight and remains effective in low-resource settings.
Critical weaknesses
-
The connection between Assumption 1 and the noise-based experiment is not convincing. Under Assumption 1, the negative transformation should act on the representation, producing inputs whose representations lack the required evidence, so that the probe's non-detection rate on can serve as a proxy for representation failures. Gaussian noise does not guarantee this: after noising, some inputs may still encode the evidence (corresponding to utilization failures) while others do not, and this mixture is unknown and varies with . As a result, the lower bound is largely controlled by the noise level. Notably, is selected because the answer remains identifiable, yet if humans can still identify the attribute, the representation may still encode it as well. The authors should provide human verification of what the noised images retain, report the bounds as a function of , and consider counterfactual edits that remove the evidence by construction (e.g., recoloring the object). Although such edits change the answer, the proof of Theorem 1 relies only on Assumption 1, not on answer preservation.
-
The estimate lacks the controls needed to interpret it. The paper should report the probe's detection rate on correctly answered samples, where the evidence is almost surely encoded, as an estimate of the probe's sensitivity. In addition, depends on the probe thresholds, yet these thresholds are not reported and no sensitivity analysis over them is provided.
-
The practical claim that the diagnosis can guide error mitigation is not actionable. The probe evaluates , i.e., evidence for the ground-truth answer, and relies on the annotated bounding box to select visual tokens; neither is available at inference time. Thus, per-instance diagnosis and the targeted attention intervention cannot be applied in practice.
-
The experiments are insufficient to support the general conclusion. The study covers only two 7B models, one dataset, three local attributes, and a multiple-choice format, but concludes that utilization is "the dominant bottleneck in VLMs". Supporting this claim would require larger and more recent models, additional datasets, and open-ended question formats; otherwise, the conclusion should be scoped to the evaluated setting.
Questions for authors
Please refer to the weakness.
2 comments
comment 2/2
Regarding the actionability of the proposed diagnosis and intervention, Weakness 3
VELens is designed for annotated evaluation data. As §6.3 states, the attention intervention is a proof of concept testing whether evidence-positive errors are easier to correct by attending to the target region (compared to evidence-negative). The ground-truth answer identifies errors and defines the tested evidence, while the bounding box selects the target-object tokens (at inference time, target bounding boxes can be easily obtained using segmentation models such as SAM 3); the intervention does not supply the correct attribute value.
| Evidence subset | Region | Correction rate |
|---|---|---|
| Evidence-positive | Target | 25.2% |
| Evidence-negative | Target | 12.0% |
| Evidence-positive | Random matched-size | 6.5% ± 0.7% |
Evidence-positive errors are corrected more than twice as often as evidence-negative errors. For the random control, we sampled five matched-size regions per image on the same evidence-positive errors and obtained only correction. This shows that the 25.2% correction rate on the evidence-positive set is not incidental: directing attention specifically to the region where VELens detected the evidence is substantially more likely to correct the answer than directing it elsewhere in the image. The random-region control is added in blue to Appendix D.6.
Regarding the empirical scope of our conclusions: Weakness 4
Our original evaluation examines two models across three distinct attributes using 6,074 natural images from VAW, a widely used and diverse dataset, and we now extend it to Qwen2.5-VL-72B-Instruct. The same qualitative conclusion holds across every evaluated model and attribute. We believe this consistency provides meaningful evidence about current open-weight VLMs in attribute QA, while agreeing with the reviewer that it does not establish the same conclusion for other datasets, tasks, or question formats. We will therefore revise the abstract, discussion, and all statements of our conclusion to explicitly specify the evaluated setup.
To broaden model scale, we evaluated Qwen2.5-VL-72B-Instruct (B parameters) on the same examples, splits, questions, probes, and thresholds as Qwen2.5-VL-7B and LLaVA. Both share the same vision encoder design, while 72B has a much larger language decoder.
| Metric | Color | Material | Shape | Overall |
|---|---|---|---|---|
| Attribute AUC, 72B | 0.899 | 0.853 | 0.734 | 0.849 |
| Object Attention AUC, 72B | 0.970 | 0.976 | 0.983 | 0.976 |
| Error rate, 72B | 23.4% | 21.4% | 31.8% | 24.9% |
| Error rate, 7B | 24.4% | 24.2% | 41.8% | 28.8% |
| Utilization failures, 72B | 60.7% | 64.3% | 64.6% | 63.0% |
Without reconfiguration, VELens achieves overall attribute/object AUCs of 0.849/0.976. The 72B error rate falls from 28.8% to 24.9%, and its utilization-failure share from 70.0% to 63.0%, consistent with the larger decoder using evidence missed by 7B. The central result therefore persists at a substantially larger model scale, while VELens also captures the expected improvement in evidence utilization. We will add the 72B experiment to the revised paper. The corresponding Theorem 1 bounds are currently being computed using the same controlled transformations and will be added with the final results.
comment 1/2
Dear Reviewer j4NR,
Thank you for the clear and constructive comments. We are glad that you found the research question relevant, and VELens’s intuitive, low-resource approach.
Regarding the lower bounds, Weakness 1; Question 1
We agree that Gaussian noise does not guarantee complete evidence removal. However, our analysis does not require this. As stated in §3.2, ψ− is “designed to make the visual evidence relevant to the query less salient,” rather than eliminate it from every representation. Appendix A formalizes how the transformed probe behavior provides a conservative reference; if evidence remains detectable, the resulting lower bound becomes more conservative.
In the paper we selected 30% noise through manual inspection of validation examples: the target signal was visibly weakened while the object and attribute remained identifiable, whereas at higher noise levels the answer was often no longer recoverable. Following the reviewer’s suggestion, we conducted an additional human verification with five independent annotators on 50 randomly sampled images transformed using the paper’s 30% target-region Gaussian noise. For each image, annotators answered “What is the color of the [object]?” using four options: the ground-truth color and three distractors (25% chance). Mean accuracy was 90.0% (95% CI: 84%-95%), and a strict majority selected the correct color for 48/50 images (96%). Thus, the 30% transformation weakens the target-region signal while preserving the color answer in nearly all evaluated images.
To address the reviewer’s sensitivity concern, we also evaluated 10% noise. LLaVA's lower bounds remain above 50% (59.2%–76.2% across attributes), supporting the stability of the majority-utilization conclusion. We added these results to Appendix D.4.
Counterfactual recoloring is complementary but changes the correct answer, falling outside our answer-preserving framework.
Regarding controls for interpreting μ^ϕ, Weakness 2
Following the reviewer’s suggestions, we add two controls: detection rate on correct answers and threshold sensitivity. As stated in Appendix D.4, both probes use threshold 0.5 (we additionally report below the results using thresholds of 0.4 and 0.6 demonstrating that our conclusions remain stable across these choices); we will clarify that it is used for all estimates and bounds on clean and transformed images.
Detection on correct answers and errors:
Correct answers are an approximate sensitivity control because they can also result from guessing or prior knowledge. Joint detection is 8.1 points lower on errors for LLaVA and 10.3 for Qwen, mainly due to the attribute probe; object detection changes little.
Sensitivity to the probe thresholds
We varied both thresholds over {0.4,0.5,0.6}, applying the same thresholds to the clean, X+, and 30%-noise images. Each cell reports the utilization failure estimate [lower bound, upper bound].
LLaVA
Qwen
The conclusion is stable across all nine threshold settings: the lowest pooled lower bound is 69.4% for LLaVA and 52.4% for Qwen. In the revised paper, we clarify in §6.2 and Appendix D.1 that the fixed 0.5 threshold is used for all calculations. We also add the threshold-sensitivity analysis as Tables 7–8 in Appendix D.4, highlighted in purple, holding the probes and transformations from Figure 4 fixed while varying only the thresholds.