- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
VLMs frequently prioritize textual context over visual evidence. The paper investigates how language bias propagates through VLM inference on word completion tasks. The authors split VLM inference into 4 stages: Visual Evidence Identification (S1), Visual Relationship Identification (S2), Misleading Visual Cue Exclusion (S3), and Decision (S4). They also track two main factors: linguistic prior and cross-modal coverage. The authors show that while higher cross-modal coverage helps models see the image better, a strong language prior can still override correct visual clues at the decision stage ("perceptron-decision gap"). They test this on 5 VLMs using prompt-based questions and 500 controlled rewrite tests.
Critical strengths
The paper clearly demonstrates the "Perceptron-Decision Gap" (Table 3, Sec. 3.2). Even when models pass all visual checks (S1-S3), a strong text prior still overrides the visual evidence at the decision stage (S4). This insight is important because it shows models fail not from "visual blindness", but from poor arbitration at the decision boundary, suggesting that simply boosting visual attention is not enough.
Critical weaknesses
-
The paper aims to investigate the mechanisms underlying language bias in VLMs broadly (Abstract, Sec. 1), but all experiments are built solely on a single-word completion task. This evaluation setting limits the extent to which the proposed diagnostic framework and findings can be generalized to other vision-language tasks, such as visual question answering or image-text reasoning, where the interaction between linguistic priors and visual evidence may differ. The authors should test their 4-stage framework on at least one other task with a different output format. Otherwise, the scope of the claims should be narrowed down to word completion rather than VLMs as a whole.
-
The expected answers are always Yes for S1 and S2 and always No for S3, so a yes-bias alone could produce the pattern in Table 1. Balanced probes are needed.
-
Key results are confounded or too thin.
- S2’s expected answer is “Yes” and S3’s is “No”. A general tendency to say yes would by itself make S3 look like the hardest stage, so the headline S3 gap may reflect answer bias rather than misleading-cue exclusion. I didn’t see a control for this.
- “No-bias” cases are defined as the ones where the VLM did use the image to correct itself. Finding that they also do better on visual probes is close to built into the definition.
- The Table 3 drops (−1.8 to −7.8 points) rest on 43-185 cases in the low-prior bin, with no significance tests, and the 0.99 cutoff looks arbitrary.
- The rewrites also change syntax and word order, and they were kept only if they lowered the prior and preserved S1–S3. With no base rate for how often a prior-neutral rewrite would also flip the answer, this shows the effect can happen, not that the prior causes it.
- Table 3 implies only about 7–15 gap cases for most models, so it’s unclear how 100 pairs per model were built. All 10 appendix examples come from Qwen2.5-VL.
- Higher coverage means the caption already says more, so those items may simply be easier. For Qwen the coverage effect isn’t significant ().
- There are also citation and presentation problems:
- Lin et al. 2024a and 2024b are the same paper listed twice.
- If I'm not missing something, some citations don’t support the claims they’re attached to. Jeong et al. 2026 is a dataset-distillation paper. Feng et al. 2023 is about political bias in NLP. PostAlign is cited as evidence that its own approach fails.
- Table 2’s caption promises low- and high-prior results for Qwen that aren’t in the table.
- Gemma 3 12B is missing from Table 1.
- "Perceptron" appears where "perception" is meant.
Questions for authors
- When probing Stages S1-S3, the questions explicitly mention the target and distractor words (for example: "Is soccer being played...?" or "Are the women playing volleyball?"). Does including these words in the prompt accidentally trigger the VLM's language prior before it even looks at the image?
- Can you control for yes-bias? For example, balance S2 and S3 so each has both Yes and No targets, or report each model’s base yes-rate on matched control questions. If the S3 gap mostly disappears, the main diagnostic claim weakens a lot.
- How exactly were the 100 reformulation pairs per model built, given the small number of gap cases? What fraction of attempted rewrites flip the answer, and what is the flip rate for control rewrites that leave the prior roughly unchanged?
- Can you report significance for Table 3? Ideally use a single model with prior and coverage as continuous variables plus their interaction, instead of a single 0.99 cutoff.
- What is the inter-annotator agreement for the coverage labels, and how many annotators labeled each item? Does the coverage effect hold after controlling for item difficulty, such as text-only accuracy or caption length?
- What makes these findings new compared with Mousi et al. (2026) and Nooralahzadeh et al. (2026)?