acceptodds
Withdrawn from ICLR 2027

When Words Speak Louder than Images: Towards Understanding Language Bias in Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite substantial progress across downstream applications, vision–language models (VLMs) remain susceptible to language bias, often prioritizing linguistic cues over visual evidence and consequently producing incorrect predictions. Prior studies have proposed various approaches to understanding and mitigating language bias in VLMs, yet their findings often conflict due to the difficulty of tracing how language bias propagates within black-box VLMs. Building on the word completion task, we trace how language bias propagates through VLM inference by (1) proposing a diagnostic framework that decomposes the inference process into four distinct yet interdependent stages to trace the propagation of language bias; and (2) examining how two key factors underlying language bias, i.e., linguistic priors and cross-modal coverage, evolve across these stages and ultimately give rise to incorrect predictions. The linguistic prior captures the strength of statistical bias induced by the language model component of a VLM and represents the origin of language bias, whereas cross-modal coverage measures the extent to which linguistic cues cover the visual content. By decomposing inference into four stages and characterizing the interplay between linguistic priors and cross-modal coverage across these stages, we propose a systematic framework for tracing the propagation of language bias throughout the inference process; and uncover the underlying mechanism of language bias by revealing the interplay between linguistic priors and cross-modal coverage.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Language Bias, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J6hyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VLMs frequently prioritize textual context over visual evidence. The paper investigates how language bias propagates through VLM inference on word completion tasks. The authors split VLM inference into 4 stages: Visual Evidence Identification (S1), Visual Relationship Identification (S2), Misleading Visual Cue Exclusion (S3), and Decision (S4). They also track two main factors: linguistic prior and cross-modal coverage. The authors show that while higher cross-modal coverage helps models see the image better, a strong language prior can still override correct visual clues at the decision stage ("perceptron-decision gap"). They test this on 5 VLMs using prompt-based questions and 500 controlled rewrite tests.

Critical strengths

The paper clearly demonstrates the "Perceptron-Decision Gap" (Table 3, Sec. 3.2). Even when models pass all visual checks (S1-S3), a strong text prior still overrides the visual evidence at the decision stage (S4). This insight is important because it shows models fail not from "visual blindness", but from poor arbitration at the decision boundary, suggesting that simply boosting visual attention is not enough.

Critical weaknesses
  1. The paper aims to investigate the mechanisms underlying language bias in VLMs broadly (Abstract, Sec. 1), but all experiments are built solely on a single-word completion task. This evaluation setting limits the extent to which the proposed diagnostic framework and findings can be generalized to other vision-language tasks, such as visual question answering or image-text reasoning, where the interaction between linguistic priors and visual evidence may differ. The authors should test their 4-stage framework on at least one other task with a different output format. Otherwise, the scope of the claims should be narrowed down to word completion rather than VLMs as a whole.

  2. The expected answers are always Yes for S1 and S2 and always No for S3, so a yes-bias alone could produce the pattern in Table 1. Balanced probes are needed.

  3. Key results are confounded or too thin.

  • S2’s expected answer is “Yes” and S3’s is “No”. A general tendency to say yes would by itself make S3 look like the hardest stage, so the headline S3 gap may reflect answer bias rather than misleading-cue exclusion. I didn’t see a control for this.
  • “No-bias” cases are defined as the ones where the VLM did use the image to correct itself. Finding that they also do better on visual probes is close to built into the definition.
  • The Table 3 drops (−1.8 to −7.8 points) rest on 43-185 cases in the low-prior bin, with no significance tests, and the 0.99 cutoff looks arbitrary.
  • The rewrites also change syntax and word order, and they were kept only if they lowered the prior and preserved S1–S3. With no base rate for how often a prior-neutral rewrite would also flip the answer, this shows the effect can happen, not that the prior causes it.
  • Table 3 implies only about 7–15 gap cases for most models, so it’s unclear how 100 pairs per model were built. All 10 appendix examples come from Qwen2.5-VL.
  • Higher coverage means the caption already says more, so those items may simply be easier. For Qwen the coverage effect isn’t significant (βC=0.080β_C = 0.080).
  1. There are also citation and presentation problems:
  • Lin et al. 2024a and 2024b are the same paper listed twice.
  • If I'm not missing something, some citations don’t support the claims they’re attached to. Jeong et al. 2026 is a dataset-distillation paper. Feng et al. 2023 is about political bias in NLP. PostAlign is cited as evidence that its own approach fails.
  • Table 2’s caption promises low- and high-prior results for Qwen that aren’t in the table.
  • Gemma 3 12B is missing from Table 1.
  • "Perceptron" appears where "perception" is meant.
Questions for authors
  1. When probing Stages S1-S3, the questions explicitly mention the target and distractor words (for example: "Is soccer being played...?" or "Are the women playing volleyball?"). Does including these words in the prompt accidentally trigger the VLM's language prior before it even looks at the image?
  2. Can you control for yes-bias? For example, balance S2 and S3 so each has both Yes and No targets, or report each model’s base yes-rate on matched control questions. If the S3 gap mostly disappears, the main diagnostic claim weakens a lot.
  3. How exactly were the 100 reformulation pairs per model built, given the small number of gap cases? What fraction of attempted rewrites flip the answer, and what is the flip rate for control rewrites that leave the prior roughly unchanged?
  4. Can you report significance for Table 3? Ideally use a single model with prior and coverage as continuous variables plus their interaction, instead of a single 0.99 cutoff.
  5. What is the inter-annotator agreement for the coverage labels, and how many annotators labeled each item? Does the coverage effect hold after controlling for item difficulty, such as text-only accuracy or caption length?
  6. What makes these findings new compared with Mousi et al. (2026) and Nooralahzadeh et al. (2026)?
Reviewer_FKChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates language bias in vision–language models (VLMs), namely their tendency to prioritize linguistic cues over visual evidence. Using a word completion task, the authors propose a four-stage diagnostic framework comprising visual evidence identification, visual relationship identification, misleading visual cue exclusion, and decision, and analyze the interplay between linguistic priors and cross-modal coverage across these stages. Experiments show that higher cross-modal coverage is generally associated with better performance on visual-evidence probes. However, strong linguistic priors may still bias the final decision, leading to incorrect completions even when the model correctly answers the S1–S3 probes.

Critical strengths

1.Clear diagnostic framework. The four-stage framework distinguishes visual evidence perception from final decisions, providing a more fine-grained view of language bias than final-answer accuracy alone. 2.Evaluation across multiple models. The authors analyze linguistic priors and cross-modal coverage across five models from three model families, observing several consistent patterns. 3.Evidence from controlled reformulations. The authors construct 500 reformulation pairs, providing examples in which final completions change while cross-modal coverage and successful S1–S3 judgments remain unchanged.

Critical weaknesses

1.Potential confounding in probe design. The expected answers are “Yes” for S1/S2 and “No” for S3. No evaluation with balanced answer polarity is reported, potentially confounding comparisons across stages. 2.Limited causal attribution. As the authors acknowledge, reformulation also changes the textual surface form. Without control reformulations that keep the linguistic prior approximately unchanged, general prompt sensitivity remains a possible alternative explanation. 3.Limited quantitative reporting. Table 3 lacks uncertainty estimates, and aggregate outcomes for the 500 reformulation pairs would clarify the consistency of decision changes within the selected set. 4.Unclear annotation reliability. The cross-modal coverage annotation pipeline is described, but neither inter-annotator agreement nor the sensitivity of the results to the granularity of visual cues is reported.

Questions for authors

1.Clarification regarding diagnostic validity: Do the S3 performance gap between biased and no-bias cases and the occurrence of incorrect final completions despite correct S1–S3 judgments persist when probe wording is varied and the expected “Yes/No” answers are balanced? 2.Clarification regarding the reformulation experiment: Could the authors report the numbers of candidate reformulations generated and retained after filtering, together with aggregate statistics and uncertainty estimates for changes in final completions across all retained reformulation pairs? Are there meaning-preserving control reformulations that keep the linguistic prior approximately unchanged, to distinguish effects of linguistic priors from general prompt sensitivity?

Reviewer_8qWXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies language-biased predictions in visual word completion. Noun-target diagnostics identify misleading-cue rejection as a prominent difficulty. The authors also construct 100 selected caption-reformulation pairs per model, preserving coverage and diagnostic success while weakening competing priors, to demonstrate cases where final completions change.

Critical strengths
  1. The diagnostic separates useful failure modes beyond final-answer accuracy.

  2. Several complementary observations support investigating prior-sensitive behavior. Reformulation witnesses provide concrete cases where answers change while coverage and probe success remain fixed.

Critical weaknesses
  1. The probe design does not yet separate the proposed diagnostic failure from question-format and response-polarity effects. S2 asks true relationships, with expected answers Yes, whereas S3 swaps the same entities/concepts into false relationships, with expected answers No. The large S3 gap can reflect difficulty rejecting propositions or a Yes-response tendency. Correct responses to separate, answer-suggesting probes do not establish that the corresponding information was available or used during the original completion. Balanced true/false controls across stages, or matched paraphrases or answer-format controls would materially strengthen the central conclusion.

  2. Coverage depends on which cues annotators count and their relevance judgments; Does not report annotator counts, agreement, or sensitivity to alternative valid cue sets. Table 2 reports coefficients/significance but no confidence intervals or controls for caption length, target type, or item/image difficulty.

Questions for authors
  1. Does the S3 disadvantage persist when truth polarity and answer format are balanced across stages, for example with matched true/false propositions for both S2 and S3 or equivalent non-Yes/No probes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.