acceptodds
Under review as a conference paper at ICLR 2027

When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

Abstract

Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input. We investigate the root causes of these failure modes with a mechanistic analysis focusing on the decoder-based VLMs. We trace these failure modes to a geometric over-alignment: to bridge the modality gap required by attention mechanisms, decoder-based VLMs over-align visual embeddings with the text manifold, injecting a statistical linguistic bias that systematically overshadows fine-grained visual evidence. While prior work either aggressively closes this gap or suppresses hallucinations through expensive black-box decoding strategies, none addresses the underlying geometric cause. We provide the first quantitative characterization of this over-alignment, demonstrating that linguistic bias concentrates in the top principal components of a universal, dataset-agnostic text subspace. Building on this insight, we propose two complementary remedies: a training-free inference strategy and a bias-aware fine-tuning paradigm, both of which explicitly project out this subspace from visual representations. Our methods significantly reduce hallucinations across POPE, CHAIR, and AMBER benchmarks, and improve CLAIR scores on long-form captioning tasks, with the training-free variant adding no computational overhead over the base model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Modality Gap, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oiesOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies over-alignment between visual representations and the text manifold in decoder-based VLMs, arguing that excessive alignment injects statistical linguistic bias into visual features and suppresses fine-grained visual information. The authors construct a text-derived subspace using PCA on pretraining-caption token embeddings, and analyze it through cross-dataset subspace similarity, layer-wise alignment, linear probing, and co-occurrence hallucination studies. Based on this analysis, they propose two geometric debiasing approaches: a training-free projection-removal method and a bias-aware projector fine-tuning strategy, both removing visual components aligned with the top text principal components. Experiments on Qwen2.5-VL and LLaVA models across POPE, AMBER, CHAIR, MME, MMBench, VRSBench, and TextCaps show consistent hallucination reduction and performance gains, supported by rank/layer ablations, subspace controls, and qualitative analyses.

Critical strengths
  1. The mechanistic analysis is comprehensive. Beyond downstream hallucination metrics, the paper includes cross-dataset subspace similarity, layer-wise alignment, linear probing, co-occurrence hallucination analysis, and logit-lens studies.

  2. The experimental evaluation is broad and considers both hallucination reduction and capability preservation. The authors evaluate multiple models and datasets, and also report CHAIR recall to reveal the trade-off between hallucination reduction and information loss.

Critical weaknesses
  1. The method relies on an offline global text-PCA subspace, which limits both novelty and adaptivity. The subspace is estimated from caption token embeddings and may depend on the corpus distribution and data quality. Compared with offline subspace methods such as Nullu and sample-adaptive online methods such as HulluEdit, the paper does not clearly establish the unique advantage of using a fixed global text-PCA subspace.

  2. The mechanistic interpretation may be overstated. The paper interprets the leading text PCs as hallucination-causing linguistic bias and further treats visual-text over-alignment as a key cause of hallucination. However, the current evidence mainly shows that these directions capture frequent linguistic structure, while the causal link to hallucination remains insufficiently validated.

  3. The practical efficiency and offline construction cost are not sufficiently quantified. The paper does not report the time or sample cost required to build the text-PCA subspace, nor actual inference latency. It also lacks a preprocessing-cost comparison with offline subspace methods such as Nullu, making the claimed low-overhead advantage difficult to assess.

Questions for authors
  1. How does the proposed fixed text-PCA subspace fundamentally differ from existing offline subspace methods such as Nullu, and what advantage does it offer over sample-adaptive methods such as HulluEdit?

  2. What evidence shows that the leading text PCs specifically capture hallucination-causing linguistic bias, rather than generic high-frequency linguistic or anisotropic directions?

  3. What are the offline construction cost, required corpus size, and actual inference latency of the proposed method, and how do these costs compare with Nullu or other subspace-based methods?

Reviewer_YEV7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on mitigating hallucinations in decoder-based vision-language models (VLMs). These hallucinations can arise when models rely on learned linguistic correlations (e.g., words that frequently co-occur) rather than visual evidence in the image.

The proposed method uses Principal Component Analysis (PCA) to identify dominant directions in text embeddings that capture statistical patterns in language. It then removes these directions from visual representations to reduce language-driven hallucinations.

Critical strengths

Figure 3 provides interesting empirical evidence supporting the motivation behind the proposed method.

The experimental results demonstrate that this simple, ad hoc, and easy-to-implement method can effectively reduce hallucinations. The findings may also inspire future research in this direction.

Critical weaknesses

Clarification of terminology: The paper introduces the term over-alignment without clearly defining it. If this is not a commonly established term, the authors should provide an explicit definition. From my understanding, it refers to a situation where VLMs rely excessively on learned linguistic correlations (e.g., frequently co-occurring words) rather than actual visual evidence when interpreting images. If this interpretation is correct, the authors should clarify it in the paper.

Organization of Section 3: Section 3 would benefit from a brief overview of the proposed framework at the beginning. Without an overview, it is easy for readers to lose track of how the different components fit together.

Theoretical justification and choice of (k): Although I find the proposed framework valuable, some of the claims appear overstated. There is no theoretical justification establishing that the leading principal components necessarily capture harmful linguistic biases responsible for hallucinations, nor is there a guarantee that removing these components will reduce hallucinations. In fact, removing these directions may also discard useful information, suggesting a potential trade-off. A more detailed discussion of this trade-off, how much of the identified subspace actually corresponds to harmful linguistic biases, and how the hyperparameter (k) should be selected would strengthen the paper.

Interaction with fine-tuning: The fine-tuning scheme introduces a potential theoretical limitation: the model may learn to encode the same linguistic biases in alternative directions, effectively bypassing the removed subspace. In other words, fine-tuning could allow the model to recover the information that the proposed method aims to suppress. It would be valuable to discuss how the proposed intervention interacts with fine-tuning and whether the method remains effective after the model adapts.

Distinction from Nullu (https://arxiv.org/abs/2412.13817): The proposed methodology appears closely related to Nullu, yet the paper does not provide a sufficiently detailed discussion of their differences. From my understanding, the main distinction is that the proposed method uses an unsupervised intervention, whereas Nullu relies on a different mechanism for identifying the subspace to remove. The authors should explicitly compare the two approaches and clarify the methodological novelty of their work.

Questions for authors

See above

Reviewer_NQJ2OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates object hallucination in decoder-based Vision-Language Models (VLMs) through a geometric lens. The authors claim that decoder-based VLMs "over-align" visual embeddings to a text manifold, injecting statistical linguistic bias that overshadows visual evidence. They characterize this bias via PCA over caption token embeddings, argue the top principal components (PCs) carry the linguistic bias, and propose two interventions: (1) a training-free inference-time projection that removes the top-k text-PC components from visual embeddings, and (2) a bias-aware fine-tuning paradigm that applies the same projection at the projector output during SFT. Experiments on Qwen2.5-VL-7B, LLaVA-1.5-7B, and LLaVA-1.5-13B report improvements on POPE, AMBER, CHAIR, MME, MMBench, and CLAIR-based long-form captioning.

Critical strengths
  • The training-free intervention (subtracting top text-PC projections from vision embeddings) is easy to implement, adds no inference overhead, and is broadly compatible with decoder-based VLMs.
  • Evaluation spans three models, several hallucination benchmarks (POPE, AMBER, CHAIR), general-capability benchmarks (MME, MMBench), and long-form captioning (VRSBench, TextCaps). The paper is honest in places about not being uniformly best (e.g., SID and DeCo win some metrics on LLaVA-1.5-7B).
Critical weaknesses
  • Removing top text-aligned directions from visual features is closely related to VTI (Li et al., 2025), Nullu, and DeCo — all of which are already baselines in Table 1 and operate on the same principle of editing visual representations to reduce text-prior dominance. The paper does not clearly articulate what its method does that these do not (e.g., Nullu also identifies and nulls a hallucination-related subspace). The empirical margins over these baselines are small (often within or near the reported standard deviations), so the contribution reduces to a slightly different choice of subspace with a mechanistic story attached.

  • The text manifold is built from "captions directly from the pretraining dataset" — but for LLaVA-1.5 and Qwen2.5-VL the actual pretraining captions are not publicly available; what was actually used, how many captions, and how sensitive results are to this choice is relegated to an appendix or unspecified.

  • Table 3 is explicitly noted to exclude the actual proposed top-text-PC removal condition, and uses a different baseline from Table 1 — making it impossible to tell from the controls whether the proposed subspace is better than bottom-PC removal, which in Table 3 appears to improve several hallucination metrics comparably. This undercuts the specificity claim that the top text PCs are the culprit. Similarly, the claim that removing top PCs reduces hallucination "by over 50% on average" (Fig. 3c) is based on a single hand-picked co-occurrence probe ("dining table"), which is anecdotal.

  • Section 3.1's "mathematical necessity of alignment" argument (attention requires a shared coordinate system) is trivially true and does not support the stronger claim that current training produces over-alignment; no formal or quantitative definition of "over-alignment" versus "necessary alignment" is ever given, so the paper's central concept remains rhetorically defined.

  • Figure 2's axis labels and legends are partially garbled in the submission; the DeCo/ICT rows of Table 1 (13B) have typographical corruption; several claims ("first quantitative characterization", "universal, dataset-agnostic") are overstated relative to the evidence. The AI-use statement is appreciated but the writing shows signs of unedited LLM polish in places (e.g., duplicated periods, "as mentioned in paper").

Questions for authors

Please refer to weaknesses.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewers NQJ2, YEV7, and oies for their careful and constructive feedback. The reviews raised four main themes: (1) a precise definition of over-alignment, (2) whether the top text PCs are specifically linked to hallucination, (3) the difference from related subspace methods (Nullu, VTI, DeCo, HulluEdit), and (4) practical details, including corpus choice, the selection of k, cost, and interaction with fine-tuning. We added analyses for each, as summarized below. Full results are in the individual responses and will be included in the revision.

1. Defining over-alignment (NQJ2, YEV7, oies).

  • We now define over-alignment in terms of the modality gap. A decoder VLM must partly close this gap so that attention can operate; we call this necessary alignment. Over-alignment is closing the gap too far along the dominant directions of caption text.
  • We measure it as the excess alignment of real vision tokens over random vectors that share their mean and spread.
  • When we train a projector from scratch between a frozen vision encoder and a frozen LLM, this excess is absent at initialization and grows during training. This suggests that projector training creates the over-alignment.
  • Section 3.1 will be shortened to a short motivation that points to this definition. Section 3 will open with an overview of the framework.

2. Link between the top text PCs and hallucination (NQJ2, YEV7, oies).

  • Matched-subspace controls. We added our method to the control table. Removing the top text PCs reduces hallucination more than removing random, bottom, or middle subspaces of the same dimension, on both CHAIR and AMBER.
  • Population-level analysis. Apart from the "dining table" probe, we added an analysis of all (image, absent object) pairs in CHAIR. Hallucination rises steadily with the object's linguistic co-occurrence prior. Our removal reduces hallucinations in the high-prior group, while random removal does not.
  • Not generic anisotropy. Every removal is centered on the same mean, so the effect cannot come from removing the modality offset or generic anisotropy.
  • Scope of the claim. We do not claim a theoretical guarantee. The claim is empirical and controlled, and we will tone down the overstated wording.

3. Difference from related methods (NQJ2, YEV7, oies).

  • A mechanistic contribution. Prior methods find their subspace from downstream symptoms: hallucinated vs. truthful pairs, perturbations, or decoding signals. We derive ours from the geometric cause. We show where it originates (projector training), what it encodes (the linguistic prior), and that removing it, unlike any equal-sized control, reduces prior-driven hallucination.
  • Data and placement. Our subspace comes from unsupervised PCA of caption text alone, with no images, hallucination labels, or paired data. It acts only on vision tokens at the projector output and can be used inside the training loop (bias-aware SFT).
  • Comparison with Nullu. We re-implemented Nullu's HalluSpace on the same model. The two subspaces are nearly orthogonal, and their logit-lens readouts differ: ours decodes to function words and frequent caption nouns, Nullu's to descriptive and spatial content.
  • Comparison with HulluEdit. HulluEdit edits output-side states at every decoding step with per-token SVDs. We act once, at the vision–language interface where the over-alignment is created, so the two approaches are complementary.

4. Corpus, choice of k, cost, and fine-tuning (NQJ2, YEV7, oies).

  • Corpus and k. We specify the caption source, which is COCO train captions disjoint from all evaluation images, and report sensitivity to corpus size and domain. A few thousand captions suffice. The choice of k scales roughly with hidden size, and removing too many directions begins to hurt recall. We discuss this trade-off explicitly.
  • Cost. Building the subspace takes under two minutes of text-only forward passes plus one SVD. The removal folds exactly into the projector weights, so measured inference latency and FLOPs are unchanged. Language-modeling perplexity is preserved.
  • Interaction with fine-tuning. Fine-tuning cannot place signal in the removed directions, but it can partly shift its updates to neighboring directions. With a narrow removed band, bias-aware SFT improves both in-domain captioning and out-of-domain hallucination. With a wide band, out-of-domain hallucination worsens. We report this as a limitation and recommend a narrow band during training, with any wider removal applied only at inference.

Revisions. We will add all new analyses, update Fig. 2 and replace "first quantitative characterization" and "universal, dataset-agnostic" with claims scoped to what we measured.

We would be grateful if the reviewers could look over the responses and let us know of any remaining concerns before the discussion period ends.

AuthorsOpenReview ↗

Polite Nudge

We thank all reviewers again for their thoughtful feedback. We have added additional experiments, and clarified the contribution of the work which have thus made the positioning of the paper much stronger.

As the discussion period ends soon, we would be grateful if the reviewers could read our responses and, if their concerns have been addressed, consider updating their scores. We are happy to answer any remaining questions before the deadline.

Thank you for your response. While the authors have supplemented relevant experiments, definitions and analyses, the core concerns have only been partially addressed. The overall paradigm of this method is similar to existing representation‑editing baselines with limited empirical gains. Causal evidence for over‑alignment remains indirect, its quantitative definition depends on the chosen baseline, and the original manuscript suffers from risks of cherry‑picked examples and overstated claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.