- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies over-alignment between visual representations and the text manifold in decoder-based VLMs, arguing that excessive alignment injects statistical linguistic bias into visual features and suppresses fine-grained visual information. The authors construct a text-derived subspace using PCA on pretraining-caption token embeddings, and analyze it through cross-dataset subspace similarity, layer-wise alignment, linear probing, and co-occurrence hallucination studies. Based on this analysis, they propose two geometric debiasing approaches: a training-free projection-removal method and a bias-aware projector fine-tuning strategy, both removing visual components aligned with the top text principal components. Experiments on Qwen2.5-VL and LLaVA models across POPE, AMBER, CHAIR, MME, MMBench, VRSBench, and TextCaps show consistent hallucination reduction and performance gains, supported by rank/layer ablations, subspace controls, and qualitative analyses.
Critical strengths
-
The mechanistic analysis is comprehensive. Beyond downstream hallucination metrics, the paper includes cross-dataset subspace similarity, layer-wise alignment, linear probing, co-occurrence hallucination analysis, and logit-lens studies.
-
The experimental evaluation is broad and considers both hallucination reduction and capability preservation. The authors evaluate multiple models and datasets, and also report CHAIR recall to reveal the trade-off between hallucination reduction and information loss.
Critical weaknesses
-
The method relies on an offline global text-PCA subspace, which limits both novelty and adaptivity. The subspace is estimated from caption token embeddings and may depend on the corpus distribution and data quality. Compared with offline subspace methods such as Nullu and sample-adaptive online methods such as HulluEdit, the paper does not clearly establish the unique advantage of using a fixed global text-PCA subspace.
-
The mechanistic interpretation may be overstated. The paper interprets the leading text PCs as hallucination-causing linguistic bias and further treats visual-text over-alignment as a key cause of hallucination. However, the current evidence mainly shows that these directions capture frequent linguistic structure, while the causal link to hallucination remains insufficiently validated.
-
The practical efficiency and offline construction cost are not sufficiently quantified. The paper does not report the time or sample cost required to build the text-PCA subspace, nor actual inference latency. It also lacks a preprocessing-cost comparison with offline subspace methods such as Nullu, making the claimed low-overhead advantage difficult to assess.
Questions for authors
-
How does the proposed fixed text-PCA subspace fundamentally differ from existing offline subspace methods such as Nullu, and what advantage does it offer over sample-adaptive methods such as HulluEdit?
-
What evidence shows that the leading text PCs specifically capture hallucination-causing linguistic bias, rather than generic high-frequency linguistic or anisotropic directions?
-
What are the offline construction cost, required corpus size, and actual inference latency of the proposed method, and how do these costs compare with Nullu or other subspace-based methods?
General Response to All Reviewers
We thank Reviewers NQJ2, YEV7, and oies for their careful and constructive feedback. The reviews raised four main themes: (1) a precise definition of over-alignment, (2) whether the top text PCs are specifically linked to hallucination, (3) the difference from related subspace methods (Nullu, VTI, DeCo, HulluEdit), and (4) practical details, including corpus choice, the selection of k, cost, and interaction with fine-tuning. We added analyses for each, as summarized below. Full results are in the individual responses and will be included in the revision.
1. Defining over-alignment (NQJ2, YEV7, oies).
2. Link between the top text PCs and hallucination (NQJ2, YEV7, oies).
3. Difference from related methods (NQJ2, YEV7, oies).
4. Corpus, choice of k, cost, and fine-tuning (NQJ2, YEV7, oies).
Revisions. We will add all new analyses, update Fig. 2 and replace "first quantitative characterization" and "universal, dataset-agnostic" with claims scoped to what we measured.
We would be grateful if the reviewers could look over the responses and let us know of any remaining concerns before the discussion period ends.