acceptodds
Withdrawn from ICLR 2027

Many Paths, One Coordinate: Semantic Mediation and Readout Dissociation in Vision-Language Hallucinations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–language models can hallucinate through diverse upstream failures, from missing visual evidence to relying on misleading contextual cues. We ask whether these distinct failure paths nevertheless converge on a common internal computation. Across multiple VLMs, we find that object omissions and fabrications become opposite movements along a shared semantic coordinate representing object presence. In Qwen3-VL and InternVL2, causal interventions on unseen clean–error transitions show that independently estimated versions of this coordinate mediate a large fraction of the late-layer behavioral effect, while matched controls have little influence. The semantic mediator is nevertheless distinct from the model's direct output readout: it remains stable across changes in answer surface even when the corresponding readout changes substantially, and removing its sample-specific coupling to the readout nearly eliminates direct behavioral control. Additional composition and trajectory experiments show that heterogeneous perturbations can follow nonlinear upstream paths while converging onto a compact downstream computation. Together, these results suggest a layered mechanism for vision–language hallucination in which many upstream paths converge on a shared semantic state that is subsequently translated into behavior by a distinct low-dimensional readout.

Primary area:
interpretability and explainable AI
Keywords:
ision-Language Models, Object Hallucination, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oYUYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discovers a shared semantic state within VLMs that explains diverse object hallucination patterns. The authors also show that this hidden state is distinct from the VLMs’ direct readout. These results demonstrate a shared hidden state within VLMs that routes diverse hallucination causes to diverse surface forms of the model answers.

Critical strengths

(1) The finding of a shared hidden semantic coordinate that explains different visual hallucination patterns is interesting.

(2) The control studies validating the presence and mechanistic interpretability of such an internal state are also thoroughly designed.

Critical weaknesses

(1) The paper cannot reliably claim "cross-model" signs based on just three specific small VLMs (Qwen3-VL-2B, Qwen2.5-VL-7B and InternVL2-7B), and these interpretations might not hold for larger VLMs. Additionally, apart from Fig. 2, the results in Tab. 2–3 are reported only for the 2B Qwen3-VL. Tab. 4 is also missing variants tested on InternVL. With most analyses done only on Qwen3-VL-2B, the conclusion doesn’t apply across models.

(2) Throughout the paper, only numeric statistics are reported. Since this paper is about a "semantic mediator" for visual hallucinations, one would expect to see qualitative examples demonstrating the effects. Moreover, the experiments are done on binary classification tasks only, where the reasoning of VLMs might be buried under the binary readout label.

Questions for authors

(1) Related to weakness (2), how does the interpretability finding transfer to, e.g., open-ended visual reasoning?

(2) The citation for CountBench is missing (Line 200).

Reviewer_kvShOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the mechanisms behind object hallucination in VLMs. It mainly analyzes the different mechanistic paths that lead to hallucination and finds that these paths converge onto a shared semantic coordinate.

Critical strengths

S1. The paper's analysis of hallucination involves a substantial amount of work.

S2. The proposed framework is valuable. It categorizes hallucinations by their mechanistic pathways and then looks for what these pathways have in common.

Critical weaknesses

W1. For an analysis paper, the experiments are seriously lacking in qualitative results, case studies, statistical analyses and interpretability analyses. Instead, the paper is filled with large quantitative tables, which makes the presentation quality poor. Figure 1 deserves particular mention. It should be the key figure that conveys the motivation and the logic of the work, but in its current form it is very rudimentary and looks like an AI-generated flowchart.

W2. The analysis relies on POPE, a benchmark that is now dated. The reliability and generality of the findings should be validated on a broader range of more recent hallucination benchmarks.

W3. The paper claims that û is a semantic variable encoding whether an object is present, but the evidence does not distinguish it from the model's generic binary affirmative/negative decision. Without such a distinction, û may simply be a re-description of the paper's narrative rather than a genuine novelty.

W4. The evidence for "multiple paths converging on a single coordinate" is far from sufficient and lacks rigor. The current sample sizes are very small, and it is unclear whether the examples were deliberately selected. The analysis should be extended to a larger sample, with confidence intervals reported. In addition, the paper's central conclusion is that "multiple paths converge on a single coordinate", yet the causal tests for each model cover only a single perturbation path.

W5. In its current form, the paper does not meet the ICLR bar in presentation quality, experimental thoroughness, theoretical depth, reliability of the analysis, or writing quality.

W6. The scope of the study does not match the title. Work on hallucination typically covers multiple evaluation dimensions, such as open-ended generation and free-form VQA, which are not addressed here; attribute and relation hallucinations are examined only as binary yes/no judgments.

Questions for authors

The current reference list is too limited. The authors should conduct a more thorough survey of related work on hallucination in MLLMs, which would help improve the quality of this work.

Reviewer_KHS9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether different routes to vision–language hallucination converge on a common internal representation. It studies binary object-presence judgments, contrasting omissions of real objects with fabrications of absent ones. Across several VLMs, these errors develop opposite directions of change along a shared late-layer presence coordinate. The authors estimate this coordinate on separate examples, freeze it, and intervene on unseen correct–error transitions. In Qwen3-VL and InternVL2, the coordinate recovers a substantial fraction of the answer-margin change produced by replacing the full hidden state, while matched controls have much weaker effects.

The paper also distinguishes this semantic coordinate from the output readout—the model's sensitivity to changes that affect its answer. Equivalent answer wordings preserve the coordinate while changing the readout, and the same general pattern appears in binary counting, attribute, and relation judgments in Qwen3-VL. Further experiments show that nonlinear image-perturbation trajectories can coexist with a comparatively simple downstream mapping to the answer. The proposed account is that varied upstream failures converge on a shared semantic state that is subsequently translated into an output.

Critical strengths

The paper offers an interesting account of how different errors can share a common point of influence. Missing an object and inventing an absent object correspond to opposite changes in the model's representation of presence. This connects the two behaviors without assuming they have the same origin. It also explains why strengthening a presence representation can help omissions while making fabrications worse.

The intervention experiments give this account substantial support. A direction estimated from separate examples recovers much of the difference between correct and incorrect judgments on unseen cases, while matched controls have little effect. Replication in another model family, reverse interventions, and transfer to naturally occurring errors strengthen the result. These checks make the common presence representation a credible empirical finding.

Critical weaknesses

The readout analysis supports a more limited conclusion than the shared-coordinate intervention results. Preserving the meaning of a judgment while changing its answer wording shows that semantic representation and output control need not coincide. The evidence is less direct about how these roles interact during the model's computation. Related distinctions appear in Causality ≠ Invariance [1] and TruthLens[2], although they study different variables. The clearest contribution here is the causal role of the shared presence representation; the broader account of how it is translated into an answer should be stated more cautiously.

The scope of the common-coordinate account is less developed than the evidence for its existence. The main controlled transitions concern errors induced by degrading images. The contextual-prior comparison broadens the analysis, but changing the image also changes visual evidence, making it harder to isolate a distinct contextual cause. The additional binary tasks broaden the kinds of judgments studied, rather than resolving this issue about error origins. Examining which existing errors are poorly captured by the coordinate, and what distinguishes them from successful cases, would help define where the proposed explanation applies.

[1]https://arxiv.org/abs/2602.22424 [2]https://arxiv.org/abs/2608.05616

Questions for authors

What characterizes the cases in which intervention along the shared coordinate recovers little of the correct–error difference? Examples or an analysis of these cases would help establish whether the account has identifiable limits across error types and perturbations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.