- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates how vision-language models transform different semantic edits into visually grounded truth judgments. Using balanced question quartets that vary negation and object identity, the authors report a transition from predominantly edit-specific representations in early layers to a shared truth-related component in intermediate layers. The main contribution is a mechanistic account linking this component to answer formation, together with an activation intervention for correcting inconsistent responses. The study evaluates three VLMs on RePOPE and AMBER. Validation includes cross-edit probing, attention and MLP activation patching, truth-axis decomposition, localized visual-evidence removal, reversed response codebooks, and frozen cross-dataset transfer. Additional experiments examine visual-only changes, paraphrases, and color attributes. Finally, source-calibrated interventions improve accuracy on the target dataset without target-label fitting; the strongest result raises LLaVA’s AMBER accuracy from 79.0% to 87.6%, correcting 103 errors without introducing new ones.
Critical strengths
++ Interesting mechanistic finding. The emergence of a shared truth-related component across different semantic edits offers a useful perspective on how VLMs combine language with visual evidence.
++ Clear empirical effects. The interventions produce measurable behavioral improvements, notably raising LLaVA’s accuracy from 79.0% to 87.6%, supporting the functional relevance of the identified direction.
Critical weaknesses
-- The relationship to purely linguistic truth representations remains unexplored. I find the central observation interesting, but would appreciate a deeper investigation of whether the identified direction is shared with truth directions derived from purely linguistic assertions within the same VLM. A controlled comparison, ideally using matched propositions and cross-modal transfer tests, would help clarify the scope and interpretation of the proposed mechanism. This is my primary concern and an important factor in my acceptance recommendation. If the authors can provide convincing experiments during rebuttal that characterize this relationship—whether shared or distinct—I would be open to raising my score.
-- The limited complexity of the evaluated judgments leaves broader implications unclear. The experiments primarily address object existence and negation, with a small color-attribute extension. Given the promising starting point, I believe investigating more demanding reasoning tasks could reveal substantially richer findings. One potentially useful resource is the contrastive VQA pairing framework underlying VisCoR-55K, introduced in Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs, ICLR2026. Its pairing of semantically similar questions with different images and corresponding answers could help extend the analysis beyond simple existence judgments.
To clarify the relative importance of these points: the first is central to my current recommendation. The second is primarily an opportunity to strengthen the paper toward oral-level distinction or broader impact. I hope this distinction helps the authors prioritize their rebuttal efforts.
Questions for authors
See the weaknesses part.
6 comments
Thank you for the prompt and thorough experiments. These results convincingly address my primary concern regarding whether the truth representation is shared across or distinct between modalities. However, two minor points remain slightly puzzling: Q1-Model Substitution: Qwen3-VL-4B from the main paper is replaced by InternVL2-8B here. Could you clarify this change and maintain model consistency in the revision? Q2-Directional Asymmetry: In L18 probe transfer, Qwen3-VL-2B and LLaVA-1.5-7B show strong Text → Vision but weak Vision → Text accuracy (despite strong Vision → Text causal effects), while InternVL2-8B shows the opposite trend. I hope you can include these experiments and briefly discuss these two nuances in the revised manuscript to benefit future research. Given these solid results, I am already leaning toward raising my score to 3, and I look forward to your follow-up experiments on the second concern.
Response to W1
Thank you for your response.
- The reason we added InternVL2 is that we wanted to test whether our findings also generalize to a different architecture. We have also added Qwen3-VL-4B back here. We will include these results in the paper.
Qwen3-VL-4B
Cosine similarity
| Model | L14 | L16 | L18 | L24 |
|---|---|---|---|---|
| Qwen3-VL-4B | 0.111 | 0.372 | 0.497 | 0.755 |
| Model | Vision vs. text with scene facts | Vision vs. question only |
|---|---|---|
| Qwen3-VL-4B | 0.497 | 0.218 |
Cross-modal probe transfer
| Model | Text → Vision | Vision → Text | Vision → Vision | Text → Text |
|---|---|---|---|---|
| Qwen3-VL-4B | 97.38% | 78.50% | 97.38% | 100.00% |
Causal transfer
| Layer | Text → Vision | Vision → Vision | Random → Vision | Vision → Text | Text → Text | Random → Text |
|---|---|---|---|---|---|---|
| L24 | +12.49 | +11.39 | −1.33 | +15.18 | +16.29 | −1.26 |
- Directional asymmetry
We further examined InternVL2. Text → Vision has an AUROC of 0.979 despite an accuracy of only 75.00%: the probe ranks true and false statements well but classifies too many as true. Correcting the score offset using training scenes only, while keeping the probe weights fixed, raises accuracy to 92.13%.
This suggests that the observed directional asymmetry is partly due to the probe's decision boundary, instead of solely reflecting a difference between visual and textual representations.
Furthermore, intervening along the identified truth direction improves LLaVA's accuracy from 79.0% to 87.6%.
Response to Reviewer gWa7
Thank you for your suggestion. We hope these experiments address your first concern. We will try to address you second concern next.
We added two experiments using the same 200 scenes in RePOPE.
1. Matched visual–textual comparison
Each scene was presented either as an image or as an explicit textual description without an image. Both modalities received identical A/B/C/D questions. We independently estimated $ d_l=\mathbb{E}[h_l\mid A,D]-\mathbb{E}[h_l\mid B,C] $ and compared directions across layers.
Cosine similarity between visual and textual truth directions
Table 1 shows the text and image directions become more similar in the middle and later layers across models.
In table 2, the second column reports the cosine similarity between truth directions from visual inputs and textual descriptions of the same scenes. The third column reports truth directions from visual inputs with directions from the questions alone (without an image or a scene description). This suggests that the similarity depends on the scene information, rather than just the shared question wording.
2. Cross-modal transfer
We further tested whether truth representations learned from textual descriptions transfer to visual inputs, and vice versa.
We trained a linear truth probe in one modality and tested it in the other. We used five-fold splits by scene, keeping all questions and both modalities of each scene in the same fold. Within-modality tests provide reference accuracies. The following results are at L18.
Truth probes transfer across modalities, although transfer accuracy differs by model and direction.
Causal transfer. We estimated truth directions from 50 separate training scenes and tested them on the 200 held-out scenes. All interventions used L18, with equal perturbation norms within each model.
We found causal effects in both directions across all three models. Cross-modal effects were similar to within-modality effects and much larger than those of the tested random controls.
The experiments suggest that visual and textual truth representations are partly shared