acceptodds
Under review as a conference paper at ICLR 2027

Different Semantic Edits Converge to Grounded Truth in Vision-Language Models

Abstract

Vision-language models (VLMs) must determine how a semantic change alters the truth of a proposition given visual evidence, rather than respond to surface cues such as negation or object identity. Yet how this grounded truth computation emerges inside a VLM remains unclear. We study this question using answer-balanced natural-image quartets in which negation and object substitution each reverse the correct answer in both directions. Across three VLMs and two natural-image benchmarks, we identify a common edit-to-truth transition: early representations preserve edit-specific structure, whereas intermediate layers organize distinct edits according to the truth change required by the image. This shared representation is both causally answer-relevant and visually grounded. At the transition, a one-dimensional truth-axis projection retains over 80% of the full attention-swap effect for both edits, while the orthogonal residual has little effect; the same geometric progression also appears when only the visual evidence changes. Localized removal of visual evidence suppresses the truth representation and its causal effect but largely preserves linguistic polarity. The resulting representation follows proposition truth rather than the literal output symbol under response-code reversal, and frozen RePOPE→AMBER probes transfer without target fitting, reaching 91–93% accuracy over all target images. Finally, source-derived truth directions selectively repair naturally occurring errors: on LLaVA, they correct 103 AMBER errors without introducing new ones, raising accuracy from 79.0% to 87.6%. Together, these results reveal a shared grounded-truth stage through which distinct semantic edits are transformed into visually grounded decisions in VLMs.

Primary area:
interpretability and explainable AI
Keywords:
Vision-language models, Visual grounding, Truth representations

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_gWa7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates how vision-language models transform different semantic edits into visually grounded truth judgments. Using balanced question quartets that vary negation and object identity, the authors report a transition from predominantly edit-specific representations in early layers to a shared truth-related component in intermediate layers. The main contribution is a mechanistic account linking this component to answer formation, together with an activation intervention for correcting inconsistent responses. The study evaluates three VLMs on RePOPE and AMBER. Validation includes cross-edit probing, attention and MLP activation patching, truth-axis decomposition, localized visual-evidence removal, reversed response codebooks, and frozen cross-dataset transfer. Additional experiments examine visual-only changes, paraphrases, and color attributes. Finally, source-calibrated interventions improve accuracy on the target dataset without target-label fitting; the strongest result raises LLaVA’s AMBER accuracy from 79.0% to 87.6%, correcting 103 errors without introducing new ones.

Critical strengths

++ Interesting mechanistic finding. The emergence of a shared truth-related component across different semantic edits offers a useful perspective on how VLMs combine language with visual evidence.

++ Clear empirical effects. The interventions produce measurable behavioral improvements, notably raising LLaVA’s accuracy from 79.0% to 87.6%, supporting the functional relevance of the identified direction.

Critical weaknesses

-- The relationship to purely linguistic truth representations remains unexplored. I find the central observation interesting, but would appreciate a deeper investigation of whether the identified direction is shared with truth directions derived from purely linguistic assertions within the same VLM. A controlled comparison, ideally using matched propositions and cross-modal transfer tests, would help clarify the scope and interpretation of the proposed mechanism. This is my primary concern and an important factor in my acceptance recommendation. If the authors can provide convincing experiments during rebuttal that characterize this relationship—whether shared or distinct—I would be open to raising my score.

-- The limited complexity of the evaluated judgments leaves broader implications unclear. The experiments primarily address object existence and negation, with a small color-attribute extension. Given the promising starting point, I believe investigating more demanding reasoning tasks could reveal substantially richer findings. One potentially useful resource is the contrastive VQA pairing framework underlying VisCoR-55K, introduced in Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs, ICLR2026. Its pairing of semantically similar questions with different images and corresponding answers could help extend the analysis beyond simple existence judgments.

To clarify the relative importance of these points: the first is central to my current recommendation. The second is primarily an opportunity to strengthen the paper toward oral-level distinction or broader impact. I hope this distinction helps the authors prioritize their rebuttal efforts.

Questions for authors

See the weaknesses part.

6 comments
AuthorsOpenReview ↗

Response to Reviewer gWa7

Thank you for your suggestion. We hope these experiments address your first concern. We will try to address you second concern next.

We added two experiments using the same 200 scenes in RePOPE.

1. Matched visual–textual comparison

Each scene was presented either as an image or as an explicit textual description without an image. Both modalities received identical A/B/C/D questions. We independently estimated $ d_l=\mathbb{E}[h_l\mid A,D]-\mathbb{E}[h_l\mid B,C] $ and compared directions across layers.

Cosine similarity between visual and textual truth directions

ModelL14L16L18L24
Qwen3-VL-2B0.2990.7040.6900.721
LLaVA-1.5-7B0.4610.5700.7520.786
InternVL2-8B0.3590.3500.5750.656

Table 1 shows the text and image directions become more similar in the middle and later layers across models.

ModelVision vs. text with scene factsVision vs. question only
Qwen3-VL-2B0.6900.267
LLaVA-1.5-7B0.752−0.143
InternVL2-8B0.5750.247

In table 2, the second column reports the cosine similarity between truth directions from visual inputs and textual descriptions of the same scenes. The third column reports truth directions from visual inputs with directions from the questions alone (without an image or a scene description). This suggests that the similarity depends on the scene information, rather than just the shared question wording.

2. Cross-modal transfer

We further tested whether truth representations learned from textual descriptions transfer to visual inputs, and vice versa.

We trained a linear truth probe in one modality and tested it in the other. We used five-fold splits by scene, keeping all questions and both modalities of each scene in the same fold. Within-modality tests provide reference accuracies. The following results are at L18.

ModelText → VisionVision → TextVision → VisionText → Text
Qwen3-VL-2B95.25%65.88%96.63%100.00%
LLaVA-1.5-7B88.13%55.75%93.50%100.00%
InternVL2-8B75.00%91.38%93.38%100.00%

Truth probes transfer across modalities, although transfer accuracy differs by model and direction.

Causal transfer. We estimated truth directions from 50 separate training scenes and tested them on the 200 held-out scenes. All interventions used L18, with equal perturbation norms within each model.

We found causal effects in both directions across all three models. Cross-modal effects were similar to within-modality effects and much larger than those of the tested random controls.

ModelText → VisionVision → VisionRandom → VisionVision → TextText → TextRandom → Text
Qwen3-VL-2B+7.67+9.04+0.31+15.44+14.48+0.59
LLaVA-1.5-7B+3.46+2.89+0.06+4.08+4.75+0.08
InternVL2-8B+3.21+3.26−0.04+1.88+2.17+0.01
(Values are differences in the Yes–No logit margin between positive and negative interventions, not accuracy gains.)

The experiments suggest that visual and textual truth representations are partly shared

Thank you for the prompt and thorough experiments. These results convincingly address my primary concern regarding whether the truth representation is shared across or distinct between modalities. However, two minor points remain slightly puzzling: Q1-Model Substitution: Qwen3-VL-4B from the main paper is replaced by InternVL2-8B here. Could you clarify this change and maintain model consistency in the revision? Q2-Directional Asymmetry: In L18 probe transfer, Qwen3-VL-2B and LLaVA-1.5-7B show strong Text → Vision but weak Vision → Text accuracy (despite strong Vision → Text causal effects), while InternVL2-8B shows the opposite trend. I hope you can include these experiments and briefly discuss these two nuances in the revised manuscript to benefit future research. Given these solid results, I am already leaning toward raising my score to 3, and I look forward to your follow-up experiments on the second concern.

AuthorsOpenReview ↗

Response to W1

Thank you for your response.

  1. The reason we added InternVL2 is that we wanted to test whether our findings also generalize to a different architecture. We have also added Qwen3-VL-4B back here. We will include these results in the paper.

Qwen3-VL-4B

Cosine similarity

ModelL14L16L18L24
Qwen3-VL-4B0.1110.3720.4970.755
ModelVision vs. text with scene factsVision vs. question only
Qwen3-VL-4B0.4970.218

Cross-modal probe transfer

ModelText → VisionVision → TextVision → VisionText → Text
Qwen3-VL-4B97.38%78.50%97.38%100.00%

Causal transfer

LayerText → VisionVision → VisionRandom → VisionVision → TextText → TextRandom → Text
L24+12.49+11.39−1.33+15.18+16.29−1.26
  1. Directional asymmetry

We further examined InternVL2. Text → Vision has an AUROC of 0.979 despite an accuracy of only 75.00%: the probe ranks true and false statements well but classifies too many as true. Correcting the score offset using training scenes only, while keeping the probe weights fixed, raises accuracy to 92.13%.

This suggests that the observed directional asymmetry is partly due to the probe's decision boundary, instead of solely reflecting a difference between visual and textual representations.

Furthermore, intervening along the identified truth direction improves LLaVA's accuracy from 79.0% to 87.6%.

Reviewer_CxENOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how different interventions, such as negation and object substitution, impact visually grounded representations in VLMs through layer-wise analysis.

Critical strengths

Understanding how VLMs integrate visual and linguistic information is an important and interesting research topic.

Critical weaknesses

W1. The paper is generally hard to read and follow, especially for readers who are not familiar with the unexplained terms or concepts.

  • In the introduction, many terms are used but not explained, such as what an "edit"is (probably the most important concept), as well as concepts including "compositional reasoning", "semantic operations", "decodability", "grounded proposition truth", "linguistic polarity", "pending output symbo". In line 53, it is hard to understand what an "edit identity", a "response-code", and a "proposition truth" are. In lines 54-56, it's hard to follow what "early change vectors", "answer-direction classifiers", and "decodable edit identity" are.
  • The related work introduces prior works associated with some of the above concepts, but do not clearly explain their definitions and motivations. I cannot find discussion on the importance of the listed techniques in the related work.
  • Another major concern is: from the introduction to the method section I cannot easily identify the objectives of the work, thus, I am unable to fully appreciate the designed evaluation and the result claims. In lines 65-68, the paper states "On LLaVA, internal displacement fixes 103 errors without introducing new ones, raising accuracy from 79.0% to 87.6%", which seemingly implies one of the objective is to fix VLMs' errors. If this is the case, the work should better situate itself in the broader literature of model editing, which aims at targeted error-correction for LLMs and VLMs.

W2. The layer-wise analysis does not sufficiently account for the multimodal architecture of VLMs.

  • The paper primarily analyzes representation changes from early to late language decoder layers (e.g., Figure 2). However, VLMs consist of distinct components, including a vision encoder, a multimodal projector/merger, and language decoder blocks. Since the paper's main claim concerns the emergence of visually grounded truth representations, I would expect a more architecture-aware analysis that distinguishes the roles of these components.
  • Although the authors investigate visual evidence removal in Section 5.1, this experiment does not directly explain where or how visual and linguistic representations interact. Additional analysis or discussion of the vision encoder, projector/merger, and language decoder would help substantiate the proposed mechanism.
Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer CxEN

Thanks for your feedback. We hope the following clarifications and additional experiments address your concerns. We kindly ask you to reconsider your assessment of the paper. If you have any further questions, please feel free to ask.

Response to W1.

We apologize that the motivation and terminology were not sufficiently clear. We agree that the research question should be stated more explicitly and key terms should be defined upon first use. We will revise these parts in the revised paper.

We clarify the main terms below and will define them at first use in the revised paper.

TermMeaning
EditChanging the question by adding negation or replacing an object word.
Linguistic polarityWhether the question is positive or negative.
Grounded proposition truthWhether the statement is true given the image.
Response codeThe mapping between answers and output symbols, such as A/B.
DecodabilityWhether a probe can predict information from hidden states.

We clarify our main objective here. Our goal is to understand whether VLMs use a common internal representation of truth across negation and object substitution, and whether this representation influences their answers.

We use the repair experiment to test whether intervening on this representation can correct errors. We change activations while keeping model weights fixed. We will also explain the relationship to model editing in the related work.

Response to W2

We added experiments to figure out the role of each component.

1. Vision encoder: object information is already available.

We trained probes using visual features to predict each object's presence. On 200 multi-object images from COCO, we obtained the following results:

ModelPresent object 1 (n=200)Present object 2 (n=200)Present object 3 (n=26)Random absent object (n=200)
Qwen3-VL-2B80.0%86.0%100.0%89.0%
Qwen3-VL-4B81.5%85.5%100.0%91.0%
LLaVA-1.5-7B76.0%84.0%76.9%89.5%

The table shows that the vision encoder already contains representations of multiple objects' presence. However, it contains information about a combination of objects rather than just the one we test.

2. Merger: A translation-like role between visual and language representations.

We compared visual representations before the merger, after normalization, and at the output using objects from the same 200 images. Object-word representations were extracted from text-only inputs. We used CKA to measure agreement between visual and linguistic similarity structures. Higher CKA indicates greater agreement in similarity structure.

ModelBefore mergerAfter native normalizationMerger output
Qwen3-VL-2B0.0200.3880.441
Qwen3-VL-4B0.0290.4770.526

The table shows that vsual and linguistic similarity structures become substantially more aligned after native normalization, with an additional increase at the merger output.

3. Language decoder: visual evidence is used according to the question.

We then traced how the decoder uses visual representations in Qwen3-VL-4B. We crossed each object question with each object region. We replaced only the selected region's visual activations with those from an image where that region was locally filled. We then calculated how much the Yes-minus-No score decreased (log⁡P(Yes)−log⁡P(No)\log P(\mathrm{Yes})-\log P(\mathrm{No})).

Intervention locationQueried-object regionOther-object region
Merger output5.6950.312
Decoder image tokens, L86.8810.374
Decoder image tokens, L163.8110.302

This supports question-dependent use of visual evidence downstream of the merger, but does not by itself identify where proposition truth is formed.

We then tested how NOT changes the use of visual evidence. In decoder layers 1–16, we blocked later text positions from attending to "no". We repeated the same intervention on "there" as a control.

For negative questions, we calculated the Yes-minus-No score difference between the intact image and the image with the queried-object region damaged.

ModelOriginalAccess to "no" blockedAccess to "there" blocked
Qwen3-VL-2B, 198 images−5.064+6.286−4.974
Qwen3-VL-4B, 200 images−6.673+9.814−6.803

Negative values mean that intact visual evidence favors No relative to the damaged image; positive values mean it favors Yes. Blocking access to "no" reversed this effect in both models, while blocking "there" did not. This supports a functional role for negation in changing how visual evidence affects the answer.

Thank you for the response and additional experiments. I appreciate the authors' efforts to clarify the presentation and provide additional analyses of the vision encoder, multimodal merger, and language decoder. However, given the substantial revisions needed to improve the presentation, particularly for readers without a strong background in this topic, and to address experimental concerns in the original submission, I am inclined to maintain my original rating.

Reviewer_iUDuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

By adding additional structured prompts, it utilizes vision language model as visually grounded truth judgement tool. The following open source VLM models are used: Qwen3-VL-2b, Qwen3-VL-4B and LLaVA-1.5-7B. The authors found clue of edit-to-truth in the intermediate-layer. This observation has been confirmed in different ways.

Critical strengths
  1. The experiment in this paper is designed very cleanly. The quarter construction decouples edit identity from answer direction. The results show the impact clearly.
  2. Convergent causal evidence supports the interpretation through patching, truth-axis decomposition, visual-evidence removal, and codebook reversal.
Critical weaknesses

The claimed scope of the grounded truth is not fully covered by the experiment. The title and abstract describe a general grounded-truth stage. The experiment is a binary existence question whose truth is the product of two binary factors.

Questions for authors
  1. Transition-layer selection. How exactly was the per-model transition layer chosen?
  2. Do the authors claim that the representation extends beyond binary existence questions, where truth is the product of two binary factors? If so, what evidence supports this?
2 comments
AuthorsOpenReview ↗

Response to Reviewer iUDu

Thanks for you suggestion, we hope following experiment could address you concerns. If you have any further questions, please feel free to ask.

Response to Q1

We found that the layer-wise RePOPE profiles differ across architectures. We used RePOPE to select representative transition layers based on cross-edit answer-direction accuracy and same-edit, opposite-answer cosine similarity (Fig. 2). We then fixed these layers for evaluation on AMBER, without using any AMBER data for layer selection. These results demonstrate cross-dataset generalizability of the selected transition layers.

Response to Q2

To test the ability more than binary existence task we conduct following experiment.

Extension to logical composition

We conduct logical combinations of multiple visual conditions using Qwen3-VL-4B.

1. AND/OR transfer

We evaluated 100 source-held-out RePOPE images with 800 compound questions. The truth direction learned on RePOPE was frozen and applied directly to these questions.

TaskFrozen truth-direction AUROC
AND0.982
OR0.980

The same direction distinguishes true and false statements under both logical rules.

2. Harder logical judgments

We use the above truth direction in this experiment. We tested five logical rules on 100 source-held-out COCO images (1600 questions per task). We supplemented positive object labels with COCO instance annotations.

The router applies each logical rule to the model’s own atomic object-presence predictions. When this candidate answer disagrees with the original response, we rerun the compound question with an internal intervention. Target labels are not used for routing.

TaskOriginalRouted intervention
Exactly two52.00%74.06%
At least two77.94%82.75%
Majority62.94%84.19%
Odd parity48.88%72.25%
(A AND B) OR (C AND D)78.31%85.06%

According to the table, The frozen RePOPE truth direction improves accuracy across all five logical tasks.

Extension to more reasoning tasks

We use the same source-calibrated intervention from RePOPE and evaluate it on VisCoR. Original presents the accuracy of the model's original outputs. The VisCoR questions are reformulated as binary judgments of answer correctness, but not object existence.

Qwen3-VL-2B

TaskOriginalOursGain
General51.2552.50+1.25
Reasoning46.2551.50+5.25
Math47.0056.25+9.25
Graph/Chart48.5062.25+13.75
OCR56.0071.25+15.25
Overall49.8058.75+8.95

Qwen3-VL-4B

TaskOriginalOursGain
General55.0061.50+6.50
Reasoning63.0069.75+6.75
Math60.2570.50+10.25
Graph/Chart62.0078.25+16.25
OCR74.2585.75+11.50
Overall62.9073.15+10.25

LLaVA-1.5-7B

TaskOriginalOursGain
General50.2555.75+5.50
Reasoning50.2554.50+4.25
Math50.2551.25+1.00
Graph/Chart50.2550.75+0.50
OCR46.7554.50+7.75
Overall49.5553.35+3.80

Our method consistently improves performance across all five task categories. The improvements are especially encouraging for OCR. These results suggest that our method can generalize beyond RePOPE to different visual reasoning tasks.

The explanation and additional experiment partially addressed my major concern. My final rating is 3 (Weak acceptance).

Reviewer_9avTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how negation and object substitution become represented as visually grounded truth judgments. It constructs balanced natural-image quartets that separate linguistic polarity, object presence, and proposition truth, then combines cross-edit probes, component interventions, visual-evidence removal, response-code reversal, and cross-dataset transfer. A source-calibrated activation intervention further tests whether the inferred truth direction can repair naturally occurring errors.

Critical strengths
  1. Well-controlled experimental design. The balanced question quartets separate negation, object presence, and truth, reducing simple shortcuts. Splitting data by image also prevents closely related examples from appearing in both training and testing.

  2. Multiple complementary checks. Component interventions, visual-evidence removal, reversed answer labels, and cross-dataset transfer jointly support the grounded-truth interpretation, going beyond showing that a probe can predict the answer.

  3. On held-out Qwen3-VL-2B images, the one-dimensional truth component preserves 88.6% and 82.3% of the full intervention effect for the two edits, while the remaining component has much smaller effects.

  4. The source-calibrated intervention corrects 103 LLaVA errors without introducing new ones on AMBER, demonstrating selective repair without using target labels to choose the intervention.

  5. All-image evaluations, alternative question wording, visual-only changes, and color-attribute experiments extend the findings beyond the original object-existence questions.

Critical weaknesses
  1. The strongest evidence that both edits act through the same one-dimensional direction comes from Qwen3-VL-2B on only 50 held-out images. Repeating this analysis across the other models would strengthen the claim of a shared mechanism. Response-code reversal should also be evaluated beyond examples answered correctly under both mappings.

  2. Removing the queried object introduces unusual image corruption (section 5.1). Applying the same corruption to an equally sized, unrelated region would help establish whether the observed effects specifically reflect loss of relevant evidence rather than general sensitivity to image damage.

  3. The “bottleneck” interpretation needs stronger evidence. The interventions show that the identified component influences answers, but do not establish that answers must pass through it. Additional tests should examine whether other pathways can preserve performance when this component is removed.

  4. Evaluation covers only 2B, 4B, and 7B models, with the strongest causal decomposition limited to 2B. Testing larger models would clarify whether the truth representation, transition depth, and repair benefits remain consistent as model size increases.

Questions for authors
  1. Does the one-dimensional causal decomposition replicate in Qwen3-VL-4B and LLaVA, and under reversed response codebooks?
  2. How do matched non-target-region corruptions affect the truth projection and attention-swap effect?
  3. Why does internal repair outperform hard complementation on LLaVA but underperform it on Qwen3-VL-4B?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9avT

Thanks for your suggestion. We hope these experiments address your concerns. We will conduct experiments on models with larger parameter counts next.

Response to Q1

We extended the one-dimensional causal decomposition to Qwen3-VL-4B and LLaVA-1.5-7B, using the attention layers reported in the paper (L23 and L17).

ModelAttention layerNegationObject substitution
Qwen3-VL-2B (original)1688.6%82.3%
Qwen3-VL-4B2374.0%86.2%
LLaVA-1.5-7B1789.3%95.0%

These results support the replication of causal concentration along a shared truth direction across the three models.

We also extended the cross-codebook experiment in Section 5.2 to Qwen3-VL-4B and LLaVA-1.5-7B.

Following Section 5.2, we train a proposition-truth probe under one response codebook and test it under the other. Cross-codebook truth accuracy reaches 96.6% for Qwen3-VL-4B and 93.5% for LLaVA. Proposition truth remains decodable across response codebooks beyond the jointly correct examples used in the original analysis.

ModelCross-codebook truth accuracyLayer
Qwen3-VL-4B96.6%24
LLaVA-1.5-7B93.5%10

Response to Q2

We applied the same corruption to the queried-object region and an equally sized non-target region that did not overlap with any annotated object. (Values are mean changes from the intact image at full corruption.)

ModelTruth projection: targetTruth projection: non-targetAttention-swap effect: targetAttention-swap effect: non-target
Qwen3-VL-2B−23.233−0.419−4.343−0.044
Qwen3-VL-4B−12.128−0.342−3.813−0.012
LLaVA-1.5-7B−15.065+0.190−1.500+0.036

Across all three models, non-target-region corruption had little effect. Corrupting the queried-object region substantially reduced both measures. This supports dependence on relevant visual evidence rather than general sensitivity to image damage.

Response to Q3

The models have different error distributions on AMBER:

ConditionLLaVA accuracyQwen3-VL-4B accuracy
A: positive, object present99.0%98.0%
B: positive, object absent53.7%95.0%
C: negative, object present64.3%99.3%
D: negative, object absent99.0%66.7%

For LLaVA, negation errors mainly occur in C, and D is already mostly correct. Internal repair fixes 103 errors without changing correct answers. Hard complementation could turn 71 correct answers into errors. For Qwen4B, errors mainly occur in D. Its router selects 59 incorrect answers and only one correct answer, so simply flipping them works well. Internal repair fixes 39 errors. For the remaining 20, the margin moves in the right direction but does not cross zero. LLaVA benefits from avoiding harmful flips, while Qwen4B benefits from flipping a set of answers that are almost all wrong.

Thanks to the authors for the responses. However, the investigations are still limited to at most 7B model. Testing at least one larger model would clarify whether the truth representation, transition depth, and repair benefits remain valid as model scales up. I am inclined to keep my current rating.

AuthorsOpenReview ↗

Response to Reviewer 9avT

We have now extended the experiments to Qwen3-VL-32B and Qwen3-VL-235B.

Response to Q1

We extended the one-dimensional causal decomposition to the new models.

ModelAttention layerNegationObject substitution
Qwen3-VL-32B5081.77%79.46%
Qwen3-VL-235B7949.87%74.58%

We also extended the cross-codebook experiment in Section 5.2.

ModelCross-codebook truth accuracyLayer
Qwen3-VL-32B95.00%50
Qwen3-VL-235B96.29%79

Response to Q2

We applied the same corruption to the queried-object region and an equally sized non-target region that did not overlap with any annotated object.

ModelTruth projection: targetTruth projection: non-targetAttention-swap effect: targetAttention-swap effect: non-target
Qwen3-VL-32B−64.927−1.126−6.074−0.183
Qwen3-VL-235B−11.383−0.257−1.391+0.293

Response to Q3

We evaluated RePOPE-to-AMBER transfer. We show that the large models already perform very well on AMBER. However, RePOPE can still improve their results.

ModelOriginal accuracyInternal repairHard complementation
Qwen3-VL-32B92.75%93.42%94.58%
Qwen3-VL-235B92.75%93.58%94.75%

We further evaluated transfer from RePOPE to four-object logical reasoning on Qwen3-VL-32B and Qwen3-VL-235B.

Qwen3-VL-32B

RuleOriginalOursGain
Exactly two63.3974.29+10.89
At least two80.1885.54+5.36
More than half65.3686.61+21.25
Odd parity51.0767.14+16.07
Nested pairs84.6486.25+1.61
Overall68.9379.96+11.04

Qwen3-VL-235B

RuleOriginalOursGain
Exactly two67.5078.93+11.43
At least two78.2188.75+10.54
More than half66.4388.75+22.32
Odd parity51.0772.86+21.79
Nested pairs86.0789.11+3.04
Overall69.8683.68+13.82

We show that large models such as Qwen3-VL-32B and Qwen3-VL-235B also exhibit this mechanism, and our method also works on them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.