acceptodds
Under review as a conference paper at ICLR 2027

Who Is the Real Witness? Localizing Attention Heads through Image-Induced Semantic Shifts in LVLMs

Abstract

Large vision-language models (LVLMs) have demonstrated strong performance on multimodal tasks. However, they may overrely on prior knowledge when it conflicts with visual information, producing plausible predictions that are inconsistent with the input image. Previous studies have sought to mitigate this issue by identifying and enhancing attention heads that are sensitive to visual input. We observe that visually sensitive heads do not always capture the key cues needed for the current prediction. In this paper, we propose Witness Identification Through Next-token Evidence of Semantic Shifts (WITNESS), a training-free method. It projects each head's outputs into the vocabulary space and compares the resulting predictive distributions under text-only and image-text conditions over the same candidate token set. The resulting image-induced semantic shifts are quantified using Jensen-Shannon (JS) divergence. WITNESS then selects and enhances heads exhibiting larger shifts and combines this intervention with contrastive decoding. We conduct extensive experiments on the counterfactual datasets WHOOPS-AHA! and SynConFact using four LVLMs. Compared with the base models, WITNESS achieves relative improvements of 6.49% and 16.16% in accuracy averaged across models on the two datasets, respectively. Our code is available at https://anonymous.4open.science/r/WITNESS-57B7.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large vision-language models, Visual knowledge conflicts, Attention heads, Semantic shifts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZJzSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WITNESS is a training-free intervention for LVLM predictions when images conflict with familiar textual associations. It forms a candidate-token union from image-text and text-only predictions, projects individual final-layer attention-head outputs into vocabulary space, and measures their candidate-restricted Jensen-Shannon divergence. High-scoring heads are amplified before contrastive decoding (Section 4, Equations 1-11). The paper also introduces SynConFact, comprising 420 synthetic counterfactual instances in six categories (Section 5.1; Appendix A.1). Evaluation covers four 7B/8B LVLMs on WHOOPS-AHA! and SynConFact, component and selection ablations, and two models on MME (Tables 1-5). The method is clearly specified, and the distinction between visual sensitivity and prediction relevance is interesting; the empirical evidence for reliable head localization is less complete than the evidence for improved task accuracy.

Critical strengths
  1. A concrete, prediction-conditioned selection mechanism. The golf example connects attention maps, projected token preferences, and individual-head interventions: enhancing Head 20 raises the image-consistent answer, whereas enhancing Head 16 primarily strengthens the familiar answer (Section 3.1, Figure 3). Restricting the vocabulary separates their otherwise nearly saturated JS divergences (Figure 4). This is consequential because it motivates a specific alternative to indiscriminately reinforcing visually sensitive heads. The candidate-mask ablation decreases accuracy for all four models by an average of 1.10 percentage points (Table 5), providing quantitative support beyond the illustration.

  2. Consistent transfer across model families and counterfactual categories. WITNESS exceeds the base models and both reported intervention baselines, VHR and CK-PLUG, in aggregate accuracy for every model on both counterfactual datasets (Table 1). Importantly, the WHOOPS-AHA!-selected settings are transferred to SynConFact without further tuning (Section 5.3). Qwen2-VL's Scene accuracy rises from 24.3% to 68.6%, versus 57.1% for CK-PLUG (Table 1), suggesting that the method can address substantial failures rather than only marginally adjust already-correct predictions. The MME results also transparently report a decrease for Qwen2-VL, appropriately limiting broader generalization claims (Section 6.2, Table 2).

Critical weaknesses
  1. The incremental advantage lacks sufficiently independent, uncertainty-aware evaluation. Section 5.3 selects the head threshold and model-specific enhancement strengths through sweeps on WHOOPS-AHA!, the same benchmark used for the reported comparison (Table 4; Figure 5); no separate validation partition is described, so overlap between tuning and evaluation examples remains unclear. Gains over CK-PLUG there are 0.6-0.8 percentage points, equivalent to a net increase of three or four correct predictions out of 500 examples (Sections 5.1 and 6.1). Tables 1 and 3 provide no paired uncertainty estimates, and the random-head control has no reported seed count or variability (Section 6.3; Appendix A.5). All methods use greedy decoding, but the random-head control still needs a reproducible stochastic assessment (Appendix A.5; Table 3). Moreover, correctness means a reference answer appears anywhere within an up-to-eight-token response (Section 5.1; Appendix A.5), potentially crediting negated or contradictory mentions. These fixable protocol issues matter because the claimed advantage over a strong baseline is small, although untuned SynConFact transfer is encouraging.

  2. Semantic shift is not yet established as reliable localization of useful visual evidence. JS divergence measures change, not whether that change favors the correct visual answer (Equations 5-7). Appendix A.3's same-index correlations support representation correspondence, rather than demonstrating that the isolated-head projection faithfully predicts intervention effects (Equation 4; Table 6). The evidence includes individual-head enhancement in the golf example (Figure 3), golf/chess attention and token-shift visualizations (Appendices A.2 and A.4), a factual/counterfactual candidate-mask illustration (Appendix A.7, Figure 10), and four qualitative output comparisons (Appendix A.8, Figure 11). Table 3 also compares random-head selection, but reports aggregate accuracy rather than how individual shift scores relate to causal benefit. Table 5 shows 0.2-0.8-point gains over base models without contrastive decoding. Figure 5 includes a no-enhancement control on WHOOPS-AHA!; no corresponding SynConFact component analysis is reported. Extending these existing controls to SynConFact and measuring individual-head usefulness across examples would better establish whether the larger transferred gains support the localization claim. This fixable evidential gap, together with the first concern, supports weak rejection rather than dismissing the approach.

Questions for authors
  1. [Clarification] Were the Section 5.3 sweeps separated from the 500 examples reported in Table 1? Please clarify the split and provide paired uncertainty for the CK-PLUG comparison and repeated-seed results for Table 3's random control.
  2. [Clarification] Under Section 5.1's answer-anywhere rule, how many credited responses contain negation or both factual and counterfactual answers? Does first-answer or manually adjudicated scoring preserve Table 1's improvements?
  3. [Future-work] Building on Table 3's random-head comparison and Figure 5's no-enhancement control, can you extend these analyses to SynConFact and measure whether individual heads' shift scores predict their causal effects on correct-answer preference? This would clarify whether the larger transferred gains support useful-head localization rather than primarily contrastive decoding.
6 comments
AuthorsOpenReview ↗

Response to weakness 1

We sincerely thank the reviewer for his/her time and their detailed, insightful feedback. The comments are extremely valuable and have helped us identify areas to clarify and strengthen in our paper. We address the primary concerns below.

Weakness 1. Evaluation Independence, Uncertainty, and Scoring Rules

Thank you for highlighting these protocol issues. We address four aspects:

  1. Parameter selection: Original tuning used all 500 examples. The supplementary 100/400 split is not an untouched independent test (Question 1, part (1)).

  2. Paired uncertainty: We report paired 95% CIs against CK-PLUG on all four models, distinguishing positive estimates from statistical significance (Question 1, part (2)).

  3. Random-selection reproducibility: Five-seed experiments report mean accuracy and sample SD, including SynConFact (Question 1, part (3)).

  4. Scoring review: AI-assisted screening and targeted manual review correct identified false positives while preserving positive accuracy gaps on all four models. Question 2 details the scope and limitations.

AuthorsOpenReview ↗

Response to weakness 2

Weakness 2. Semantic Shifts and Useful Visual Information

Thank you for pointing out this gap in the evidence. We added the following analyses and clarified the scope of the conclusions accordingly:

  1. Controlled interventions: On WHOOPS-AHA!, high-scoring head sets yield the largest mean answer-margin increases on both models under matched position, head count, and strength, without contrastive decoding. This supports their use of key visual cues (Question 3, part (3)).

  2. SynConFact random controls: Five-seed mean ± sample SD results favor WITNESS over the random mean on both models (Question 3, part (1)).

  3. SynConFact ablations: No-head-enhancement, no-contrastive-decoding, and no-candidate-mask controls show additional gains from head selection and enhancement (Question 3, part (2)).

  4. Scope of the evidence: In Table 3 of the original paper, the Avg. #Heads values for Janus-Pro and Qwen2-VL are both close to 2. Therefore, this controlled experiment simultaneously enhances two heads for each example, evaluates the joint intervention effect of the selected head set, and averages the results across examples. This setting more closely reflects the intervention scale of our method and directly tests the practical value of different head-selection strategies under the same intervention budget; its conclusions concern the joint effect of the selected head set. (Question 3, part (3)).

AuthorsOpenReview ↗

Response to question 1

Question 1. Parameter Selection, Paired Uncertainty, and Random Repeats

(1) Parameter selection and evaluation overlap.

Section 5.3 tuned parameters on the same 500 WHOOPS-AHA! examples as Table 1, without a separate validation set.

We added a fixed 100/400 split (seed 42): select τ and λ on 100 validation examples, freeze them, then evaluate on 400 examples. With α = 0.4, we searched τ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1.0} and λ ∈ {1, 2, 3}. Both Janus-Pro and Qwen2-VL selected τ = 0.8, λ = 1; Qwen2-VL's λ differs from the original main experiment's λ = 2.

Because all 500 examples had already informed parameter exploration, this is a supplementary reevaluation, not an independent test on untouched data.

(2) Paired uncertainty against CK-PLUG.

Using all 500 original examples and the original scoring rule, we report accuracy, correctness transitions, and paired 95% CIs.

ModelCK-PLUG Accuracy (%)WITNESS Accuracy (%)Newly correct / Correct-to-incorrectAccuracy difference (percentage points, 95% CI)
Janus-Pro68.2069.005/1+0.80 [−0.16, 1.76]
LLaVA-NeXT71.6072.2012/9+0.60 [−1.20, 2.40]
Qwen2-VL75.2075.8010/7+0.60 [−1.02, 2.22]
Qwen3-VL77.6078.205/2+0.60 [−0.44, 1.64]

“Newly correct” means CK-PLUG is wrong and WITNESS correct; “Correct-to-incorrect” means the reverse.

With frozen parameters on the 400-example evaluation split:

Method / ComparisonJanus-ProQwen2-VL
CK-PLUG Accuracy (%)68.5073.75
WITNESS Accuracy (%)69.2574.75
WITNESS − CK-PLUG (percentage points, 95% CI)+0.75 [0.00, 1.75]+1.00 [0.25, 2.00]

Janus-Pro gain 0.75 percentage points (3 correct predictions) over CK-PLUG. Qwen2-VL gain 1.00 percentage points (4 correct predictions) over CK-PLUG. The paired 95% CIs are [0.00, 1.75] and [0.25, 2.00].

(3) Repeated random head selection.

We use five seeds (42, 43, 44, 45, 46) on both datasets, matching WITNESS's head-count setting and keeping other intervention/decoding settings unchanged. Results report mean accuracy ± sample standard deviation (SD) across seeds (n = 5), measuring random-selection variability.

DatasetHead-selection strategyJanus-Pro Accuracy (%)Qwen2-VL Accuracy (%)
WHOOPS-AHA!Random68.36 ± 0.2274.96 ± 0.30
Fixed Top-268.6075.20
WITNESS69.0075.80
SynConFactRandom78.52 ± 0.4688.05 ± 0.39
Fixed Top-279.7688.33
WITNESS80.2488.81

Accuracies are rounded; gains use unrounded accuracies.

WITNESS exceeds the random mean by 0.64–1.71 percentage points in all four model–dataset combinations, preserving the advantage while quantifying random-selection variability.

Reviewer_HRFBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes WITNESS, a training-free inference-time method to mitigate conflicts between visual evidence and text biased prior knowledge in MLLM. At each decoding step, the model runs both an image-text and a text-only forward pass, localizes attention heads in the last layer whose predictions change significantly among the two runs and scales up their influence. The resulting enhanced image-text logits are combined with the text-only logits via contrastive decoding. The authors also introduce SynConFact, a 420-sample synthetic counterfactual next-token benchmark, and evaluate WITNESS on four 7B–8B LVLMs across WHOOPS-AHA!, SynConFact, and MME, improving significantly on the first two and being neutral or slightly negative on MME in some runs.

Critical strengths
  • Intuitive Motivation: Identifying attention heads whose logits prediction change with or without an image in the context is an intuitive and sound idea.

  • Training-free and simple pipeline: The method requires no model retraining or external calibration data and consistently improves over base greedy decoding across four recent LVLM families.

Critical weaknesses

A. Somehow limited novelty and weak experimental support: The method reads as a combination of previously proposed techniques. The contrastive decoding is heavily inspired by CK-PLUG and VCD, while VHR highlighted attention heads as a good place for intervention to mitigate visual hallucinations. The paper does combine all of them in a consistent framework, but the technical novelty is in my opinion limited.

B. Experiments not supporting the paper narrative: The work emphasizes how the intervention on the attention head is the main component for mitigating visual hallucinations (as this part is also the main novelty of the work). However the results in Tab. 5 show how the boost in performance is actually dominated by the contrastive decoding technique which is heavily lifted from previous works. TThe candidate token mask comparatively contributes minimally.

D. Last-layer-only intervention: Applying head logit projection and enhancement only in the final LLM layer ignores middle-to-late layers where according to previous literature cross-modal conflict resolution largely takes place.

E. Missing experimental analysis: Regarding competitors, Appendix A.5 indicates that the VHR baseline was also restricted to the final layer rather than its original multi-layer setup, and the evaluation omits a direct comparison with the closely related head-steering method of Ortu et al. (2026) on WHOOPS-AHA!. The paper should have also been evaluated on more commonly used visual hallucination benchmarks such as POPE, CHAIR, or AMBER.

F. Inference overhead: The method requires at least 2x compute flop at inference time.

Questions for authors
  • Out-of-sample hyperparameter selection: How does WITNESS perform on WHOOPS-AHA! when tau and lambda are selected on a held-out validation split (or transferred from SynConFact) rather than tuned directly on the WHOOPS-AHA! test set?

  • Multi-layer intervention and baselines: Why was the intervention (and the VHR baseline reproduction) restricted only to the last LLM layer, and how does WITNESS compare against multi-layer VHR and the head-intervention method of Ortu et al. (2026)?

15 comments
AuthorsOpenReview ↗

Response to weakness A

We sincerely thank the reviewer for taking the time to provide detailed and insightful feedback. These comments are extremely valuable and have helped us identify aspects of the paper that require clarification and improvement. Below, we respond to the reviewer's main concerns one by one.

A. Technical Novelty and Experimental Support

Thank you for pointing out this issue. We were indeed inspired by CK-PLUG and related work, but the core contribution of this paper is to enable the model to rely more on the image while further focusing on key visual cues in the image that are relevant to the current answer.

  1. Contrastive decoding increases the model's reliance on visual information. Drawing on CK-PLUG's approach to adjusting knowledge reliance, we adjust the relative influence of visual information and language priors in the image-text setting. We do not claim contrastive decoding itself as a novel contribution; instead, we further investigate which attention heads can help the model use the key visual cues needed for the current question.

  2. WITNESS identifies these heads using a prediction-relevant selection criterion and verifies their practical contribution. Within the shared candidate token set constructed from image-text and text-only predictions, we renormalize each head's projected distributions and compute JS divergence, focusing the selection criterion on changes in relative preferences over the current candidate tokens.

To test their contribution beyond semantic shifts, we compare high-scoring, random, and low-scoring head pairs on WHOOPS-AHA!: last-layer intervention, 2× output scaling, and no contrastive decoding. Evaluation retains complete single-token answer labels, yielding 441 Janus-Pro and 439 Qwen3-VL examples. ΔM measures the intervention-induced change in the image-consistent versus commonsense answer margin.

We define:

-\log\sum_{w\in A_{\mathrm{prior}}}\exp(\ell_w), \qquad \Delta M=M_{\mathrm{intervention}}-M_{\mathrm{original}}.$$ $A_{\mathrm{img}}$ and $A_{\mathrm{prior}}$ are image-consistent and commonsense answer token sets, respectively. | Model | Head-selection strategy | Mean ΔM (95% CI) | Proportion with ΔM > 0 | | --------- | ----------------------- | --------------------------: | ---------------------: | | Janus-Pro | Bottom-2 | 0.0248 [0.0126, 0.0369] | 60.54% | | | Random-2 | 0.0449 [0.0325, 0.0575] | 64.63% | | | **Top-2** | **0.1225** [0.0976, 0.1484] | **65.76%** | | Qwen3-VL | Bottom-2 | 0.0285 [0.0086, 0.0489] | 51.48% | | | Random-2 | 0.0692 [0.0278, 0.1124] | 54.67% | | | **Top-2** | **0.2827** [0.2241, 0.3404] | **65.83%** | **High-scoring head sets yield larger mean positive contributions on both models, supporting WITNESS's use of key visual cues relevant to the current answer.** This effect does not rely on contrastive decoding or imply that every high-scoring head is beneficial. 3. **VHR differs in its selection criterion**. VHD measures Euclidean distances between head outputs without explicitly assessing current candidate preferences. Keeping intervention position, head count, and strength fixed, we vary only the criterion without contrastive decoding: | Head-selection criterion | Janus-Pro Accuracy (%) | Qwen2-VL Accuracy (%) | | ------------------------------------- | ---------------------: | --------------------: | | Baseline | 62.2 | 73.4 | | VHD | 62.6 | 72.6 | | Full-vocabulary JS | 62.4 | 73.2 | | **JS within the candidate token set** | **62.8** | **73.8** | JS divergence within the candidate token set achieves the highest accuracy on both models. Although differences are small, the intervention analysis further supports its value. Therefore, the additional contribution of this paper is a prediction-relevant head-selection criterion and its validation through intervention. Contrastive decoding makes the model rely more on the image, while head selection and enhancement further guide the model to focus on key visual cues that help answer the current question—identifying such “witnesses” is precisely the motivation behind the name WITNESS.
AuthorsOpenReview ↗

Response to weakness B

B. Experimental Results and the Paper's Narrative

We acknowledge contrastive decoding's contribution to accuracy. Our head-selection and enhancement mechanism further focuses on key visual cues relevant to the current answer. We added an “Only Contrastive Decoding (no head enhancement)” control and extended the ablations to SynConFact. Accuracy (%):

MethodWHOOPS-AHA! · Janus-ProWHOOPS-AHA! · Qwen2-VLSynConFact · Janus-ProSynConFact · Qwen2-VL
Baseline62.273.467.174.5
Only Contrastive Decoding68.275.078.687.9
w/o Contrastive Decoding62.673.667.978.3
w/o Candidate Token Mask68.274.479.087.6
Full Method69.075.880.288.8

“Only Contrastive Decoding” disables head enhancement within WITNESS but retains its candidate token set construction and final decoding settings; it is distinct from the independently implemented CK-PLUG baseline. Accuracies are rounded; gains use unrounded sample-count accuracies.

  1. Head enhancement adds gains beyond contrastive decoding: 0.8, 0.8, 1.7, and 1.0 percentage points across the four settings.

  2. Gains remain without contrastive decoding: 0.4, 0.2, 0.7, and 3.8 percentage points over the original models. The mechanism is not entirely dependent on contrastive decoding, although its effect varies by model and dataset.

  3. The candidate token set constraint adds small, consistently positive gains. Removing it changes only head scoring to full-vocabulary JS, leaving final decoding unchanged. The full method gains 0.8, 1.4, 1.2, and 1.2 percentage points, respectively.

These results support additional gains from prediction-relevant head selection and enhancement through better use of key visual cues. We will distinguish the full method's overall gains from this mechanism's incremental contribution, aligning the narrative with the ablations.

AuthorsOpenReview ↗

Response to weakness D and question 2

D. Intervention Only in the Last Layer (Also Addressing Question 2)

  1. The roles of layers and attention heads may differ across models. We added comparisons of the intervention configurations of different methods, including a multi-layer variant of WITNESS. Accuracy (%) is shown below:
MethodIntervention scopeJanus-ProLLaVA-NeXTQwen2-VLQwen3-VL
Baseline (no enhancement)—62.265.473.476.2
VHRLayer 2 + last 14 layers, 2×60.865.872.673.4
CausalLensLayers 11–19 (zero-based)62.866.073.275.8
SPINAll layers57.057.867.073.8
HGALayers 5–17 (zero-based)59.262.271.075.8
Ortu et al. (2026)Union of zero-based layer indices of heads selected through calibration60.465.473.476.4
WITNESS (multi-layer variant)Last 14 layers69.269.075.275.6
WITNESSLast layer69.072.275.878.2

For the method of Ortu et al. (2026), the intervention heads and intervention strength λ are calibrated on SynConFact. Heads are selected from all decoder layers of the language model, with a default maximum of 20 heads in each of the counterfactual and factual groups. The actual intervention layers are the union of the zero-based layer indices of the heads selected in these two groups after calibration; at inference time, intervention is applied only to the selected heads in those layers. The intervention strengths λ for Janus-Pro, LLaVA-NeXT, Qwen2-VL, and Qwen3-VL are −2, −3, −1, and −1.5, respectively.

The gains from existing methods are not consistent on the current task. This may relate to differences in the roles of layers and attention heads across models, as well as differences in the visual information required by different tasks. For example, object existence judgments in POPE and image captioning in CHAIR differ from counterfactual next-token prediction in WHOOPS-AHA!; interventions that are effective for the former two tasks may not directly improve the latter.

  1. The choice of the last layer considers both interpretability and empirical performance. WITNESS intervenes in the last layer because it is closer to the model's output, making the candidate token preferences reflected in its head outputs after vocabulary projection easier to interpret. In the experiments, the full WITNESS method with last-layer intervention improves performance on all tested models; in contrast, expanding the intervention to the last 14 layers does not yield consistent additional gains.

Based on these two points, we choose the last layer as a design that considers both interpretability and empirical performance, without claiming that middle layers have no role or that the last layer is necessarily optimal for every model.

Reviewer_ogdKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates visual faithfulness in large vision-language models (LVLMs) by proposing a training-free method, WITNESS. It identifies and enhances attention heads based on image-induced shifts in next-token prediction distributions, measured by Jensen–Shannon divergence, and combines this intervention with contrastive decoding to improve visual faithfulness. Extensive experiments on 2 datasets verify the effectiveness of the proposed method.

Critical strengths
  1. The paper addresses an important problem of visual hallucinations in LVLMs, particularly when visual evidence conflicts with prior knowledge.
  2. The proposed WITNESS method is training-free and can be applied at inference time without additional model training.
  3. The method achieves consistent accuracy improvements across four LVLMs on two counterfactual datasets, demonstrating its potential to improve visual faithfulness.
Critical weaknesses
  1. My major concern is the technical novelty/contribution of this paper. Because there are already quite a lot existing works on identifying and enhancing visually sensitive attention heads to mitigate hallucinations. The novelty of using JS divergence to measure image-induced semantic shifts appears limited, and its advantages over existing head-selection criteria are unclear for me.
  2. Another issues is that a large semantic shift between text-only and image-text conditions does not necessarily indicate that an attention head captures task-relevant visual evidence. Hence more analysis is needed to validate the reliability of the proposed criterion.
  3. Experiments need more insightful analysis. For example, in Table 3, it seems the head selection strategy has limited influences on the results. Because even for random, the accuracy only drops 0.8%. The authors need to provide more insight analysis.
  4. The compared methods is limited, as it only compared with 2 methods on this problem. Since there are already quite a lot of work on this problem, the authors need to compare with them. For example, "Sensitivity-Guided Multi-Head Causal Intervention, CVPR 2026" "Image-Guided Head Suppression, EMNLP 2025" "Devils in Middle Layers of Large Vision-Language Models" etc.
Questions for authors

Please refer to the weakness part.

4 comments
AuthorsOpenReview ↗

Response to weakness 1

1. Technical Novelty and Contributions

Thank you for pointing out this issue. In the revised manuscript, we will clarify the technical contributions of our method and how it differs from existing head-selection criteria.

We agree with the reviewer that simply using JS divergence to measure image-induced semantic shifts is insufficient to constitute the novelty of this work.

JS divergence is only one computational step in our method. Our core innovation is to measure how introducing an image changes each attention head's relative preferences over candidate tokens within a shared candidate token set determined by the current next-token prediction, and to select heads accordingly.

  1. Applying JS divergence directly over the full vocabulary: This reflects distributional changes across the entire vocabulary and may not highlight differences relevant to the current prediction. An attention head's response to visual information may lead to large distributional changes between the text-only and image-text conditions, but these changes are not necessarily relevant to the current prediction.

  2. Our head-selection method based on JS divergence within the candidate token set: We renormalize each head's predictive distributions within the shared candidate token space determined by the current prediction before computing JS divergence. This makes the selection criterion more focused on changes in relative preferences among the current candidate tokens.

  3. The role of JS divergence itself: Our motivation is that heads exhibiting larger distributional differences within the candidate token set may carry visual information relevant to the current prediction, and JS divergence can quantify the difference between these two probability distributions.

The magnitude of changes in hidden space measured by existing methods is not equivalent to changes in answer preferences for the current prediction. For example, VHR's VHD measures the Euclidean distance between head output vectors under the image-text and text-only conditions, but does not explicitly distinguish whether these changes affect the candidate tokens involved in the current prediction.

To examine the practical value of head selection within the candidate token set, we added a controlled experiment: we fixed the intervention to the last layer, selected two heads, uniformly applied 2× enhancement, and disabled contrastive decoding, changing only the head-selection criterion. Here, the outputs of the two selected heads are directly multiplied by 2, which differs from the main method's threshold-based head selection and enhancement using 1+λs^h1+\lambda\hat{s}_h.

Head-selection criterionJanus-Pro Accuracy (%)Qwen2-VL Accuracy (%)
Baseline62.273.4
VHD62.672.6
Full-vocabulary JS62.473.2
JS within the candidate token set62.873.8

Under the same intervention budget, head selection using JS divergence within the candidate token set achieves the highest accuracy on both models. We acknowledge that these accuracy differences are small, and this table alone is insufficient to establish the general superiority of one selection criterion. Therefore, in our response to Comment 2, we further provide an intervention analysis of answer preferences to examine whether high-scoring heads make a greater practical contribution to image-consistent predictions.

We will clarify the above technical contributions in the revised manuscript.

AuthorsOpenReview ↗

Response to weakness 2

2. Relationship Between Semantic Shifts and Task-Relevant Visual Information

Thank you for pointing out this important distinction. We agree that a large semantic shift between the image-text and text-only conditions does not, by itself, guarantee that an attention head provides useful task-relevant visual information. To examine this, we added controlled intervention experiments to directly investigate whether heads selected by this criterion can more effectively increase the model's support for image-consistent answers.

On WHOOPS-AHA!, Top-2 and Bottom-2 select heads with the two highest and lowest JS scores within the candidate token set, respectively; Random-2 selects two random heads(seed 42). We test Janus-Pro-7B and Qwen3-VL-8B with identical interventions: 2× output scaling at the final input position in the last layer, without contrastive decoding.

From first-step logits, we define the prediction margin of image-consistent over commonsense answers:

-\log\sum_{w\in A_{\mathrm{prior}}}\exp(\ell_w), \qquad \Delta M=M_{\mathrm{intervention}}-M_{\mathrm{original}}.$$ Here, $A_{\mathrm{img}}$ and $A_{\mathrm{prior}}$ are image-consistent and commonsense answer token sets. Labels are used only for evaluation, not selection; positive $\Delta M$ favors image-consistent answers. **Strict subset**. All answer labels on both sides must have complete single-token representations, with nonempty, disjoint token sets. Filtering uses only labels and the tokenizer, not predictions or intervention outcomes, and never substitutes the first token of a multi-token answer. This yields 148 examples for Janus-Pro and 126 for Qwen3-VL: | Model | Head-selection strategy | Mean ΔM (95% CI) | Proportion with ΔM > 0 | | --------- | ----------------------- | --------------------------: | ---------------------: | | Janus-Pro | Bottom-2 | 0.0333 [0.0138, 0.0528] | 62.84% | | | Random-2 | 0.0257 [0.0076, 0.0439] | 61.49% | | | **Top-2** | **0.1336** [0.0915, 0.1768] | **69.59%** | | Qwen3-VL | Bottom-2 | 0.0111 [−0.0256, 0.0504] | 45.24% | | | Random-2 | 0.1023 [0.0164, 0.1914] | 62.70% | | | **Top-2** | **0.3082** [0.2068, 0.4106] | **67.46%** | **Broader subset**. We retain only complete single-token answer labels, allowing other reference labels to lack single-token representations without truncating them. This covers 441 examples for Janus-Pro and 439 for Qwen3-VL, again giving Top-2 the largest mean margin increase: | Model | Head-selection strategy | Mean ΔM (95% CI) | Proportion with ΔM > 0 | | --------- | ----------------------- | --------------------------: | ---------------------: | | Janus-Pro | Bottom-2 | 0.0248 [0.0126, 0.0369] | 60.54% | | | Random-2 | 0.0449 [0.0325, 0.0575] | 64.63% | | | **Top-2** | **0.1225** [0.0976, 0.1484] | **65.76%** | | Qwen3-VL | Bottom-2 | 0.0285 [0.0086, 0.0489] | 51.48% | | | Random-2 | 0.0692 [0.0278, 0.1124] | 54.67% | | | **Top-2** | **0.2827** [0.2241, 0.3404] | **65.83%** | These results provide intervention evidence that was missing from observations of semantic shifts alone: **under the same enhancement strength and with contrastive decoding disabled, sets of high-scoring heads make a larger average positive contribution to image-consistent answers**. We again thank the reviewer for these constructive comments.
AuthorsOpenReview ↗

Response to weakness 3

3. Additional Experimental Analysis

A single random run is insufficient to establish the role of head selection. We added three analyses:

  1. Repeated random controls and transfer to SynConFact. We use five seeds (42–46), matching our method's head-count setting on both datasets. Results report mean accuracy ± sample standard deviation (SD) across seeds (n = 5):
DatasetHead-selection strategyJanus-Pro Accuracy (%)Qwen2-VL Accuracy (%)
WHOOPS-AHA!Random68.36 ± 0.2274.96 ± 0.30
Fixed Top-268.6075.20
WITNESS69.0075.80
SynConFactRandom78.52 ± 0.4688.05 ± 0.39
Fixed Top-279.7688.33
WITNESS80.2488.81

Accuracies are rounded; gains are computed from unrounded sample-count accuracies before rounding.

WITNESS exceeds the random mean by 0.64–1.71 percentage points across all four model–dataset combinations, preserving the direction of the advantage in repeated experiments and on another dataset.

  1. Answer preferences beyond accuracy. Accuracy misses preference shifts that leave answer correctness unchanged. Response 2 shows that Top-2 yields the largest mean increase in the image-consistent answer margin on both models without contrastive decoding.

  2. Head enhancement's contribution. We added a contrastive-decoding-only control without head enhancement alongside the component ablations. Accuracy (%):

MethodWHOOPS-AHA! · Janus-ProWHOOPS-AHA! · Qwen2-VLSynConFact · Janus-ProSynConFact · Qwen2-VL
Baseline62.273.467.174.5
Only Contrastive Decoding68.275.078.687.9
w/o Contrastive Decoding62.673.667.978.3
w/o Candidate Token Mask68.274.479.087.6
Full Method69.075.880.288.8

“Only Contrastive Decoding” disables head enhancement within WITNESS but retains its candidate token set construction and final decoding settings; it is distinct from the independently implemented CK-PLUG baseline. Accuracies are rounded; gains use unrounded sample-count accuracies.

Beyond contrastive decoding's increased reliance on images, prediction-relevant head selection and enhancement guide the model toward key visual cues and improve accuracy in all four settings.

We will add these analyses and tables to clarify random-selection variability, selection-strategy effects, and component contributions.

AuthorsOpenReview ↗

Response to weakness 4

4. Scope of Comparisons

Thank you for the recommendations. We expanded the WHOOPS-AHA! comparison beyond VHR and CK-PLUG to include CausalLens, SPIN, HGA, Ortu et al. (2026), and multi-layer VHR. Accuracy (%):

MethodJanus-ProLLaVA-NeXTQwen2-VLQwen3-VL
Baseline62.265.473.476.2
SPIN57.057.867.073.8
HGA59.262.271.075.8
VHR, multi-layer configuration60.865.872.673.4
VHR, last-layer variant61.665.072.675.2
Ortu et al. (2026)60.465.473.476.4
CausalLens62.866.073.275.8
CK-PLUG68.271.675.277.6
WITNESS69.072.275.878.2

WITNESS achieves the highest accuracy in this expanded baseline comparison on all four models. Other methods' gains vary, possibly reflecting model-specific layer/head roles and task differences: POPE tests object existence, CHAIR image captioning, and WHOOPS-AHA! counterfactual next-token prediction. Effective interventions may not transfer directly between these tasks.

The original VHR baseline used a last-layer variant to match WITNESS's intervention position. This does not replace evaluation of multi-layer VHR; we will distinguish and report both configurations.

Ortu et al.'s (2026) heads and intervention strength λ are calibrated on SynConFact. Heads are selected across all decoder layers, with at most 20 counterfactual and 20 factual heads by default. Intervention layers are the union of their zero-based layer indices, and only selected heads are modified at inference. λ is −2, −3, −1, and −1.5 for Janus-Pro, LLaVA-NeXT, Qwen2-VL, and Qwen3-VL, respectively.

We will report each method's citation, implementation source, intervention layers, hyperparameters, and model/task adaptations.


We hope these clarifications address the reviewer's concerns. We are confident that these additions will strengthen the paper and more clearly highlight our novel contributions.

We thank the reviewer again for their constructive feedback.

Comments on the paper

1 comment
Area_Chair_TtFXOpenReview ↗

Reminder: Please Review the Author Rebuttal

Dear Reviewers,

The authors have submitted their rebuttal and addressed the concerns raised in the reviews. Please take some time to carefully examine their responses and participate in the discussion. We would appreciate your updated assessment, including any potential rating adjustments, where appropriate.

Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.