- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WITNESS is a training-free intervention for LVLM predictions when images conflict with familiar textual associations. It forms a candidate-token union from image-text and text-only predictions, projects individual final-layer attention-head outputs into vocabulary space, and measures their candidate-restricted Jensen-Shannon divergence. High-scoring heads are amplified before contrastive decoding (Section 4, Equations 1-11). The paper also introduces SynConFact, comprising 420 synthetic counterfactual instances in six categories (Section 5.1; Appendix A.1). Evaluation covers four 7B/8B LVLMs on WHOOPS-AHA! and SynConFact, component and selection ablations, and two models on MME (Tables 1-5). The method is clearly specified, and the distinction between visual sensitivity and prediction relevance is interesting; the empirical evidence for reliable head localization is less complete than the evidence for improved task accuracy.
Critical strengths
-
A concrete, prediction-conditioned selection mechanism. The golf example connects attention maps, projected token preferences, and individual-head interventions: enhancing Head 20 raises the image-consistent answer, whereas enhancing Head 16 primarily strengthens the familiar answer (Section 3.1, Figure 3). Restricting the vocabulary separates their otherwise nearly saturated JS divergences (Figure 4). This is consequential because it motivates a specific alternative to indiscriminately reinforcing visually sensitive heads. The candidate-mask ablation decreases accuracy for all four models by an average of 1.10 percentage points (Table 5), providing quantitative support beyond the illustration.
-
Consistent transfer across model families and counterfactual categories. WITNESS exceeds the base models and both reported intervention baselines, VHR and CK-PLUG, in aggregate accuracy for every model on both counterfactual datasets (Table 1). Importantly, the WHOOPS-AHA!-selected settings are transferred to SynConFact without further tuning (Section 5.3). Qwen2-VL's Scene accuracy rises from 24.3% to 68.6%, versus 57.1% for CK-PLUG (Table 1), suggesting that the method can address substantial failures rather than only marginally adjust already-correct predictions. The MME results also transparently report a decrease for Qwen2-VL, appropriately limiting broader generalization claims (Section 6.2, Table 2).
Critical weaknesses
-
The incremental advantage lacks sufficiently independent, uncertainty-aware evaluation. Section 5.3 selects the head threshold and model-specific enhancement strengths through sweeps on WHOOPS-AHA!, the same benchmark used for the reported comparison (Table 4; Figure 5); no separate validation partition is described, so overlap between tuning and evaluation examples remains unclear. Gains over CK-PLUG there are 0.6-0.8 percentage points, equivalent to a net increase of three or four correct predictions out of 500 examples (Sections 5.1 and 6.1). Tables 1 and 3 provide no paired uncertainty estimates, and the random-head control has no reported seed count or variability (Section 6.3; Appendix A.5). All methods use greedy decoding, but the random-head control still needs a reproducible stochastic assessment (Appendix A.5; Table 3). Moreover, correctness means a reference answer appears anywhere within an up-to-eight-token response (Section 5.1; Appendix A.5), potentially crediting negated or contradictory mentions. These fixable protocol issues matter because the claimed advantage over a strong baseline is small, although untuned SynConFact transfer is encouraging.
-
Semantic shift is not yet established as reliable localization of useful visual evidence. JS divergence measures change, not whether that change favors the correct visual answer (Equations 5-7). Appendix A.3's same-index correlations support representation correspondence, rather than demonstrating that the isolated-head projection faithfully predicts intervention effects (Equation 4; Table 6). The evidence includes individual-head enhancement in the golf example (Figure 3), golf/chess attention and token-shift visualizations (Appendices A.2 and A.4), a factual/counterfactual candidate-mask illustration (Appendix A.7, Figure 10), and four qualitative output comparisons (Appendix A.8, Figure 11). Table 3 also compares random-head selection, but reports aggregate accuracy rather than how individual shift scores relate to causal benefit. Table 5 shows 0.2-0.8-point gains over base models without contrastive decoding. Figure 5 includes a no-enhancement control on WHOOPS-AHA!; no corresponding SynConFact component analysis is reported. Extending these existing controls to SynConFact and measuring individual-head usefulness across examples would better establish whether the larger transferred gains support the localization claim. This fixable evidential gap, together with the first concern, supports weak rejection rather than dismissing the approach.
Questions for authors
- [Clarification] Were the Section 5.3 sweeps separated from the 500 examples reported in Table 1? Please clarify the split and provide paired uncertainty for the CK-PLUG comparison and repeated-seed results for Table 3's random control.
- [Clarification] Under Section 5.1's answer-anywhere rule, how many credited responses contain negation or both factual and counterfactual answers? Does first-answer or manually adjudicated scoring preserve Table 1's improvements?
- [Future-work] Building on Table 3's random-head comparison and Figure 5's no-enhancement control, can you extend these analyses to SynConFact and measure whether individual heads' shift scores predict their causal effects on correct-answer preference? This would clarify whether the larger transferred gains support useful-head localization rather than primarily contrastive decoding.
6 comments
Response to weakness 2
Weakness 2. Semantic Shifts and Useful Visual Information
Thank you for pointing out this gap in the evidence. We added the following analyses and clarified the scope of the conclusions accordingly:
-
Controlled interventions: On WHOOPS-AHA!, high-scoring head sets yield the largest mean answer-margin increases on both models under matched position, head count, and strength, without contrastive decoding. This supports their use of key visual cues (Question 3, part (3)).
-
SynConFact random controls: Five-seed mean ± sample SD results favor WITNESS over the random mean on both models (Question 3, part (1)).
-
SynConFact ablations: No-head-enhancement, no-contrastive-decoding, and no-candidate-mask controls show additional gains from head selection and enhancement (Question 3, part (2)).
-
Scope of the evidence: In Table 3 of the original paper, the Avg. #Heads values for Janus-Pro and Qwen2-VL are both close to 2. Therefore, this controlled experiment simultaneously enhances two heads for each example, evaluates the joint intervention effect of the selected head set, and averages the results across examples. This setting more closely reflects the intervention scale of our method and directly tests the practical value of different head-selection strategies under the same intervention budget; its conclusions concern the joint effect of the selected head set. (Question 3, part (3)).
Response to question 1
Question 1. Parameter Selection, Paired Uncertainty, and Random Repeats
(1) Parameter selection and evaluation overlap.
Section 5.3 tuned parameters on the same 500 WHOOPS-AHA! examples as Table 1, without a separate validation set.
We added a fixed 100/400 split (seed 42): select τ and λ on 100 validation examples, freeze them, then evaluate on 400 examples. With α = 0.4, we searched τ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1.0} and λ ∈ {1, 2, 3}. Both Janus-Pro and Qwen2-VL selected τ = 0.8, λ = 1; Qwen2-VL's λ differs from the original main experiment's λ = 2.
Because all 500 examples had already informed parameter exploration, this is a supplementary reevaluation, not an independent test on untouched data.
(2) Paired uncertainty against CK-PLUG.
Using all 500 original examples and the original scoring rule, we report accuracy, correctness transitions, and paired 95% CIs.
| Model | CK-PLUG Accuracy (%) | WITNESS Accuracy (%) | Newly correct / Correct-to-incorrect | Accuracy difference (percentage points, 95% CI) |
|---|---|---|---|---|
| Janus-Pro | 68.20 | 69.00 | 5/1 | +0.80 [−0.16, 1.76] |
| LLaVA-NeXT | 71.60 | 72.20 | 12/9 | +0.60 [−1.20, 2.40] |
| Qwen2-VL | 75.20 | 75.80 | 10/7 | +0.60 [−1.02, 2.22] |
| Qwen3-VL | 77.60 | 78.20 | 5/2 | +0.60 [−0.44, 1.64] |
“Newly correct” means CK-PLUG is wrong and WITNESS correct; “Correct-to-incorrect” means the reverse.
With frozen parameters on the 400-example evaluation split:
| Method / Comparison | Janus-Pro | Qwen2-VL |
|---|---|---|
| CK-PLUG Accuracy (%) | 68.50 | 73.75 |
| WITNESS Accuracy (%) | 69.25 | 74.75 |
| WITNESS − CK-PLUG (percentage points, 95% CI) | +0.75 [0.00, 1.75] | +1.00 [0.25, 2.00] |
Janus-Pro gain 0.75 percentage points (3 correct predictions) over CK-PLUG. Qwen2-VL gain 1.00 percentage points (4 correct predictions) over CK-PLUG. The paired 95% CIs are [0.00, 1.75] and [0.25, 2.00].
(3) Repeated random head selection.
We use five seeds (42, 43, 44, 45, 46) on both datasets, matching WITNESS's head-count setting and keeping other intervention/decoding settings unchanged. Results report mean accuracy ± sample standard deviation (SD) across seeds (n = 5), measuring random-selection variability.
| Dataset | Head-selection strategy | Janus-Pro Accuracy (%) | Qwen2-VL Accuracy (%) |
|---|---|---|---|
| WHOOPS-AHA! | Random | 68.36 ± 0.22 | 74.96 ± 0.30 |
| Fixed Top-2 | 68.60 | 75.20 | |
| WITNESS | 69.00 | 75.80 | |
| SynConFact | Random | 78.52 ± 0.46 | 88.05 ± 0.39 |
| Fixed Top-2 | 79.76 | 88.33 | |
| WITNESS | 80.24 | 88.81 |
Accuracies are rounded; gains use unrounded accuracies.
WITNESS exceeds the random mean by 0.64–1.71 percentage points in all four model–dataset combinations, preserving the advantage while quantifying random-selection variability.
Response to weakness 1
We sincerely thank the reviewer for his/her time and their detailed, insightful feedback. The comments are extremely valuable and have helped us identify areas to clarify and strengthen in our paper. We address the primary concerns below.
Weakness 1. Evaluation Independence, Uncertainty, and Scoring Rules
Thank you for highlighting these protocol issues. We address four aspects:
Parameter selection: Original tuning used all 500 examples. The supplementary 100/400 split is not an untouched independent test (Question 1, part (1)).
Paired uncertainty: We report paired 95% CIs against CK-PLUG on all four models, distinguishing positive estimates from statistical significance (Question 1, part (2)).
Random-selection reproducibility: Five-seed experiments report mean accuracy and sample SD, including SynConFact (Question 1, part (3)).
Scoring review: AI-assisted screening and targeted manual review correct identified false positives while preserving positive accuracy gaps on all four models. Question 2 details the scope and limitations.