- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies hallucination caused by poor selection among visual tokens, even when substantial attention is allocated to the image. It associates this behavior with concentrated visual-key directions and proposes WAS, which attenuates dominant spectral directions, matches score statistics, and applies the correction through an entropy gate. Experiments on LLaVA, InstructBLIP, and Qwen3-VL show reduced hallucination. The appendix provides mechanism controls, sensitivity analyses, and statistical tests.
Critical strengths
- The distinction between visual attention mass and region selection leads to a concrete, effective intervention.
- The analysis associates concentrated visual-key spectra with weak query-dependent selection, identifying a geometric property that aggregate attention statistics can overlook. This offers a useful diagnostic perspective beyond simply increasing attention to image tokens.
- Reverse interventions, matched controls, paired statistical tests, and modest measured overhead strengthen the empirical contribution.
Critical weaknesses
-
The visual attention budget is only approximately controlled. Matching score means and variances does not preserve visual probability mass under the joint softmax, and sharpening introduces an additional change. A small average difference can conceal substantial changes in individual heads or decoding steps. Exact mass-preserving controls are needed to attribute the improvement specifically to better visual selection.
-
The theory does not establish the central link from low effective rank to failed query-dependent selection. Low-rank keys can still produce different attention patterns for different queries. The analysis therefore does not identify sufficient conditions under which dominant-direction suppression should improve grounding. The intervention experiments provide useful support, but the theoretical contribution is narrower than a general explanation of the proposed mechanism.
-
The strongest matched mechanism controls are concentrated on LLaVA. Performance improvements on other architectures support transfer of the method, but do not establish that the same geometric mechanism explains those gains. Repeating the matched direction and head controls on another backbone would strengthen the central claim. Spatial-grounding measurements would also help distinguish failed selection from appropriate attention to multiple regions.
-
The appendix demonstrates local parameter stability, but the configuration-selection protocol remains insufficiently specified. It is unclear whether the rank, gate, sharpening strength, and layer placement were chosen on held-out images. Since the detailed sensitivity analysis is concentrated on LLaVA/CHAIR, it provides limited evidence about stability across architectures and image distributions.
-
The statistical comparisons primarily establish improvement over greedy decoding, rather than over the strongest competing method. WAS is not uniformly best across model–metric combinations. Paired comparisons against the strongest baseline would better establish the extent of its advantage.
Questions for authors
- Does the hallucination reduction persist when visual attention mass is preserved exactly after correction and blending?
- Under what conditions does low effective rank imply reduced query-dependent selection? Can the proposed information-theoretic account predict which heads benefit from correction?
- Can the matched direction and head controls be replicated on InstructBLIP or Qwen, with a direct measure of spatial grounding?
- Were the configuration and intervention layers selected on held-out data? Does the reported parameter stability extend to another architecture?
8 comments
W2 / Q2. Under what condition does low effective rank reduce query-dependent selection, and can the account predict which heads benefit?
W2. The theory does not establish the central link from low effective rank to failed query-dependent selection. Low-rank keys can still produce different attention patterns for different queries. The analysis therefore does not identify sufficient conditions under which dominant-direction suppression should improve grounding. The intervention experiments provide useful support, but the theoretical contribution is narrower than a general explanation of the proposed mechanism.
Q2. Under what conditions does low effective rank imply reduced query-dependent selection? Can the proposed information-theoretic account predict which heads benefit from correction?
A2. Thank you for this question; answering it has made the theory more complete. The new Proposition 5 (App. F.6) states when collapse lowers query dependence and predicts the gain of each head.
- Split. Write the centered visual scores at step as , the parts along and outside the dominant directions (Eq. (24)). Because consists of right singular vectors of the centered keys, the two parts are orthogonal at all pairs of steps, and the correlation between two steps is exactly (Eq. (25))
where is the dominant share of the score variance.
- Condition. For near-uniform attention and a common score scale and share , a second-order expansion of the JSD (median relative error 7% on gated heads) gives (Eq. (26))
A larger lowers Q if and only if , i.e. the dominant part is more stable across generated words than the residual part. This identifies precisely when low-rank keys reduce query dependence, which covers the case you raise.
- Per-head prediction. Under a full correction (), WAS removes nearly all the weight of the dominant directions (App. F.2) and sets the score scale to , so a head gains (Eq. (27))
We tested this on all 384 heads in layers 9–20 of LLaVA-1.5: captions are teacher-forced, the measured ΔQ of a head comes from correcting that head alone (g = 1, α = 1.2), and gated heads are those on which the gate fires in more than 10% of steps (Figure 3b).
Table R2. Test of Proposition 5, LLaVA-1.5-7B (revised PDF: Table 32).
| Quantity | Value |
|---|---|
| Mean dominant-part correlation across steps, gated heads | 0.8 |
| Mean residual-part correlation across steps, gated heads | 0.4 |
| Gated heads satisfying the condition | 88% (21 of 24) |
| Spearman correlation, predicted vs. measured ΔQ (384 heads) | 0.7 |
| Gated heads among the 10% with the largest predicted gain | 18 of 24 (chance: 2.4) |
- The condition holds on the heads where WAS acts, and the formula ranks heads by their gain in query dependence.
- This fits the much smaller reduction when the same correction is applied to non-collapsed heads (Table 21, −6.4 against −24.0).
- That the added query dependence lands on the described objects is measured directly in A3 (Table R4).
Changes in the revised PDF (blue):
- App. F.6: Proposition 5 with its proof.
- App. Q.4: the test on 384 heads (Table 32).
- Sec. 3.3: one sentence stating the condition.
W3 / Q3. Repeat the matched direction and head controls on another backbone, with a direct measure of spatial grounding.
W3. The strongest matched mechanism controls are concentrated on LLaVA. Performance improvements on other architectures support transfer of the method, but do not establish that the same geometric mechanism explains those gains. Repeating the matched direction and head controls on another backbone would strengthen the central claim. Spatial-grounding measurements would also help distinguish failed selection from appropriate attention to multiple regions.
Q3. Can the matched direction and head controls be replicated on InstructBLIP or Qwen, with a direct measure of spatial grounding?
A3. Thank you for this suggestion; both additions strengthen the central claim. We used Qwen3-VL-4B, whose visual tokens correspond to image patches, so that grounding can be measured; the 32 Q-Former tokens of InstructBLIP have no spatial position.
Matched controls, built as in Table 21, with the rank and correction scale of WAS on Qwen3-VL-4B and random directions averaged over five seeds.
Table R3. Matched controls on Qwen3-VL-4B, CHAIR (revised PDF: Table 24).
| Configuration | CHAIR_S ↓ | Δ from greedy [95% CI] |
|---|---|---|
| Greedy | 26.0 | – |
| Reverse (λ = −1) | 27.4 | +1.4 [−1.4, +4.4] |
| Random directions (5 seeds) | 23.5 | −2.5 [−5.4, +0.4] |
| Weakest directions | 24.0 | −2.0 [−4.9, +0.9] |
| Non-collapsed heads | 23.0 | −3.0 [−6.0, −0.1] |
| Sharpening only | 22.6 | −3.4 [−6.3, −0.3] |
| WAS | 17.8 | −8.2 [−12.0, −4.4] |
- The strongest control, sharpening alone, reaches 41% of the reduction of WAS (−3.4 of −8.2), close to the 39% on LLaVA-1.5 (−9.4 of −24.0).
- Every paired comparison with WAS remains significant after Holm correction (p < 0.01).
- The reverse correction raises CHAIR_S as on LLaVA-1.5, by a smaller amount, in line with the milder collapse of Qwen3-VL-4B (Table 18).
Spatial grounding. At each correctly mentioned object word, we take the heads on which the gate fires in the unmodified teacher-forced pass and use the same heads and words in every setting. The ratio divides their visual attention inside the union of all ground-truth boxes of that category by the share of the image the union covers (1 = random placement). Taking the union counts attention spread over several instances as correct selection, which separates the two cases you mention.
Table R4. Spatial grounding ratio with 95% CI (revised PDF: Table 25).
| Setting | LLaVA-1.5 | Qwen3-VL-4B |
|---|---|---|
| Greedy | 1.0 [0.97, 1.03] | 1.1 [1.07, 1.13] |
| Sharpening only | 1.1 [1.07, 1.13] | 1.1 [1.08, 1.14] |
| WAS | 1.4 [1.34, 1.46] | 1.3 [1.25, 1.35] |
| Reverse | 0.9 [0.87, 0.93] | 1.0 [0.96, 1.04] |
- WAS raises the ratio from 1.0 to 1.4 on LLaVA-1.5 and from 1.1 to 1.3 on Qwen3-VL-4B; sharpening alone moves it much less, so the gain comes from the spectral correction.
- The reverse correction brings attention to or below the random level.
- Fewer hallucinations thus go together with more attention on the described objects.
Changes in the revised PDF (blue):
- App. P: new paragraphs “Second backbone” (Table 24) and “Spatial grounding” (Table 25).
Thank you for your time and for the constructive comments on the budget control, the theory, the second backbone, the configuration protocol and the baselines; addressing them has strengthened the paper. We address each point below. Changes in the revised PDF are marked in blue; Tables 1–21 keep their original numbers, and new tables start at Table 22. Each table in this response (R1–R7) gives its location in the revised PDF.
W1 / Q1. Does the reduction persist when the visual attention mass is preserved exactly after correction and blending?
A1. Yes, almost unchanged. Thank you for this suggestion, which makes the attribution exact. WAS already keeps the visual log-sum-exp (LSE) close to its original value (average shift 0.06, App. F.3); the control below fixes it at every corrected head and step.
Table R1. Exact attention budget, LLaVA-1.5-7B, CHAIR (revised PDF: Table 22).
Changes in the revised PDF (blue):