acceptodds
Under review as a conference paper at ICLR 2027

Beyond Visual Attention Magnitude: Visual Sink Reweighting for Mitigating Multimodal Hallucination

12 views

Abstract

Multimodal Large Language Models (MLLMs) have demon￾strated strong vision-language capabilities, yet they remain prone to multimodal hallucination, where generated responses are inconsistent with visual content Recent studies have iden￾tified the Visual Attention Sink phenomenon, showing that MLLMs may allocate excessive attention to visually uninfor￾mative or semantically irrelevant visual tokens. While exist￾ing work has analyzed sink tokens and begun to distinguish sinkprone attention heads, the relationship between visual attention magnitude and the structural quality of headlevel visual attention remains insufficiently understood. As a result, highvisualattention heads may be mistakenly regarded as effective visual heads even when they exhibit sinklike patterns, whereas lowvisualattention heads with meaningful visual structures may be overlooked. In this paper, we conduct a structureaware headlevel analysis of Visual Sink and show that visual attention magnitude alone is insufficient for identifying effective visual heads. Motivated by this observation, we propose Visual Sink Reweighting (VSR), a trainingfree method for mitigating multimodal hallucination. VSR constructs a prototypical Visual Sink pattern and introduces SinkScore to measure the structural similarity between each attention head and the prototype. Based on SinkScore, VSR suppresses sink-prone heads while enhancing non-sink heads whose visual attention structures deviate from the Visual Sink prototype during generation. Extensive experiments on multiple benchmarks and MLLMs demonstrate that VSR consis￾tently reduces hallucination and improves visual grounding without additional training or parameter updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Multimodal Hallucination, Visual Attention, Attention Head Analysis, Visual Grounding, Training-Free Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Rrk7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multimodal hallucination in MLLMs from the perspective of head-level visual attention sinks. The authors argue that the visual attention ratio, which is commonly used to identify “visual heads,” does not reflect whether a head’s attention is spatially meaningful: high-ratio heads may collapse onto fixed, semantically irrelevant tokens, while some low-ratio heads still attend to meaningful regions. Based on this observation, they propose Visual Sink Reweighting (VSR), a training-free inference-time method. VSR (i) builds a Visual Sink prototype by averaging mutually similar, ℓ2-normalized attention maps drawn from low-ratio heads, (ii) scores each head by its cosine similarity to this prototype (SinkScore), and (iii) uses two thresholds to down-weight high-SinkScore heads and up-weight low-SinkScore heads. Experiments on CHAIR, POPE, and MME with LLaVA-1.5-7B/13B and Qwen-VL-7B show consistent gains over regular decoding, VCD, and VAF. The appendix adds threshold sensitivity, head-masking, and qualitative analyses.

Critical strengths

1. Interesting and well-motivated observation, turned into a simple training-free method. The paper makes a nice point that the spatial structure of a head’s visual attention matters more than its visual attention ratio when deciding which heads actually help grounding. This challenges a common assumption in prior head-selection work. The authors then build a typical Visual Sink pattern in an unsupervised way and use similarity to it to reweight heads at inference time, which reduces hallucination fairly effectively without any training. I find this perspective useful for the community, and it is the main reason I see value in the paper.

Critical weaknesses

1. Several key claims are not supported by experiments.

Some important statements in the paper lack empirical evidence. For example, Sec. 3.1 claims that, for the same image, the tokens attended by sink-like heads stay highly consistent across prompts and generation steps. This is the premise for aggregating maps across contexts to build the prototype, but no quantitative verification is provided. More importantly, the central claim that selecting heads by SinkScore is better than selecting them by visual attention ratio is never tested in a controlled way: the only comparison is end-to-end against VAF, so the gain cannot be attributed to the selection criterion itself. This weakens both the interpretability and the generality of the method.

2. The prototype construction is in tension with the paper’s own motivation.

The paper stresses that high-ratio heads can also be sink heads, yet the prototype is built only from low-ratio heads. If high-ratio sink heads have different spatial patterns (e.g., a different number or location of peaks), the prototype would be biased, and such heads could be underestimated or misclassified. This issue is neither discussed nor verified.

3. Inconsistent descriptions and missing details hurt reproducibility.

The candidate set is defined as r<γr<\gamma in Eq. (2) but as ∣r−rˉ∣≤τr|r-\bar r|\le\tau_r in Fig. 3, which mean different things; the weighting function in Eq. (8) also differs from Fig. 3. The values of γ, a, and b, and the exact procedure for selecting CsinkC_{\text{sink}}, are not given. In addition, the caption of Fig. 5 does not match its content, and a method label in Fig. 7 appears to be wrong.

Questions for authors
  1. (Follow-up on W1) Could the authors provide quantitative evidence that sink locations are consistent across prompts and generation steps (e.g., IoU of top-k sink tokens across prompts, or cosine similarity between prototypes)? Also, could you run a controlled comparison where the reweighting scheme is identical and only the head-selection criterion is switched from SinkScore to visual attention ratio?

  2. (Follow-up on W2) If prototypes are built separately from high-ratio and low-ratio sink heads, how similar are they? Does a prototype built from all heads perform differently? Please also clarify how CsinkC_{\text{sink}} is selected from the pairwise similarities (threshold, top-k, or clustering) and what value of γ is used.

  3. Where does the improvement mainly come from? Table 4 shows that masking sink-prone heads barely changes performance, so what does down-weighting them contribute? Could you report results for “only down-weighting sink-prone heads” and “only up-weighting non-sink heads” separately? Also, what fraction of heads is classified as non-sink under the default thresholds, and could you add a random-masking control with the same number of heads to Table 4?

Reviewer_eGVCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that the visual attention ratio (how much attention a head puts on image tokens) is a poor indicator of whether that head actually grounds the model in the image. Some low-ratio heads still have spatially meaningful attention maps, while some high-ratio heads collapse onto sink tokens. The authors build a "Visual Sink prototype" from normalized attention maps of low-ratio heads, score every head by cosine similarity to it (SinkScore), and then, at inference time, scale each head's output: high-SinkScore heads get discounted, low-SinkScore heads get boosted. They test on CHAIR, POPE and four MME perception subsets, using LLaVA-1.5-7B/13B and Qwen-VL-7B, compared with regular decoding, VCD and VAF.

Critical strengths
  • Separating attention magnitude from spatial structure is a sensible framing, and Fig. 2(a) does show the two are not interchangeable.
  • The method is simple: one scalar per head, no training, easy to combine with other decoding-time tricks.
  • The Mask-SPH result in Table 4 (zeroing sink-prone heads barely changes POPE on two backbones) is interesting on its own.
Critical weaknesses
  • Overlap with VAR. VAR already selects "image-centric" heads by a visual non-sink ratio, i.e., visual attention computed after excluding sink tokens, so the confound the paper says has been overlooked is precisely what VAR addresses. The related work mentions VAR only as token-level redistribution. What remains new is soft template matching rather than explicit sink identification, plus output scaling rather than attention editing; neither is tested against the simpler alternative (e.g., a head's share of visual attention on VAR's sink tokens is also magnitude-free).

  • The decisive ablation is missing. The claim is that SinkScore beats ratio for choosing heads. The direct test — the same reweighting rule with heads chosen by ratio (or VAR's non-sink ratio), matched head counts, plus suppression-only and amplification-only variants — is not run. The comparison is against VAF, a different intervention. Table 4 also lacks controls: Mask-NSH removes far more heads than Mask-SPH (~half the network if Fig. 2(c) is representative), so falling to near-chance POPE accuracy shows the model broke, not that those heads carry grounding.

Questions for authors
  • Is the prototype per image or global, and at which pass/step is it computed? What is the overhead relative to regular decoding and VAR?
  • What exactly is the C_sink selection rule, and what are γ, a, b? Which is correct, Fig. 3 or Eq. 8?
Reviewer_EyvZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Visual Sink Reweighting (VSR), a training-free method that uses the structural similarity between attention maps and a visual sink prototype to reweight attention heads during decoding. Experiments are conducted on three hallucination benchmarks based on three base MLLMs.

Critical strengths

This paper is easy to follow.

Critical weaknesses
  1. Inference-time attention reweighting has already been widely explored for mitigating multimodal hallucination. The paper does not sufficiently establish that SinkScore-based head selection (the main novelty of VSR) is more effective or generalizable than existing attention- and activation-based selection strategies.
  2. Eq.(2) selects candidates with low visual attention ratios (r<γ), whereas Fig.3 selects candidates whose ratios are close to a reference value (|r-\bar{r}|<τ). These criteria are not generally equivalent. The authors should clarify which rule is used in the experiments and define the corresponding parameters.
  3. The experiments rely on LLaVA-v1.5 and Qwen-VL-7B, while the baselines are limited to relatively early methods such as VCD and VAR. Recent attention-based or activation-based methods should be included, and the method should be evaluated on more recent MLLMs (e.g., QWen3-VL) to substantiate its claimed generality.
  4. The proposed SinkScore may introduce nontrivial inference and memory overhead. However, Tthe paper does not report runtime, memory consumption, or the cost of prototype construction.
Questions for authors

Please refer to the issues raised in the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.