- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies multimodal hallucination in MLLMs from the perspective of head-level visual attention sinks. The authors argue that the visual attention ratio, which is commonly used to identify “visual heads,” does not reflect whether a head’s attention is spatially meaningful: high-ratio heads may collapse onto fixed, semantically irrelevant tokens, while some low-ratio heads still attend to meaningful regions. Based on this observation, they propose Visual Sink Reweighting (VSR), a training-free inference-time method. VSR (i) builds a Visual Sink prototype by averaging mutually similar, ℓ2-normalized attention maps drawn from low-ratio heads, (ii) scores each head by its cosine similarity to this prototype (SinkScore), and (iii) uses two thresholds to down-weight high-SinkScore heads and up-weight low-SinkScore heads. Experiments on CHAIR, POPE, and MME with LLaVA-1.5-7B/13B and Qwen-VL-7B show consistent gains over regular decoding, VCD, and VAF. The appendix adds threshold sensitivity, head-masking, and qualitative analyses.
Critical strengths
1. Interesting and well-motivated observation, turned into a simple training-free method. The paper makes a nice point that the spatial structure of a head’s visual attention matters more than its visual attention ratio when deciding which heads actually help grounding. This challenges a common assumption in prior head-selection work. The authors then build a typical Visual Sink pattern in an unsupervised way and use similarity to it to reweight heads at inference time, which reduces hallucination fairly effectively without any training. I find this perspective useful for the community, and it is the main reason I see value in the paper.
Critical weaknesses
1. Several key claims are not supported by experiments.
Some important statements in the paper lack empirical evidence. For example, Sec. 3.1 claims that, for the same image, the tokens attended by sink-like heads stay highly consistent across prompts and generation steps. This is the premise for aggregating maps across contexts to build the prototype, but no quantitative verification is provided. More importantly, the central claim that selecting heads by SinkScore is better than selecting them by visual attention ratio is never tested in a controlled way: the only comparison is end-to-end against VAF, so the gain cannot be attributed to the selection criterion itself. This weakens both the interpretability and the generality of the method.
2. The prototype construction is in tension with the paper’s own motivation.
The paper stresses that high-ratio heads can also be sink heads, yet the prototype is built only from low-ratio heads. If high-ratio sink heads have different spatial patterns (e.g., a different number or location of peaks), the prototype would be biased, and such heads could be underestimated or misclassified. This issue is neither discussed nor verified.
3. Inconsistent descriptions and missing details hurt reproducibility.
The candidate set is defined as in Eq. (2) but as in Fig. 3, which mean different things; the weighting function in Eq. (8) also differs from Fig. 3. The values of γ, a, and b, and the exact procedure for selecting , are not given. In addition, the caption of Fig. 5 does not match its content, and a method label in Fig. 7 appears to be wrong.
Questions for authors
-
(Follow-up on W1) Could the authors provide quantitative evidence that sink locations are consistent across prompts and generation steps (e.g., IoU of top-k sink tokens across prompts, or cosine similarity between prototypes)? Also, could you run a controlled comparison where the reweighting scheme is identical and only the head-selection criterion is switched from SinkScore to visual attention ratio?
-
(Follow-up on W2) If prototypes are built separately from high-ratio and low-ratio sink heads, how similar are they? Does a prototype built from all heads perform differently? Please also clarify how is selected from the pairwise similarities (threshold, top-k, or clustering) and what value of γ is used.
-
Where does the improvement mainly come from? Table 4 shows that masking sink-prone heads barely changes performance, so what does down-weighting them contribute? Could you report results for “only down-weighting sink-prone heads” and “only up-weighting non-sink heads” separately? Also, what fraction of heads is classified as non-sink under the default thresholds, and could you add a random-masking control with the same number of heads to Table 4?