- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes the anchoring effect of attention sinks (AS) and reveal that masking the AS can mitigate hallucinations, but it may degrade visual reasoning and broader model capabilities. Motivated by these findings, this author propose gated sink attention redistribution, which uses a simple MLP as the gate that learns the fraction of attention retained by each sink. Experiments on diverse general visual understanding, hallucination, and visual reasoning benchmarks, show that the proposed method consistently improves overall performance while reducing hallucination.
Critical strengths
-
Unlike prior works that consider the AS as detrimental artifacts, this paper systematically investigates the anchoring effect of AS and further reveal that suppressing them leads to a hallucination-reasoning trade-off.
-
Extensive experiments show that the proposed method consistently improves overall performance while reducing hallucination.
Critical weaknesses
The novelty of the proposed G-SAR needs further clarification. Since the sink-to-visual attention redistribution is similar to VAR and the learning of gating modules is related to LSG. Could the authors more clearly explain the unique technical contributions?
Questions for authors
-
G-SAR distribute the attention mass released from AS according to the original query-visual attention, but this does not seem to guarantee directly enhancing visual evidence. Several methods (e.g., [1,2]) directly identify and enhance the salient visual evidence. Could the authors compare these methods to better show the effectiveness of the redistribution strategy?
-
The paper lacks an inference time comparison with the baseline methods. Do the gating module and redistribution cause significant inference delay?
[1] Qi et al., Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation, ICML 2026
[2] Xie et al., Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy, ICML 2026
4 comments
Response to Reviewer BT1K (Continue)
2. About the inference efficiency.
The paper lacks an inference time comparison with the baseline methods. Do the gating module and redistribution cause significant inference delay?
We thank the reviewer for this valuable suggestion. We evaluate the inference efficiency of all methods using Qwen3-VL-8B on the MME benchmark under identical settings. As shown in the table below, G-SAR achieves an average inference time of 0.356 s/sample, which is 56.5% lower than the average of the five baseline methods (0.818 s/sample). Its inference cost is comparable to existing attention regulation methods, including VAR (0.319 s), DIYSink (0.389 s), and LSG (0.351 s).
Importantly, G-SAR achieves the best performance across representative benchmarks covering general visual understanding (GQA: 74.2), hallucination mitigation (CHAIR-S: 20.7), and visual reasoning (SciQA: 70.8; MathVista: 40.9). These results demonstrate that G-SAR achieves a favorable balance between inference efficiency and performance across diverse visual tasks.
Table: Performance and inference efficiency of Qwen3-VL-8B. Inference time is measured on MME.
| Method | GQA ↑ | CHAIR-S ↓ | SciQA ↑ | MathVista ↑ | Time (s/sample) ↓ | Relative Latency ↓ |
|---|---|---|---|---|---|---|
| Greedy | 72.4 | 24.3 | 68.0 | 37.4 | 0.164 | 1.00× |
| OPERA | 72.3 | 22.5 | 63.0 | 36.0 | 2.830 | 17.26× |
| TAME | 71.7 | 20.8 | 65.9 | 33.4 | 0.201 | 1.23× |
| VAR | 72.3 | 24.7 | 68.5 | 38.2 | 0.319 | 1.95× |
| DIYSink | 73.0 | 23.8 | 69.0 | 39.9 | 0.389 | 2.37× |
| LSG | 73.9 | 24.0 | 68.1 | 40.0 | 0.351 | 2.14× |
| Average (w/o Greedy & Ours) | 72.6 | 23.2 | 66.9 | 37.5 | 0.818 | 4.99× |
| G-SAR (Ours) | 74.2 | 20.7 | 70.8 | 40.9 | 0.356 | 2.17× |
3. Clarifying the Novelty of G-SAR.
The novelty of the proposed G-SAR needs further clarification.
We thank the reviewer for this important observation. While G-SAR shares certain design elements with VAR and LSG, its novelty lies in translating our findings on the structural anchoring effect of attention sinks into a unified, anchoring-aware regulation mechanism, rather than introducing gating or redistribution as isolated components.
First, G-SAR is motivated by a distinct empirical finding. Our analysis reveals that attention sinks are not merely redundant attention attractors but also exert substantial influence on the model's internal organization. Specifically, sink-targeted perturbations induce 3.4–4.0× greater representation drift than matched random controls (Table 2), while excessive sink suppression can severely degrade visual understanding and reasoning (Table 3). These findings reveal a critical design consideration: effective sink regulation should alleviate excessive attention concentration without indiscriminately disrupting the structural role of attention sinks.
Second, G-SAR addresses this challenge through three coordinated designs. (1) Unlike VAR's fixed redistribution, G-SAR learns input-adaptive, sink-specific retention gates to determine how much attention should be preserved. (2) Unlike LSG's gating-based adjustment, G-SAR explicitly conserves and redistributes the released attention mass toward visual non-sink tokens, reducing excessive sink dominance while strengthening visual information utilization. (3) Crucially, an anchoring-aware retention constraint discourages excessive sink suppression, directly reflecting our empirical findings.
Third, G-SAR achieves a better hallucination–reasoning trade-off. On LLaVA-1.5-7B (Table 4), G-SAR achieves 46.6 CHAIR-S / 28.9 MathVista, outperforming VAR (48.1 / 25.6) and LSG (51.2 / 24.9). These results demonstrate that anchoring-aware regulation improves hallucination mitigation without sacrificing reasoning performance.
In summary, G-SAR's contribution lies in connecting the structural role of attention sinks to their principled regulation. We have clarified this motivation–design connection and further distinguished G-SAR from prior methods in the revised manuscript.
Reference
[1] Qi et al., Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation, ICML 2026
[2] Xie et al., Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy, ICML 2026
Thanks to the authors for responding to my questions. The authors addressed all my concerns, and I am willing to raise my score.
Thank you for your thoughtful reassessment and continued support! We are delighted that our revisions have addressed your concerns and sincerely appreciate your willingness to raise your score. We are grateful for your constructive feedback and encouragement!
Response to Reviewer BT1K: Clarifications and Additional Experimental Evidence
Thank you for your insightful comments and for highlighting the issues that could inform a more favorable assessment. We appreciate your concerns regarding the effectiveness of G-SAR's attention redistribution strategy, its inference efficiency, and its technical novelty. We address each of these points in detail below.
1. On the effectiveness of G-SAR's attention redistribution strategy
We appreciate this important observation. We agree that redistributing attention according to the pretrained query–visual attention distribution does not theoretically guarantee identification of the most semantically relevant visual evidence.
Our objective is different from explicitly estimating visual saliency: G-SAR aims to alleviate excessive sink dominance while preserving the pretrained attention structure, rather than introducing an additional visual evidence selection mechanism. Our redistribution ablations provide empirical support for this design choice. As shown in Table 5, query-guided visual redistribution improves GQA by 2.0 points and MathVista by 3.5 points on Qwen3-VL-8B, but worsens CHAIR-S by 2.4 points relative to the baseline. In contrast, redistribution to visual non-sink tokens improves GQA by 1.8 points and MathVista by 3.5 points while reducing CHAIR-S by 3.6 points.
These findings suggest that stronger attention concentration on query-relevant visual tokens does not necessarily yield a better balance between hallucination mitigation and reasoning. The visual non-sink redistribution strategy provides a more balanced alternative in our experiments.
That's a good idea. Thank you! Importantly, G-SAR is complementary to GIFT [1] and ILVAD [2] rather than an alternative to them. While these methods explicitly identify and enhance salient visual evidence, G-SAR addresses a distinct issue: excessive attention concentration on structurally influential attention sinks. Rather than selecting visual evidence, G-SAR adaptively regulates sink attention while preserving its anchoring role.
To verify this complementarity, we integrate G-SAR with both methods on Qwen3-VL-8B. As shown below, adding G-SAR improves all nine evaluated metrics for both GIFT and ILVAD. In particular, ILVAD + G-SAR improves MMMU by +5.4, ScienceQA by +8.0, and MathVista by +5.7 points, while consistently improving all three POPE subsets. Similar gains are observed with GIFT.
Table: Complementarity of G-SAR with visual evidence enhancement methods on Qwen3-VL-8B.
These consistent improvements demonstrate that enhancing salient visual evidence alone does not exhaust the benefits of attention regulation. G-SAR provides complementary gains across visual understanding, hallucination mitigation, and reasoning, further validating the importance and general applicability of sink regulation.
We have incorporated these additional experiments and corresponding analyses into the experimental section of the revised manuscript. We sincerely thank the reviewer for these insightful suggestions, which have helped us strengthen our work.