acceptodds
Under review as a conference paper at ICLR 2027

The Anchoring Effect of Attention Sinks

27 views

Abstract

Attention Sinks (AS), tokens that accumulate disproportionately high attention weights, are commonly regarded as detrimental artifacts in Large Vision-Language Models (LVLMs), often associated with hallucination and attention redundancy. In this work, we show that this view is incomplete: AS also serve as structural anchors in LVLMs. Specifically, AS persist across layers and attention heads, while targeted perturbations induce disproportionate changes in attention organization and representation geometry. Further masking experiments reveal their dual role: Masking sink attention can mitigate hallucinations, but this benefit may come at the cost of degraded visual reasoning and broader model capabilities. Motivated by these findings, we propose Gated Sink Attention Redistribution (G-SAR), a simple input-adaptive gating mechanism that learns how much attention to release from sink tokens and redistributes the released mass toward visual non-sink tokens. Across three LVLMs and diverse general visual understanding, hallucination, and visual reasoning benchmarks, G-SAR consistently improves overall performance while reducing hallucination. Our results suggest that AS should be regulated rather than eliminated, highlighting adaptive sink control as a more principled strategy to balance visual grounding and reasoning in LVLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention Sinks, Anchoring Effect, Large Vision-Language Models, Attention Redistribution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BT1KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the anchoring effect of attention sinks (AS) and reveal that masking the AS can mitigate hallucinations, but it may degrade visual reasoning and broader model capabilities. Motivated by these findings, this author propose gated sink attention redistribution, which uses a simple MLP as the gate that learns the fraction of attention retained by each sink. Experiments on diverse general visual understanding, hallucination, and visual reasoning benchmarks, show that the proposed method consistently improves overall performance while reducing hallucination.

Critical strengths
  1. Unlike prior works that consider the AS as detrimental artifacts, this paper systematically investigates the anchoring effect of AS and further reveal that suppressing them leads to a hallucination-reasoning trade-off.

  2. Extensive experiments show that the proposed method consistently improves overall performance while reducing hallucination.

Critical weaknesses

The novelty of the proposed G-SAR needs further clarification. Since the sink-to-visual attention redistribution is similar to VAR and the learning of gating modules is related to LSG. Could the authors more clearly explain the unique technical contributions?

Questions for authors
  1. G-SAR distribute the attention mass released from AS according to the original query-visual attention, but this does not seem to guarantee directly enhancing visual evidence. Several methods (e.g., [1,2]) directly identify and enhance the salient visual evidence. Could the authors compare these methods to better show the effectiveness of the redistribution strategy?

  2. The paper lacks an inference time comparison with the baseline methods. Do the gating module and redistribution cause significant inference delay?

[1] Qi et al., Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation, ICML 2026

[2] Xie et al., Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy, ICML 2026

4 comments
AuthorsOpenReview ↗

Response to Reviewer BT1K: Clarifications and Additional Experimental Evidence

Thank you for your insightful comments and for highlighting the issues that could inform a more favorable assessment. We appreciate your concerns regarding the effectiveness of G-SAR's attention redistribution strategy, its inference efficiency, and its technical novelty. We address each of these points in detail below.

1. On the effectiveness of G-SAR's attention redistribution strategy

G-SAR distribute the attention mass released from AS according to the original query-visual attention, but this does not seem to guarantee directly enhancing visual evidence.

We appreciate this important observation. We agree that redistributing attention according to the pretrained query–visual attention distribution does not theoretically guarantee identification of the most semantically relevant visual evidence.

Our objective is different from explicitly estimating visual saliency: G-SAR aims to alleviate excessive sink dominance while preserving the pretrained attention structure, rather than introducing an additional visual evidence selection mechanism. Our redistribution ablations provide empirical support for this design choice. As shown in Table 5, query-guided visual redistribution improves GQA by 2.0 points and MathVista by 3.5 points on Qwen3-VL-8B, but worsens CHAIR-S by 2.4 points relative to the baseline. In contrast, redistribution to visual non-sink tokens improves GQA by 1.8 points and MathVista by 3.5 points while reducing CHAIR-S by 3.6 points.

These findings suggest that stronger attention concentration on query-relevant visual tokens does not necessarily yield a better balance between hallucination mitigation and reasoning. The visual non-sink redistribution strategy provides a more balanced alternative in our experiments.

Several methods (e.g., [1,2]) directly identify and enhance the salient visual evidence. Could the authors compare these methods to better show the effectiveness of the redistribution strategy?

That's a good idea. Thank you! Importantly, G-SAR is complementary to GIFT [1] and ILVAD [2] rather than an alternative to them. While these methods explicitly identify and enhance salient visual evidence, G-SAR addresses a distinct issue: excessive attention concentration on structurally influential attention sinks. Rather than selecting visual evidence, G-SAR adaptively regulates sink attention while preserving its anchoring role.

To verify this complementarity, we integrate G-SAR with both methods on Qwen3-VL-8B. As shown below, adding G-SAR improves all nine evaluated metrics for both GIFT and ILVAD. In particular, ILVAD + G-SAR improves MMMU by +5.4, ScienceQA by +8.0, and MathVista by +5.7 points, while consistently improving all three POPE subsets. Similar gains are observed with GIFT.

Table: Complementarity of G-SAR with visual evidence enhancement methods on Qwen3-VL-8B.

MethodGQA ↑MME-P ↑MME-C ↑MMMU ↑POPE-R ↑POPE-P ↑POPE-A ↑ScienceQA ↑MathVista ↑
Greedy72.41716.7613.929.092.790.190.568.037.4
G-SAR74.21735.0619.929.993.290.990.970.840.9
GIFT72.21725.0615.634.492.190.090.374.040.6
GIFT + G-SAR72.81736.1620.535.692.390.190.576.841.2
ILVAD68.61691.7583.333.892.789.290.072.038.6
ILVAD + G-SAR71.61709.4602.839.293.890.491.080.044.3

These consistent improvements demonstrate that enhancing salient visual evidence alone does not exhaust the benefits of attention regulation. G-SAR provides complementary gains across visual understanding, hallucination mitigation, and reasoning, further validating the importance and general applicability of sink regulation.

We have incorporated these additional experiments and corresponding analyses into the experimental section of the revised manuscript. We sincerely thank the reviewer for these insightful suggestions, which have helped us strengthen our work.

AuthorsOpenReview ↗

Response to Reviewer BT1K (Continue)

2. About the inference efficiency.

The paper lacks an inference time comparison with the baseline methods. Do the gating module and redistribution cause significant inference delay?

We thank the reviewer for this valuable suggestion. We evaluate the inference efficiency of all methods using Qwen3-VL-8B on the MME benchmark under identical settings. As shown in the table below, G-SAR achieves an average inference time of 0.356 s/sample, which is 56.5% lower than the average of the five baseline methods (0.818 s/sample). Its inference cost is comparable to existing attention regulation methods, including VAR (0.319 s), DIYSink (0.389 s), and LSG (0.351 s).

Importantly, G-SAR achieves the best performance across representative benchmarks covering general visual understanding (GQA: 74.2), hallucination mitigation (CHAIR-S: 20.7), and visual reasoning (SciQA: 70.8; MathVista: 40.9). These results demonstrate that G-SAR achieves a favorable balance between inference efficiency and performance across diverse visual tasks.

Table: Performance and inference efficiency of Qwen3-VL-8B. Inference time is measured on MME.

MethodGQA ↑CHAIR-S ↓SciQA ↑MathVista ↑Time (s/sample) ↓Relative Latency ↓
Greedy72.424.368.037.40.1641.00×
OPERA72.322.563.036.02.83017.26×
TAME71.720.865.933.40.2011.23×
VAR72.324.768.538.20.3191.95×
DIYSink73.023.869.039.90.3892.37×
LSG73.924.068.140.00.3512.14×
Average (w/o Greedy & Ours)72.623.266.937.50.8184.99×
G-SAR (Ours)74.220.770.840.90.3562.17×

3. Clarifying the Novelty of G-SAR.

The novelty of the proposed G-SAR needs further clarification.

We thank the reviewer for this important observation. While G-SAR shares certain design elements with VAR and LSG, its novelty lies in translating our findings on the structural anchoring effect of attention sinks into a unified, anchoring-aware regulation mechanism, rather than introducing gating or redistribution as isolated components.

First, G-SAR is motivated by a distinct empirical finding. Our analysis reveals that attention sinks are not merely redundant attention attractors but also exert substantial influence on the model's internal organization. Specifically, sink-targeted perturbations induce 3.4–4.0× greater representation drift than matched random controls (Table 2), while excessive sink suppression can severely degrade visual understanding and reasoning (Table 3). These findings reveal a critical design consideration: effective sink regulation should alleviate excessive attention concentration without indiscriminately disrupting the structural role of attention sinks.

Second, G-SAR addresses this challenge through three coordinated designs. (1) Unlike VAR's fixed redistribution, G-SAR learns input-adaptive, sink-specific retention gates to determine how much attention should be preserved. (2) Unlike LSG's gating-based adjustment, G-SAR explicitly conserves and redistributes the released attention mass toward visual non-sink tokens, reducing excessive sink dominance while strengthening visual information utilization. (3) Crucially, an anchoring-aware retention constraint discourages excessive sink suppression, directly reflecting our empirical findings.

Third, G-SAR achieves a better hallucination–reasoning trade-off. On LLaVA-1.5-7B (Table 4), G-SAR achieves 46.6 CHAIR-S / 28.9 MathVista, outperforming VAR (48.1 / 25.6) and LSG (51.2 / 24.9). These results demonstrate that anchoring-aware regulation improves hallucination mitigation without sacrificing reasoning performance.

In summary, G-SAR's contribution lies in connecting the structural role of attention sinks to their principled regulation. We have clarified this motivation–design connection and further distinguished G-SAR from prior methods in the revised manuscript.

Reference

[1] Qi et al., Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation, ICML 2026

[2] Xie et al., Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy, ICML 2026

Thanks to the authors for responding to my questions. The authors addressed all my concerns, and I am willing to raise my score.

Thank you for your thoughtful reassessment and continued support! We are delighted that our revisions have addressed your concerns and sincerely appreciate your willingness to raise your score. We are grateful for your constructive feedback and encouragement!

Reviewer_txJwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates attention sinks in vision-language models, showing that they are not merely harmful artifacts but structural anchors supporting attention organization and representation geometry. Although suppressing them reduces hallucinations, it can weaken visual reasoning. The authors propose G-SAR, an adaptive gating and redistribution method that regulates sink attention, improving visual understanding and reasoning while reducing hallucinations across three models.

Critical strengths

1.Clear writing and systematic analysis.

2.G-SAR is simple and parameter-efficient.

3.Broad cross-model evaluation: three representative LVLMs and multiple benchmarks covering understanding, hallucination, and reasoning, which offers preliminary evidence of consistency across architectures and task types.

Critical weaknesses

1.Hallucination-reasoning trade-off is not significant enough. 2.Hallucination baselines are outdated. The paper does not compare against SHIELD, CausalLens, and PTI, which are new training-free methods with strong POPE, CHAIR, and MME-H results. 3.No case study or good visualization.

Questions for authors

Can the authors compare G-SAR with recent hallucination baselines and provide qualitative visualizations demonstrating a better hallucination–reasoning trade-off across models?

3 comments
AuthorsOpenReview ↗

Response to Reviewer txjw: Baseline Clarifications and Additional Experiments

Thank you for your insightful comments. We believe that some concerns may stem from misunderstandings, which we aim to clarify in detail below. We hope that our response can clarify the misunderstandings and you can consider our work more favorably.

1. About the baseline selection.

Hallucination baselines are outdated. The paper does not compare against SHIELD, CausalLens, and PTI, which are new training-free methods with strong POPE, CHAIR, and MME-H results.

We apologize for the confusion regarding our baseline selection. Our work specifically investigates the structural role of attention sinks (AS) in LVLMs and proposes G-SAR to adaptively regulate sink attention while preserving its anchoring effect. Accordingly, we primarily select baselines that directly suppress or regulate AS, including OPERA, TAME, VAR, DIYSink, and LSG. Notably, OPERA and TAME were deliberately selected from hallucination mitigation methods because they explicitly intervene in sink attention, rather than because they are older baselines. While SHIELD [1], CausalLens [2], and PTI [3] are effective recent hallucination mitigation methods, they do not explicitly target AS regulation. Thus, our baseline selection reflects methodological relevance rather than recency, ensuring a focused comparison of different AS regulation strategies.

Importantly, SHIELD, CausalLens, and PTI address hallucination through mechanisms distinct from AS regulation, making them complementary to G-SAR rather than direct alternatives. To verify this complementarity, we integrate G-SAR with all three methods on Qwen3-VL-8B. As shown below, there are consistent improvements in general visual understanding and reasoning. Specifically, adding G-SAR improves all three methods on GQA (+1.2/+2.1/+4.4), MMMU (+1.8/+4.5/+2.5), ScienceQA (+3.8/+4.0/+4.3), and MathVista (+2.6/+2.6/+2.3), respectively, with gains reported over SHIELD, CausalLens, and PTI. Meanwhile, POPE performance is largely preserved or improved, with changes ranging from -0.3 to +0.9 points across its three metrics.

Table: Complementarity of G-SAR with SHIELD, CausalLens, and PTI on Qwen3-VL-8B.

MethodGQA ↑MME-P ↑MME-C ↑MMMU ↑POPE-R ↑POPE-P ↑POPE-A ↑ScienceQA ↑MathVista ↑
Greedy72.41716.7613.929.092.790.190.568.037.4
G-SAR74.21735.0619.929.993.290.990.970.840.9
SHIELD71.81798.2580.031.692.790.590.965.637.0
SHIELD + G-SAR73.01790.8600.133.493.290.890.669.439.6
CausalLens68.31720.5612.328.893.091.291.062.035.7
CausalLens + G-SAR70.41729.1614.333.393.991.491.066.038.3
PTI70.41699.2613.730.592.790.290.566.137.6
PTI + G-SAR74.81715.4612.033.092.790.490.470.439.9

These results provide empirical evidence that G-SAR is complementary to existing hallucination-mitigation approaches: it consistently strengthens general visual capabilities while largely retaining their hallucination-related performance. This composability further demonstrates that G-SAR addresses an independent source of performance limitations and can be integrated into existing mitigation frameworks. We have added complementary experiments in the experimental section to demonstrate that G-SAR can complement existing hallucination-mitigation methods.

2. On visualization and the hallucination-reasoning trade-off.

No case study or good visualization. Can the authors provide qualitative visualizations demonstrating a better hallucination–reasoning trade-off across models?

We sincerely thank the reviewer for the constructive suggestions that have helped improve our paper. In response, we have (1) added visualizations (Fig. 3) to better illustrate the hallucination–reasoning trade-off across models; (2) conducted three additional experiments (Lines 508–531) to investigate complementarity with visual evidence enhancement and hallucination mitigation methods, as well as inference efficiency; and (3) included an illustrative example of AS in the Introduction (Fig. 1) to improve clarity. We sincerely appreciate the reviewer's valuable feedback.

Reference

[1] Huang et al., SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense, ICLR 2026.

[2] Ji et al., CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models, CVPR 2026.

[3] Zhang et al., Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models, CVPR 2026.

AuthorsOpenReview ↗

Kind Reminder on Your Questions and Our Response

Dear Reviewer txJw,

Thank you again for your time and constructive feedback on our paper.

We have carefully addressed your concerns in our rebuttal, including additional comparisons with recent hallucination mitigation methods (SHIELD, CausalLens, and PTI), new experiments, and improved visualizations.

We would greatly appreciate it if you could take another look at our response and let us know whether any concerns remain. We sincerely hope that these clarifications and additional results may encourage you to reconsider your assessment of our work.

We would be more than happy to address any further questions or concerns you may have.

Best regards,
Authors

AuthorsOpenReview ↗

Additional Evidence of G-SAR's Complementarity and Generalizability

We sincerely thank the reviewer for their constructive suggestions, as well as the valuable time they have dedicated to reviewing our work. To further strengthen our experimental evidence, we additionally evaluate the integration of G-SAR with SHIELD, CausalLens, and PTI on LLaVA-1.5-7B.

As shown in the table, integrating G-SAR with SHIELD, CausalLens, and PTI consistently improves visual understanding and reasoning performance across GQA, MME, MMMU, ScienceQA, and MathVista. Meanwhile, hallucination mitigation performance is preserved or further improved across POPE and CHAIR. These results further demonstrate the complementarity and generalizability of G-SAR across different LVLM architectures.

Table: Complementarity of G-SAR with SHIELD, CausalLens, and PTI on LLaVA-1.5-7B. POPE reports the average accuracy across the Random, Popular, and Adversarial subsets.

MethodGQA ↑MME-P ↑MME-C ↑MMMU ↑POPE ↑CHAIR-S ↓CHAIR-I ↓ScienceQA ↑MathVista ↑
Greedy61.91466.9282.927.386.751.014.364.824.4
G-SAR63.51492.6301.027.987.546.412.865.328.9
SHIELD60.11453.0288.327.187.635.89.962.224.3
SHIELD + G-SAR62.11458.9297.327.387.635.29.964.025.1
CausalLens61.01480.2294.127.387.038.111.360.924.1
CausalLens + G-SAR62.21499.0294.327.787.136.311.163.624.5
PTI61.31478.3285.127.287.515.25.861.922.0
PTI + G-SAR62.91496.2293.027.687.515.05.563.223.1
Reviewer_2gV4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the role of attention sinks as structural anchors in large vision-language models. It finds that masking attention to sink tokens can mitigate hallucinations but may also impair visual reasoning and general visual understanding. Motivated by this trade-off, the authors propose Gated Sink Attention Redistribution (G-SAR), which adaptively retains part of the attention allocated to sink tokens and redistributes the remaining mass to visual non-sink tokens. Experiments across multiple models and benchmarks show improvements in hallucination mitigation and overall task performance.

Critical strengths
  1. The paper presents detailed intervention analyses of attention sinks, providing evidence for their anchoring role and their influence on hidden representations, internal attention distributions, and downstream task performance.
  2. G-SAR is well motivated by the observed trade-off between hallucination mitigation and capability preservation. It adaptively balances sink retention and attention redistribution, with experiments showing improvements across multiple models and benchmarks.
  3. The paper is well organized, clearly written, and easy to follow.
Critical weaknesses
  1. My main concern is that it doesn’t really push things much further conceptually or methodologically compared to prior work. The structural role of attention sinks has already been discussed before, VAR already introduced attention-proportional redistribution to visual non-sink tokens, and LSG already did input-dependent sink gating. G-SAR basically extends that line with finer-grained gates and a retention regularizer, but the current ablations don’t really show what’s distinctly valuable about these additions. The TAS/VAS intervention analysis is useful, but stronger evidence of new mechanistic insights and controlled comparisons against simpler learned redistribution variants would make the contribution a lot more convincing.
  2. The computational overhead of G-SAR is insufficiently characterized. The method relies on attention-based sink identification and post-softmax redistribution, but the paper does not quantify the resulting end-to-end overhead.
Questions for authors
  1. Could you provide an experiment demonstrating that a simple moderate redistribution sink token attention mechanism does not require a gate mechanism, or other experiments proving the effectiveness of this gate design?
  2. Why are the baselines reported by MMMU and MathVista for Qwen3-VL-8B and InternVL3.5-8B so low, with a significant performance gap compared to the official reports?
  3. I'm curious about the ratio of TAS to VAS used in the final design. Will a hybrid approach offer an improvement over using only TAS?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 2gV4: Clarifications and Additional Experimental Evidence

Thank you for your thoughtful and constructive feedback. We believe that several concerns can be addressed through further clarification and supporting evidence, which we provide below. We sincerely hope our responses will resolve your concerns and encourage you to reconsider your evaluation more favorably.

1. On the novelty concerns.

The structural role of attention sinks has already been discussed before, VAR already introduced attention-proportional redistribution to visual non-sink tokens, and LSG already did input-dependent sink gating. G-SAR basically extends that line with finer-grained gates and a retention regularizer, but the current ablations don’t really show what’s distinctly valuable about these additions.

We thank the reviewer for this constructive suggestion. We agree that the novelty of G-SAR should be established through controlled comparisons rather than the individual novelty of gating or redistribution. While VAR employs attention-proportional redistribution and LSG introduces input-dependent gating, G-SAR connects the structural anchoring effect of attention sinks to fine-grained, retention-aware regulation. We further clarify this distinction through mechanistic analysis and additional controlled experiments.

First, G-SAR is motivated by distinct mechanistic findings. Beyond identifying the functional importance of attention sinks, our analysis reveals their non-local influence on model representations. Sink-targeted perturbations induce 3.4–4.0× greater representation drift than random controls (Table 2 in the manuscript). Moreover, textual sinks predominantly influence attention organization, whereas visual sinks exhibit stronger effects on representation geometry. These findings motivate selective regulation that alleviates excessive attention concentration while preserving the structural contributions of sinks.

Second, controlled comparisons validate fine-grained gating. We additionally compare fixed-ratio redistribution, input-adaptive group-level gating, and sink-specific gating, keeping the backbone, sink detection, redistribution targets, and applicable training settings consistent.

Table R1. Controlled ablation of fine-grained gating on Qwen3-VL-8B.

MethodGQA ↑CHAIR-S ↓CHAIR-I ↓SciQA ↑MathVista ↑
Greedy72.424.36.868.037.4
Fixed Ratio72.922.96.568.738.2
Group Gate73.422.26.269.039.0
Group Gate + Ret.73.722.56.370.140.0
Sink-specific Gate73.921.25.969.439.5
G-SAR (Ours)74.220.75.870.840.9

As shown in Table R1, sink-specific gating reduces CHAIR-S from 22.2 to 21.2 and improves MathVista from 39.0 to 39.5 compared with group-level gating. Furthermore, G-SAR outperforms group-level gating with retention across all reported benchmarks. These results demonstrate the benefits of fine-grained sink regulation beyond simpler learned redistribution.

Third, the retention constraint improves the hallucination–reasoning trade-off. We further compare our retention constraint with unconstrained gating and conventional L2 regularization.

Table R2. Ablation of the retention constraint on Qwen3-VL-8B.

MethodGQA ↑CHAIR-S ↓CHAIR-I ↓SciQA ↑MathVista ↑
w/o Retention73.921.25.969.439.5
L2 Regularization74.021.15.970.040.2
G-SAR (Ours)74.220.75.870.840.9

As shown in Table R2, Our retention constraint improves MathVista to 40.9 while maintaining competitive hallucination mitigation (20.7 CHAIR-S), outperforming w/o Retention and conventional L2 regularization on both metrics.

AuthorsOpenReview ↗

Response to Reviewer 2gV4 (Continue)

To further investigate its structural effects, we measure sink attention retention and representation drift.

Table R3. Mechanistic analysis of sink retention on Qwen3-VL-8B.

MethodMean Retention ↑Low-retention Rate ↓Representation Drift ↓
w/o Retention0.5812.4%0.030
L2 Regularization0.688.3%0.026
G-SAR (Ours)0.681.1%0.019

As shown in Table R3, G-SAR reduces representation drift from 0.026 to 0.019 compared with L2 regularization under matched average attention retention. It also substantially reduces the frequency of excessively suppressed sink attention. These observations suggest that the proposed constraint better preserves internal representations rather than merely increasing overall attention retention.

Overall, G-SAR is not merely a combination of existing components. Its contribution lies in connecting the structural role of attention sinks to selective regulation, with controlled experiments demonstrating the complementary benefits of fine-grained gating and sink retention. We will incorporate these analyses and further clarify the distinctions from VAR and LSG in the revised manuscript.

2. On the inference efficiency.

The computational overhead of G-SAR is insufficiently characterized. The method relies on attention-based sink identification and post-softmax redistribution, but the paper does not quantify the resulting end-to-end overhead.

We thank the reviewer for highlighting the importance of inference efficiency. To address this concern, we conduct an additional end-to-end latency evaluation of all methods on Qwen3-VL-8B using MME under identical inference settings. The measured latency accounts for the additional computations introduced by G-SAR, including attention sink identification and redistribution.

As shown below, G-SAR achieves an inference latency of 0.356 s/sample, comparable to other attention regulation methods, including VAR (0.319 s), DIYSink (0.389 s), and LSG (0.351 s). Notably, G-SAR incurs only 0.005 s/sample additional latency compared with LSG, while consistently outperforming it across all four benchmarks.

Moreover, G-SAR achieves the best performance across all evaluated tasks, covering general visual understanding (GQA: 74.2), hallucination mitigation (CHAIR-S: 20.7), and visual reasoning (SciQA: 70.8; MathVista: 40.9). These results demonstrate that G-SAR delivers consistent performance gains at a computational cost comparable to existing attention regulation methods, achieving a favorable trade-off between effectiveness and inference efficiency. We have incorporated this additional analysis into the revised manuscript (Lines 525–531).

Table: Performance and end-to-end inference efficiency on Qwen3-VL-8B. Latency is measured on MME.

MethodGQA ↑CHAIR-S ↓SciQA ↑MathVista ↑Time (s/sample) ↓
Greedy72.424.368.037.40.164
OPERA72.322.563.036.02.830
TAME71.720.865.933.40.201
VAR72.324.768.538.20.319
DIYSink73.023.869.039.90.389
LSG73.924.068.140.00.351
G-SAR (Ours)74.220.770.840.90.356
AuthorsOpenReview ↗

Response to Reviewer 2gV4 (Continue)

3. On the necessity of gated redistribution

Could you provide an experiment demonstrating that a simple moderate redistribution sink token attention mechanism does not require a gate mechanism, or other experiments proving the effectiveness of this gate design?

We thank the reviewer for this insightful suggestion. Our motivation for introducing gated redistribution stems from the limitations of fixed-ratio attention redistribution. Although a moderate redistribution ratio can be selected for different tasks, such task-level adjustments remain too coarse to accommodate variations across individual inputs.

For instance, even within the same visual question-answering task, an image may contain a single salient object or a cluttered scene with multiple small objects and complex spatial relationships. These inputs may exhibit different attention distributions and require different degrees of visual evidence utilization. Applying a uniform redistribution ratio overlooks such input-dependent differences and may either insufficiently alleviate excessive sink attention or unnecessarily disrupt its structural contributions.

To validate this motivation, we conduct controlled comparisons with fixed-ratio and learned gating strategies (Table R1). Under identical redistribution targets and evaluation settings, fixed-ratio redistribution achieves 72.9 GQA, 22.9 CHAIR-S, and 38.2 MathVista on Qwen3-VL-8B, whereas G-SAR achieves 74.2, 20.7, and 40.9, respectively. Furthermore, sink-specific gating improves upon simpler input-adaptive group-level gating, highlighting the benefits of finer-grained regulation.

These results support the effectiveness of adaptive, sink-specific redistribution. Rather than applying a uniform suppression strength, G-SAR dynamically determines how much attention to retain at each sink based on the input representations. This design is further motivated by our structural analysis (Tables 2–3), which reveals the heterogeneous influences of attention sinks on attention organization, representation geometry, and downstream capabilities. Together, these findings support fine-grained gating as a more flexible approach to balancing visual evidence enhancement and structural preservation.

4. On the discrepancy with official benchmark results.

Why are the baselines reported by MMMU and MathVista for Qwen3-VL-8B and InternVL3.5-8B so low, with a significant performance gap compared to the official reports?

We thank the reviewer for pointing out this discrepancy. In our experiments, we evaluate MathVista on the MathVista-mini split and MMMU on its test split. For Qwen3-VL-8B, we consistently set max_new_tokens=128 across all benchmarks to maintain a unified evaluation configuration.

5. On the combination of TAS and VAS

I'm curious about the ratio of TAS to VAS used in the final design. Will a hybrid approach offer an improvement over using only TAS?

We thank the reviewer for this insightful question. G-SAR does not impose a predefined TAS-to-VAS ratio. Instead, TAS and VAS are identified using modality-specific criteria, and each sink is assigned an input-dependent gate that independently determines how much attention to retain or redistribute. Consequently, the relative contributions of TAS and VAS to attention redistribution are adaptively determined rather than manually specified.

This design is motivated by their distinct structural roles. As shown in Sec. 4.2, TAS predominantly influence attention organization, whereas VAS have stronger effects on representation geometry. Moreover, our masking analysis (Table 3) reveals different functional effects: on Qwen3-VL-8B, masking TAS improves ScienceQA from 68.0 to 75.0, whereas masking VAS reduces CHAIR-S from 24.3 to 22.9 but decreases ScienceQA to 59.6. Jointly masking both types further causes severe performance degradation. These observations highlight the risks of uniform suppression and motivate selective regulation of both modalities.

Thus, G-SAR enables adaptive and complementary regulation of TAS and VAS, selectively redistributing excessive attention while preserving their structural contributions. Rather than relying on a fixed mixing ratio or exclusively targeting one modality, this design provides the flexibility to balance hallucination mitigation with general visual understanding and reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.