acceptodds
Under review as a conference paper at ICLR 2027

SSA: Scaled Sink Attention for Large Language Models

31 views

Abstract

As large language models (LLMs) attract growing interest, the attention mechanism at their core has become one of their most critical components. Yet Softmax attention still exhibits attention sink: a large share of the probability mass is placed on a few uninformative tokens, usually the first one. Recent remedies relax the sum-to-one constraint of Softmax with a learnable denominator bias (Sink Attention) or a sigmoid output gate (Gated Attention) and report much lower first-token scores. We point out, however, that any such per-query rescaling, including ours, leaves the normalized attention distribution unchanged for fixed logits; whether a method removes the sink is decided by what the trained model learns and must be measured after normalization. Under this measure, existing remedies still leave a substantial share of the attention on the first token and, in addition, suppress attention on informative tokens. We propose (SSA), which combines two components: (i) an input-dependent virtual sink, a per-head logit projected from the hidden state that competes inside the Softmax and is then discarded, so that the drained mass is decided token by token against the attention evidence; and (ii) a fixed gain on the surviving attention, parameterized by a reference surviving mass , which restores the output scale of heavily draining heads and sharpens contextual attention elsewhere. SSA substantially lowers the normalized first-token share and redistributes attention to informative context tokens. SSA is also practical: it adds only one projection per layer, and we have integrated it into FlashAttention-3 as a ready-to-use fused kernel, so that it inherits the efficiency of state-of-the-art attention kernels and incurs an end-to-end training slowdown of less than 1%. In extensive experiments on a from-scratch pre-trained Mixture-of-Experts model, SSA improves the fourteen-benchmark average by 2.74 points over Softmax attention, reaches the baseline's final pre-training loss with about fewer tokens, and never triggers the QK-clip safeguard that all baselines trigger hundreds to thousands of times.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Attention, Attention Sink

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qczxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the attention‑sink problem in LLMs, where substantial attention mass is assigned to uninformative early tokens under standard Softmax attention. The authors prove a key invariance property: uniform per‑query rescaling (used by prior fixes Sink Attention and Gated Attention) cannot alter the normalized attention distribution given fixed logits. This exposes a critical evaluation pitfall: reduced absolute attention scores do not prove sink mitigation, and existing methods cause collateral damage by suppressing attention over informative tokens.

The authors propose Scaled Sink Attention (SSA): (1) an input‑dependent virtual‑sink logit competing inside Softmax, whose probability is discarded to dynamically drain attention mass based on Q‑K matching evidence; (2) a fixed gain g=1/ρg=1/\rho to counteract output‑magnitude dilution. SSA is lightweight and integrated into FlashAttention‑3 with <1% training overhead.

Extensive from‑scratch MoE pre‑training experiments show SSA lowers normalized first‑token attention, improves average benchmark performance (+2.74), reaches baseline pre‑training loss with ~1.3× fewer tokens, and eliminates QK‑clip triggers. It also yields strong results on math reasoning and 128K long‑context RULER benchmarks.

Critical strengths
  1. Important theoretical diagnosis. The invariance proposition clearly points out that absolute attention scores are insufficient for judging attention‑sink mitigation. The argument and normalized‑measurement protocol provide useful guidance for future research in this direction. The analysis of collateral damage (suppression of informative-token attention) explains why prior methods often yield limited loss improvement.

  2. Good engineering practicality. The method incurs negligible parameter and compute overhead, integrated as a drop‑in replacement for standard attention inside FlashAttention‑3, compatible with MoE and context‑parallel training. The authors provide pseudocode and kernel‑modification details and promise to release code for reproducibility.

  3. Extensive experiments. Most experiments are from‑scratch pre‑training (not fine‑tuning only), with controlled variables where only attention normalization differs. Evaluations cover knowledge, math, code benchmarks, larger‑scale MoE scaling tests, long‑context continued pre‑training, rich ablations, and mechanistic analysis. Observed benefits span downstream accuracy, pre‑training convergence speed, and training numerical stability.

Critical weaknesses
  1. Key hyper‑parameter ρ=0.2\rho=0.2 is empirically tuned. There is no principled way to select ρ\rho for new model settings. Ablations show bad choice of g (e.g. g=10g=10) causes severe performance degradation at later training stages. This creates adoption risk for practitioners porting SSA to new setups.

  2. Insufficient evaluation on inference‑centric properties. The paper focuses heavily on pre‑training dynamics. Important attention-sink-related scenarios, including streaming inference, KV‑cache eviction, and quantization (INT8 / INT4), are not evaluated.

  3. The proposition about per‑query rescaling invariance applies only for fixed logits in forward pass. The manuscript could better emphasize this boundary condition to prevent readers from misinterpreting that Sink / Gated Attention are completely useless (they still modify gradients during training). Potential failure modes of large gain g>1g>1 (e.g., noise amplification) are not discussed.

Questions for authors
  1. Attention sink is originally identified in streaming LLM inference with KV‑cache. How does SSA behave in streaming decoding and KV‑cache eviction setups? Do you observe any benefits or pitfalls?

  2. Why does learnable / input‑dependent gain underperform? Your ablation shows learnable or input‑dependent gates on top of ASA tend to degrade back toward Gated‑Attention‑like behavior. Could you give a more intuitive or analytical explanation of why the fixed gain acts as such a strong structural prior? What drives learnable g to shrink itself during training?

  3. Could you briefly discuss how you expect SSA would compare against other sink‑mitigating attention variants (Softpick, sigmoid‑attention, etc.) under controlled from‑scratch pretraining?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qczx

We conducted additional experiments in response to your feedback; please see the General Response above for the results.

Q1: How does SSA behave in streaming-style decoding and KV-cache eviction?

A1: SSA needs no stored virtual K/V entry because its sink logit comes from the current hidden state. In the teacher-forced 16K/2,048-slot eviction test, recent-only caching raises NLL by 0.0362 nats/token for SSA versus 0.2116 for Gated Attention. Pinning the first four tokens helps SSA by just 0.0015, versus 0.1795 for Gated; a 4K/512-slot test also favors SSA. See General Response A5 for details and the separate full-cache RULER-128K results.

Q2: Why choose fixed g = 5 (ρ = 0.2) over learned, input-dependent, or larger gains, and what does per-query rescaling imply?

A2: We set ρ = 0.2 from gain-free ASA's surviving real-token mass m: fixed g = 1/ρ = 5 gives gm = 1 at m = 0.2, while the sink still adjusts m per query. In the 20B runs, fixed g = 5 has the highest observed final 14-task mean (36.10 versus 35.73 for learned gain and 33.46–34.61 for input-dependent gains), though validation PPL is close to learned gain (3.7718 versus 3.7704). Mean learned per-head gain falls from 4.888 to 4.418 between 101B and 524B tokens; shared-gradient pressure could contribute. At g = 10, median effective scale reaches 4.06–4.47 and the final task mean is about six points lower, consistent with over-amplification. For new models, calibrate ρ using gain-free mass and a small held-out sweep.

At fixed logits, SSA, Sink Attention, and headwise Gated Attention rescale a query's real-token weights by the same positive factor, leaving their normalized shares unchanged. Training can change logits, so SSA's lower first-token share is empirical (General Response A4).

Q3: How does SSA compare with Softpick and sigmoid attention under controlled pre-training?

A3: In the controlled 1B comparison, gain-free ASA reaches 8.8997 overall PPL versus 9.0190 for Softpick [1] and 9.0826 for sigmoid attention [2]. Full SSA reaches 8.6781. Both ASA and SSA have lower PPL in each of wiki, math, and code. Please see General Response A1 for the full results.

Q4: What do the added inference analyses show?

A4: SSA scores higher than Gated Attention and Softmax on full-cache RULER-128K and shows a smaller teacher-forced NLL penalty than Gated under fixed-size KV-cache eviction. These are separate evaluations; please see General Response A5 for the setup and full results.

  • [1] Zuhri et al. (2025). Softpick: No Attention Sink, No Massive Activations with Rectified Softmax, arXiv:2504.20966.
  • [2] Ramapuram et al. (2025). Theory, Analysis, and Best Practices for Sigmoid Self-Attention, ICLR.
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer qczx,

Thank you once again for your time and thoughtful effort in reviewing our paper. As the author-reviewer discussion period draws to a close today, we would like to kindly follow up on whether our response has adequately addressed your concerns. Should you have any remaining questions, please do not hesitate to let us know, and we will be glad to clarify them further.

If you find our responses satisfactory and have no further concerns, we would be deeply grateful if you could consider raising your score. Your recognition and support mean a great deal to us. Thank you again for your valuable feedback and consideration.

Best regards, The Authors

Thank the authors for the response. Most of my concerns have been addressed, and I have decided to maintain my original score of weak acceptance.

Reviewer_r2isOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the attention sink phenomenon in large language models, arguing that a reduction in the absolute attention weight assigned to the first token does not necessarily imply a reduction in its relative importance among real tokens. By renormalizing attention weights over real tokens, the authors show that Sink Attention and Gated Attention reduce absolute first-token attention but still retain substantial normalized attention on the first token. The authors further propose Scaled Sink Attention (SSA), which generates a virtual sink logit through a linear projection of the hidden state. The virtual sink participates in Softmax normalization and absorbs part of the attention mass but is excluded from value aggregation, assigning it a zero value vector. A fixed gain (g=5) is then applied to the remaining attention weights to compensate for the reduced output magnitude. Large-scale pre-training experiments demonstrate improvements in downstream performance and training stability.

Critical strengths

The experiments include pre-training a 20B MoE model on 524B tokens and evaluating multiple downstream tasks, demonstrating improvements in performance and QK-logit stability. The FlashAttention-3 implementation also keeps the reported training overhead below 1%, supporting the practical applicability of SSA.

Critical weaknesses
  1. Missing comparisons with closely related methods. Although Section 4.1 discusses the relationship to the virtual key/value mechanism of [1], the evaluation does not include direct comparisons with that mechanism or the zero-value virtual-key variant of [2]. Both already yield query-dependent virtual sink logits. Controlled comparisons under matched training conditions and gain settings would better isolate the benefit of SSA’s independent hidden-state projection.

  2. Limited statistical validation of normalized attention analysis. The normalized attention results in Figure 1 are based on a single short English probe passage. Given that renormalized attention is a central contribution of the paper, a more systematic evaluation across diverse text samples, context lengths, and attention heads is needed to establish the generality of the observed reduction in attention sinks.

[1] Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. In Conference on Language Modeling (COLM), 2024.

[2] Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In International Conference on Learning Representations (ICLR), 2025.

Questions for authors
  1. Could the authors provide direct comparisons with the virtual key/value mechanism of [1] and the zero-value virtual-key mechanism of [2]? What specific advantages does the independent hidden-state projection offer over these approaches?

  2. The paper does not compare Softmax, Sink Attention, or Gated Attention with the same fixed gain (g=5). How much of SSA's improvement would remain after applying this gain to these baselines?

  3. Recent work [3] has shown that sink value representations can contain useful information for reasoning. Could the proposed virtual sink be extended to aggregate or preserve useful contextual information rather than discarding its value contribution entirely?

[1] Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. In Conference on Language Modeling (COLM), 2024.

[2] Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In International Conference on Learning Representations (ICLR), 2025.

[3] Suho Yoo, Youngjoon Jang, and Joon Son Chung. On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs. arXiv preprint arXiv:2603.14337, 2026.

2 comments
AuthorsOpenReview ↗

Response to Reviewer r2is

We conducted additional experiments in response to your feedback; please see the General Response above for the results.

Q1: How does SSA compare directly with learned virtual K/V and zero-value virtual-K methods, and what does independent projection offer?

A1: We compared a learned virtual K with zero V (Gu-style) [1], a learned virtual K/V slot (Sun-style) [2], and our independent hidden-state sink under the same 1B recipe. The table reports overall validation PPL and mean first-token share among real tokens S0S_0 on 192 paired 4096-token probes.

Virtual slotGainOverall PPLRelative first-token score S0S_0
Learned virtual K, zero V (Gu-style)19.01000.1339
Learned virtual K/V (Sun-style)18.99660.1792
Independent hidden-state sink, zero V (ASA)18.89970.0228
Learned virtual K, zero V (Gu-style)58.69360.0952
Learned virtual K/V (Sun-style)58.70640.0930
Independent hidden-state sink, zero V (SSA)58.67810.0207

At g=1g=1, ASA has 8.8997 PPL and S0=0.0228S_0=0.0228, versus the controls' best PPL of 8.9966 and smallest S0S_0 of 0.1339. At g=5g=5, SSA has 8.6781 and 0.0207, versus the controls' best PPL of 8.6936 and smallest S0S_0 of 0.0930. The independent hidden-state sink thus has the lowest PPL and relative first-token score at both gains; its separate projection permits independent sink routing.

Q2: How much of SSA's improvement remains when Softmax, Sink Attention, and Gated Attention all receive g=5g=5?

A2: At g=5g=5, SSA records 8.6781 overall PPL versus 8.7312 for Softmax, 8.7255 for Sink, and 8.6917 for headwise Gated (General Response A2). SSA leads all tested gain-matched 1B methods in overall PPL and in each validation domain.

Q3: Can the virtual sink preserve useful sink-token information or carry contextual information?

A3: This is a good idea. We agree that real sink-token values may carry useful information [3]. SSA sets only the added virtual slot's value to zero; every real token, including the first, retains its K/V path. The query-dependent sink logit competes with the real-token log-sum-exp, so when real-token scores dominate, little mass goes to the zero-value slot; the fixed gain rescales the surviving real-token attention. This can relieve forced attention to uninformative tokens while retaining useful value contributions. SSA's lower PPL and higher full-cache RULER-128K scores are consistent with this behavior (General Response A1 and A5).

Q4: Is the normalized-attention result based only on one short English passage?

A4: Figure 1 shows one illustrative passage. Across 192 paired held-out 20B probes, SSA's relative first-token score is 0.0207 versus 0.1236 for headwise Gated Attention. Therefore, this conclusion also holds across diverse datasets. The definitions and full results are in General Response A4.

  • [1] Gu et al. (2025). When Attention Sink Emerges in Language Models: An Empirical View, ICLR.
  • [2] Sun et al. (2024). Massive Activations in Large Language Models, COLM.
  • [3] Suho Yoo, Youngjoon Jang, and Joon Son Chung (2026). On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs, arXiv:2603.14337.
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer r2is,

Thank you once again for your time and thoughtful effort in reviewing our paper. As the author-reviewer discussion period draws to a close today, we would like to kindly follow up on whether our response has adequately addressed your concerns. Should you have any remaining questions, please do not hesitate to let us know, and we will be glad to clarify them further.

If you find our responses satisfactory and have no further concerns, we would be deeply grateful if you could consider raising your score. Your recognition and support mean a great deal to us. Thank you again for your valuable feedback and consideration.

Best regards, The Authors

Reviewer_sSoyOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Scaled Sink Attention (SSA), which combines an input-dependent virtual sink with a fixed gain on the attention output. It distinguishes absolute attention weights from the normalized distribution over real tokens and evaluates how these distributions change after training. Experiments report improvements in pretraining loss, downstream performance, long-context modeling, and QK-logit stability, with negligible training overhead.

Critical strengths
  1. The experimental evaluation is substantial. The main comparison trains a 20B MoE model on 524B tokens, with additional larger-model and 128K-context experiments. The mathematical reasoning and long-context results are particularly encouraging.

  2. The method is simple and practical. The FlashAttention-3 implementation incurs less than 1% reported overhead, while the combination of lower training loss and fewer QK-clipping events makes the approach useful beyond benchmark improvements alone.

Critical weaknesses
  1. The contribution of the fixed gain is not sufficiently isolated. The paper attributes much of the loss improvement to the gain, but the gain ablations are applied only on top of the proposed adaptive sink. Applying the same gain to Gated Attention or Sink Attention would help establish whether the improvement specifically requires the proposed mechanism.

  2. The interpretation of attention-sink measurements needs more care. A high normalized first-token share does not necessarily imply strong functional dependence on that token when the head is nearly switched off. Also, the released Gated Attention visualization already uses normalized Softmax weights, with gating applied separately to the output. The discussion should distinguish findings from the authors’ retrained baselines from claims about how prior work measured attention sinks.

  3. The advantage over the strongest baseline is less uniform than the overall framing suggests. SSA improves the average over Gated Attention by 0.74 points, with gains concentrated in mathematics and several knowledge and code tasks favoring Gated Attention. The 264B-model experiment is also limited to 44B training tokens, so it mainly demonstrates an early-training advantage.

Questions for authors
  1. How much of SSA’s improvement remains when the Gated Attention baseline receives the same fixed output gain?
  2. The explanation for the fixed gain relies partly on Muon and weight decay. Does the benefit persist under a different optimizer or after adjusting the initialization and update scale of the output projection?
  3. How does SSA compare with elementwise Gated Attention? The headwise baseline is a reasonable parameter-matched control, but it does not cover the more expressive variant of the original method.
2 comments
AuthorsOpenReview ↗

Response to Reviewer sSoy

We conducted additional experiments in response to your feedback; please see the General Response above for the results.

Q1: How much of SSA's improvement remains when Gated Attention receives the same fixed output gain?

A1: With g=5g=5, headwise Gated reaches 8.6917 overall PPL, down from 9.0014; SSA reaches 8.6781, another 0.0136 lower, and leads in all three domains (General Response A2). SSA thus retains a modest gain-matched validation advantage.

Q2: Does the gain persist under another optimizer or after changing output-projection initialization and update scale?

A2: Under Adam, SSA records 9.5386 overall PPL versus 9.9262 for ASA, so the ASA-to-SSA gain persists (General Response A3). Under Muon, we also tested output-projection initialization at ×5:

MethodWiki PPLMath PPLCode PPLOverall PPL
ASA + muon27.28096.37223.98948.8997
SSA + muon26.59796.20403.89678.6781
ASA + muon + Wₒ initialization × 526.49776.22463.91078.6870
SSA + muon + Wₒ initialization × 526.65026.24943.92548.7261

This is a good idea. We agree with the reviewer that SSA’s fixed g=5 factor can be fused into the output projection in the forward pass: Wₒ(5u)=(5Wₒ)u. Thus, initializing ASA’s output projection at five times its usual scale reproduces SSA’s initial forward scaling. Applying both interventions gives (5Wₒ)(5u)=Wₒ(25u), an effective g=25 at initialization. Under Muon, the larger initialization lowers ASA’s overall PPL from 8.8997 to 8.6870, close to SSA’s 8.6781. Combining the larger initialization with g=5 raises overall PPL to 8.7261, consistent with Figure 4(b), which shows that an overly large fixed factor can hurt performance.

Q3: How does SSA compare with elementwise Gated Attention?

A3: Elementwise Gated records 8.9616 overall PPL versus 9.0014 for headwise Gated; SSA reaches 8.6781 and has lower PPL in wiki, math, and code (General Response A1). SSA therefore outperforms both gated variants on 1B validation.

Q4: The interpretation of attention-sink measurements needs more care.

A4: Thank you for pointing this out. We will revise the manuscript to specify these definitions and how both scores are computed.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer sSoy,

Thank you once again for your time and thoughtful effort in reviewing our paper. As the author-reviewer discussion period draws to a close today, we would like to kindly follow up on whether our response has adequately addressed your concerns. Should you have any remaining questions, please do not hesitate to let us know, and we will be glad to clarify them further.

If you find our responses satisfactory and have no further concerns, we would be deeply grateful if you could consider raising your score. Your recognition and support mean a great deal to us. Thank you again for your valuable feedback and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.