- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the attention‑sink problem in LLMs, where substantial attention mass is assigned to uninformative early tokens under standard Softmax attention. The authors prove a key invariance property: uniform per‑query rescaling (used by prior fixes Sink Attention and Gated Attention) cannot alter the normalized attention distribution given fixed logits. This exposes a critical evaluation pitfall: reduced absolute attention scores do not prove sink mitigation, and existing methods cause collateral damage by suppressing attention over informative tokens.
The authors propose Scaled Sink Attention (SSA): (1) an input‑dependent virtual‑sink logit competing inside Softmax, whose probability is discarded to dynamically drain attention mass based on Q‑K matching evidence; (2) a fixed gain to counteract output‑magnitude dilution. SSA is lightweight and integrated into FlashAttention‑3 with <1% training overhead.
Extensive from‑scratch MoE pre‑training experiments show SSA lowers normalized first‑token attention, improves average benchmark performance (+2.74), reaches baseline pre‑training loss with ~1.3× fewer tokens, and eliminates QK‑clip triggers. It also yields strong results on math reasoning and 128K long‑context RULER benchmarks.
Critical strengths
-
Important theoretical diagnosis. The invariance proposition clearly points out that absolute attention scores are insufficient for judging attention‑sink mitigation. The argument and normalized‑measurement protocol provide useful guidance for future research in this direction. The analysis of collateral damage (suppression of informative-token attention) explains why prior methods often yield limited loss improvement.
-
Good engineering practicality. The method incurs negligible parameter and compute overhead, integrated as a drop‑in replacement for standard attention inside FlashAttention‑3, compatible with MoE and context‑parallel training. The authors provide pseudocode and kernel‑modification details and promise to release code for reproducibility.
-
Extensive experiments. Most experiments are from‑scratch pre‑training (not fine‑tuning only), with controlled variables where only attention normalization differs. Evaluations cover knowledge, math, code benchmarks, larger‑scale MoE scaling tests, long‑context continued pre‑training, rich ablations, and mechanistic analysis. Observed benefits span downstream accuracy, pre‑training convergence speed, and training numerical stability.
Critical weaknesses
-
Key hyper‑parameter is empirically tuned. There is no principled way to select for new model settings. Ablations show bad choice of g (e.g. ) causes severe performance degradation at later training stages. This creates adoption risk for practitioners porting SSA to new setups.
-
Insufficient evaluation on inference‑centric properties. The paper focuses heavily on pre‑training dynamics. Important attention-sink-related scenarios, including streaming inference, KV‑cache eviction, and quantization (INT8 / INT4), are not evaluated.
-
The proposition about per‑query rescaling invariance applies only for fixed logits in forward pass. The manuscript could better emphasize this boundary condition to prevent readers from misinterpreting that Sink / Gated Attention are completely useless (they still modify gradients during training). Potential failure modes of large gain (e.g., noise amplification) are not discussed.
Questions for authors
-
Attention sink is originally identified in streaming LLM inference with KV‑cache. How does SSA behave in streaming decoding and KV‑cache eviction setups? Do you observe any benefits or pitfalls?
-
Why does learnable / input‑dependent gain underperform? Your ablation shows learnable or input‑dependent gates on top of ASA tend to degrade back toward Gated‑Attention‑like behavior. Could you give a more intuitive or analytical explanation of why the fixed gain acts as such a strong structural prior? What drives learnable g to shrink itself during training?
-
Could you briefly discuss how you expect SSA would compare against other sink‑mitigating attention variants (Softpick, sigmoid‑attention, etc.) under controlled from‑scratch pretraining?
3 comments
Official Comment by Authors
Dear Reviewer qczx,
Thank you once again for your time and thoughtful effort in reviewing our paper. As the author-reviewer discussion period draws to a close today, we would like to kindly follow up on whether our response has adequately addressed your concerns. Should you have any remaining questions, please do not hesitate to let us know, and we will be glad to clarify them further.
If you find our responses satisfactory and have no further concerns, we would be deeply grateful if you could consider raising your score. Your recognition and support mean a great deal to us. Thank you again for your valuable feedback and consideration.
Best regards, The Authors
Thank the authors for the response. Most of my concerns have been addressed, and I have decided to maintain my original score of weak acceptance.
Response to Reviewer qczx
We conducted additional experiments in response to your feedback; please see the General Response above for the results.
Q1: How does SSA behave in streaming-style decoding and KV-cache eviction?
A1: SSA needs no stored virtual K/V entry because its sink logit comes from the current hidden state. In the teacher-forced 16K/2,048-slot eviction test, recent-only caching raises NLL by 0.0362 nats/token for SSA versus 0.2116 for Gated Attention. Pinning the first four tokens helps SSA by just 0.0015, versus 0.1795 for Gated; a 4K/512-slot test also favors SSA. See General Response A5 for details and the separate full-cache RULER-128K results.
Q2: Why choose fixed g = 5 (ρ = 0.2) over learned, input-dependent, or larger gains, and what does per-query rescaling imply?
A2: We set ρ = 0.2 from gain-free ASA's surviving real-token mass m: fixed g = 1/ρ = 5 gives gm = 1 at m = 0.2, while the sink still adjusts m per query. In the 20B runs, fixed g = 5 has the highest observed final 14-task mean (36.10 versus 35.73 for learned gain and 33.46–34.61 for input-dependent gains), though validation PPL is close to learned gain (3.7718 versus 3.7704). Mean learned per-head gain falls from 4.888 to 4.418 between 101B and 524B tokens; shared-gradient pressure could contribute. At g = 10, median effective scale reaches 4.06–4.47 and the final task mean is about six points lower, consistent with over-amplification. For new models, calibrate ρ using gain-free mass and a small held-out sweep.
At fixed logits, SSA, Sink Attention, and headwise Gated Attention rescale a query's real-token weights by the same positive factor, leaving their normalized shares unchanged. Training can change logits, so SSA's lower first-token share is empirical (General Response A4).
Q3: How does SSA compare with Softpick and sigmoid attention under controlled pre-training?
A3: In the controlled 1B comparison, gain-free ASA reaches 8.8997 overall PPL versus 9.0190 for Softpick [1] and 9.0826 for sigmoid attention [2]. Full SSA reaches 8.6781. Both ASA and SSA have lower PPL in each of wiki, math, and code. Please see General Response A1 for the full results.
Q4: What do the added inference analyses show?
A4: SSA scores higher than Gated Attention and Softmax on full-cache RULER-128K and shows a smaller teacher-forced NLL penalty than Gated under fixed-size KV-cache eviction. These are separate evaluations; please see General Response A5 for the setup and full results.