- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes token-gated spars attention (TGSA), adding a small learnable gate head for each KV group to make binary decisions per query-key pair to enable each token to attend to a variable length subset of KV cache. The gates need to be jointly trained during pre-training of LLM, with sparsity loss and attention-matching KL loss. Pre-training models at 600M, 1.7B and 4B scales, the authors report sparsity over 80% achieved with modest model quality loss, while achieving up to 3.5$\times$ decoding throughput.
Sparse attention is also applied to prefill but prefill gets no speedup but just latency overhead. TGSA needs to be applied from the initial stage of pre-training since PPL from continued pre-training is worse than DSA. Comparsion with top-p oracle is son
Critical strengths
S1. Well-motivated mechanism with a principled recipe.
Small per-KV-group gate heads make token-level, variable-length selection learnable during pre-training, and the gate-key cache lets most of the KV load be skipped.
S2. Real systems results with honest reporting.
Authors implemented custom kernels for training, decode, and prefill and integrated TGSA into vLLM, with measured decode speedups of 1.6$\times$ at 16K and 3.5$\times$ at 64K.
The prefill overhead, the quadratic gate cost in the FLOP accounting, and the CPT limitation are all reported openly.
Critical weaknesses
W1. No from-scratch pre-trained sparse baselines.
Adding comparison with DSA/MSA-style is strongly recommended and should add NSA and DashAttention if available. DSA only appears in CPT comparison with 600M model, so the source of the gains (variable-length selection rule or training approach) cannot be separated.
W2. No real post-hoc baselines.
There is no comparison with post-hoc baselines like QUEST, Loki, Twilight at matched sparsity. The regular top-p baseline loads all keys and therefore is structurally disadvantaged at iso-data and iso-FLOPs settings.
W3. Insufficient analysis and ablations.
Training wall-clock (GPU-hours) and memory overhead are not reported. No ablations on d_g selection even though it is a critical hyperparameter for token selection quality.
W4. Overstatement of novelty.
The abstract’s “fixed sparsity” claim is overstated relative to other adaptive budget works [1][2]. Most of individual loss terms are from existing techniques (MSA/DSA, MoE), so the contribution is the recipe.
[1] Lin, Chaofan, et al. "Twilight: Adaptive attention sparsity with hierarchical top-$ p $ pruning." Advances in Neural Information Processing Systems 38 (2026): 119406-119432.
[2] Zhu, Kan, et al. "Tactic: Adaptive sparse attention with clustering and distribution fitting for long-context LLMs." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
Q1. How does TGSA compare with from-scratch DSA/MSA-style indexers under same token budget and matched memory movement, including indexer loads?
Q2. How do post-hoc sparse attention methods compare in benchmarks that are reported in the paper, when applied to the same ungated model?
Q3. Could the TGSA model be trained to be robust to dense prefill? Since TGSA introduces non-negligible overhead to prefill attention latency, if would be helpful if you can just adapt model to robust to prefill without gating.
Q4. Could you provide ablations with varying d_g and justify the selection of 16?
Q5. Could you report training wall-clock, throughput and memory against ungated model at 8K and at 64K LCE, plus iso-FLOP results at 1.7B and 4B?
2 comments
(continued from above)
The abstract’s “fixed sparsity” claim is overstated relative to other adaptive budget works [1][2]. Most of individual loss terms are from existing techniques (MSA/DSA, MoE), so the contribution is the recipe.
We will clarify the novelty claim. Our intent was to say that there is comparatively less work that explores how to architecturally change the attention mechanism (and auxiliary losses) such that we are able to come up with naturally adaptive sparse attention. Twilight and Tactic are very relevant baselines, but we class these as taking existing attention mechanisms and learning how to approximate attention on the fly (and, as we mention above, these should be upper-bounded by the oracle top-p baseline).
We also agree that many of the loss terms are inspired by other papers (e.g., MoEs), and coincide with concurrent work (e.g., MSA). We tried to mention this explicitly in the paper (e.g., reference to MoE load balancing in training section, references to MSA/DSA attention matching), but will make it clearer.
Overall, we believe that our main contribution is a new mechanism and recipe to train sparse attention models that have innately adaptive sparsity, and showing the effects of the resulting adaptive sparsity. Concretely: per-query, token-level adaptive gating trained from scratch, with decode kernels that realize throughput gains at 64K context.
Q3 - Could the TGSA model be trained to be robust to dense prefill? Since TGSA introduces non-negligible overhead to prefill attention latency, if would be helpful if you can just adapt model to robust to prefill without gating.
This is a great question! We explored this, but unfortunately we found that TGSA is not robust to dense prefill out-of-the-box. We did find that LCE training made the model more robust to dense prefill, but there is still non-trivial degradation (~0.3 PPL). We will include these results in the next version of the paper.
We do believe that more careful tuning could address this (e.g., changing STE so that it forces dense prefill during the forward pass at random); we intend to explore this in future work.
Thank you for your review! We address the individual questions below (broken down into two comments).
Refer to general response, section 1 for a comparison against DSA/top-k style indexers. To summarize: we find that the from-scratch DSA indexer, under the same token budget, matches TGSA PPL with 5.89 percentage points less memory movement (which includes indexer loads; this is the case for all memory movement numbers), but in long-context tasks we find that TGSA outperforms DSA in terms of HELMET recall, also in the memory-matched setting.
We include the oracle top-p as a practical upper bound for attention-mass-based selection baselines like Quest, Loki and Twilight: it assumes that the top-p set can be found “for free” (i.e., with no memory movement whatsoever), which allows us to only load exactly the top-p keys and values. This is more optimistic than the methods described above, which generally try to make that initial lookup cheap (but not free). Thus, we feel this baseline actually provides a structural advantage to these methods, and even so TGSA outperforms it.
To confirm: the current top-p numbers are exactly the top-p baselines applied on top of the same ungated KV8 model at each scale.
We found that a gate dimension of size 16 balanced quality and memory movement. We conducted this ablation at 600M scale. We will include this table in the next version of the paper:
Due to the high compute requirements, we were unable to run both iso-data and iso-FLOP experiments at 1.7B and 4B, but we note that the FLOP difference goes down across scales as we increase in size, which means that differences in the two settings should get milder as we scale up:
That said, we will try our best to include iso-FLOP results in the next version of the paper.
In FLOPs, TGSA training costs only 2.7–6.4% more than KV8 (table above). Regarding training time, across scales 8K TGSA pretraining was 1.72–1.99× more expensive than regular KV8 training, while 64K LCE was 4.86–5.42× more expensive. This is because our training kernels are currently not as optimized as regular kernels, especially in long-context settings. Most of our effort went towards developing good inference kernels, and for training we included an iso-FLOP study that controls for training FLOPs. That said, we will report these numbers and be clear about this limitation in the next version of the paper.
Peak memory, on the other hand, was the same (within 0.5%) across both TGSA and KV8.