acceptodds
Under review as a conference paper at ICLR 2027

Token-Gated Sparse Attention

Abstract

Attention-based architectures struggle in long-context decode settings due to the need to load and process the full set of attention keys and values for the sequence (the KV cache) for each newly generated token. Sparse attention approaches alleviate this bottleneck by computing attention over a subset of the KV cache. Prior sparse attention approaches generally impose restrictive structure on the sparsity pattern, such as targeting a fixed amount of sparsity within a given attention head or at any given context length. We propose token-gated sparse attention (TGSA), a new variant of sparse attention that allows each token within a given attention head to choose which elements of the KV cache it reads, resulting in a dynamic and non-uniform allocation of sparsity across heads, layers, and contexts. TGSA introduces an auxiliary set of small gating heads, which control whether or not a given element of the regular attention KV cache must be loaded. Computing the gating heads before regular attention allows TGSA to skip chunks of the (much larger) standard attention KV cache load. To evaluate TGSA, we pretrain models at 600M, 1.7B and 4B scales and find that its textual quality is comparable to regular attention and post-trained sparse attention baselines, and that TGSA leads to up to 3.5x faster decoding than regular attention at 64K context length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
nlp, pretraining, sparse attention, kernels, cuda, dsa, inference, architecture, llm

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_wbhKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes token-gated spars attention (TGSA), adding a small learnable gate head for each KV group to make binary decisions per query-key pair to enable each token to attend to a variable length subset of KV cache. The gates need to be jointly trained during pre-training of LLM, with sparsity loss and attention-matching KL loss. Pre-training models at 600M, 1.7B and 4B scales, the authors report sparsity over 80% achieved with modest model quality loss, while achieving up to 3.5$\times$ decoding throughput.

Sparse attention is also applied to prefill but prefill gets no speedup but just latency overhead. TGSA needs to be applied from the initial stage of pre-training since PPL from continued pre-training is worse than DSA. Comparsion with top-p oracle is son

Critical strengths

S1. Well-motivated mechanism with a principled recipe.

Small per-KV-group gate heads make token-level, variable-length selection learnable during pre-training, and the gate-key cache lets most of the KV load be skipped.

S2. Real systems results with honest reporting.

Authors implemented custom kernels for training, decode, and prefill and integrated TGSA into vLLM, with measured decode speedups of 1.6$\times$ at 16K and 3.5$\times$ at 64K.

The prefill overhead, the quadratic gate cost in the FLOP accounting, and the CPT limitation are all reported openly.

Critical weaknesses

W1. No from-scratch pre-trained sparse baselines.

Adding comparison with DSA/MSA-style is strongly recommended and should add NSA and DashAttention if available. DSA only appears in CPT comparison with 600M model, so the source of the gains (variable-length selection rule or training approach) cannot be separated.

W2. No real post-hoc baselines.

There is no comparison with post-hoc baselines like QUEST, Loki, Twilight at matched sparsity. The regular top-p baseline loads all keys and therefore is structurally disadvantaged at iso-data and iso-FLOPs settings.

W3. Insufficient analysis and ablations.

Training wall-clock (GPU-hours) and memory overhead are not reported. No ablations on d_g selection even though it is a critical hyperparameter for token selection quality.

W4. Overstatement of novelty.

The abstract’s “fixed sparsity” claim is overstated relative to other adaptive budget works [1][2]. Most of individual loss terms are from existing techniques (MSA/DSA, MoE), so the contribution is the recipe.

[1] Lin, Chaofan, et al. "Twilight: Adaptive attention sparsity with hierarchical top-$ p $ pruning." Advances in Neural Information Processing Systems 38 (2026): 119406-119432.

[2] Zhu, Kan, et al. "Tactic: Adaptive sparse attention with clustering and distribution fitting for long-context LLMs." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

Q1. How does TGSA compare with from-scratch DSA/MSA-style indexers under same token budget and matched memory movement, including indexer loads?

Q2. How do post-hoc sparse attention methods compare in benchmarks that are reported in the paper, when applied to the same ungated model?

Q3. Could the TGSA model be trained to be robust to dense prefill? Since TGSA introduces non-negligible overhead to prefill attention latency, if would be helpful if you can just adapt model to robust to prefill without gating.

Q4. Could you provide ablations with varying d_g and justify the selection of 16?

Q5. Could you report training wall-clock, throughput and memory against ungated model at 8K and at 64K LCE, plus iso-FLOP results at 1.7B and 4B?

2 comments

Thank you for your review! We address the individual questions below (broken down into two comments).

Q1 - How does TGSA compare with from-scratch DSA/MSA-style indexers under same token budget and matched memory movement, including indexer loads?

Refer to general response, section 1 for a comparison against DSA/top-k style indexers. To summarize: we find that the from-scratch DSA indexer, under the same token budget, matches TGSA PPL with 5.89 percentage points less memory movement (which includes indexer loads; this is the case for all memory movement numbers), but in long-context tasks we find that TGSA outperforms DSA in terms of HELMET recall, also in the memory-matched setting.

There is no comparison with post-hoc baselines like QUEST, Loki, Twilight at matched sparsity. The regular top-p baseline loads all keys and therefore is structurally disadvantaged at iso-data and iso-FLOPs settings.

We include the oracle top-p as a practical upper bound for attention-mass-based selection baselines like Quest, Loki and Twilight: it assumes that the top-p set can be found “for free” (i.e., with no memory movement whatsoever), which allows us to only load exactly the top-p keys and values. This is more optimistic than the methods described above, which generally try to make that initial lookup cheap (but not free). Thus, we feel this baseline actually provides a structural advantage to these methods, and even so TGSA outperforms it.

Q2 - How do post-hoc sparse attention methods compare in benchmarks that are reported in the paper, when applied to the same ungated model?

To confirm: the current top-p numbers are exactly the top-p baselines applied on top of the same ungated KV8 model at each scale.

Q4 - Could you provide ablations with varying d_g and justify the selection of 16?

We found that a gate dimension of size 16 balanced quality and memory movement. We conducted this ablation at 600M scale. We will include this table in the next version of the paper:

λ_sparseGate dimensionPaloma PPLEstimated memory movement (% relative to KV8)
1421.59416.12
1821.27614.90
11621.13116.63
13221.13322.34
0.2421.17551.09
0.2821.15134.00
0.21621.11233.75
0.23221.14538.96

Q5 - Could you report training wall-clock, throughput and memory against ungated model at 8K and at 64K LCE, plus iso-FLOP results at 1.7B and 4B?

Due to the high compute requirements, we were unable to run both iso-data and iso-FLOP experiments at 1.7B and 4B, but we note that the FLOP difference goes down across scales as we increase in size, which means that differences in the two settings should get milder as we scale up:

ScaleDense KV8 FLOPs in iso-data settingTGSA FLOPs in iso-data settingIncrease in TGSA FLOPs in iso-data setting
600M1.41×10201.41 \times 10^{20}1.50×10201.50 \times 10^{20}6.4%
1.7B9.68×10209.68 \times 10^{20}9.98×10209.98 \times 10^{20}3.1%
4B5.48×10215.48 \times 10^{21}5.63×10215.63 \times 10^{21}2.7%

That said, we will try our best to include iso-FLOP results in the next version of the paper.

In FLOPs, TGSA training costs only 2.7–6.4% more than KV8 (table above). Regarding training time, across scales 8K TGSA pretraining was 1.72–1.99× more expensive than regular KV8 training, while 64K LCE was 4.86–5.42× more expensive. This is because our training kernels are currently not as optimized as regular kernels, especially in long-context settings. Most of our effort went towards developing good inference kernels, and for training we included an iso-FLOP study that controls for training FLOPs. That said, we will report these numbers and be clear about this limitation in the next version of the paper.

Peak memory, on the other hand, was the same (within 0.5%) across both TGSA and KV8.

(continued from above)

The abstract’s “fixed sparsity” claim is overstated relative to other adaptive budget works [1][2]. Most of individual loss terms are from existing techniques (MSA/DSA, MoE), so the contribution is the recipe.

We will clarify the novelty claim. Our intent was to say that there is comparatively less work that explores how to architecturally change the attention mechanism (and auxiliary losses) such that we are able to come up with naturally adaptive sparse attention. Twilight and Tactic are very relevant baselines, but we class these as taking existing attention mechanisms and learning how to approximate attention on the fly (and, as we mention above, these should be upper-bounded by the oracle top-p baseline).

We also agree that many of the loss terms are inspired by other papers (e.g., MoEs), and coincide with concurrent work (e.g., MSA). We tried to mention this explicitly in the paper (e.g., reference to MoE load balancing in training section, references to MSA/DSA attention matching), but will make it clearer.

Overall, we believe that our main contribution is a new mechanism and recipe to train sparse attention models that have innately adaptive sparsity, and showing the effects of the resulting adaptive sparsity. Concretely: per-query, token-level adaptive gating trained from scratch, with decode kernels that realize throughput gains at 64K context.

Q3 - Could the TGSA model be trained to be robust to dense prefill? Since TGSA introduces non-negligible overhead to prefill attention latency, if would be helpful if you can just adapt model to robust to prefill without gating.

This is a great question! We explored this, but unfortunately we found that TGSA is not robust to dense prefill out-of-the-box. We did find that LCE training made the model more robust to dense prefill, but there is still non-trivial degradation (~0.3 PPL). We will include these results in the next version of the paper.

We do believe that more careful tuning could address this (e.g., changing STE so that it forces dense prefill during the forward pass at random); we intend to explore this in future work.

Reviewer_ydaoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Token-Gated Sparse Attention, which uses lightweight binary gates to dynamically select kv entries for each query and kv group. Experiments from 600m to 4b parameters show competitive model quality, while a VLLM implementation achieves up to 3.5× higher decode throughput at 64k context on a b300 GPU.

Critical strengths
  • TGSA supports token-level, variable-budget sparsity and translates this flexibility into practical decoding gains. The VLLM implementation demonstrates up to 3.5× higher throughput than dense attention at 64k context.
  • Experiments across three model scales show competitive language-modeling and downstream performance. The iso-FLOP comparison and long-context evaluation strengthen the empirical evidence.
Critical weaknesses
  • Comparisons with other trainable sparse-attention methods are limited to a modified DSA baseline at 600m. Runtime comparisons against sparse baselines are missing, leaving TGSA's practical advantage over competing methods unclear.
  • The benefit of variable-budget gating is not directly isolated. A fixed-budget selector using the same gate architecture and a matched average memory budget would help establish whether dynamic allocation is essential.
Questions for authors
  • Can you compare TGSA with a fixed-budget selector using the same gating architecture at matched average kv memory movement? This would clarify the contribution of variable-budget allocation.
  • Do you have measured decode-throughput comparisons against DSA or another learned sparse-attention baseline under comparable quality and hardware settings?
  • How does the speedup vary with batch size and output length? Since prefill is slower, an end-to-end measurement including prefill would clarify which workloads benefit.
  • What happens when all gates for a query and kv group are closed? Please clarify the safeguard used during stochastic training and deterministic inference.
1 comment

Thank you for your review! We address your questions below.

Q2 - Do you have measured decode-throughput comparisons against DSA or another learned sparse-attention baseline under comparable quality and hardware settings?

We could not measure DSA decode throughput directly because the original DSA kernels use MLA and our models use GQA. For decode, especially in the long-context regime, we believe that memory movement is a reasonable proxy for throughput since decode is memory-bound anyway.

We also note that all throughput numbers we do report are measured on the exact same hardware setup (i.e., one B300, same machine, same vLLM version).

The benefit of variable-budget gating is not directly isolated. A fixed-budget selector using the same gate architecture and a matched average memory budget would help establish whether dynamic allocation is essential.

Q1 - Can you compare TGSA with a fixed-budget selector using the same gating architecture at matched average kv memory movement? This would clarify the contribution of variable-budget allocation.

Refer to general response section 1 for a comparison against DSA, including a comparison where the indexer top-k set size is memory-matched to the budget that TGSA uses on HELMET. There, we also discuss that a top-k variant of TGSA substantially underperforms the adaptive version.

Q3a - How does the speedup vary with batch size and output length?

We include further batch sizes for prefill and decode below (with the same setup as Figure 3a in the paper). Note that TGSA decode shines at high batch settings (to amortize the cost of parameter loading), but even at batch size 1 (@ 64K) and batch size 8 (across most sequence lengths) we see some throughput improvements relative to the KV8 model.

Prefill (attention operation, ms)

BatchAttention kernel16K32K64K
1Dense FA41.415.5221.56
1TGSA FA42.038.2932.80
8Dense FA411.5342.43184.20
8TGSA FA416.7561.35234.08
24Dense FA433.64139.55566.07
24TGSA FA448.20179.80714.96

Decode (throughput, tok/s)

Batch4B model16K32K64K
1Dense KV8343308256
1TGSA (λ_sparse = 1)295288283
1TGSA (λ_sparse = 0.2)287274258
8Dense KV81,5811,041616
8TGSA (λ_sparse = 1)1,7511,5231,145
8TGSA (λ_sparse = 0.2)1,4171,116708
24Dense KV82,0651,167624
24TGSA (λ_sparse = 1)3,5223,0002,248
24TGSA (λ_sparse = 0.2)2,8682,2641,516

Q3b - Since prefill is slower, an end-to-end measurement including prefill would clarify which workloads benefit.

We opted to keep prefill and decode numbers separate since the performance characteristics in each scenario are very different (i.e., prefill is consistently slower, decode gets faster as we increase sequence length). We explored a few different ratios of prefill vs. decode length and report the speedup (relative to the KV8 baseline) below:

Prompt tokensOutput tokensTGSA (λ_sparse = 1)TGSA (λ_sparse = 0.2)
8K1281.00×0.95×
8K2K1.24×1.08×
8K8K1.46×1.23×
32K1281.03×0.99×
32K2K1.96×1.63×
32K8K2.52×1.94×

Q4 - What happens when all gates for a query and kv group are closed? Please clarify the safeguard used during stochastic training and deterministic inference.

If all gates for a query and KV group are closed, that group’s attention output is zero. Since training relies on the soft value of the gates, there is no “division-by-zero” error for the gradients. We will clarify this in the next version of the paper.

Reviewer_MwaCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes token gated sparse attention (TGSA), which adds a 16 dimensional gating head to each KV group. For every query and key pair the gate decides whether the key and value need to be loaded, so the number of attended tokens can vary across heads, layers, and queries. The gates are trained during pretraining with a straight through estimator, a global sparsity loss with a scheduled target, and a KL loss that keeps the gated attention close to dense attention. The authors pretrain models at 600M, 1.7B, and 4B, extend them to 64K context, and implement gated decode and prefill kernels in vLLM on B300 GPUs. TGSA reaches a better tradeoff between Paloma perplexity and estimated KV memory movement than GQA variants and post hoc top-p attention, and the 4B model decodes 1.6x faster than dense attention at 16K and 3.5x faster at 64K.

Critical strengths
  1. Simple design, evaluated with real pretraining runs. The gate only adds small query and key projections per KV group plus two auxiliary losses, and the main experiments train it from scratch at three scales with about 50 tokens per parameter. In Figure 2, TGSA has lower perplexity than KV1 and KV2 at similar memory traffic and also beats the top-p oracle. The lm-eval-harness average in Table 4 drops by at most 1.78% relative to dense KV8.
  2. Convincing efficiency measurements and a clear analysis of the losses. The authors measure decode throughput in vLLM instead of relying only on memory estimates. Figure 3 reports the decode gain together with the attention kernel overhead during prefill of about 29% at 32K and 26% at 64K, as well as the short context range where TGSA is slower. Appendix D explains how the sparsity loss and the KL loss push the gates in opposite directions, and Figure 6 confirms that lambda_sparse mainly sets the final sparsity once lambda_strength is not too small.
Critical weaknesses
  1. It is not shown that threshold gating beats a fixed top-k budget. This is the main argument of the paper, but Figure 2 only compares against dense GQA and post hoc top-p. None of NSA, MoBA, MSA or DSA is pretrained with the same data and budget. DSA shows up only in the 600M continued pretraining study in Figure 4. There, starting from the same dense checkpoint, DSA gets a better perplexity versus memory movement tradeoff than TGSA. The paper's claim that TGSA is ahead puts a model trained with TGSA from the start against a dense model converted to DSA through CPT. So the gain may come from having sparsity during pretraining rather than from the gating rule itself. The discussion of prior learned sparse attention overlooks Dynamic Context Pruning (Anagnostidis et al., 2023). That work already uses sparse sigmoid gates computed from additional query and key projections plus a learned bias. It combines these gates into a cumulative attention mask and penalizes the average mask value to encourage sparsity, allowing sparsity to vary across layers and context lengths. A key difference is that a token removed from a layer remains unavailable to subsequent queries in that layer, while TGSA makes a fresh selection for each query.
  2. The decode comparison only uses dense KV8. Figure 2 shows that KV2 already reads about 25% of the KV8 cache with ordinary contiguous loads, yet Figure 3 has no KV1 or KV2 runs and no top-k sparse decode kernel. Without them it is hard to tell what the 3.5x means in practice. The paper also does not report the gate sparsity reached during the 16K to 64K throughput runs, and Figure 5, measured on 8K documents, cannot stand in for it. Long context quality is a concern too. At 64K, where the speedup is largest, the 4B TGSA model has lower HELMET recall than the dense model. HELMET here covers only recall tasks. Appendix H adds language modeling loss on 64K Longmino documents, but other long context tasks are not tested.
Questions for authors
  1. How would TGSA compare with a top-k selector pretrained under the same setup? For example, the TGSA gate with top-k selection, or a DSA or MSA style indexer, at matched average sparsity. Paloma perplexity and HELMET recall on one scale would be enough.
  2. What sparsity does the 4B model actually reach at 16K, 32K and 64K, and how does its decode throughput compare with dense KV1 and KV2 in the same vLLM setup? Please report model quality next to the speed numbers and say whether the baselines went through the same long context extension.
1 comment

Thank you for your review! We respond to your questions and concerns below.

The paper's claim that TGSA is ahead puts a model trained with TGSA from the start against a dense model converted to DSA through CPT. So the gain may come from having sparsity during pretraining rather than from the gating rule itself.

We opted for the CPT setup since this was the original setting in which DSA was proposed. During the rebuttal we conducted a non-CPT study between DSA and TGSA; we included those results in the general response (section 1). In short, we find that especially in LCE settings TGSA obtains favorable results against DSA-from-scratch.

The decode comparison only uses dense KV8. Figure 2 shows that KV2 already reads about 25% of the KV8 cache with ordinary contiguous loads, yet Figure 3 has no KV1 or KV2 runs and no top-k sparse decode kernel.

Q2b - [...] how does [the 4B model] decode throughput compare with dense KV1 and KV2 in the same vLLM setup?

We chose to use the ungated KV8 model as the reference because it performed similarly in PPL to the TGSA models we are comparing; so Figure 3 can be roughly interpreted as “practical throughput gains when moving to TGSA, while keeping PPL matched”.

With this in mind, we conducted a throughput study at sequence length 64K (batch size 24, 128-token continuations). The results are as follows:

4B model / methodTokens/s× densePaloma PPL (pre-LCE)
Dense KV8550.71.0013.735
Dense KV12,566.94.6614.065
Dense KV21,786.03.2413.953
TGSA (λ_sparse = 1)1,853.13.3713.738
TGSA (λ_sparse = 0.2)1,325.22.4113.754
KV8 + top-k (K=2048)435.10.79–
KV8 + top-p (p=0.95)186.80.3413.766

Note that the KV1 and KV2 baselines underperform TGSA and the KV8 baseline in terms of textual quality. We also note that top-k and top-p are slower than the dense KV8 baseline: this goes to show that it is hard to obtain practical performance speed-ups with off-the-shelf kernels.

Q2a - What sparsity does the 4B model actually reach at 16K, 32K and 64K

Refer to section 2 of the general response for the memory movement numbers at each context length and the additional 4B long-context quality results. In terms of what sparsity these models achieve, in the original 4B decode workloads, observed sparsity at 16K, 32K, and 64K was 92.43%, 93.42%, and 94.56%, respectively.

Q2c - Please report model quality next to the speed numbers

For our throughput study, we force each model to sample the same fixed text, since TGSA's sparsity depends on the content being generated and generations from all these non-instruction-tuned models (TGSA and baselines alike) quickly diverge, which would make throughput measurements inconsistent across models. We will clarify this in the paper.

Q2d - [can you] say whether the baselines went through the same long context extension.

Yes! Every model underwent the exact same long-context extension: same number of tokens, same YaRN recipe, etc.

HELMET here covers only recall tasks. Appendix H adds language modeling loss on 64K Longmino documents, but other long context tasks are not tested.

We opted for a recall-based benchmark since, for models that have not been instruction-tuned, we believe it provides more signal than more complex benchmarks.

For the camera ready, we will explore instruction-tuning and evaluating the resulting model on other, more realistic long-context settings.

The discussion of prior learned sparse attention overlooks Dynamic Context Pruning (Anagnostidis et al., 2023). That work already uses sparse sigmoid gates computed from additional query and key projections plus a learned bias. It combines these gates into a cumulative attention mask and penalizes the average mask value to encourage sparsity, allowing sparsity to vary across layers and context lengths. A key difference is that a token removed from a layer remains unavailable to subsequent queries in that layer, while TGSA makes a fresh selection for each query.

Thanks for flagging Dynamic Context Pruning, we were unaware of it prior to this! As you mentioned, their approach is similar but (i) focused on KV cache eviction rather than reducing KV fetches during decode, and (ii) DCP’s main experiments consider fine-tuning a pre-trained model rather than pre-training from scratch. We will include a discussion of it in the next version of the paper.

Q1 - How would TGSA compare with a top-k selector pretrained under the same setup?

We trained the TGSA gate with top-k selection in place of thresholding, as you suggest in Q1; it substantially underperformed the adaptive version (see the end of general response section 1).

Reviewer_CMnNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TGSA uses learned 16-dimensional query/key gates to select a variable number of KV tokens for each query, with selection shared within each KV group. Joint pretraining combines stochastic binary gates, straight-through gradients, global sparsity regularization, and attention-mass matching; inference uses deterministic thresholds. A producer/consumer implementation reads gate keys before loading selected K/V entries. Experiments cover 600M, 1.7B, and 4B models, extension to 64K, and vLLM decode measurements reporting up to approximately 3.5× throughput over dense attention on a B300 GPU.

Critical strengths
  1. The method provides a practical way to learn heterogeneous attention budgets. Token-level thresholding and global sparsity regularization permit different budgets across queries, KV groups, and layers. Gate analyses show both reusable local patterns and document-dependent selection. Coupled with actual decode acceleration, this makes the flexibility consequential beyond a theoretical sparsity claim.
  2. The engineering implementation and experimental controls are strong. The authors extend FlashInfer's XQA decode kernel with a producer/consumer design and integrate it into vLLM, demonstrating throughput gains against optimized dense baselines including TRTLLM-GEN. They also adapt FlashAttention-4 for gated prefill, explicitly reporting its overhead without claiming sparse-computation savings there. Three pretrained scales, iso-FLOP and wider-QK controls, loss-weight sweeps, CPT comparisons, task breakdowns, and a larger-batch dense baseline provide substantial evidence for feasibility and help explain the costs.
Critical weaknesses
  1. Comparisons with closely related sparse attention methods are limited across model scales and in measured runtime. Figure 2 provides valuable multi-scale GQA and top-p comparisons, but the learned DSA-style baseline in Figure 4 is restricted to 600M CPT and reports PPL versus estimated memory movement, without measured decode speed. Thus, relative quality gains over learned sparse alternatives are not validated at 1.7B or 4B. MoBA is discussed but not evaluated, and SeerAttention-R is a relevant missing decode baseline. No low-precision indexer ablation is reported, and the DSA-style baseline's precision is unspecified. A representative learned-sparse comparison at an additional scale, together with actual quality–latency measurements under stated training budgets and matched hardware/workloads, would better establish the benefits of TGSA's token-level dynamic gates.
  2. The quality–sparsity advantage is not consistently established as model size increases. Although three scales are evaluated, the quality gains vary with scale. In Table 1 at sparse=1, 1.7B improves 64K recall from 0.454 to 0.476, whereas 4B decreases it from 0.593 to 0.563. Figure 2 also reports PPL improvements over the dense KV8 baseline at smaller scales that are not similarly evident at 4B. These observations raise uncertainty about preserving quality at larger scales, although they do not establish monotonically worsening performance, and other 4B loss settings remain competitive in Table 4. Analysis at matched realized sparsity or memory-movement budgets would help distinguish a scaling limitation from operating-point or training effects. The absence of larger-scale learned-sparse comparisons further limits this assessment.
Questions for authors
  1. Indexer precision and accounting: For the Figure 4 DSA-style baseline, what precision is used for indexer computation and its key cache, and does estimated memory movement include those cache reads at that precision?
1 comment

Thank you for your review! We address each point below.

Relative quality gains over learned sparse alternatives are not validated at 1.7B or 4B. MoBA is discussed but not evaluated.

The absence of larger-scale learned-sparse comparisons [...].

We constrained our analysis to DSA since this is the key baseline that uses token-level (as opposed to block-level) sparsity, which is one of the key characteristics of our approach. Importantly, we include further experiments against DSA in general response section 1.

[...] the learned DSA-style baseline in Figure 4 is restricted to 600M CPT and reports PPL versus estimated memory movement, without measured decode speed

The original DSA method uses MLA, which makes an apples-to-apples throughput comparison infeasible (TGSA is based on GQA). For decode, especially in the long-context regime, we believe that memory movement is a reasonable proxy for throughput since decode is memory-bound.

Although three scales are evaluated, the quality gains vary with scale. In Table 1 at sparse=1, 1.7B improves 64K recall from 0.454 to 0.476, whereas 4B decreases it from 0.593 to 0.563.

Refer to general response section 2 for 4B LCE results on a slightly less sparse 4B model, where 64K performance is comparable (0.590), and shorter-context performance is stronger than the baseline.

Figure 2 also reports PPL improvements over the dense KV8 baseline at smaller scales that are not similarly evident at 4B. [...]. Analysis at matched realized sparsity or memory-movement budgets would help distinguish a scaling limitation from operating-point or training effects.

The Pareto frontiers show what quality would look like as we sweep different memory-movement budgets. We see that (in Figure 2) TGSA dominates over the baselines we consider. Furthermore, in all settings except the iso-FLOP setting, the best-performing TGSA model is comparable to or better than the corresponding non-TGSA (KV8) model. The reduction in the iso-FLOP setting is not that surprising in our view: we spend additional FLOPs for the sparse attention forward and backward passes; we interpret the iso-FLOP study as showing that, in spite of the additional FLOPs going towards sparsity (i.e., omitting context from the model), they still allow us to outperform iso-FLOP GQA in the memory movement–quality frontier overall.

Q1 - Indexer precision and accounting: For the Figure 4 DSA-style baseline, what precision is used for indexer computation and its key cache, and does estimated memory movement include those cache reads at that precision?

No low-precision indexer ablation is reported, and the DSA-style baseline's precision is unspecified.

We try to be fair to both methods in the DSA study, and have the TGSA gates have the same memory footprint (per-token) as the DSA model, i.e., the gates/indexer head keys are the same size and BF16 precision. We account for this in the memory movement plot for both methods.

At 600M scale, we find that we can quantize TGSA gates from BF16 to INT4 with negligible change to PPL (+0.004) while further reducing memory movement from 16.63% to 12.42%. This suggests that some indexer quantization techniques should transfer to the TGSA case, and could provide further throughput improvements.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

General response

We thank the reviewers for their careful feedback. We respond to common points below; reviewer-specific questions are addressed via direct replies to reviews.

1. Comparison to other trainable sparse attention methods

We chose to compare against DSA since this was the closest token-level learned-sparsity baseline. MoBA and MSA are block-level approaches, not token-level like ours. NSA's recipe combines several techniques (e.g., compressed attention). Among these, DSA is the most popular approach that has a pretrainable token-level selection mechanism. We compared it to TGSA in the CPT regime since this was the original DSA setup, and we wanted to be fair to the original method’s design.

We additionally pretrained DSA and TGSA on the same number of tokens (we used a ~20 tokens/parameter budget due to rebuttal period timelines): while DSA-from-scratch matches or outperforms TGSA in short-context settings, TGSA substantially outperforms DSA in long-context tasks. Specifically, on our main Paloma benchmark, we find that TGSA and DSA are comparable in terms of PPL (25.684 and 25.671, respectively), though DSA has an advantage in terms of memory movement (15.94% of dense for DSA vs. 21.83% for TGSA). Note that TGSA memory movement should improve with more training, as our ~50 tokens/parameter TGSA models reach 16.63% of dense memory movement, whereas DSA memory movement would stay the same.

However, in the long-context evaluation (column 2 below), we find that TGSA yields better performance than DSA, and this is true even when we adjust the DSA indexer head’s budget (e.g., by increasing k in the top-k selection step) so that DSA and TGSA get the same memory movement budget; this strongly suggests that TGSA’s adaptive budgets are directly contributing to its stronger performance.

We also include results when models are LCE’d with YaRN (columns 3–5; due to time limitations, for these runs we did not conduct long-context fine-tuning); we found that TGSA is very amenable to training-free LCE, whereas DSA’s performance degrades very rapidly. We suspect this may be because of two separate effects: (i) DSA’s fixed k is especially ineffective for tasks that require long-range understanding (which is why the DSA row sees such a sharp drop), and (ii) adapting k on-the-fly helps a bit (e.g., 2.65% → 33.33% at 16K), but is still much more rigid than TGSA’s ability to have each head, token, and sequence allocate budget as it sees fit.

HELMET Macro @ 8KHELMET Macro @ YaRN 8K → 16KHELMET Macro @ YaRN 8K → 32KHELMET Macro @ YaRN 8K → 64K
TGSA43.04%42.26%36.73%31.33%
DSA38.96%2.65%0.00%0.00%
DSA (memory movement-matched to TGSA)35.18%33.33%11.16%6.57%

We will include all of these results in the final version of the paper.

We also trained a version of TGSA where the thresholding was replaced with top-k selection, but we found that this severely underperformed our current TGSA recipe (e.g., 39.901 PPL vs 34.918 PPL at an intermediate checkpoint, with the gap remaining constant for several thousand steps at that point). This suggests that our TGSA recipe works only in the adaptive token selection regime.

2. 4B long-context performance

Some reviewers highlighted that 64K results at 4B were slightly under the baseline. We initially had only included the λ_sparse = 1 experiment; we therefore conducted LCE on the λ_sparse = 0.2 model as in the 600M and 1.7B scale LCE experiments.

We also collected memory movement statistics at each point. With this regularizer configuration, TGSA uses more memory movement but outperforms or matches the baseline (HELMET macro-average shown, with relative memory movement in parentheses).

4B model8K16K32K64K
Ungated0.847 (100%)0.783 (100%)0.693 (100%)0.593 (100%)
TGSA, λ_sparse = 10.838 (12.6%)0.791 (11.3%)0.669 (10.9%)0.563 (10.3%)
TGSA, λ_sparse = 0.20.868 (25.2%)0.802 (22.4%)0.709 (20.4%)0.590 (19.9%)

We will add the above results to Table 1 in the final version of the paper.

When running decode, we find that we are still able to obtain a speedup of about 2.4× in throughput, despite the reduced sparsity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.