acceptodds
Under review as a conference paper at ICLR 2027

Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding

Abstract

Massive KV caches can cause severe memory-bandwidth bottlenecks during long-context decoding. Sparse attention methods mitigate this problem, but often drop necessary context, leading to quality degradation. We introduce **Elastic Threshold Attention (ETA)**, an end-to-end trainable architecture that rivals dense model quality under hardware-aligned block-sparse decoding. ETA predicts dynamic, contextual thresholds directly from query representations, adjusting context retention depending on the task at hand. To learn this policy from scratch while enabling near lossless KV cache pruning at inference time, we filter attention logits through a shifted SiLU gate during training. We show theoretically and empirically that this creates a smooth, near-uniform attention floor that neutralizes sub-threshold value contributions while simultaneously causing localized attention sinks on initial tokens to disappear. To materialize these advantages, we implement a fused inference-time kernel in Triton that screens KV blocks in time using cached geometric-probabilistic bounds. Across language modeling, reasoning, and RULER benchmarks, our 1.45B ETA model matches or exceeds dense quality, outperforming alternative fast decoding methods (Quest, HO, NSA) while achieving higher sparsity levels. Our kernel also achieves up to end-to-end speedups over FlashAttention-2 at context lengths of up to K tokens. Finally, we introduce an offline calibration algorithm for domain-specific deployments that freezes per-head constant thresholds, cutting attention compute by an additional 27% at no quality cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
inference optimization, sparse attention, deep learning, transformer

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_RnLGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ETA, a trainable sparse attention mechanism. A per-layer linear predictor maps queries to dynamic thresholds, and attention logits are gated multiplicatively through a shifted SiLU. Sub-threshold logits are pulled toward zero rather than −∞-\infty, which forms a near-uniform background floor. The paper argues that this floor makes pruned and co-admitted tail keys negligible after RMSNorm, and that it removes attention sinks. At decode time, a fused Triton kernel screens KV blocks with a centroid, variance, and norm bound. A 1.45B model trained at 2048 tokens matches dense perplexity and outperforms Quest, H2O, and NSA. The kernel reaches up to 2.36 times speedup over FlashAttention-2, and the full model up to 2.15 times.

Critical strengths

The multiplicative SiLU gate is a simple and well motivated design that makes training-time and inference-time behavior consistent.

The comparison with additive masking shows a clear perplexity and density advantage, and the tail cancellation analysis is verified empirically. The method is backed by complete training and decoding kernels, a direct comparison with NSA under the same backbone, and measured end-to-end speedups.

Critical weaknesses
  1. I'm a little curious why quality and efficiency are evaluated in different systems. The model is trained at 2048 tokens, and quality is measured only up to 8192, where the dense baseline itself falls to near zero on passkey retrieval. Speedups are reported at 64K to 512K, where model quality is not evaluated. So it seems this paper does not show ETA preserves quality at the context lengths where it delivers acceleration.

  2. I think the headline speedups depend on aggressive operating points whose quality cost is not clearly reported. The 2.15 times end-to-end gain requires 16% union density, while the quality tables mostly use 50% to 70% union density. And at 51.2% union density, the kernel is slower than FlashAttention-2 for batch size 32.

  3. In the ARC-Easy and HellaSwag tasks, the prompts are shorter than the 256-token windows used by the heuristic baseline methods, which means that these baseline methods may degenerate into standard dense attention mechanisms. And this may also explain why the confidence intervals are so large.

Questions for authors
  1. What are the perplexity and RULER results at 16% and 35.5% union density, the operating points used for the speedup claims?

  2. Can ETA be evaluated after continued pretraining at 32K or longer, so that quality and speed are measured at the same context length?

  3. Which NSA implementation was used for the decode latency comparison, and is it the official optimized kernel?

  4. How sensitive is the learned threshold predictor to the sparsity regularization schedule at larger scales?

2 comments
AuthorsOpenReview ↗

Response to Reviewer RnLG

We thank the reviewer for their constructive feedback! We provide detailed analyses of the points the reviewer raised below.

W1, W2, Q1 & Q2: A common ground for Quality and Speedup Evaluations

The reviewer is right in pointing out this inconsistency in our reported evaluations. We performed a continual pretraining experiment and evaluated ETA jointly for both the quality improvements and the speed-ups. In our General Response above, we provide additional details. Specifically:

  • Unified 32K & 64K Evaluation: After 32K continual pretraining, ETA-32K preserves C4 and FineWeb PPL up to 64K across all three speedup budgets — staying within +3.5%–+6.2% of Dense-32K at 35.5% union and +6.6%–+7.1% at 64K even at 16.0% union.
  • Faster Than FA2 Across All Three Union Tiers (W2): Our fp16 grouped kernel (B = 64, 64K–512K) yields 1.20×–1.48× speedups even at 51.2% union, 1.53×–1.94× at 35.5% union, and 2.10×–3.31× at 16.0% union.

We will incorporate these evaluations into our main manuscript, emphasizing a common-ground comparison between quality and speed-ups.

W3: Full-Split 15-Shot ARC-Easy and HellaSwag (L >> 256)

The reviewer is right in their assessment. The ARC-Easy and HellaSwag benchmarks were tested with short sequence length questions, prompting high density metrics. To ensure prompts substantially exceed the 256-token heuristic window and shrink confidence intervals, we re-evaluated all methods on the full validation splits of ARC-Easy (n = 570) and HellaSwag (n = 10,042) in both 0-shot and 15-shot (~468 and ~1,344 avg tokens):

Method (1.45B)ARC-Easy 0-shot / 15-shotHellaSwag 0-shot / 15-shot (n=10,042)15-shot HellaSwag D_u
Dense-1.45B42.98% / 50.53%42.40% / 42.71%100.0%
ETA-1.45B (b=4, VC)43.33% / 50.88%42.14% / 42.89%52.9% (22.1% D_h)
Quest / H2O43.16% / 50.35% | 42.98% / 50.70%37.55% / 42.92% | 42.37% / 42.84%65.9% / ~60%
SWA-256 (Window 256)42.98% / 49.65%42.37% / 41.25%18.9%

In 0-shot, window baselines indeed degenerate toward dense (~42.4% HellaSwag); in 15-shot (~1,344 tokens), SWA-256 loses distant exemplars and drops below 0-shot (41.25%), whereas sparse ETA preserves full 15-shot gains (42.89% vs. 42.71% Dense at 52.9% union / 22.1% head density). We will update our manuscript to reflect these changes.

Q3: NSA Decode Implementation and Kernel Comparison

We thank the reviewer for the helpful catch! Indeed, our deployment of NSA did have some un-optimized code paths that we have promptly fixed. Because DeepSeek did not release official NSA code, our submission used the standard open-source XunhaoLai/native-sparse-attention-triton repository, whose decode path ran the compressed branch in eager PyTorch with ~20 CPU–GPU syncs per step (3.3–6.1 ms). Replacing that bottleneck with a zero-sync Triton compressed_attention_decode kernel speeds up NSA's full step by 2.1×–4.6× (1.15–2.23 ms, and 2.1× end-to-end). Isolating pure Triton kernel time, NSA's sparse kernels land in the same 0.15–0.38 ms regime as ETA's single-pass kernel (ETA is 1.1×–1.9× faster for B=1). Thus, with fully optimized kernels, raw compute time is comparable; ETA's structural advantage is in the simplicity of the architecture and executing one fused pass over one standard KV cache rather than orchestrating three attention branches and three separate caches.

At the same time, ETA still outperforms NSA in the retrieval benchmarks we ran.

Model (126M, 1.31B tok)LD_uSingleMultiKeyMultiValVTFWE5-Task Mean
Dense-126M2K100.0%2.0%12.0%5.0%0.0%0.0%3.80%
ETA-126M (d50)2K30.3%14.0%16.0%9.0%0.0%0.0%7.80%
NSA-126M2K82.9%8.0%6.0%2.0%0.0%0.7%3.33%

Q4: Sensitivity to the Sparsity Regularization Schedule at Larger Scale

Sweeping λ 5× (0.01 → 0.025 → 0.05) on a 248M (16L, d=1024) model smoothly compresses soft density (45.8% → 25.2% → 16.3%) and D_union (96.2% → 84.9% → 75.3%) while changing in-distribution 1K loss/C4 PPL by only 0.003 nats / 0.7 PPL (49.89 → 50.60), and actually improving 2K–8K PPL by 3.0–27.8 points. Combined with our <300 ms post-hoc global union calibrator (<0.5% held-out budget error on 1.45B), ETA does not require fragile λ tuning to hit a desired inference budget. We will update our manuscript with these additional details.

We thank again the reviewer for all their helpful comments!

Thank you for your reply and for providing more experimental results. Regarding Q3, I didn't actually anticipate any implementation issues with the code, but the optimized kernel still supports the conclusions, so overall I still accept them. The author's response resolved most of my concerns, and I'm inclined to maintain my rating.

Reviewer_AtE3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Elastic Threshold Attention (ETA), an end-to-end trainable sparse-attention architecture for long-context decoding. ETA predicts query-conditioned, per-head score thresholds and applies a shifted-SiLU transformation to attention logits during training, producing a near-uniform sub-threshold attention floor rather than hard-masking low-scoring tokens. The paper argues that the resulting tail-value contributions approximately cancel, making the model robust to both pruning entire KV blocks and co-admitting sub-threshold tokens within selected blocks. At inference time, ETA uses cached centroid, variance, and norm metadata to screen KV blocks before loading them from HBM, together with a custom Triton decoding kernel. Experiments on a 1.45B model report competitive language-modeling, reasoning, and retrieval performance relative to dense attention and several sparse-attention baselines, along with substantial decoding speedups at long context lengths.

Critical strengths

The paper presents a well-integrated model-and-systems approach to sparse decoding. In particular, the shifted-SiLU tail-tempering formulation explicitly addresses the mismatch between token-level selection during training and block-level pruning or co-admission during inference, and the paper supports this mechanism through analytical arguments, masking ablations, representation-similarity measurements, and an end-to-end sparse decoding implementation.

Critical weaknesses

The novelty and empirical positioning relative to the closest prior work are insufficiently established. SeerAttention-R already studies learned, query-dependent block selection for autoregressive long-context decoding with a dedicated sparse kernel, while Twilight already introduces runtime-adaptive attention budgets rather than fixed top-kkk selection. These works overlap substantially with ETA’s broader claims of learned contextual sparsity, query-dependent context allocation, and hardware-aligned sparse decoding, but they are neither discussed in sufficient depth nor experimentally compared. ETA’s shifted-SiLU tail tempering and the resulting robustness to block pruning and co-admission appear potentially distinct, but the current evidence does not isolate whether this mechanism provides a meaningful quality-latency advantage over these closely related alternatives. This omission makes it difficult to assess the significance of the contribution and is the main reason for my weak-reject assessment.

Questions for authors

1. Novelty relative to the closest prior work

How does ETA differ technically from SeerAttention-R’s learned query-dependent block routing and Twilight’s adaptive attention-budget mechanism? Please distinguish which parts of ETA are genuinely new beyond replacing their selectors or budget rules with a learned scalar score threshold.

2. Matched empirical comparison

Can the authors compare ETA with SeerAttention-R and Twilight under matched model size, context length, block size, physical GQA-union density or HBM traffic, and output quality? If an exact comparison is infeasible, a detailed comparison of selector overhead, training requirements, achievable sparsity, and kernel assumptions would help establish the practical significance of ETA.

3. Contribution of tail tempering itself

Which reported gains arise specifically from shifted-SiLU tail tempering, rather than from query-conditioned selection or adaptive context allocation? A controlled comparison using the same threshold predictor and block-screening mechanism with hard masking, conventional soft gating, and the proposed tail tempering would help isolate the paper’s most distinctive component.

1 comment
AuthorsOpenReview ↗

Response to Reviewer AtE3

We thank the reviewer for their constructive evaluation. Their comments substantially improved our positioning and ablations, and we address each question below:

Q1: Technical Novelty Relative to SeerAttention-R and Twilight

Beyond predicting a scalar score threshold, ETA differs from SeerAttention-R and Twilight in three core mechanisms:

  1. Shifted-SiLU Tail Tempering (vs. Hard Truncation): Rather than hard-masking unselected blocks to −∞-\infty like SeerAttention-R or Twilight, ETA contracts sub-threshold logits toward 0 (exp(0) = 1) during pretraining. At decode time, the kernel epilogue retains the exact Eq. 4 pruned value sum (V_full - V_selected) in O(1) time with zero inner-loop HBM reads, preserving tail information without loading pruned KV blocks.
  2. Single-Pass Per-Head Dual Bound (vs. Two-Stage or Group-Pooled Routing): Twilight uses a two-stage pass (selector → 4-bit per-token top-p scan → sparse attention) and keeps 2 dense layers, while SeerAttention-R runs an external 7.94M-parameter AttnGate pooling all GQA query heads into one shared mask (D_head = D_union). ETA screens blocks in one fused pass across all 22 layers via min(Gaussian, Spherical) bounds with independent per-head thresholds (0.09M params, <0.01% overhead).
  3. End-to-End Sink-Free Pretraining (vs. Distillation): SeerAttention-R freezes a dense teacher and distills AttnGate on dense attention maxima (351M tokens), inheriting dense sinks (5.5%–15.1% sink mass; 14%–16% in Twilight). ETA trains end-to-end from the LM + sparsity loss with <1.5% sink mass and zero distillation.

Q2: Matched 1.45B Quality & Systems Comparison (L = 2,048, b = 64)

We distilled SeerAttention-R (351M FineWeb tokens) and implemented Twilight on our exact Dense-1.45B GQA backbone (b = 64), comparing at a matched ~51% GQA-union budget (DuD_u) on the identical C4 and RULER splits:

Method (1.45B, b=64, ~51% DuD_u )C4 PPLRULER (2K)Head Compute (DhD_h)Selector & Decode
Dense-1.45B (100% D_u)18.2322.8%100.0%None (1-pass dense)
ETA-1.45B19.8723.2%22.9%–31.6%0.09M params, 1 fused pass
SeerAttention-R19.0918.0%51.8%7.94M distilled gate, 2-stage
Twilight18.5021.8%31.1%4-bit K cache, 2-stage (2 dense layers)

Takeaways:

  • At matched ~51% DuD_u, SeerAttention-R edges out ETA slightly on short-2K b = 64 PPL because pooling all GQA heads into one shared mask feeds every head 1.6×–2.3× more tokens. At the same time, ETA's independent per-head thresholds preserve specialized retrieval heads, beating SeerAttention-R on RULER at both 51% DuD_u (23.2% vs. 18.0%) and 36% DuD_u (22.0% vs. 15.0%) in 1 fused pass with <1.5% sink mass.
  • Notably, swapping Quest's bound inside Twilight for ETA's Dual Bound lifts Twilight to 25.0% RULER performance!

Q3: Isolating Shifted-SiLU Tail Tempering (4-Way Gate Ablation)

We agree with the reviewer's assessment. We performed the following experiment to more effectively isolate the effect of shifted-SiLU tail tempering.

Holding the 126M backbone, threshold predictor, and b = 64 Dual-Bound screen fixed (L = 1,024, 10K steps), we ablated four gates: (i) ETA Shifted-SiLU, (ii) Strict (S + 100σ - 100), (iii) Post-Exp Log (S + log σ), and (iv) Hard+STE:

Gate Family (126M, b=64)1K C4 PPL (D_u)2K C4 PPL (2×)RULER 1K → 2K (λ_1 | λ_2)
ETA Shifted-SiLU (λ=0.05 / 0.20)86.68 (76%) / 90.06 (58%)87.28 / 91.292.4% → 3.1% | 2.5% → 3.2%
Strict (λ=0.20 / 0.40)84.66 (71%) / 86.71 (61%)93.36 / 98.810.0% → 1.6% | 0.4% → 1.3%
Post-Exp Log (λ=0.05 / 0.15)79.42 (97%) / 81.85 (93%)93.19 / 91.130.0% → 3.1% | 0.4% → 2.8%
Hard+STE (λ=0.05 / 0.15)134.79 (98%) / 173.90 (94%)145.57 / 181.800.0% → 0.1% | 0.0% → 0.0%

Hard+STE collapses (135–174 PPL) from boundary STE spikes, while Post-Exp Log resists sparsification (93%–97% D_u) and jumps +13.8 PPL at 2K. At matched 58%–76% D_u, Shifted-SiLU yields the best retrieval (2.4%–2.5% vs. 0.0%–0.4% at 1K) and flattest 2× extrapolation (86.7 → 87.3).

We hope the above points have helped clarify our contributions and promise of the ETA architecture. We thank the reviewer again for their engagement.

Reviewer_mDttOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose Elastic Threshold Attention, where the method uses a shifted SiLU in logit space, replacing the tail of sub-threshold keys as smooth, uniform attention background floor. As such, ETA can adaptively move between dense attention during training and a sparse operating mode during inference. For the latter mode, in particular, ETA uses a Dual Probabilistic-Geometric Block Index to screen KV blocks and a fused Triton decode kernel to skip HBM transfers for pruned blocks. Evaluations were conducted on relatively small scale experiments on perplexity and Ruler-like settings.

Critical strengths
  • The method is carefully designed, and I particularly appreciate the full-stack solution, from the attention formulation to block selection and the fused Triton kernel. The implementation work is substantial.

  • The paper is solid, well written, and polished. It is dense with information and technical details, though some sections take effort to follow.

  • The experimental settings are not frontier, but the experiments within the chosen scope are convincing. The matched dense baseline, ablations, and direct checks of training–inference consistency are well done.

Critical weaknesses
  • The long-context evaluation is a bit limited. Most quality tests use 2K–4K contexts, with single-needle retrieval extended to 8K, while the headline speedups use 64K–512K. It is unclear whether quality holds at the lengths where the method is most efficient.

  • The baselines are not frontier enough. The main comparisons focus on H₂O and Quest, while DSA is missing. The NSA comparison uses only a 126M model trained on 164M tokens, which is too limited to establish competitiveness with modern trainable sparse attention. I also have concerns regarding the effectiveness and training stability of the approach under scaled base model training.

  • Training remains dense and introduces overhead: the appendix reports 27.4% higher training step latency. Given that the method requires training changes, the practical benefit over alternatives that accelerate both training and inference needs stronger evidence.

  • The training–inference consistency experiments are useful, but leave questions about longer contexts and RL. High representation similarity and top-1 agreement do not automatically the elimination of training-inference mismatch for stable RL updates. This is an untested concern, rather than a demonstrated failure.

  • My biggest question to authors: This is a well-presented and interesting work. But what is the ultimate positioning of it? Is the recipe frontier enough? How would the authors persuade the LLM community switching from other sparse attention designs to the proposed ETA design?

Questions for authors

Please refer to the weakness section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mDtt

We thank the reviewer for their helpful comments and supportive assessment. We have run additional experiments and address each concern below:

W1: Quality Evaluation at the Speedup Operating Point

We agree that quality and efficiency should be evaluated in a common regime. In our General Response above, after 32K continual pretraining (1.05B tokens), ETA-32K (b = 64) stays within +2.5%–+5.4% PPL of Dense-32K at 35.5%–51.2% union (1.20×–1.53× speedup at 64K, up to 1.94× at 512K) and within +6.6%–+7.1% PPL at 64K even at 16.0% union (2.10× speedup at 64K, up to 3.31× at 512K). Though compute-limited, this confirms ETA preserves quality in the exact regime where it achieves 2×–3× faster decoding than FlashAttention-2.

W2: Frontier Baselines and Training Stability

  • Scaled Baselines: In our General Response, we scaled 126M pretraining 8× to 1.31B tokens across MoBA, SAS, and DMA—plus NSA (3.59 loss, 36.3 C4 PPL, 3.33% RULER vs. ETA's 7.80%–11.03%)—and in our response to Reviewer AtE3 we compare 1.45B ETA against SeerAttention-R and Twilight. ETA remains highly competitive across efficiency, PPL, and RULER. Conceptually, ETA learns a sink-free score geometry end-to-end from the LM loss, whereas alternatives either distill/prune dense models (inheriting sinks) or tune rigid top-k / top-p budgets.
  • On DSA: In our revision, we add DSA (DeepSeek Sparse Attention) to our Related Work taxonomy: DSA is co-designed around DeepSeek's MLA compressed latent KV cache and requires a separate KL-distillation stage for its lightning indexer before sparse training, whereas ETA targets standard GQA backbones with zero distillation stage.
  • Training Stability: Across all pretraining and continual runs (126M at 1.31B tokens, 248M, 1.45B, and 1.45B-32K), ETA trained with zero loss spikes or gradient instabilities, matching or beating dense final training loss (3.777 vs. 3.778 at 126M; 2.729 vs. 2.741 at 1.45B-32K). We will add a stability discussion in the revision.

W3: Dense Training Overhead vs. Sparse Training

Our primary goal with dense-soft pretraining is maximizing inference efficiency while keeping training simple via our fused Triton forward/backward kernel (27.4% step-time overhead over FA2, which further kernel tuning can narrow). Moreover, as shown in our response to Reviewer prDH (W3), ETA can also train directly in block-sparse mode with our b = 64 Dual-Bound selector: it converges cleanly (-5.8 PPL gain at 2× extrapolation), offering a path to sparse training speedups at ultra-long pretraining contexts while dense-soft remains 1.47× faster at 1K–2K windows.

W4: Compatibility with RL Updates

The reviewer raises an important point regarding downstream RL stability. To test whether block-sparse inference stays close enough in distribution to dense-soft training for PPO/GRPO updates, we evaluated teacher-forced token probability ratios p_train / p_infer on ETA-1.45B (L = 2,048) between the training forward pass and our sparse decode kernel. Across block sizes, 90.9% (b = 4) and 89.3% (b = 64) of token importance ratios fall inside the standard [0.8, 1.2] PPO clip window, with a token-level KL divergence of just 0.0091–0.010 nats. While full multi-step RL post-training remains future work, these tight ratios indicate minimal off-policy drift between sparse rollouts and training updates.

Positioning Question: When & Why Should Practitioners Choose ETA?

For serving an already-trained frozen dense LLM, post-hoc selectors like Twilight or Quest are probably best for practitioners. However, we are proposing ETA as a new transformer-based architecture that couples inference-time sparsity with training. It offers three three concrete advantages:

  1. Simplicity and a minimal systems footprint: With only 0.1% additional parameters and a one-line change to the transformer architecture, ETA is a simple and powerful improvement to dense attention.
  2. Adaptive per-query context: Dynamic per-head thresholds yield 2.3×–7.6× higher RULER retrieval than fixed top-k trainable routers (MoBA, SAS, NSA).
  3. Sink-free geometry (<1.5% sink mass) & RL-ready alignment: Eliminating attention sinks aids interpretability and KV compression, while preserving tight token probability ratios (~90% in [0.8, 1.2]) for downstream RL.

We thank the reviewer again for their engagement and hope that our exposition has adequately addressed their concerns. We invite any additional comments or insights the reviewer may have in the subsequent discussion.

Reviewer_prDHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Elastic Threshold Attention (ETA), a trainable attention mechanism for accelerating long-context autoregressive decoding. ETA predicts per-head, query-conditioned thresholds and applies a differentiable multiplicative gate to attention logits. Rather than deleting low-scoring tokens during training, it contracts their logits toward zero, creating a diffuse background distribution. Training combines the language-modeling objective with a sparsity regularizer. At inference, cached block statistics support approximate screening of KV blocks, implemented through a custom Triton kernel. The paper evaluates a 1.45B model on language modeling, commonsense reasoning, retrieval, and decoding efficiency.

Critical strengths
  1. Simple and lightweight architectural modification. ETA introduces a small linear threshold predictor while largely preserving the standard Transformer structure. The thresholds receive gradients directly from the language-modeling loss, without requiring attention-distribution distillation.

  2. Interesting treatment of low-scoring context. Contracting sub-threshold logits toward zero provides a distinctive alternative to support truncation. The resulting background distribution offers a concrete hypothesis for tolerating both skipped blocks and low-scoring tokens co-admitted within retained blocks.

  3. Attention to practical implementation. The paper develops fused training and block-sparse decoding kernels, including GQA-aware loading and compact metadata. This connects the proposed attention formulation to the memory-bandwidth bottleneck of decoding.

  4. Useful empirical and mechanistic analysis. The matched dense pretraining baseline provides a meaningful reference, and fine-grained ETA configurations retain near-dense perplexity. Analyses of attention sinks, layer-wise density, contextual thresholds, and static calibration provide additional insight beyond aggregate benchmark results.

Critical weaknesses
  1. Indirect budget control and insufficient sensitivity analysis. ETA encourages sparsity through a penalty on the average soft inclusion gate. However, this surrogate does not directly constrain the number of loaded KV blocks or the actual GQA-union memory traffic. The relationship between the regularization coefficient, training-time soft density, and inference-time computational cost therefore remains unclear. More sensitivity analysis is needed to establish whether the proposed schedule provides reliable budget control across model sizes and context lengths.

  2. Limited comparisons with trainable sparse attention methods. Much of the evaluation compares ETA, which receives full pretraining adaptation, against post-hoc sparsification methods. Although NSA is included, that comparison is restricted to a 126M model trained on approximately 164M tokens. Controlled comparisons with simple trainable alternatives such as MoBA and DMA, as well as LM-loss-based routing such as SAS, would better establish the contribution of ETA’s particular formulation. Such comparisons could be conducted at a smaller scale, provided that the backbone, data, training budget, and inference cost are appropriately controlled.

  3. The necessity of dense-soft training is not sufficiently established. Training evaluates all causal query–key pairs with continuous gating, whereas inference performs hard block selection. The paper does not sufficiently isolate the benefits of this discrepancy relative to training with the inference-time block selector while retaining continuous gates within selected blocks. The additive “Strict Mode” ablation does not directly answer this question. Consequently, it remains unclear whether dense-soft training is necessary for quality, or whether a more consistent sparse-training formulation could achieve comparable results at lower training cost.

  4. The RMSNorm-based invariance argument does not directly extend to the actual residual block. The analysis argues that pruning primarily rescales the attention output and that RMSNorm absorbs this change. However, the architecture adds the projected attention output to the residual stream before normalization. In general,

    \neq \operatorname{RMSNorm}(x+W_Oo).$$ Moreover, different heads may undergo different rescalings, which need not produce a common scalar change after concatenation and output projection. The reported empirical alignment is useful evidence, but the theoretical claim should be qualified or extended to account for these architectural details.
Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer prDH

We thank the reviewer for their questions and comments! Their insights have helped us substantially improve our exposition, and we address each concern below:

W1: Sensitivity analysis & inference budget control

We performed a 5× sweep of λ\lambda (0.01 → 0.025 → 0.05) on a 248M model (alongside an analogous 8× sweep on 126M). On 248M, λ\lambda smoothly dials soft density (45.8% → 25.2% → 16.3%) and natural union density (96.2% → 84.9% → 75.3%), while in-distribution 1K loss / C4 PPL varies by just 0.003 nats / 0.7 PPL (49.89 → 50.01 → 50.60). Furthermore, while λ\lambda sets the training sparsity prior, we can use a lightweight 1D margin-bisection step at deployment-time to map any trained ETA model to an exact target HBM budget Dunion∗D_{\text{union}}^*. We will include these findings and additional notation in our revision.

W2: Controlled comparisons with trainable sparse attention methods

We agree that positioning and comparing ETA against trainable sparse attention methods is vital. In our General Response above, we trained Dense, ETA, SAS, MoBA, and DMA (as well as NSA) from scratch on the identical 126M GQA backbone for 1.31B FineWeb tokens (8× longer than our submission) and evaluated them at matched b = 64 union budgets (35% and 50%). Among single-branch architectures, ETA achieves the lowest training loss (3.777), matches SAS and beats MoBA/DMA on sparse PPL, and delivers 2.3×–7.6× higher RULER retrieval than all four trainable baselines (including 1.31B-token NSA-126M at 3.33% RULER vs. ETA's 7.80%–11.03%).

W3: On Block-Sparse Training

The reviewer raises an important point on whether we can train directly with our Dual-Bound block selector. To examine this, we trained a 126M model (L = 1,024, 10K steps, λ = 0.05) directly using our inference-time b = 64 Dual-Bound block selector (both from scratch and after 20% dense warmup) and compared against standard dense_soft ETA:

Training Mode (126M, b=64)Train LossStep Time (L=1K)1K C4 PPL (D_u)2K C4 PPLRULER (1K / 2K)
Dense-Soft ETA (Default)4.4010.229 s80.28 (78%)92.392.40% / 2.53%
Block-Sparse (scratch)4.4330.336 s81.99 (79%)86.551.60% / 2.13%
Block-Sparse (20% warmup)4.4330.336 s82.04 (79%)86.871.60% / 2.00%

Inference-consistent block-sparse training converges cleanly without instability and improves 2× length consistency (-5.8 PPL at 2K), while dense_soft retains a slight in-distribution and RULER edge (-1.7 PPL, +0.8% RULER) and is 1.47× faster per step at L = 1K because it avoids per-step causal block-metadata computation.

We thank the reviewer for highlighting this approach and we will include this supplementary finding in our revision.

W4: Replacing the RMSNorm argument

The reviewer is completely right: per-head scalar rescaling does not pass through multi-head concatenation (W_O) and the pre-norm residual addition. We have thus retired the RMSNorm invariance claim in the revision; that argument was aimed at building intuition and can be cleanly replaced.

Intuitively, ETA's training–inference synergy follows directly from the uniform-weight (e^0 = 1) score tail created by the training-time shifted-SiLU gate. When we reject a block at inference, we do not drop its tokens entirely; instead, we omit their individual score weightings and fold in their exact value sum at weight 1. This is what Eq. (4) was also capturing in our original submission and aligns with our algorithm's implementation. The per-block value sum updates online in O(1) time without loading pruned KV blocks from HBM, cutting 1.45B train–inference KL divergence by 4.4× (0.040 → 0.0091 nats) relative to dropping pruned blocks whenever our Dual Bound retains heavy-hitter keys.

Training with the score tail completely pruned (strict mode) also preserves inference synergy, but discarding pruned value information requires maintaining lower sparsity to match the same PPL—matching what we observe in our strict mode ablations.

We thank the reviewer again for their insightful comments and are happy to discuss further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We want to thank all the reviewers for their careful reading and insightful comments. Based on their insights we have performed certain additional experiments and investigations to address their concerns. Since many reviewers share certain overall themes in their assessments, we provide a general response here pointing to specific themes:

1. Evaluating quality at the same standard as efficiency

Addresses: Reviewer mDtt (W1), RnLG (W1, W2, Q1, Q2) and prDH (W1.1).

We agree with the reviewers that our method should be evaluated in a common regime in terms of efficiency and quality. To that end, we continually pretrained both the dense and ETA models (1.45B) at L = 32K on 1.05B tokens. ETA-32K reached lower training loss than Dense-32K (2.729 vs. 2.741) at 8.7% soft density. Keeping b=64b=64, we evaluate the PPL at 32K and 64K context lengths as well as the kernel speed-up we obtain. We observe that even with a large block size on long context evaluations the PPL metrics of ETA stay within 2.5%–6.2% of the dense model at 35.5%–51.2% budgets (<4% at 64K), corroborating the conclusions of our main paper.

MethodBudget32K C432K FW64K C464K FWSpeedup (64K)
Dense-32K (FA2)100.0%16.3415.1717.4215.101.00×
ETA-32K51.2%17.1715.6717.9515.481.20×
ETA-32K35.5%17.3515.9918.0915.631.53×
ETA-32K16.0%17.9716.8918.5716.172.10×

2. Can ETA be a post-hoc mechanism?

Addresses: Reviewer mDtt (W3, positioning question) and AtE3 (W1, Q1).

To test whether an existing dense LLM can be converted into ETA via brief continual finetuning, we attached a fresh linear threshold predictor to Dense-1.45B and trained Dense → ETA-32K for the exact same 1,000 steps. As shown below, the fine-tuned model loses its low-density behavior and its PPL immediately spikes under inference-time evaluation. This is likely because pretrained dense heads rely on attention sinks, so introducing ETA's shifted-SiLU background floor (e^0 = 1) mid-stream causes LM gradients to push τ strongly negative (τ ≈ -20).

32K Continual Run (1.05B tok)Final LossSoft Density32K C4 (51% / 35.5% / 16%)
ETA-1.45B → ETA-32K (Native)2.7298.7%17.17 / 17.35 / 17.97
Dense-1.45B → ETA-32K (Converted)2.74253.3%26.94 / 31.73 / 33.72

3. On positioning: Controlled Comparisons with Learnable & Adaptive Sparse Attention

Addresses: Reviewer prDH (W2), AtE3 (W1, Q1, Q2) and mDtt (W2).

We agree with the reviewers that we should be more careful in positioning ETA alongside other learnable and adaptive sparse attention mechanisms. Conceptually:

  • ETA is not a distillation-based method like SeerAttention-R or DSA. Those methods inherit the dense attention sinks and score geometry of the dense model, whereas ETA does not.
  • ETA does not require tuning a rigid top-k parameter like (MoBA, SAS or NSA) do. Our thresholds are adjusted dynamically, based on context.
  • ETA only requires one pass to screen tokens, as opposed to 2-stage SeerAttention-R or Twilight, while DMA prunes query-agnostically.

Ultimately, we are proposing a fundamental modification to the Transformer architecture to couple inference-time sparsity with training. ETA's design is simple, does not require more than 0.01% additional parameters, does not require additional fine-tuning and is not dependent on calibrating rigid top-k, top-p or density parameters.

Our experiments comparing ETA to the methods above show that ETA reaches a superior training loss and outperforms dense-based baselines on retrieval tasks while staying very close in PPL. Below our experiments use a 126M parameter model which we trained on 1.31B tokens, an 8x increase compared to our original submission.

ArchitectureFinal Train LossC4 PPL (50% / 35% D_u)RULER L=2K (D_u)
Dense-126M3.77844.09 (100%)3.80% (100%)
ETA-126M3.77745.81 / 46.70 (44.06 @ 85%)7.80% (30%) / 11.03% (72%)
SAS-126M3.80045.51 / 45.731.51% (50%) / 1.15% (36%)
MoBA-126M3.81446.43 / 47.780.40% (50%) / 1.03% (35%)
DMA-126M4.13960.00 / 61.860.00% (50% / 35%)

At 1.45B (L = 2K, b = 64), ETA also beats distilled SeerAttention-R on 5-task RULER (23.2% / 22.0% vs. 18.0% / 15.0%) at 2.25× lower per-head compute (22.9% vs. 51.8%) with <1.5% sink mass, and matches 2-stage Twilight (21.8%).

AuthorsOpenReview ↗

Rebuttal Manuscript Changes

We thank all the reviewers for their feedback and comments. We have uploaded a revised manuscript incorporating the discussion-period updates, including the requested additional experiments and clarifications on positioning.

  • Long-Context (32K/64K32\text{K}/64\text{K}) Extension & Speedups (§4.8, App. L): Added 1.45B1.45\text{B} continual 32K32\text{K} pre-training (32K/64K32\text{K}/64\text{K} perplexity + Dense→ETA-32K\text{Dense} \to \text{ETA-32K} conversion).
  • Modern Sparse Baselines & Retrieval (§1.1, §4.4–4.5, App. I–J): Added Related Work positioning and controlled evaluations against Twilight and SeerAttention-R.
  • Theoretical Analysis & RL Alignment (§2.2, §3.2, App. F–G): Retired the RMSNorm invariance argument, modified the per-head error bound (Prop. 1) and added PPO/GRPO importance-ratio and residual alignment results.
  • Density Metrics & Budget Calibration (§2.2, §3.2, App. M.1): Explicitly separated soft gate density (mˉ\bar{m}), per-head compute density (DheadD_{\text{head}}), and GQA-union HBM density (DunionD_{\text{union}}).
  • Full-Split 00/1515-Shot Reasoning (§4.2): Updated ARC-Easy (n=570n=570) and HellaSwag (n=10,042n=10{,}042) with 1515-shot evaluations beyond the 256256-token local window.
  • Pre-Training Ablations & Limitations (§4.1, §5, App. M): Added the 248M248\text{M} λ\lambda-sensitivity sweep, 4-way gate ablation, and Dual-Bound block-sparse training in Appendix M, and expanded §5 to discuss native vs. post-hoc applicability, training overhead, and O(L)\mathcal{O}(L) floor dilution.

We hope these clarifications have addressed the reviewers concerns and thank the reviewers again for their time and engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.