- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ETA, a trainable sparse attention mechanism. A per-layer linear predictor maps queries to dynamic thresholds, and attention logits are gated multiplicatively through a shifted SiLU. Sub-threshold logits are pulled toward zero rather than , which forms a near-uniform background floor. The paper argues that this floor makes pruned and co-admitted tail keys negligible after RMSNorm, and that it removes attention sinks. At decode time, a fused Triton kernel screens KV blocks with a centroid, variance, and norm bound. A 1.45B model trained at 2048 tokens matches dense perplexity and outperforms Quest, H2O, and NSA. The kernel reaches up to 2.36 times speedup over FlashAttention-2, and the full model up to 2.15 times.
Critical strengths
The multiplicative SiLU gate is a simple and well motivated design that makes training-time and inference-time behavior consistent.
The comparison with additive masking shows a clear perplexity and density advantage, and the tail cancellation analysis is verified empirically. The method is backed by complete training and decoding kernels, a direct comparison with NSA under the same backbone, and measured end-to-end speedups.
Critical weaknesses
-
I'm a little curious why quality and efficiency are evaluated in different systems. The model is trained at 2048 tokens, and quality is measured only up to 8192, where the dense baseline itself falls to near zero on passkey retrieval. Speedups are reported at 64K to 512K, where model quality is not evaluated. So it seems this paper does not show ETA preserves quality at the context lengths where it delivers acceleration.
-
I think the headline speedups depend on aggressive operating points whose quality cost is not clearly reported. The 2.15 times end-to-end gain requires 16% union density, while the quality tables mostly use 50% to 70% union density. And at 51.2% union density, the kernel is slower than FlashAttention-2 for batch size 32.
-
In the ARC-Easy and HellaSwag tasks, the prompts are shorter than the 256-token windows used by the heuristic baseline methods, which means that these baseline methods may degenerate into standard dense attention mechanisms. And this may also explain why the confidence intervals are so large.
Questions for authors
-
What are the perplexity and RULER results at 16% and 35.5% union density, the operating points used for the speedup claims?
-
Can ETA be evaluated after continued pretraining at 32K or longer, so that quality and speed are measured at the same context length?
-
Which NSA implementation was used for the decode latency comparison, and is it the official optimized kernel?
-
How sensitive is the learned threshold predictor to the sparsity regularization schedule at larger scales?
2 comments
Thank you for your reply and for providing more experimental results. Regarding Q3, I didn't actually anticipate any implementation issues with the code, but the optimized kernel still supports the conclusions, so overall I still accept them. The author's response resolved most of my concerns, and I'm inclined to maintain my rating.
Response to Reviewer RnLG
We thank the reviewer for their constructive feedback! We provide detailed analyses of the points the reviewer raised below.
W1, W2, Q1 & Q2: A common ground for Quality and Speedup Evaluations
The reviewer is right in pointing out this inconsistency in our reported evaluations. We performed a continual pretraining experiment and evaluated ETA jointly for both the quality improvements and the speed-ups. In our General Response above, we provide additional details. Specifically:
32K&64KEvaluation: After32Kcontinual pretraining,ETA-32Kpreserves C4 and FineWeb PPL up to64Kacross all three speedup budgets — staying within+3.5%–+6.2%ofDense-32Kat35.5%union and+6.6%–+7.1%at64Keven at16.0%union.W2): Ourfp16grouped kernel (B = 64,64K–512K) yields1.20×–1.48×speedups even at51.2%union,1.53×–1.94×at35.5%union, and2.10×–3.31×at16.0%union.We will incorporate these evaluations into our main manuscript, emphasizing a common-ground comparison between quality and speed-ups.
W3: Full-Split
15-Shot ARC-Easy and HellaSwag (L >> 256)The reviewer is right in their assessment. The ARC-Easy and HellaSwag benchmarks were tested with short sequence length questions, prompting high density metrics. To ensure prompts substantially exceed the
256-token heuristic window and shrink confidence intervals, we re-evaluated all methods on the full validation splits of ARC-Easy (n = 570) and HellaSwag (n = 10,042) in both0-shot and15-shot (~468and~1,344avg tokens):1.45B)0-shot /15-shot0-shot /15-shot (n=10,042)15-shot HellaSwagD_uDense-1.45B42.98%/50.53%42.40%/42.71%100.0%ETA-1.45B(b=4, VC)43.33%/50.88%42.14%/42.89%52.9%(22.1%D_h)Quest/H2O43.16%/50.35%|42.98%/50.70%37.55%/42.92%|42.37%/42.84%65.9%/~60%SWA-256(Window256)42.98%/49.65%42.37%/41.25%18.9%In
0-shot, window baselines indeed degenerate toward dense (~42.4%HellaSwag); in15-shot (~1,344tokens),SWA-256loses distant exemplars and drops below0-shot (41.25%), whereas sparseETApreserves full15-shot gains (42.89%vs.42.71%Dense at52.9%union /22.1%head density). We will update our manuscript to reflect these changes.Q3: NSA Decode Implementation and Kernel Comparison
We thank the reviewer for the helpful catch! Indeed, our deployment of NSA did have some un-optimized code paths that we have promptly fixed. Because DeepSeek did not release official NSA code, our submission used the standard open-source
XunhaoLai/native-sparse-attention-tritonrepository, whose decode path ran the compressed branch in eager PyTorch with~20CPU–GPU syncs per step (3.3–6.1 ms). Replacing that bottleneck with a zero-sync Tritoncompressed_attention_decodekernel speeds up NSA's full step by2.1×–4.6×(1.15–2.23 ms, and2.1×end-to-end). Isolating pure Triton kernel time, NSA's sparse kernels land in the same0.15–0.38 msregime as ETA's single-pass kernel (ETA is1.1×–1.9×faster forB=1). Thus, with fully optimized kernels, raw compute time is comparable; ETA's structural advantage is in the simplicity of the architecture and executing one fused pass over one standard KV cache rather than orchestrating three attention branches and three separate caches.At the same time, ETA still outperforms NSA in the retrieval benchmarks we ran.
126M,1.31Btok)LD_uSingleMultiKeyMultiValVTFWEDense-126M2K100.0%2.0%12.0%5.0%0.0%0.0%3.80%ETA-126M(d50)2K30.3%14.0%16.0%9.0%0.0%0.0%7.80%NSA-126M2K82.9%8.0%6.0%2.0%0.0%0.7%3.33%Q4: Sensitivity to the Sparsity Regularization Schedule at Larger Scale
Sweeping
λ5×(0.01 → 0.025 → 0.05) on a248M(16L, d=1024) model smoothly compresses soft density (45.8% → 25.2% → 16.3%) andD_union(96.2% → 84.9% → 75.3%) while changing in-distribution1Kloss/C4 PPL by only0.003nats /0.7PPL (49.89 → 50.60), and actually improving2K–8KPPL by3.0–27.8points. Combined with our<300 mspost-hoc global union calibrator (<0.5%held-out budget error on1.45B), ETA does not require fragileλtuning to hit a desired inference budget. We will update our manuscript with these additional details.We thank again the reviewer for all their helpful comments!