- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents EntmaxKV, a support-aware sparse decoding framework for α-entmax attention. It combines query-aware page statistics, Gaussian-based threshold estimation, and sparse paged attention to reduce decoding-time KV traffic. The key insight is that recovering the full entmax support enables mathematically exact sparse attention.
Critical strengths
- Viewing sparse decoding as entmax support recovery provides a clear motivation and a useful conditional exactness guarantee.
- The paper evaluates approximation quality, downstream tasks, and GPU kernel performance, demonstrating encouraging support recall and attention-level speedups.
Critical weaknesses
-
Table 1 compares fixed-budget methods using 1,024 tokens against Gaussian-aware selection with an adaptive budget. Table 2 reports average Gaussian coverage of 36.5% at 16K and 26.9% at 32K on captured activations, suggesting potentially substantial budget differences, although these measurements come from a different evaluation setting. Matched-coverage and matched-quality comparisons are needed to isolate the benefit of the Gaussian selector. Comparisons with adaptive alternatives on the same entmax checkpoint would further strengthen the evaluation.
-
The threshold estimator relies on Gaussian score approximations, while the page-maximum estimate in Eq. (19) additionally assumes approximately independent within-page scores. The paper does not sufficiently evaluate threshold-estimation errors, page-maximum calibration, or support-page false-negative rates. Comparisons with an oracle support-page selector and sensitivity analyses of the selection hyperparameters would help establish the reliability and efficiency of the proposed method.
-
Figure 5 reports attention-only latency rather than end-to-end decoding performance. Moreover, the full-entmax baseline materializes attention scores, potentially affecting the reported speedups. End-to-end throughput, metadata maintenance overhead, and comparisons against an optimized full-entmax implementation would strengthen the systems claims. Downstream evaluations rely primarily on a single 1B entmax checkpoint. While passkey retrieval is evaluated at substantially longer lengths, RULER and language-modeling evaluations are limited to 32K, leaving broader model-scale and task generalization insufficiently explored.
Questions for authors
-
Can the authors report actual Gaussian selection coverage in Table 1 and Figure 5, along with matched-budget quality and latency comparisons?
-
How accurately does the Gaussian selector recover support-containing pages, and how does its coverage compare with an oracle page selector?
-
What end-to-end decoding speedups are achieved, including metadata overhead, relative to an optimized full-entmax baseline?
5 comments
Hello! Thanks again for your review. If you have any further questions, we would be happy to respond during the remaining 1-2 days of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.
Thank you for the authors' response, which has resolved some of my concerns. I still have the following questions:
-
Table 2 and G3 report significantly different relative output errors at 16K and 32K. Could the authors explain the reason for these differences?
-
G5 states that EntmaxKV is an efficient and quality-preserving way to serve entmax models. However, I found that the main downstream quality evaluations, such as RULER and PG19, only cover up to 32K, while G4 reports an end-to-end decoding speedup of only 0.80× at 32K.. At longer context lengths, larger speedups are reported, but corresponding downstream quality results are limited. Moreover, G4 reports a relative attention output error of 37% at 512K with the default parameters. I believe further downstream experimental evidence is needed to support the claim of quality preservation at these longer context lengths.
I will maintain my current rating.
We thank the reviewer for recognizing the support-recovery formulation and for the constructive feedback.
1. Budgets, realized coverage, and adaptive alternatives.
Gaussian selection adapts its budget to the scores; we do not claim better token selection at a matched budget (please see G1). This is an important distinction point in our work to related methods. For example, In RULER, top-k methods use 1024 tokens, whereas Gaussian averages 36.5% coverage at 16k and 26.9% at 32k. Across downstream tasks, coverage varies by layer and head between 10% and 55%, typically higher in earlier layers. In most tasks, except RULER’s CWE and FWE, top-k and Gaussian perform comparably and closely track full entmax.
In Figure 5, top-k uses 25% coverage, chosen near Gaussian’s 26.9% at 32k. This is not an exact budget match across lengths. Gaussian coverage in the random-data efficiency experiments falls from 86% at 64k to 36% at 256k and 10% at 1M. Its coverage varies with length, and it can outperform entmax top-k in latency even at higher coverage. Table 3 (Appendix E.2.2) shows top-k’s tradeoff: larger budgets retain more mass and reduce error, but increase latency. Gaussian’s contribution is adaptive budgeting and efficiency.
We also apply Twilight-style top-k followed by top-p to the same entmax checkpoint. With p=0.95, quality experiments at 16k and 32k give:
These results extend the evidence to another adaptive selection rule. A full Twilight speed comparison requires an efficient INT4 entmax implementation. In contrast, our Gaussian selector selects pages directly from statistics and an estimated threshold, avoiding sorting.
2. Gaussian estimation, support recovery, and sensitivity.
The real-activation analysis provides direct evidence of support recovery: at 32k, the Gaussian selector retains 99.84% of support tokens while reading 26.9% of the cache.
The selector is deliberately conservative at page granularity: a page is retained whenever its estimated maximum exceeds τ^−Δ, even if only one token marginally exceeds the threshold. An oracle would select exactly the pages intersecting the true support; our practical rule favors retaining potentially relevant pages without first loading their keys. This explains why coverage can exceed the minimum needed for support recovery.
We additionally quantify threshold-estimation accuracy on RULER:
One iteration reduces mean relative output error from 29.19% to 2.776% at 16k and from 28.14% to 2.786% at 32k. Two iterations reduce these errors to 0.102% and 0.078%, respectively. Thus, refinement substantially improves the initial Gaussian threshold estimate.
For the underlying page statistics, we obtain:
The larger scale-estimation errors, particularly with ALiBi, motivate the safety margin. G3 quantifies its practical effect: increasing Δ from 0.05 to 0.20 raises coverage from 31.43% to 39.05% at 16k while reducing relative error from 3.90% to 3.08%. At 32k, coverage rises from 22.35% to 31.12%, while error falls from 6.65% to 5.29%.
Our default settings are P=16, qpage=0.995, Δ=0.15, and one Halley iteration.
3. End-to-end performance and the full-entmax reference.
G4 reports end-to-end decoding measurements including page-metadata updates. We generate 256 sequential tokens after synthetic prefill and average over 10 seeds; prefill is excluded from timing:
The long-context gains therefore persist when including the rest of the model and metadata maintenance. The context dependence is expected: page statistics and threshold estimation introduce an upfront cost, amortized as context grows.
The full-entmax reference uses AdaSplash-2’s optimized entmax kernel. During decoding, it computes a score vector for one query against the KV cache; “score matrix” in Appendix E.1 refers to this vector, and we will correct the wording. G4 also reports results at head dimension
Please see G5 for our evaluation scope and the rationale for using a matched pair of 1B models trained at 32k context.