acceptodds
Under review as a conference paper at ICLR 2027

EntmaxKV: Support-Aware Decoding for Entmax Attention

Abstract

Long-context decoding is increasingly limited by KV-cache memory traffic as each generated token attends over a cache whose size grows linearly with context length. Existing sparse decoding methods reduce this cost by selecting subsets of tokens or pages, but are designed for softmax attention, whose dense tails make any truncation discard nonzero probability mass. In contrast, -entmax produces exact zeros, turning sparse decoding from dense-tail approximation into support recovery: if the selected candidates contain the entmax support, sparse decoding remains exact. While recent entmax kernels enable efficient training, they do not address the autoregressive decoding bottleneck, where dense inference still streams the full KV cache before sparsity is known. In this work, we introduce EntmaxKV, an entmax-native sparse decoding framework that exploits sparsity before KV pages are loaded. EntmaxKV combines query-aware page scoring, support-aware candidate selection, and sparse entmax attention. We analyze truncation error through the dropped probability mass , showing that output error is controlled by and vanishes when the entmax support is recovered. We further introduce a Gaussian-aware entmax selector that estimates the entmax threshold from lightweight page statistics, adapting the selected budget to the score distribution. Empirically, EntmaxKV drops less probability mass, retains more support tokens, and achieves lower output error than softmax-based sparse decoding at matched KV budgets. On long-context and language modeling benchmarks, it closely matches full-cache entmax while using a small fraction of the KV cache, achieving up to (softmax) and (entmax) speedup over full attention baselines as well as over speedup over softmax and entmax top-k at 1M context length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, efficiency, attention, kv cache

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JCe2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EntmaxKV, a support-aware sparse decoding framework for α-entmax attention. It combines query-aware page statistics, Gaussian-based threshold estimation, and sparse paged attention to reduce decoding-time KV traffic. The key insight is that recovering the full entmax support enables mathematically exact sparse attention.

Critical strengths
  1. Viewing sparse decoding as entmax support recovery provides a clear motivation and a useful conditional exactness guarantee.
  2. The paper evaluates approximation quality, downstream tasks, and GPU kernel performance, demonstrating encouraging support recall and attention-level speedups.
Critical weaknesses
  1. Table 1 compares fixed-budget methods using 1,024 tokens against Gaussian-aware selection with an adaptive budget. Table 2 reports average Gaussian coverage of 36.5% at 16K and 26.9% at 32K on captured activations, suggesting potentially substantial budget differences, although these measurements come from a different evaluation setting. Matched-coverage and matched-quality comparisons are needed to isolate the benefit of the Gaussian selector. Comparisons with adaptive alternatives on the same entmax checkpoint would further strengthen the evaluation.

  2. The threshold estimator relies on Gaussian score approximations, while the page-maximum estimate in Eq. (19) additionally assumes approximately independent within-page scores. The paper does not sufficiently evaluate threshold-estimation errors, page-maximum calibration, or support-page false-negative rates. Comparisons with an oracle support-page selector and sensitivity analyses of the selection hyperparameters would help establish the reliability and efficiency of the proposed method.

  3. Figure 5 reports attention-only latency rather than end-to-end decoding performance. Moreover, the full-entmax baseline materializes attention scores, potentially affecting the reported speedups. End-to-end throughput, metadata maintenance overhead, and comparisons against an optimized full-entmax implementation would strengthen the systems claims. Downstream evaluations rely primarily on a single 1B entmax checkpoint. While passkey retrieval is evaluated at substantially longer lengths, RULER and language-modeling evaluations are limited to 32K, leaving broader model-scale and task generalization insufficiently explored.

Questions for authors
  1. Can the authors report actual Gaussian selection coverage in Table 1 and Figure 5, along with matched-budget quality and latency comparisons?

  2. How accurately does the Gaussian selector recover support-containing pages, and how does its coverage compare with an oracle page selector?

  3. What end-to-end decoding speedups are achieved, including metadata overhead, relative to an optimized full-entmax baseline?

5 comments

We thank the reviewer for recognizing the support-recovery formulation and for the constructive feedback.

1. Budgets, realized coverage, and adaptive alternatives.

Gaussian selection adapts its budget to the scores; we do not claim better token selection at a matched budget (please see G1). This is an important distinction point in our work to related methods. For example, In RULER, top-k methods use 1024 tokens, whereas Gaussian averages 36.5% coverage at 16k and 26.9% at 32k. Across downstream tasks, coverage varies by layer and head between 10% and 55%, typically higher in earlier layers. In most tasks, except RULER’s CWE and FWE, top-k and Gaussian perform comparably and closely track full entmax.

In Figure 5, top-k uses 25% coverage, chosen near Gaussian’s 26.9% at 32k. This is not an exact budget match across lengths. Gaussian coverage in the random-data efficiency experiments falls from 86% at 64k to 36% at 256k and 10% at 1M. Its coverage varies with length, and it can outperform entmax top-k in latency even at higher coverage. Table 3 (Appendix E.2.2) shows top-k’s tradeoff: larger budgets retain more mass and reduce error, but increase latency. Gaussian’s contribution is adaptive budgeting and efficiency.

We also apply Twilight-style top-k followed by top-p to the same entmax checkpoint. With p=0.95p=0.95, quality experiments at 16k and 32k give:

CoverageEntmax relative errorSoftmax relative errorReduction
0.1250.1054±0.10440.2145±0.04862.0×
0.250.0944±0.10580.1578±0.04551.7×
0.500.0907±0.10640.1160±0.04631.3×

These results extend the evidence to another adaptive selection rule. A full Twilight speed comparison requires an efficient INT4 entmax implementation. In contrast, our Gaussian selector selects pages directly from statistics and an estimated threshold, avoiding sorting.

2. Gaussian estimation, support recovery, and sensitivity.

The real-activation analysis provides direct evidence of support recovery: at 32k, the Gaussian selector retains 99.84% of support tokens while reading 26.9% of the cache.

The selector is deliberately conservative at page granularity: a page is retained whenever its estimated maximum exceeds τ^−Δ\hat{\tau}-\Delta, even if only one token marginally exceeds the threshold. An oracle would select exactly the pages intersecting the true support; our practical rule favors retaining potentially relevant pages without first loading their keys. This explains why coverage can exceed the minimum needed for support recovery.

We additionally quantify threshold-estimation accuracy on RULER:

ContextInitial τ MAEAfter 1 Halley iterationAfter 2 iterations
16k0.31130.014990.000784
32k0.29270.014270.000464

One iteration reduces mean relative output error from 29.19% to 2.776% at 16k and from 28.14% to 2.786% at 32k. Two iterations reduce these errors to 0.102% and 0.078%, respectively. Thus, refinement substantially improves the initial Gaussian threshold estimate.

For the underlying page statistics, we obtain:

StatisticHeads16k MAE32k MAE
μ, including ALiBiAll0.00001630.0000320
μALiBi0.00003250.0000641
μNoPE1.71e-81.69e-8
σ, QK-onlyAll0.11210.1169
σ, QK-onlyALiBi0.15690.1668
σ, QK-onlyNoPE0.06740.0671
σ, full scores including ALiBiALiBi0.56920.5748

The larger scale-estimation errors, particularly with ALiBi, motivate the safety margin. G3 quantifies its practical effect: increasing Δ\Delta from 0.05 to 0.20 raises coverage from 31.43% to 39.05% at 16k while reducing relative error from 3.90% to 3.08%. At 32k, coverage rises from 22.35% to 31.12%, while error falls from 6.65% to 5.29%.

Our default settings are P=16P=16, qpage=0.995q_{\mathrm{page}}=0.995, Δ=0.15\Delta=0.15, and one Halley iteration.

3. End-to-end performance and the full-entmax reference.

G4 reports end-to-end decoding measurements including page-metadata updates. We generate 256 sequential tokens after synthetic prefill and average over 10 seeds; prefill is excluded from timing:

ContextFull entmax msGaussian msSpeedup
32k22.4327.910.80×
64k40.1927.701.45×
128k79.2457.441.38×
256k153.6258.672.62×
512k309.5163.714.86×

The long-context gains therefore persist when including the rest of the model and metadata maintenance. The context dependence is expected: page statistics and threshold estimation introduce an upfront cost, amortized as context grows.

The full-entmax reference uses AdaSplash-2’s optimized entmax kernel. During decoding, it computes a score vector for one query against the KV cache; “score matrix” in Appendix E.1 refers to this vector, and we will correct the wording. G4 also reports results at head dimension

  1. Model scale and long-context generalization.

Please see G5 for our evaluation scope and the rationale for using a matched pair of 1B models trained at 32k context.

Hello! Thanks again for your review. If you have any further questions, we would be happy to respond during the remaining 1-2 days of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.

Thank you for the authors' response, which has resolved some of my concerns. I still have the following questions:

  1. Table 2 and G3 report significantly different relative output errors at 16K and 32K. Could the authors explain the reason for these differences?

  2. G5 states that EntmaxKV is an efficient and quality-preserving way to serve entmax models. However, I found that the main downstream quality evaluations, such as RULER and PG19, only cover up to 32K, while G4 reports an end-to-end decoding speedup of only 0.80× at 32K.. At longer context lengths, larger speedups are reported, but corresponding downstream quality results are limited. Moreover, G4 reports a relative attention output error of 37% at 512K with the default parameters. I believe further downstream experimental evidence is needed to support the claim of quality preservation at these longer context lengths.

I will maintain my current rating.

Reviewer_df4JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EntmaxKV, a sparse decoding framework that selects KV-cache pages before loading their contents for entmax attention. It combines query-dependent page statistics with either fixed-budget top-k selection or a Gaussian-based estimate of the entmax support threshold. The central observation is that recovering the full support preserves the exact entmax output. Experiments with paired 1B softmax and entmax checkpoints cover approximation quality, passkey retrieval, PG19 perplexity, and RULER. The results show that sparse entmax stays closer to its full-cache baseline than sparse softmax, while attention-kernel benchmarks demonstrate substantial speedups at very long contexts.

Critical strengths
  1. Estimating the entmax support before reading KV pages is a useful direction. The Gaussian selector connects the selection rule to the attention transformation, providing a more targeted alternative to a fixed top-k budget.

  2. The fixed-budget experiments provide meaningful evidence that entmax tolerates sparse decoding well. The real-activation analysis is also encouraging, with 99.84% support-token recall at 26.9% cache coverage for 32k contexts.

Critical weaknesses
  1. The main downstream comparison does not match the actual cost of the Gaussian selector and other methods. Table 1 fixes other methods at 1,024 tokens while allowing an adaptive Gaussian budget. The separate activation analysis shows that it can select substantially more tokens, making it difficult to separate better selection from additional cache reads.

  2. The practical speedup evidence remains limited. The headline results measure attention computation only, and the full-entmax reference materializes the score matrix. Task evaluation uses one 1B model pair, while the largest speedups occur far beyond the 32k training context. The accuracy–latency benefit for a complete model therefore remains insufficiently established.

Questions for authors
  1. Can you report realized Gaussian coverage alongside each RULER result and provide accuracy–latency or accuracy–cache-read curves against entmax top-k and the softmax baselines? Matching average coverage would help clarify how much the Gaussian selection rule contributes.

  2. What end-to-end decoding gains remain when including the rest of the model and page-metadata maintenance? How does the method compare with a fused full-entmax decoding implementation, where available, rather than a reference that materializes all scores?

  3. What page size, confidence level qpageq_{\mathrm{page}}, safety margin Δ\Delta, and refinement settings are used? Please report their sensitivity and the normalization residual after one Halley iteration. This would help distinguish errors from missed support and errors from approximate threshold computation.

3 comments

We thank the reviewer for the insightful feedback.

On the budget mismatch:

Gaussian selection adapts its budget to the scores; we do not claim better token selection at a matched budget (see G1). This is an important distinction point in our work to related methods. For example, In RULER, top-k methods use 1024 tokens, whereas Gaussian averages 36.5% coverage at 16k and 26.9% at 32k. Across downstream tasks, coverage varies by layer and head between 10% and 55%, typically higher in earlier layers. In most tasks, except RULER’s CWE and FWE, top-k and Gaussian perform comparably and closely track full entmax.

In Figure 5, top-k uses 25% coverage, chosen near Gaussian’s 26.9% at 32k. This is not an exact budget match across lengths. Gaussian coverage in the random-data efficiency experiments falls from 86% at 64k to 36% at 256k and 10% at 1M. Its coverage varies with length, and it can outperform entmax top-k in latency even at higher coverage. Table 3 (Appendix E.2.2) shows top-k’s tradeoff: larger budgets retain more mass and reduce error, but increase latency. Gaussian’s contribution is adaptive budgeting and efficiency.

The practical speedup evidence remains limited.

Please take a look at G4, where we report end-to-end speedups including metadata updates: 1.45× at 64k and 4.86× at 512k.

Task evaluation uses one 1B model pair, while the largest speedups occur far beyond the 32k training context. The accuracy–latency benefit for a complete model therefore remains insufficiently established.

Yes. We agree that 1B models are not the strongest setup in the world. We also note that with longer context models we could obtain better downstream task results. However, we show that, assuming we have good base models, applying selectors such as top-k or a (carefully designed) gaussian-based variant, we can achieve substantial speed ups over the full entmax baseline while also obtaining a smaller approximation error than whatever is done for softmax. Moreover, evaluating/targeting 1B models is a common practice in architecture research (e.g., DeltaNet, Mamba, AdaSplash), and remain a relevant model size for many people, especially academic researchers under constrained compute settings.

Can you report realized Gaussian coverage alongside each RULER result and provide accuracy–latency or accuracy–cache-read curves against entmax top-k and the softmax baselines? Matching average coverage would help clarify how much the Gaussian selection rule contributes.

This value varies greatly based on layer and task, we report examples above in the general response. We will report pareto plots in the final version to further show the speed-accuracy tradeoff between tested methods.

What end-to-end decoding gains remain when including the rest of the model and page-metadata maintenance? How does the method compare with a fused full-entmax decoding implementation, where available, rather than a reference that materializes all scores?

Optimizing the full cache entmax decode is a solid research direction and one outside the scope this paper. As of now, we use the most efficient entmax publicly available implementation for comparison (from AdaSplash-2 [2]). As for end-to-end decoding, reporting this in a fair comparison with other methods requires integrating Entmax with VLLM, we report the speedup relative to the baseline above.

[2] Gonçalves et al. "Adasplash-2: Faster differentiable sparse attention." ICML 2026.

What page size, confidence level (q_{\mathrm{page}}), safety margin (\Delta), and refinement settings are used? Please report their sensitivity and the normalization residual after one Halley iteration. This would help distinguish errors from missed support and errors from approximate threshold computation.

We use page size P=16P=16, qpage=0.995q_{page}=0.995, Δ=0.15\Delta=0.15, and one Halley refinement on selected scores. Varying Δ\Delta changes coverage and relative error adaptively; it does not enforce a fixed coverage:

KV lengthΔ\DeltaCoverageRelative error
16k0.0531.43%3.90%
16k0.1033.98%3.42%
16k0.1536.53%3.34%
16k0.2039.05%3.08%
32k0.0522.35%6.65%
32k0.1025.24%6.48%
32k0.1528.17%5.63%
32k0.2031.12%5.29%

Thanks for pointing out the normalization residual, below we report tau error and the output errors after 0,1 and 2 Halley iterations on RULER:

ContextEstimated τ directlyAfter 1 iterationAfter 2 iterations
16k τ MAE0.31130.014990.000784
32k τ MAE0.29270.014270.000464
ContextItersMean relative L2 errorp95 relative L2 error
16k029.19%65.97%
16k12.776%12.03%
16k20.102%0.340%
32k028.14%58.65%
32k12.786%10.75%
32k20.078%0.303%

This shows that one Halley step sharply reduces threshold and output errors. Two steps reduce mean output error below 0.11%.

Hello! Thanks again for your review. If you have any further questions, we would be happy to respond during the remaining 1-2 days of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.

Dear Reviewer,

Thank you again for the time and effort you have put into reviewing our paper. We have addressed each of your questions and provided additional experiments to support our responses.

As the discussion period draws to a close, we would appreciate your consideration of these responses and results. We remain happy to clarify any additional questions. If your concerns have been resolved, we would be grateful if you could reflect this in your final assessment.

Kind regards,

Reviewer_dw4fOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EntmaxKV, a sparse decoding framework for models trained with α\alpha-entmax attention. Because entmax assigns exact zeros, sparse decoding is exact whenever the selected pages contain the entmax support, and the paper shows that the output error under truncation is bounded by 2Bδ2B\delta, where δ\delta is the dropped probability mass. EntmaxKV scores KV pages with Quest-style metadata and selects pages either by top-kk or by a Gaussian-aware selector that estimates the entmax threshold τ^\hat{\tau} from page-level score moments and adapts the budget to the score distribution. Using 1B-parameter softmax and entmax checkpoints trained with 32k context, the paper reports passkey, RULER (4k–32k) and PG19 results, and attention-only latency up to 1M tokens, where the Gaussian selector takes 3.63.6 ms (2.88×2.88\times faster than full-cache softmax and 5.5×5.5\times faster than full-cache entmax).

Critical strengths

S1: The core idea is clear and well motivated: with entmax, sparse decoding becomes a support-recovery problem rather than an approximation of a dense tail. The Gaussian-aware selector is a natural way to exploit this, with closed-form threshold equations for α∈{4/3,3/2,2}\alpha \in \lbrace 4/3, 3/2, 2 \rbrace. On real model activations, it retains more than 99.899.8% of support tokens with relative output error below 11% at 2727–3737% coverage.

S2: The method preserves quality well relative to full-cache entmax and is implemented efficiently. On RULER, the Gaussian variant stays close to full entmax at 16k (44.344.3 vs. 44.844.8), while softmax top-kk (Quest) drops sharply at 32k (8.788.78 vs. 34.734.7 for full softmax). The paged Triton kernels give clear speedups at very long contexts.

Critical weaknesses

W1: The main comparisons mix the effect of the attention function (and the model) with the effect of the decoding method. The softmax and entmax results come from two different checkpoints with different full-cache quality (e.g., RULER 46.446.4 vs. 56.156.1 at 4k). Support retention ρ\rho is not informative for softmax, since every token is in its support; Table 3 accordingly shows ρ\rho equal to the coverage with zero variance. These comparisons therefore mainly show that entmax attention is easier to sparsify. On the entmax model itself, the only baselines are EntmaxKV's own variants, and no other selection rule is applied to it. The method also requires an entmax-trained model, and only one 1B checkpoint with 32k context is evaluated.

W2: Budgets are not matched, and the speed and quality results cover different regimes. On RULER, top-kk methods use 10241024 tokens (about 33% of a 32k context), whereas on activations from six RULER tasks the Gaussian selector uses on average 36.536.5% coverage at 16k and 26.926.9% at 32k. In Figure 5, top-kk methods use 2525% of the context, but the coverage of the Gaussian selector is not reported. The Gaussian selector is slower than full-cache softmax at 32k (0.60.6 vs. 0.40.4 ms) and at 128k (2.02.0 vs. 1.41.4 ms), and its clear gains appear only from 256k, which is beyond the evaluated RULER lengths and around where passkey accuracy starts to drop (about 200k). Timing is attention-only with head dimension 3232, and no variance is reported for RULER; the Gaussian variant even exceeds full entmax at 32k (41.241.2 vs. 38.538.5), which suggests noticeable evaluation noise.

Questions for authors

Q1: What coverage does the Gaussian selector use in the RULER, passkey and efficiency experiments? Can you compare it with top-kk at the same average budget?

Q2: Can you compare against other selection rules applied to the same entmax model, for example a top-pp rule as in Twilight, or the deterministic box-bound selector of Proposition 4?

Q3: How does quality hold up at the lengths where the speedups appear (256k and beyond)? How do the latency results change with larger head dimensions and in end-to-end decoding?

Q4: How many examples per task are used in RULER, and what is the variance? How do you explain the Gaussian variant exceeding full entmax at 32k, and entmax top-kk exceeding full entmax on mk1 at 32k (69.869.8 vs. 59.459.4)?

Q5: Which values of the safety margin Δ\Delta and the page-level confidence level are used, and how sensitive are the coverage and accuracy to them?

2 comments

We thank the reviewer for the detailed feedback.

W1. Attention function vs decoding method (see G2). Our central question is whether entmax models are more suitable for sparse decoding. At the same 1024-token budget, entmax top-k retains 99/96/92/94% of its full-cache RULER score at 4k/8k/16k/32k, versus 97/90/64/25% for softmax top-k. We agree that ρ\rho is uninformative for softmax; it is included only for completeness. Relative output error favors entmax at every tested coverage (512k tokens, 24 heads, head dimension 32):

CoverageEntmax errorSoftmax errorError ratioEntmax msSoftmax ms
10%74.82%±7.17%302.25%±10.73%4.04×0.641±0.0030.596±0.001
12.5%68.43%±5.86%262.37%±9.78%3.83×0.714±0.0020.673±0.001
25%47.82%±7.42%162.48%±6.91%3.40×1.209±0.0011.158±0.001
50%21.20%±5.84%85.80%±2.83%4.05×2.041±0.0031.982±0.002

The ratio is softmax/entmax relative error. These results support entmax’s suitability for approximation, and we note that its threshold structure additionally enables our adaptive Gaussian selector.

Other selectors on the same model. We apply Quest-style top-k to the entmax model, providing a comparison with a widely used selection rule. We also provide Twilight-style (top-p) results below (Q2).

Scope. We acknowledge evaluating only one matched 1B entmax/softmax pair trained at 32k. This enables a controlled comparison, and 1B models are common in architecture research (e.g., DeltaNet, Mamba, AdaSplash). ASEntmax [1] also demonstrates benefits for long-context generalization. EntmaxKV provides an efficient, quality-preserving way to serve entmax models.

[1] Vasylenko et al. “Long-context generalization with sparse attention.” ICLR 2026.

W2/Q1. Budgets and coverage. Gaussian selection adapts its budget to the scores; we do not claim better token selection at a matched budget (see G1). This is an important distinction point in our work to related methods. For example, In RULER, top-k methods use 1024 tokens, whereas Gaussian averages 36.5% coverage at 16k and 26.9% at 32k. Across downstream tasks, coverage varies by layer and head between 10% and 55%, typically higher in earlier layers. In most tasks, except RULER’s CWE and FWE, top-k and Gaussian perform comparably and closely track full entmax.

In Figure 5, top-k uses 25% coverage, chosen near Gaussian’s 26.9% at 32k. This is not an exact budget match across lengths. Gaussian coverage in the random-data efficiency experiments falls from 86% at 64k to 36% at 256k and 10% at 1M. Its coverage varies with length, and it can outperform entmax top-k in latency even at higher coverage. Table 3 (Appendix E.2.2) shows top-k’s tradeoff: larger budgets retain more mass and reduce error, but increase latency. Gaussian’s contribution is adaptive budgeting and efficiency.

Q2. Twilight-style selection. Twilight combines top-k selection, K quantization, and top-p filtering. In quality experiments at 16k and 32k, Twilight-style selection also yields lower relative error with entmax, using p=0.95p=0.95:

CoverageEntmax+TwilightSoftmax+TwilightReduction
0.1250.1054±0.10440.2145±0.04862.0×
0.250.0944±0.10580.1578±0.04551.7×
0.500.0907±0.10640.1160±0.04631.3×

These are quality comparisons. Note that an efficient INT4 implementation for entmax is not yet available for a full speed comparison, and top-p requires carefully tuned kernels (partial sorting for example), which is hard to optimize for GPUs. In contrast, our Gaussian selector uses page statistics and an estimated entmax threshold to select pages directly, avoiding sorting.

Q3. Long-context quality and efficiency. Gaussian computes page statistics and solves for τ^\hat{\tau} up front. This fixed cost dominates at short lengths and is amortized at long ones. Note that reducing coverage also accelerates refinement in our sparsity-aware implementation. We acknowledge that its clear speedups begin around 256k, beyond our models’ 32k training context and the evaluated RULER lengths.

Since our models do not achieve strong downstream scores at these extreme lengths (they were length-extended to 32K tokens). We therefore use synthetic data to measure approximation error, showing greater retained mass and lower error than softmax and entmax top-k (see our general response). These results support the good approximation’s quality, which correlate well with downstream quality. Larger head dimensions and end-to-end decoding results are reported in the W1 table above, which uses 24 heads with head dimension 32.

Q4. RULER variance. We use 500 examples per task and sequence length; the maximum observed standard error is 2.05 on MK. One possible explanation for Gaussian exceeding full entmax at 32k, and top-k exceeding it on mk1, is that truncation focuses attention on relevant pages. This remains a hypothesis.

Q5. Hyperparameters and sensitivity. See G3.

Thanks for the detailed answers and new experiments. The retention results compared to each model’s own full cache address my main concern about the different checkpoints and the RULER variance and hyperparameter details are helpful too. I still think a matched budget comparison and downstream results at the lengths where the speedups show up are missing but given the other answers I will raise my score to 3.

Comments on the paper

1 comment

We thank all reviewers for taking the time to review our paper.

G1. Budgets and Gaussian selector coverage

We do not claim the Gaussian selector picks better tokens at a matched budget. Its goal is principled, adaptive budget selection, which fixed top-k cannot provide. For example, top-p adapts the budget but requires a predefined probability-mass target, effectively tuning δ\delta directly. Our claim is that entmax is more suitable for these approximations than softmax: throughout the paper, approximated entmax outperforms at matched or smaller budgets.

The Gaussian selector is conservative: it keeps a page whenever its estimated maximum exceeds τ^−Δ\hat{\tau}-\Delta, so one token marginally above threshold retains the whole page. Sparsity therefore varies greatly across layers and heads. RULER performance retention relative to each full-attention model is:

Retention on RULER4k8k16k32k
Softmax top-k (Quest)97.2%90.3%64.3%25.3%
Entmax top-k99.3%96.0%92.0%94.0%
Entmax Gaussian98.4%98.2%98.9%107.0%

Gaussian retention improves with context length, exceeding full entmax performance at 32k. Importantly, Δ\Delta controls selection conservativeness rather than the token budget. So, even removing the safety margin entirely (Δ=0\Delta=0) yields the following results on real activations:

ContextCoverageRel. error
16k28.87%4.40%
32k19.50%7.08%

These coverages remain above the 1024-token baselines (6.25% and 3.125%). Thus, decreasing the margin cannot bring Gaussian selection to those fixed budgets. Imposing such a budget would require an additional selection constraint, changing the method’s score-adaptive rule completely.

G2. Attention function and decoding method

Different attention functions are not a confounder in our evaluation: whether entmax models are more suitable for sparse decoding is an open research question, previously unaddressed, and our main goal. We first compare both attention functions using the same approximation method, Quest, finding that entmax variants perform better in absolute and relative terms. We then exploit entmax’s structure to design a selector whose budget adapts to the scores.

G3. Gaussian hyperparameters and sensitivity

We use page size P=16P=16, qpage=0.995q_{page}=0.995, Δ=0.15\Delta=0.15, and one Halley refinement on selected scores. Varying Δ\Delta changes coverage and relative error adaptively; it does not enforce a fixed coverage:

KV lengthΔ\DeltaCoverageRelative error
16k0.0531.43%3.90%
16k0.1033.98%3.42%
16k0.1536.53%3.34%
16k0.2039.05%3.08%
32k0.0522.35%6.65%
32k0.1025.24%6.48%
32k0.1528.17%5.63%
32k0.2031.12%5.29%

G4. Efficiency evidence

Full-entmax baseline. At decode time, the reference computes a score vector (one query against the KV cache) and applies AdaSplash-2’s entmax kernel, the fastest available option. We will correct “materializes the score matrix” in Appendix E.1 to “score vector”.

Larger head dimension. At d=64d=64, trends persist: Gaussian reaches 6.2× speedup over full entmax at 1M.

Where gains appear. Fixed costs for page statistics and τ^\hat{\tau} make our current Gaussian implementation slower than top-k below roughly 256k. Top-k already approximates entmax well. At longer lengths, Gaussian is fastest: 2.3 vs 4.0 ms for entmax top-k at 512k, and 3.6 vs 8.0 ms at 1M. Coverage falls with length, as intended.

Quality at long lengths. In the 512k Δ\Delta-sweep below, error falls from 0.37 to 0.16 as coverage rises from 33% to 53%, with nearly flat latency (1.87 → 2.04 ms).

Table 1 provides an end-to-end efficiency sanity check, dependent on the entmax vLLM integration. The paper’s tasks used a Torchtitan fork not fully optimized for entmax; most engines are heavily optimized for softmax. We average end-to-end decoding times over 10 seeds, generating 256 sequential tokens with metadata updates after synthetic prefill.

Table 1. End-to-end decoding

ContextEntmax msGaussian msSpeedup
32k22.4327.910.80×
64k40.1927.701.45×
128k79.2457.441.38×
256k153.6258.672.62×
512k309.5163.714.86×

Table 2. Δ\Delta-sweep at 512k

Δ\DeltaGauss p50 (ms)Relative errorCoverage (pages selected)
0.151.869 ± 0.0290.370 ± 0.06632.8% ± 1.4
0.1751.949 ± 0.0280.300 ± 0.06839.9% ± 1.5
0.202.008 ± 0.0270.223 ± 0.03947.3% ± 1.5
0.222.044 ± 0.0250.164 ± 0.03553.3% ± 1.4

G5. Scope

We evaluate matched 1B entmax/softmax models trained at 32k context. While we do acknowledge this limitation, it provides an accurate comparison and 1B models is the current standard for scientific papers investigating architecture developments (e.g., DeltaNet, Mamba, AdaSplash, etc). We also point to ASEntmax, which shows long-context generalization with entmax. EntmaxKV is an efficient and quality-preserving way to serve entmax models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.