acceptodds
Under review as a conference paper at ICLR 2027

Weights or KV Cache? Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

Abstract

At each step, decoding one sequence with a large language model rereads the projection weights, whose traffic is fixed, and the key-value (KV) cache, whose traffic grows with context. Activation sparsity trims the first term and KV-cache sparsity the second, yet their reported speedups are hard to compare because each depends on context length and on the dense attention kernel it is measured against. We derive a byte crossover, the context length at which the two savings are equal, together with ideal speedup bounds for each branch and for their composition, from model dimensions and keep ratios alone. We then time both branches and their composition from 2K to 128K tokens on two GPUs after a dense prefill of real text, with dense and sparse modes reading the cache through the same split-K attention kernel. The projection branch leads at short context and the KV branch at long context, with speedups that follow their byte bounds up to fixed kernel costs. Adding these costs, measured in separate sweeps, lets the byte account predict the measured crossings of three keep-ratio pairs, a second model, and a second GPU to within 4.1K tokens. Timing the dense baseline with masked instead of split-K attention inflates the apparent speedup of the same KV policy about fivefold. An attention-scored KV selection answers the same passkey and multi-key placements as dense decoding up to 127K tokens, whereas a KV window misses most of them. Under matched perplexity budgets, activation sparsity composed with this selection decodes 14 to 26% faster than the best single branch on both GPUs. Code will be made publicly available.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
activation sparsity, KV cache, KV-cache sparsity, LLM inference, decoding efficiency, memory bandwidth, long context

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UyxoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks a straightforward question: when decoding one sequence at a time, does activation sparsity or KV-cache sparsity yield greater memory-traffic savings? Since weight traffic remains constant but cache traffic increases with context, the two kinds of savings must intersect at some particular context length. The authors derive this point of intersection from the model's dimensions, keeping the relevant ratios, and then measure the time taken by both approaches (TEAL for the weights and a SnapKV-type method for the cache) and their combination on the Llama-3.1-8B model, varying the context length from 2K to 128K.

Critical strengths

1-I believe that the problem is interesting to study.

2-The timing protocol is relatively good in that the dense and sparse modes share a single split-K FlashAttention kernel, proceed after a real-text prefill, operate in separate processes, and each transition is accompanied by an interval.

Critical weaknesses

1-The amount of novelty is limited since both branches involve existing methods (TEAL, SnapKV); the only genuinely new aspect is setting the two savings equal. This can be regarded as a kind of measurement study.

2-The paper makes use of a weak KV baseline and fails to include proper comparisons. The KV branch employs SnapKV with a retention rate of 20 to 30 percent and is compared only against a sink-plus-recent window, a combination which its own authors state is not intended for retrieval purposes. SnapKV is an eviction-based approach used at the end of the prefill phase. Methods that compress during both prefill and decoding, and also try to compress rather than eviction, need to be studied in depth. Some of the state-of-the-art methods that are worth examining include RocketKV (ICML 2025), TidalDecode (ICLR 2026), SeerAttention-R (ICLR 2026), and BLASST (MLSys 2026). Regarding activation methods, only TEAL is implemented; although CATS and R-Sparse are mentioned, they are not tested. The Sparse Frontier (Nawrot et al., 2025) has already looked at the trade-offs associated with sparse attention along the context axis, and this work should therefore be discussed. Because it relies on outdated, limited baseline models, the findings are not valid and cannot be applied beyond the benchmarks studied.

3-The only evaluation metrics mentioned are WikiText-2 perplexity on eight different windows, a passkey test, and a home-made multi-key test. It is possible for a model to have a slightly higher perplexity yet considerably poorer performance on actual tasks, or alternatively to have a slightly lower perplexity but much better answers on real tasks. Nothing is reported regarding RULER, LongBench, multi-needle NIAH, reasoning tasks, or any other QA or reasoning task that most of the papers cover. Although the paper includes MMLU for just one branch (and in that case it already performs poorly), it gives no task scores for the combined setup that it recommends. Therefore, we have no idea what kind of accuracy loss the fastest configuration incurs on any real task.

4-The memory numbers are not provided. It appears that the entire cache remains allocated and the selection adds a contiguous copy on top of it. So, the authors need to show memory-saving numbers.

5-The batching above 1.0 does not include sparse branches. In the single experiment they do carry out (the one in Table 15, which is dense only), they use a different and slower software stack—namely, plain Hugging Face with eager attention. All of the key results, on the other hand, are based on their faster stack: gpt-fast together with torch.compile and FlashAttention split-K. For this reason, the figures in Table 15 cannot be compared with the main results since they are not comparable. The gather-to-contiguous-buffer approach is not compatible with paged KV (vLLM, SGLang), tensor parallelism is not addressed, and no quantized model is run even though the paper states that 4-bit weights bring the crossover point four times closer, which is the typical deployment scenario.

6-The models that are timed all belong to the Llama family and have 8 KV heads with a head dimension of 128; the 'two models' refer to Llama-3.1-8B and Llama-3.2-3B. Qwen3-8B is given only a traffic profile, Mistral only the projection branch, and the figure of 70B is a prediction. There is no use of MoE, no MLA, and no different GQA ratio.

Questions for authors

Please check the weakness section.

21 comments
AuthorsOpenReview ↗

Response to Reviewer Uyxo (1/15)

We thank the reviewer for finding the problem interesting and for valuing the timing protocol, in which dense and sparse modes share one split-K FlashAttention kernel after a real-text prefill.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Table and section numbers follow the submitted PDF unless marked revised. In the revised PDF, Tables R2, R4, R5, R6, R7 and R10 are Tables 27, 15, 21, 23, 6 and 11. Table R3 is Table 26, and Table R8 is Table 31. Tables R12, R13, R14, R15, R16, R17, R18 and R19 are Tables 16, 24, 34, 36, 28, 37, 29 and 30. The submitted Tables 7, 8, 11, 13, 14, 15, 16 and 17 are Tables 8, 9, 17, 19, 22, 25, 32 and 33.


W1. Predictive account and controlled timing.

"The amount of novelty is limited since both branches involve existing methods (TEAL, SnapKV); the only genuinely new aspect is setting the two savings equal. This can be regarded as a kind of measurement study."

Answer. The paper uses the established TEAL projection branch and a SnapKV-style KV selection. It does not claim new branch algorithms, but predicts unmeasured crossings, controls dense timing and chooses configurations from single-branch measurements.

Equation 5 has no fitted parameter, the kernel costs come from separate sweeps, and Section 4.3 states that "no cell of the crossing campaigns enters any prediction". In Figure 4 the "lines are predicted without any crossing cell", and the paper places all six crossings it measures, for three keep-ratio pairs, two models and two GPUs, within 4.1K tokens of them (Table 2). To our knowledge, no prior work measures projection-side and KV-cache sparsity against each other along the context axis (Section 2), let alone predicts where they cross.

The same account, unchanged, also places crossings outside the submitted campaigns. It places both crossings of LLaMA-2-7B-32K, a model with 32 KV heads, within 0.2K tokens and predicts the crossing of Qwen2.5-7B-Instruct at 95.7K against 90.4K measured (Table R5, under W6). It places the crossing of a KV method that selects again at every decode step within 2.8K tokens (Table R4). For RocketKV with its released code at its default budget of 4,096 tokens it predicts 80.5K against 85.1K measured (Table R12, under W2).

The branches are the published methods because the question the paper answers is which of two established families a deployment should run. The paper also contributes a timing protocol that corrects how KV speedups are reported, since the attention path of the dense baseline alone moves the speedup of the same KV policy at 128K from 1.29x to 6.92x (Section 4.4, Figure 5, Table 11).

AuthorsOpenReview ↗

Response to Reviewer Uyxo (2/15)

W1. Single-branch configuration choices, continued.

"the only genuinely new aspect is setting the two savings equal."

The account also chooses the configuration to run. To test whether the best sparsity pair requires enumeration, we ran new configuration-choice experiments using only single-branch quality and latency measurements.

Table R8. The configuration chosen under each perplexity budget at 32K from single-branch measurements only, against the choice of Table 4, which enumerates all 16 settings on RTX A6000. Keep ratios read projection / KV. Measured speedups are the five-block means of Table 17.

BudgetChosen from single branchesChoice of Table 4Perplexity increase, predicted / measuredRTX A6000 speedup, predicted / measuredA100 speedup, predicted / measured
0.11.0 / 0.21.0 / 0.20.004 / 0.0041.182 / 1.2051.180 / 1.196
0.20.7 / 0.20.7 / 0.20.184 / 0.1461.467 / 1.5261.378 / 1.364
0.50.6 / 0.20.6 / 0.20.406 / 0.4021.621 / 1.6861.478 / 1.467
1.00.5 / 0.20.5 / 0.20.999 / 0.9631.892 / 1.8571.574 / 1.568

Under every perplexity budget of Table 4, single-branch measurements alone pick the configuration that enumerating the grid picks, from 7 timed settings instead of 16 (revised Appendix H), and in batched decoding the account picks the faster of composing and the selection alone in all nine measured cells (Table R13, under W5). To test whether single-branch measurements also find faster configurations outside the grid, we ran a new experiment that applies the rule to the full single-branch sweeps.

Table R14. The two budgets under which the rule, applied to the full single-branch sweeps, changes the choice of Table 4, on Llama-3.1-8B-Instruct, FP16, RTX A6000, at 32K, with keep ratios as projection / KV. Measured increases carry 95% intervals from the paired-window bootstrap of Table 16, and the budget applies to the point estimate. Retrieval follows Section 5.1, and speedups are means of five blocks with 95% intervals from all 3,125 resamples. In these blocks the choices of Table 4 run at 1.206x and 1.687x, against 1.205x and 1.686x in Table 17.

BudgetChosenChoice of Table 4Perplexity increase, predicted / measuredPasskey, multi-key at 32KSpeedup, predicted / measuredRatio to the choice of Table 4, predicted / measured
0.11.0 / 0.11.0 / 0.20.059 / 0.059 [0.018, 0.099]40/40, 40/401.229 / 1.2421.026 / 1.030 [1.029, 1.030]
0.50.6 / 0.10.6 / 0.20.460 / 0.495 [0.372, 0.605]40/40, 40/401.768 / 1.7551.037 / 1.040 [1.033, 1.052]

Under both budgets the rule lowers the KV keep of the choice of Table 4 from 0.2 to 0.1. Both choices answer all 40 placements of both retrieval tests at 32K, stay within their budgets and run 3.0% and 4.0% faster on average, gains the account predicted from the single-branch sweeps.

The paper studies unmeasured crossings and deployment configuration choices. This matters because separate single-branch sweeps predict the crossings and choose the configuration without timing those cells, as Appendices D and H explain. Revised PDF location (revised Table 2 and Appendices E and H).


W2. Dense baseline, window control and stronger methods.

"The paper makes use of a weak KV baseline and fails to include proper comparisons."

"The KV branch employs SnapKV with a retention rate of 20 to 30 percent and is compared only against a sink-plus-recent window, a combination which its own authors state is not intended for retrieval purposes."

Answer. The sink-plus-recent window tests whether meeting a perplexity budget also preserves retrieval. It is not the baseline for the paper's decoding speedups, which compare every branch and composition with dense decoding.

Every branch and composition is timed against dense decoding, on the same split-K kernel in the main results (Figure 3). The window is a control that shows why perplexity alone cannot admit a KV configuration, as Section 1 states, "a KV window can meet a perplexity budget and still fail to retrieve information outside its retained region". At 30% keep the window raises perplexity by only 0.06, yet it answers 16 of 40 passkey placements at every length. The attention-scored selection at 20% keep answers all of them at every length up to 127K (Table 3). Section 5.1 draws its rule from this result, "We therefore admit a KV configuration only if it also passes both retrieval tests."

AuthorsOpenReview ↗

Response to Reviewer Uyxo (3/15)

W2. Released RocketKV code, continued.

"Methods that compress during both prefill and decoding, and also try to compress rather than eviction, need to be studied in depth. Some of the state-of-the-art methods that are worth examining include RocketKV (ICML 2025), TidalDecode (ICLR 2026), SeerAttention-R (ICLR 2026), and BLASST (MLSys 2026)."

RocketKV compresses in both phases, since it compresses the prompt cache after the prefill and selects again at every decode step. SeerAttention-R belongs among methods that select with a learned gate, and the revised Section 2 places TidalDecode and BLASST with RocketKV among the methods that select pages, tokens or blocks again at every decode step. Each of them enters the account through the fraction of cache bytes its step reads and through the cost of its selection. To test the comparison against a stronger KV method, we ran a new experiment on RocketKV and the paper's selection.

Table R12. RocketKV with its released code against the static selection of the paper, on Llama-3.1-8B-Instruct, FP16, RTX A6000, at 30% keep and at 4,096 tokens, the default budget of the released RocketKV pipeline. RocketKV runs its released prompt compression, decode-step selection and attention after the same dense prefill, inside the paper's compiled gpt-fast harness. Its cells give released / in place, where in place reads the selected entries with the FlashInfer decode kernel instead of gathering them. Speedups over dense decoding come from one three-block sweep, the kernel cost is the step time beyond the byte time, and crossings are predicted from the sweep alone and measured in four fresh-process blocks, as in Table 2. Quality follows Table R2.

MeasureStatic selection 30%RocketKV 30%RocketKV 4,096 tokens
En.MC accuracy change at 64K−0.44 [−1.31, 0.00]0.00 [0.00, 0.00]0.00 [0.00, 0.00]
En.QA F1 change at 64K0.31 [−0.50, 1.21]−0.19 [−1.07, 0.55]−0.80 [−2.32, 0.57]
Speedup at 4K1.0280.917 / 0.9240.829 / 0.876
Speedup at 16K1.0910.845 / 0.8880.869 / 0.903
Speedup at 32K1.1750.812 / 0.8980.950 / 1.000
Speedup at 64K1.3320.730 / 0.8631.119 / 1.168
Kernel cost at 64K (ms)0.1921.67 / 14.328.15 / 6.85
Crossing with 50% projection keep, predicted / measured68.5K / 68.7K [68.5K, 68.9K] (Table 2)no crossing up to 96K / no crossing from 68K to 96K, both variants80.5K / 85.1K [84.9K, 85.2K] released, 76.4K / 73.2K [72.9K, 73.5K] in place

In this harness the static selection is faster than RocketKV at both budgets at every measured length from 4K to 64K. At 64K it spends 0.19 ms beyond its byte time, since it scores once after prefill, while RocketKV repeats its selection, gather and attention at every step, which costs 21.67 ms beyond its bytes at 30% keep. At 4,096 tokens the speedup of RocketKV over dense decoding grows with the context and is 1.400x at 96K, or 1.474x in place. For this budget the byte crossover alone gives 56.0K, while with the kernel cost timed in the sweep the account predicts 80.5K against 85.1K measured. Near the predicted and the measured crossing the two step times differ by 0.23 ms at 80K. At 84K they differ by 0.37 ms. Both differences are at most 1.3% of the 29 ms step. On InfiniteBench at 64K every interval against dense decoding contains zero for the selection and for RocketKV.

Reviewer_8W5COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares the two per-step memory savings available to single-sequence LLM decoding along the context axis: activation sparsity, which trims the context-independent projection term, and KV-cache sparsity, which trims the context-linear cache term.

Critical strengths
  1. The illustration and the pipeline figures are clear. The method is easy to follow or implement.

  2. The problem setting is interesting and the motivation are inspiring.

Critical weaknesses
  1. The byte account maybe not independently tested, because the kernel cost is a residual of the account itself. "Quality-admissible" is not defined where it is used, and the evidence behind it is narrower than the phrase.

  2. "Quality-admissible" is not defined where it is used, and the evidence behind it is narrower than the phrase.

Questions for authors
  1. Can you validate the byte counts directly, either with hardware DRAM counters per mode or by regressing measured step time on nominal bytes removed across the five keep ratios at fixed context?

  2. Can you add at least one real long-context task at 64K or beyond for the selection and for the two compositions you recommend, and give the selection's perplexity increase at 64K and 127K in a table?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 8W5C (1/6)

We thank the reviewer for finding the figures clear, the method easy to follow, and the problem setting interesting.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

We answer each point in the reviewer's order with the measurement it asks for. For W1 and Q1 we ran the regression the reviewer names and a second regression along the context axis, and neither fits a kernel cost (Table R1). For W2 the revised PDF defines quality-admissible at first use, and its evidence now includes a long decode and a model outside the Llama family (Tables R16 and R18). For Q2 we ran two real long-context tasks at 64K and give the selection's perplexity at 64K and 127K in a table (Tables R2 and R3).

Table and section numbers follow the submitted PDF unless marked as revised. In the revised PDF, Tables R1, R16 and R18 are Tables 12, 28 and 29, and Tables R2 and R3 are Tables 27 and 26.

AuthorsOpenReview ↗

Response to Reviewer 8W5C (2/6)

W1, Q1. Independent byte-count tests.

"The byte account maybe not independently tested, because the kernel cost is a residual of the account itself."

"Can you validate the byte counts directly, either with hardware DRAM counters per mode or by regressing measured step time on nominal bytes removed across the five keep ratios at fixed context?"

Answer. The byte account counts decode reads from model dimensions and keep ratios, then uses separately timed kernel costs to predict latency. It does not fit those costs to the crossing cells used to test the predictions. We ran the regression the reviewer names, and each byte the selection removes saves the time a dense step spends reading one byte, within 3%.

To test byte counts independently, we ran a regression of step time on nominal bytes removed at fixed context, with an intercept for each block. The reference is the time a dense step spends reading one more byte. It comes from the dense cells of the context sweep of Figure 3 alone, so this test contains no kernel cost and no quantity fitted by the account. We also ran the same regression on the projection branch.

The top part of Table R1 gives a second regression along the context axis, which also fits no kernel cost and is described after the bottom part.

Table R1. Tests of the byte account on Llama-3.1-8B and Llama-3.2-3B that fit no kernel cost. Top, growth of the gap between the projection and selection step times for every 1K tokens in each crossing campaign of the submitted Table 2 and the two new H100 campaigns against the byte account s. Bottom, the regression Q1 names, step time regressed on nominal bytes removed at the fixed context marked in each row, with an intercept for each block, given as the time saved for each removed byte relative to a dense step reading one more byte in the dense context sweep of Figure 3 on the same GPU. Brackets give 95% intervals of the gap growth.

GPUMeasurementMeasuredByte account
RTX A6000gap growth, keep 0.7 / 0.3, 28K to 52K, 4 blocks0.1327 [0.1256, 0.1399] ms0.134
RTX A6000gap growth, keep 0.5 / 0.3, 56K to 84K, 4 blocks0.1326 [0.1229, 0.1423] ms0.134
A100-SXM4gap growth, keep 0.7 / 0.3, 20K to 52K, 5 blocks0.0514 [0.0474, 0.0553] ms0.059
A100-SXM4gap growth, keep 0.5 / 0.3, 36K to 80K, 5 blocks0.0549 [0.0514, 0.0585] ms0.059
A100-SXM4gap growth, keep 0.5 / 0.5, 56K to 128K, 5 blocks0.0395 [0.0369, 0.0421] ms0.042
A100-SXM4gap growth, Llama-3.2-3B, keep 0.5 / 0.3, 4K to 32K, 5 blocks0.0459 [0.0436, 0.0482] ms0.053
H100gap growth, keep 0.7 / 0.3, 2K to 28K, 4 blocks0.0271 [0.0251, 0.0291] ms0.031
H100gap growth, keep 0.5 / 0.3, 28K to 54K, 4 blocks0.0256 [0.0227, 0.0284] ms0.031
A100-SXM4selection, keep 0.7, 0.5, 0.3, 0.2, 0.1, 32K, 3 blocks0.97 of a dense byte1
RTX A6000selection, keep 0.7, 0.5, 0.3, 0.2, 0.1, 32K, 4 blocks0.97 of a dense byte1
RTX A6000selection, keep 0.7, 0.5, 0.3, 0.2, 0.1, 64K, 4 blocks1.02 of a dense byte1
RTX A6000projection, keep 0.7, 0.6, 0.5, 32K, 5 blocks1.04 of a dense byte1
A100-SXM4projection, keep 0.7, 0.6, 0.5, 32K, 3 blocks0.70 of a dense byte1

In the bottom part, each removed byte saves 0.97 of the time a dense step spends reading one byte at 32K on both GPUs, and the RTX A6000 fit at 64K gives 1.02. The selection fits have R² of 0.99 on A100 and of 0.996 and 0.999 on RTX A6000. On RTX A6000 the projection ratio is 1.04 between 70% and 50% keep, with R² of 0.99, so there both branches match the byte counts within 4%.

On A100 the projection ratio is 0.70. The difference lies in the sparse matrix-vector kernel on that GPU, which in the 32K keep-ratio campaign costs 1.30 ms at 70% keep and 2.03 ms at 50% keep. At 50% keep submitted Table 10 gives this cost between 1.88 and 2.25 ms from 2K to 128K, so it barely changes along the context axis, the axis of every crossing prediction. The byte counts depend only on model dimensions and keep ratios, so they are the same on every GPU.

AuthorsOpenReview ↗

Response to Reviewer 8W5C (3/6)

W1, Q1. Growth of the latency gap, continued.

"The byte account maybe not independently tested, because the kernel cost is a residual of the account itself."

The second regression uses each crossing campaign of Llama-3.1-8B and Llama-3.2-3B, and none of their cells enters a prediction. In each campaign we regress the gap between the projection and selection step times on context, with an intercept for each block, so every cost that does not change with context cancels. The byte account predicts that this gap grows by s for every 1K tokens (Section 4.3). Here s is the share of the added KV bytes that the selection skips, times the increase of the dense step time for every added 1K tokens in the dense context sweep of the same GPU and model.

On RTX A6000 at 30% keep that share is 70%, and the increase is 0.191 ms, the same value that the dense steps printed in revised Table 6 give. In that table a dense step takes 28.839 ms at 32K and 34.939 ms at 64K. For both RTX A6000 keep pairs the byte account gives s = 0.134, and the gap grows by 0.1327 and 0.1326 ms for every 1K tokens. Both slopes are within 1.1% of s, and s lies inside both 95% intervals. In all eight campaigns, on three GPUs and two models, the measured growth is 82% to 99% of s, a value that contains no kernel cost.

Every slope can be recomputed within 0.0001 ms from the mean differences printed in the submitted Tables 12 and 13 (revised Tables 18 and 19) and, for H100, in revised Table 20.

In the six campaigns of Llama-3.1-8B at 30% KV keep, every shortfall from s has the direction of one separately measured effect and is no larger than it. That effect is the growth of the selection's own cost with context, in the separate sweep of the same GPU, and a growing cost lowers the gap growth. This growth is taken between 16K and 64K, and all growths and shortfalls here are for every 1K tokens. It is 0.0125 ms on A100, 0.002 ms on RTX A6000 and 0.007 ms on H100, and the A100 and RTX A6000 values come from submitted Table 10. The shortfalls are 0.008 and 0.004 ms on A100, 0.001 ms on RTX A6000 and 0.004 and 0.005 ms on H100.

The submitted paper already contains two tests that a residual cannot pass by construction. First, the byte bounds of Figure 3 contain no kernel cost and no fitted parameter. Second, the kernel costs are never fitted to the cells on which the account is tested. They are timed once in separate sweeps, and Section 4.3 states that "no cell of the crossing campaigns enters any prediction". They are listed in the revised Table 2. Costs timed on A100 were also combined with only the dense step times of RTX A6000.

This combination places both Llama-3.1-8B crossings on RTX A6000 within 3.5K tokens (Section 4.3). With the separately timed costs the account places the eight A100 and RTX A6000 crossings within 4.1K tokens, and Section 4.2 reports that "The selection runs within 1% of its bound on RTX A6000 and within 6% on A100".

The revised PDF adds the two crossing campaigns on H100 80 GB HBM3, a third GPU. There the account predicts the step latency of all 56 cells, with a mean error of 0.4% in one campaign and 0.9% in the other and a largest error of 2.8% (revised Table 13). Two further measurements of the revised PDF test the byte counts without any sparse projection kernel. At 16K a dense step with eight sequences reads 1.876 times the bytes of a dense step with one sequence and takes 1.884 times as long (revised Appendix F). With int4 weights the weight bytes of a step fall from 15.01 to 3.99 GB. The 30% selection still saves the same step time as with FP16 weights, within 0.07 ms at every length from 8K to 64K. The kernel cost cannot act as a residual in any of these tests.

The paper studies whether bytes and separately timed costs predict decoding outside calibration. This matters because independent latency predictions and measurements without a sparse projection kernel support the byte account, as Appendix D, “Cost-aware crossings,” and Appendix C, “Quantized weights,” explain.

Revised PDF location (revised Appendices C and D).

Reviewer_uAcFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares activation sparsity and KV-cache sparsity for LLM decoding. It develops a byte-traffic model to derive their crossover length and ideal speedups, then incorporates measured kernel costs to predict latency crossings. It also examines attention baseline implementations, evaluates quality under different sparsity settings, and studies the combination of both approaches.

Critical strengths

S1. The controlled comparison using a common attention kernel is useful. The experiments demonstrate how the attention implementation can substantially affect the reported speedup of the same KV retention policy.

S2. Evaluating both perplexity and retrieval accuracy provides a more informative quality assessment than perplexity alone. The comparison of individual and combined methods under a common quality budget is also valuable.

Critical weaknesses

W1. The narrative and central objective are unclear. The paper moves between byte crossovers, latency crossovers, quality constraints, and composition without clearly connecting them. It is difficult to determine whether the primary goal is to explain crossover behavior, predict execution costs, or guide deployment decisions.

W2. The motivation for accurately locating the crossover is insufficient, particularly given the substantial calibration required. Composition often outperforms either branch alone, and Table 5 shows only marginal gains from calibrated dispatch over always composing both methods. Meanwhile, latency prediction requires measured dense latencies and branch-specific cost sweeps, and admissible sparsity settings still require quality evaluation. The paper does not establish what practical benefit the crossover model provides over directly profiling configurations and selecting from a lookup table, or how much profiling effort it saves.

W3. Crossover accuracy is insufficient to establish the accuracy of the performance model. Although Figures 3 and 4 provide supporting curves, the evaluation emphasizes crossover error rather than systematically reporting latency prediction errors for individual and combined configurations. Two inaccurate latency predictions can still produce the correct crossing. Comparisons against simple empirical interpolation, with clearly separated calibration and evaluation settings, would better establish the model’s predictive value.

W4. Generalization across sparsity methods remains limited. Despite experiments with multiple keep ratios, the main validation centers on TEAL and a SnapKV-style selection with one-time scoring and contiguous KV storage. It remains unclear whether the model remains useful for methods with different selection, packing, access, and update costs, particularly those performing dynamic selection during decoding.

W5. The model provides limited guidance for jointly configuring the two branches. Section 5.3 evaluates a composition grid, but does not establish how the model can identify the best pair of sparsity ratios under a quality constraint without enumerating configurations. Since composition accounts for most of the observed benefit, this joint configuration problem appears more relevant to deployment than the crossover between two standalone methods.

W6. The quality-constrained conclusions have limited coverage. The quality-matched crossover uses keep ratios derived from perplexity measurements at 32K to predict crossings at other context lengths, without adequately validating the same quality budgets near those crossings. Composition quality is evaluated primarily on one model using WikiText-2 perplexity and synthetic retrieval tasks. Broader long-context generation tasks are needed to substantiate the claimed quality–performance tradeoffs.

W7. The main performance validation is restricted to batch-size-one decoding. Section 3.3 and Appendix F discuss batching and measure activation-channel unions and dense throughput, but do not directly validate the latency model or composition benefits for concurrently decoded requests. Because batching changes weight reuse, effective activation sparsity, and the compute–memory balance, the batch-size-one results provide limited evidence for applicability to concurrent serving.

W8. The deployment evaluation relies on simulated costs rather than actual request execution. Section 5.4 estimates costs for a synthetic request mix using measured latency tables, and its gate uses both prompt and output lengths. The paper does not demonstrate how this selection works when output length is unknown or validate the resulting benefit through long-context end-to-end execution including prefill, KV selection, and decoding.

Questions for authors

see weaknesses

14 comments
AuthorsOpenReview ↗

Response to Reviewer uAcF (1/12)

We thank the reviewer for finding the controlled comparison on a common attention kernel useful and for valuing the comparison of single and composed branches under a common quality budget.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

We answer each point in the reviewer's order. Table and section numbers follow the submitted PDF unless marked revised, and the last section gives the revised table numbers. For two points the submitted paper itself already reports the result, Section 5.4 for W2 and Section 5.1 with Table 3 for W6.


W1. The deployment decision.

"It is difficult to determine whether the primary goal is to explain crossover behavior, predict execution costs, or guide deployment decisions."

Answer. The paper uses byte crossovers, latency predictions and quality tests to choose which sparse configuration a deployment should run. It does not treat crossover explanation, execution-cost prediction and deployment guidance as separate objectives.

The parts serve this decision in order. The byte crossover orders the branches from model dimensions and keep ratios alone (Section 3.3), and the kernel cost of each branch locates the latency crossing on a given GPU (Section 4.3). The quality tests decide which keep ratios are admissible (Section 5.1), and composition under a quality budget gives the configuration to run (Section 5.3). Section 6 collects these steps into one procedure.

The revised abstract begins its method sentence with "To guide which sparse configuration a deployment runs", and the revised introduction states "Together these results serve one goal, guiding which sparse configuration a deployment runs" and then this order. We would be grateful if the reviewer read the revised abstract on page 1, the last paragraph before the contributions on page 2 and the first sentences of Sections 4.2, 4.4, 5.1, 5.3 and 5.4 again with this goal in mind.

What this means. The paper studies the sparse configuration a deployment should run. This matters because byte crossovers, latency prediction, quality tests and composition together determine that choice, as Section “Implications for Deployment” explains.

Revised PDF location (revised abstract and Section 1).


W2. Profiling saved over a lookup table.

"The paper does not establish what practical benefit the crossover model provides over directly profiling configurations and selecting from a lookup table, or how much profiling effort it saves."

Answer. Table 5 compares calibrated dispatch with fixed sparse configurations for single-sequence requests. It does not claim a large gain from calibrated dispatch over always composing both methods. The account removes the compositions from the profiling, and on another GPU it needs only dense timings.

The projection branch skips weight bytes and the selection skips cache bytes, so the account adds their savings, a rule a lookup table does not have. On the grid of Table 4 it times 7 settings where a table times 16 and still picks the configuration of Table 4 under every budget (Table R8, under W5). On the full 32K keep-ratio sweeps it times 10 instead of 30 and finds a composition that neither the grid nor the sweeps contain, faster than the choice of Table 4 in each of five blocks (Table R14).

"Composition often outperforms either branch alone, and Table 5 shows only marginal gains from calibrated dispatch over always composing both methods."

The submitted paper reports and explains this small gain. Table 5 is the case of batch size one, where the bound of the composition is the largest of the three at every length (Section 3.3), so always composing gives 1.401x on A100 and 1.694x on RTX A6000, against 1.404x and 1.716x for the gate. Section 5.4 states it itself, "Always composing the two branches already reaches 1.40× and 1.69×, so nearly all of the gate's gain comes from composition."

Under batching the projection branch has less to skip, and always composing is no longer the fastest policy. There the account chooses correctly between composing and the selection alone in all nine batched cells (Table R13, under W7), and its picks finish an executed batched mix 1.062x faster than always composing 50% projection keep with the selection, the best fixed configuration (Table R15, under W8).

What this means. The paper studies which sparse configuration to run as batching changes the available savings. This matters because separately measured kernel costs identify when always composing loses to selection alone, as Appendix F, “Batched projection branch and composition,” explains.

Revised PDF location (revised Appendices D, F and H).

AuthorsOpenReview ↗

Response to Reviewer uAcF (2/12)

W3. Latency errors outside calibration.

"systematically reporting latency prediction errors for individual and combined configurations."

"Two inaccurate latency predictions can still produce the correct crossing."

Answer. Table R9 scores the predicted step time itself, so it does not depend on where two predictions cross.

It covers each branch at every context of every crossing campaign and the three compositions of Table 17, none of which enters a prediction. Calibration uses only the cells behind the predictions of Table 2. To test latency prediction errors for individual and combined configurations, we ran new prediction tests on single branches and compositions outside calibration.

Table R9. Step-latency error of the account on cells outside its calibration, as the mean and largest absolute error in percent of the measured step, with the predicted and measured crossing. The byte bound alone drops the kernel costs. For the RTX A6000 compositions, costs of keep ratios outside the context sweep come from the one-block screen of Table 16, not from the scored Table 17 campaign. In the transfer rows the account uses A100 kernel costs with the RTX A6000 dense times, and the brackets apply interpolated A100 speedups of the same calibration cells to those dense times.

GPUCells evaluatedAccount, mean / largestByte bound alone, mean / largestCrossing, predicted / measured
A100-SXM48B, 0.7 / 0.3, 18 cells1.3% / 2.4%8.0% / 13.3%28.6K / 25.1K
A100-SXM48B, 0.5 / 0.3, 24 cells1.5% / 3.9%10.4% / 19.2%51.9K / 50.4K
A100-SXM48B, 0.5 / 0.5, 20 cells1.6% / 3.1%8.7% / 17.9%72.4K / 68.3K
A100-SXM43B, 0.5 / 0.3, 16 cells1.8% / 4.2%11.7% / 29.1%13.9K / 16.5K
A100-SXM48B, three 32K compositions2.0% / 2.8%24.2% / 29.5%not a crossing
RTX A60008B, 0.7 / 0.3, 14 cells0.7% / 1.4%1.9% / 4.2%38.7K / 39.3K
RTX A60008B, 0.5 / 0.3, 16 cells4.1% / 10.1%6.4% / 12.7%68.5K / 68.7K
RTX A60008B, three 32K compositions3.5% / 3.5%11.3% / 15.0%not a crossing
H100 80 GB HBM38B, 0.7 / 0.3, 28 cells0.4% / 1.6%9.9% / 20.7%15.3K / 14.2K
H100 80 GB HBM38B, 0.5 / 0.3, 28 cells0.9% / 2.8%12.9% / 25.5%41.5K / 35.4K
RTX A6000 from A100 costs8B, 0.7 / 0.3, 14 cells1.6% / 2.2% [A100 speedups 5.4% / 8.3%]1.9% / 4.2%38.5K / 39.3K [29.0K]
RTX A6000 from A100 costs8B, 0.5 / 0.3, 16 cells3.8% / 9.0% [A100 speedups 5.9% / 13.8%]6.4% / 12.7%65.2K / 68.7K [50.3K]

Apart from one RTX A6000 pair beyond the end of its sweep, the error of the account is at most 4.2% of the measured step, against up to 29.5% for the byte bound alone, and on every set it errs less than the byte bound on average and at the largest error. So the crossings of Table 2 do not come from two inaccurate predictions that cross at the right length.

On H100 the account places the crossing of the pair 0.5 / 0.3 at 41.5K against 35.4K measured. Near this crossing the measured gap changes sign three times, between 35.4K and 39.6K. The two step times stay within 0.11 ms of each other from 36K to 42K. Switching branches at the prediction therefore makes a step at most 0.09 ms, 1.3%, slower than the faster branch at any context of the campaign.

AuthorsOpenReview ↗

Response to Reviewer uAcF (3/12)

W3. Interpolation and predictions beyond the sweep, continued.

"Comparisons against simple empirical interpolation, with clearly separated calibration and evaluation settings, would better establish the model’s predictive value."

We compared the account with linear interpolation in keep ratio through the same calibration cells on the A100 crossing cells of Llama-3.1-8B. Table 12 times the 30% selection in two campaigns at 36K to 52K that differ by 0.16 to 0.48 ms. There the mean error of the account is 0.16 to 0.25 ms. Between timed cells both stay comparable to this repeatability, with a largest error of 1.8% to 4.4% for interpolation and 1.4% to 4.2% for the account. Interpolation in keep ratio has no rule for a composition.

Past the 64K end of the RTX A6000 sweep, interpolation keeps its 64K values and finds no crossing for the pair 0.5 / 0.3. The account, whose byte ratios keep changing with the context, places it at 68.5K against 68.7K measured. The larger errors of that pair lie at 72K to 84K, and up to 68K its largest error is 2.4%. Extending the dense time log-log brings the largest errors of the two RTX A6000 rows of this pair to 3.0% and 4.2%. It leaves the predicted crossings at 68.6K and 65.3K.

"latency prediction requires measured dense latencies and branch-specific cost sweeps"

On RTX A6000 the account used kernel costs timed on A100 and only its 18 dense cells, where a table needs 54 cells for one keep pair and 72 for both. For a keep ratio that the sweep lacks, the account needs 3 cells at 32K. A table needs 24, at the eight sweep lengths in three blocks. For a projection keep ratio this is 0.04 against 0.38 GPU-hours. In the transfer rows of Table R9 the interpolated A100 speedups err by up to 13.8% of a step, against 9.0% for the account. On H100 it used only the 24 dense cells of its sweep. In all four cases its prediction lies closer to the measured H100 crossing than the source-GPU speedups applied to the H100 dense times and the source crossings used unchanged, and choosing the branch from the source-GPU costs makes a step at most 5.5% slower than the faster branch.

Outside calibration the account errs less than the byte bound on every set, and beyond the sweep and on another GPU it is closer than interpolation.

Revised PDF location (revised Section 4.3 and Appendices D and E).

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response (1/5)

We thank the reviewers for their careful reading. Reviewers Uyxo and 8W5C find the problem interesting. Reviewer Uyxo values the timing protocol, in which dense and sparse modes share one split-K FlashAttention kernel after a real-text prefill. Reviewer uAcF finds the controlled comparison on a common attention kernel useful and values the comparison of single and composed branches under a common quality budget. Reviewer 8W5C also finds the figures clear and the method easy to follow.

Answer. The submitted results stand unchanged, and the revised PDF extends the same account and timing protocol.

Several concerns are answered by results already in the submitted paper, which the replies quote with their locations. The KV window tests whether low perplexity preserves retrieval and fails the retrieval tests, while every branch and composition is timed against dense decoding rather than the window (Figure 3, Tables 3 and 4, Uyxo W2). The predicted crossings of Table 2 and Figure 4 use no crossing cell (Section 4.3, Uyxo W1, 8W5C W1). Table 3 tests the selection, alone and composed with 50% projection keep, the fastest configuration of Table 4, on both retrieval tests at 64K and 127K (uAcF W6, 8W5C W2). Section 5.4 states that composition gives nearly all of the gate's gain (uAcF W2). The paper's goal (uAcF W1) is to choose a sparse deployment configuration using established branches, and its novelty (Uyxo W1) is prediction and controlled timing, not new branch algorithms.

The other points fall into five groups, and each is answered below with new measurements that are all in the revised PDF. The table lists each group, its new tables and its main result.

GroupNew tablesMain result
(1) Stronger and dynamic KV methods (Uyxo W2 and W5, uAcF W4)Tables R4, R12, R17 and R20The static selection is faster than RocketKV, a dynamic selection and TidalDecode at every length compared up to 64K.
(2) Byte tests and prediction (8W5C W1 and Q1, uAcF W2, W3 and W5)Tables R1, R8, R9 and R14, revised Table 2Byte tests that fit no kernel cost validate the counts. The crossings on a third GPU and on a model outside the Llama family fall within 6.1K tokens of the predictions.
(3) Real tasks and quality beyond 32K (Uyxo W3, uAcF W6, 8W5C W2 and Q2)Tables R2, R3, R16, R18 and R19Every configuration that Table 4 selects stays within 2.1 points of dense decoding at 64K, and the budgets set at 32K hold at 64K.
(4) Batching and executed requests (Uyxo W5, uAcF W7 and W8)Tables R6, R11, R13 and R15The account picks correctly in all nine batched cells, and its pick is the fastest of four configurations in all six request shapes.
(5) Memory, quantization and layouts (Uyxo W4, W5 and W6)Tables R5, R7 and R10Releasing the unselected cache lowers the peak at 64K without slowing the step, and int4 weights raise the speedup of the selection.

In the revised PDF, Table R1 is Table 12, R2 is Table 27, R3 is Table 26, R4 is Table 15, R5 is Table 21, R6 is Table 23, R7 is Table 6, R8 is Table 31, R9 is Table 13 and R10 is Table 11. Table R11 is Table 35, R12 is Table 16, R13 is Table 24, R14 is Table 34, R15 is Table 36, R16 is Table 28, R17 is Table 37, R18 is Table 29 and R19 is Table 30. The two H100 crossing campaigns are Table 20. The submitted appendix tables keep their content. In the revised PDF, Tables 6 to 9 appear as Tables 7 to 10, and Tables 10 to 17 appear as Tables 14, 17, 18, 19, 22, 25, 32 and 33, in that order. Text and table cells that are new or changed in the revised PDF are in blue.

The same account answers the reviewers' questions across hardware, methods, tasks and deployment settings.

AuthorsOpenReview ↗

General Response (2/5)

Purpose and novelty.

(uAcF W1 and W2, Uyxo W1)

Answer. The paper uses TEAL projection sparsity and a SnapKV-style KV selection as established decoding branches. It does not claim new branch algorithms, but predicts their crossings and chooses which branch or composition a deployment should run.

What the paper is for (uAcF W1 and W2). The paper guides which sparse configuration a deployment runs. The byte crossover of Equation 5 orders the two branches from model dimensions and keep ratios alone (Section 3.3). The kernel cost of each branch, timed in separate sweeps, locates the latency crossing on a given GPU. Retrieval tests decide which keep ratios are admissible, because a KV window can stay within a perplexity budget and still miss most passkey placements (Section 5.1, Table 3). Composition under a quality budget then gives the configuration to run (Section 5.3, Table 4). The account makes this choice without enumeration. The revised abstract states this goal, and the revised introduction states it and how each result serves it.

What is new (Uyxo W1).

A measurement study reports the cells it timed. The submitted paper predicts latency crossings from sweeps that contain no crossing cell and then measures them (Section 4.3, Table 2, Figure 4).

The new part is the account that answers the question. Equation 5 gives the byte crossover with no fitted parameter (Section 3.3). With kernel costs timed in separate sweeps, and without using any crossing cell, the account places the six submitted crossings on A100 and RTX A6000 and the two new crossings of a Llama model with 32 KV heads within 4.1K tokens (Section 4.3, revised Table 2). On a third GPU, H100 80 GB HBM3, and on Qwen2.5-7B-Instruct, a model outside the Llama family, it places their three crossings within 6.1K tokens, and the branch chosen from the prediction is at most 1.3% slower than the faster branch at any context of these campaigns. On H100 the two step times of the pair 0.5 / 0.3 stay within 0.11 ms of each other from 36K to 42K. Switching branches at the predicted 41.5K makes a step at most 0.09 ms slower than the faster branch. It also places the crossings of two KV methods that select again at every decode step, one of them RocketKV with its released code (Tables R4 and R12). On the projection side the predictions hold with TEAL thresholds from three sources, the Llama-3-8B and Llama-2-7B thresholds of TEAL and the activations of Qwen2.5-7B-Instruct itself, and with our own batched kernel in place of the TEAL kernel (Tables R5 and R13).

From single-branch measurements alone the account also picks the configuration to run. Under every budget of Table 4 it picks the one that enumerating all 16 settings picks, and it times 7 settings instead of 16 (Table R8). Over the full keep-ratio sweeps it times 10 settings where enumeration would need 30, and it finds configurations beyond the grid that run faster than the choice of Table 4 (Table R14). On a second GPU, with kernel costs taken from the first, it needs only the 18 dense cells of one sweep, where a lookup table needs 54 to 72 (Table R9).

The timing protocol is the second new part. It times both families against a dense step on the same split-K kernel. Section 4.4 changes only the attention path of the dense baseline, for the same KV policy, and at 128K this alone moves the speedup from 1.29x to 6.92x. The released timing path of RocketKV, run unchanged, times its dense baseline with masked attention and gives RocketKV three speedups above the byte bounds of the tokens RocketKV attends to, which Section 4.4 takes as the sign of wasted execution in a dense baseline (Table R17). To our knowledge, no prior work measures projection-side and KV-cache sparsity against each other along the context axis (Section 2).

The paper studies how predictive accounting and controlled timing guide the deployment configuration. This matters because quality tests and composition connect those measurements to the configuration to run, as the Introduction and Appendix J explain. Revised PDF location (revised Tables 2 and 21, Appendices D, E, H and J).

AuthorsOpenReview ↗

General Response (3/5)

(1) Stronger KV methods and their timing.

(Uyxo W2 and W5, uAcF W4)

Answer. The static selection of the paper is faster than RocketKV, a dynamic selection and TidalDecode at every length compared up to 64K, and the account locates the crossings of methods that select again during decoding from their own sweeps. To test comparisons with stronger KV methods, we ran all three in the same harness on RTX A6000, RocketKV with its released code and TidalDecode with its released selection.

We compared the static selection at 30% keep with RocketKV at 30% keep and at 4,096 tokens, the default budget of its released evaluation pipeline (Table R12). The static selection is faster than both at every length from 4K to 64K. At 64K it runs 1.332x over dense decoding, against 0.730x for RocketKV at 30% keep and 1.119x at its default budget. At its default budget the speedup of RocketKV grows to 1.400x at 96K. When RocketKV reads its selected entries in place, the two budgets give 0.863x and 1.168x. The static selection scores once after prefill and spends 0.19 ms beyond its byte time at 64K. RocketKV, which selects again at every step, spends 6.85 to 21.67 ms. On InfiniteBench at 64K the static selection and RocketKV both stay within one multiple-choice question and 0.80 F1 points of dense decoding, with every interval containing zero. From the sweep of RocketKV alone, the account places the crossings of RocketKV at its default budget with 50% projection keep within 4.6K tokens, for reading in place and for the released gather. With the released gather the two step times stay within 0.37 ms of each other from 80K to 84K, at most 1.3% of the 29 ms step.

Run unchanged on one RTX A6000, the released timing path of RocketKV times its dense step with masked attention (Table R17). At 16K that dense step takes 1.15x as long as the split-K dense step. At 32K and 44K it takes 1.38x and 1.64x as long. Against its own baseline the path gives RocketKV 0.967x to 1.678x. Against the split-K dense step, with no code of ours in its step, RocketKV at 30% keep is slower than dense decoding at all three lengths, at 0.838x, 0.811x and 0.797x. The static selection, timed in the same blocks on the same card, is faster than the released RocketKV step at both budgets and every length.

The dynamic selection is a query-aware selection of 256-token pages at every step and layer in the style of Quest, and it reads the selected pages in place through the paged path of the same FlashAttention kernel (Table R4). From its sweep alone the account predicts its crossing with 50% projection keep at 75.6K tokens, against 72.8K [71.3K, 74.4K] measured. This error is within the largest error of the account on the A100 and RTX A6000 crossings of Table 2, 4.1K tokens. The static selection is faster at every length from 2K to 64K. At 64K it decodes 1.334x faster than dense decoding, against 1.285x for the dynamic selection.

The revised Section 2 places TidalDecode, BLASST and SeerAttention-R in the same class, methods that select again at every decode step or with a learned gate.

To compare TidalDecode with the static selection in the same harness, we ran TidalDecode's released selection on the paper's branches.

Table R20 (Uyxo W2). TidalDecode against the static selection of the paper, Llama-3.1-8B-Instruct, FP16, RTX A6000, speedup over dense decoding with the byte bound in parentheses, and the crossing with 50% projection keep, predicted from the sweep alone / measured.

MeasureStatic selection 30%TidalDecode 30%TidalDecode 4,096 tokens
Speedup at 4K1.030 (1.025)1.010 (0.993)0.949 (0.920)
Speedup at 8K1.051 (1.049)1.004 (0.987)0.967 (0.946)
Speedup at 16K1.090 (1.096)0.983 (0.978)1.014 (1.001)
Speedup at 32K1.174 (1.184)0.959 (0.962)1.109 (1.108)
Speedup at 64K1.328 (1.342)0.922 (0.939)1.294 (1.314)
Crossing, predicted / measured68.5K / 68.7K [68.5K, 68.9K] (Table 2)no crossing up to 256K / no crossing from 68K to 96K71.5K / 70.8K [70.5K, 71.1K]

The static selection is faster than TidalDecode at both budgets at every length from 4K to 64K.

The static selection stays faster over the compared lengths, and methods that select during decoding enter the unchanged account through their cache-read fraction and measured selection cost. Revised PDF location (Section 2, Appendices D and J, revised Table 19).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.