- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks a straightforward question: when decoding one sequence at a time, does activation sparsity or KV-cache sparsity yield greater memory-traffic savings? Since weight traffic remains constant but cache traffic increases with context, the two kinds of savings must intersect at some particular context length. The authors derive this point of intersection from the model's dimensions, keeping the relevant ratios, and then measure the time taken by both approaches (TEAL for the weights and a SnapKV-type method for the cache) and their combination on the Llama-3.1-8B model, varying the context length from 2K to 128K.
Critical strengths
1-I believe that the problem is interesting to study.
2-The timing protocol is relatively good in that the dense and sparse modes share a single split-K FlashAttention kernel, proceed after a real-text prefill, operate in separate processes, and each transition is accompanied by an interval.
Critical weaknesses
1-The amount of novelty is limited since both branches involve existing methods (TEAL, SnapKV); the only genuinely new aspect is setting the two savings equal. This can be regarded as a kind of measurement study.
2-The paper makes use of a weak KV baseline and fails to include proper comparisons. The KV branch employs SnapKV with a retention rate of 20 to 30 percent and is compared only against a sink-plus-recent window, a combination which its own authors state is not intended for retrieval purposes. SnapKV is an eviction-based approach used at the end of the prefill phase. Methods that compress during both prefill and decoding, and also try to compress rather than eviction, need to be studied in depth. Some of the state-of-the-art methods that are worth examining include RocketKV (ICML 2025), TidalDecode (ICLR 2026), SeerAttention-R (ICLR 2026), and BLASST (MLSys 2026). Regarding activation methods, only TEAL is implemented; although CATS and R-Sparse are mentioned, they are not tested. The Sparse Frontier (Nawrot et al., 2025) has already looked at the trade-offs associated with sparse attention along the context axis, and this work should therefore be discussed. Because it relies on outdated, limited baseline models, the findings are not valid and cannot be applied beyond the benchmarks studied.
3-The only evaluation metrics mentioned are WikiText-2 perplexity on eight different windows, a passkey test, and a home-made multi-key test. It is possible for a model to have a slightly higher perplexity yet considerably poorer performance on actual tasks, or alternatively to have a slightly lower perplexity but much better answers on real tasks. Nothing is reported regarding RULER, LongBench, multi-needle NIAH, reasoning tasks, or any other QA or reasoning task that most of the papers cover. Although the paper includes MMLU for just one branch (and in that case it already performs poorly), it gives no task scores for the combined setup that it recommends. Therefore, we have no idea what kind of accuracy loss the fastest configuration incurs on any real task.
4-The memory numbers are not provided. It appears that the entire cache remains allocated and the selection adds a contiguous copy on top of it. So, the authors need to show memory-saving numbers.
5-The batching above 1.0 does not include sparse branches. In the single experiment they do carry out (the one in Table 15, which is dense only), they use a different and slower software stack—namely, plain Hugging Face with eager attention. All of the key results, on the other hand, are based on their faster stack: gpt-fast together with torch.compile and FlashAttention split-K. For this reason, the figures in Table 15 cannot be compared with the main results since they are not comparable. The gather-to-contiguous-buffer approach is not compatible with paged KV (vLLM, SGLang), tensor parallelism is not addressed, and no quantized model is run even though the paper states that 4-bit weights bring the crossover point four times closer, which is the typical deployment scenario.
6-The models that are timed all belong to the Llama family and have 8 KV heads with a head dimension of 128; the 'two models' refer to Llama-3.1-8B and Llama-3.2-3B. Qwen3-8B is given only a traffic profile, Mistral only the projection branch, and the figure of 70B is a prediction. There is no use of MoE, no MLA, and no different GQA ratio.
Questions for authors
Please check the weakness section.
21 comments
Response to Reviewer Uyxo (2/15)
W1. Single-branch configuration choices, continued.
"the only genuinely new aspect is setting the two savings equal."
The account also chooses the configuration to run. To test whether the best sparsity pair requires enumeration, we ran new configuration-choice experiments using only single-branch quality and latency measurements.
Table R8. The configuration chosen under each perplexity budget at 32K from single-branch measurements only, against the choice of Table 4, which enumerates all 16 settings on RTX A6000. Keep ratios read projection / KV. Measured speedups are the five-block means of Table 17.
| Budget | Chosen from single branches | Choice of Table 4 | Perplexity increase, predicted / measured | RTX A6000 speedup, predicted / measured | A100 speedup, predicted / measured |
|---|---|---|---|---|---|
| 0.1 | 1.0 / 0.2 | 1.0 / 0.2 | 0.004 / 0.004 | 1.182 / 1.205 | 1.180 / 1.196 |
| 0.2 | 0.7 / 0.2 | 0.7 / 0.2 | 0.184 / 0.146 | 1.467 / 1.526 | 1.378 / 1.364 |
| 0.5 | 0.6 / 0.2 | 0.6 / 0.2 | 0.406 / 0.402 | 1.621 / 1.686 | 1.478 / 1.467 |
| 1.0 | 0.5 / 0.2 | 0.5 / 0.2 | 0.999 / 0.963 | 1.892 / 1.857 | 1.574 / 1.568 |
Under every perplexity budget of Table 4, single-branch measurements alone pick the configuration that enumerating the grid picks, from 7 timed settings instead of 16 (revised Appendix H), and in batched decoding the account picks the faster of composing and the selection alone in all nine measured cells (Table R13, under W5). To test whether single-branch measurements also find faster configurations outside the grid, we ran a new experiment that applies the rule to the full single-branch sweeps.
Table R14. The two budgets under which the rule, applied to the full single-branch sweeps, changes the choice of Table 4, on Llama-3.1-8B-Instruct, FP16, RTX A6000, at 32K, with keep ratios as projection / KV. Measured increases carry 95% intervals from the paired-window bootstrap of Table 16, and the budget applies to the point estimate. Retrieval follows Section 5.1, and speedups are means of five blocks with 95% intervals from all 3,125 resamples. In these blocks the choices of Table 4 run at 1.206x and 1.687x, against 1.205x and 1.686x in Table 17.
| Budget | Chosen | Choice of Table 4 | Perplexity increase, predicted / measured | Passkey, multi-key at 32K | Speedup, predicted / measured | Ratio to the choice of Table 4, predicted / measured |
|---|---|---|---|---|---|---|
| 0.1 | 1.0 / 0.1 | 1.0 / 0.2 | 0.059 / 0.059 [0.018, 0.099] | 40/40, 40/40 | 1.229 / 1.242 | 1.026 / 1.030 [1.029, 1.030] |
| 0.5 | 0.6 / 0.1 | 0.6 / 0.2 | 0.460 / 0.495 [0.372, 0.605] | 40/40, 40/40 | 1.768 / 1.755 | 1.037 / 1.040 [1.033, 1.052] |
Under both budgets the rule lowers the KV keep of the choice of Table 4 from 0.2 to 0.1. Both choices answer all 40 placements of both retrieval tests at 32K, stay within their budgets and run 3.0% and 4.0% faster on average, gains the account predicted from the single-branch sweeps.
The paper studies unmeasured crossings and deployment configuration choices. This matters because separate single-branch sweeps predict the crossings and choose the configuration without timing those cells, as Appendices D and H explain. Revised PDF location (revised Table 2 and Appendices E and H).
W2. Dense baseline, window control and stronger methods.
"The paper makes use of a weak KV baseline and fails to include proper comparisons."
"The KV branch employs SnapKV with a retention rate of 20 to 30 percent and is compared only against a sink-plus-recent window, a combination which its own authors state is not intended for retrieval purposes."
Answer. The sink-plus-recent window tests whether meeting a perplexity budget also preserves retrieval. It is not the baseline for the paper's decoding speedups, which compare every branch and composition with dense decoding.
Every branch and composition is timed against dense decoding, on the same split-K kernel in the main results (Figure 3). The window is a control that shows why perplexity alone cannot admit a KV configuration, as Section 1 states, "a KV window can meet a perplexity budget and still fail to retrieve information outside its retained region". At 30% keep the window raises perplexity by only 0.06, yet it answers 16 of 40 passkey placements at every length. The attention-scored selection at 20% keep answers all of them at every length up to 127K (Table 3). Section 5.1 draws its rule from this result, "We therefore admit a KV configuration only if it also passes both retrieval tests."
Response to Reviewer Uyxo (3/15)
W2. Released RocketKV code, continued.
"Methods that compress during both prefill and decoding, and also try to compress rather than eviction, need to be studied in depth. Some of the state-of-the-art methods that are worth examining include RocketKV (ICML 2025), TidalDecode (ICLR 2026), SeerAttention-R (ICLR 2026), and BLASST (MLSys 2026)."
RocketKV compresses in both phases, since it compresses the prompt cache after the prefill and selects again at every decode step. SeerAttention-R belongs among methods that select with a learned gate, and the revised Section 2 places TidalDecode and BLASST with RocketKV among the methods that select pages, tokens or blocks again at every decode step. Each of them enters the account through the fraction of cache bytes its step reads and through the cost of its selection. To test the comparison against a stronger KV method, we ran a new experiment on RocketKV and the paper's selection.
Table R12. RocketKV with its released code against the static selection of the paper, on Llama-3.1-8B-Instruct, FP16, RTX A6000, at 30% keep and at 4,096 tokens, the default budget of the released RocketKV pipeline. RocketKV runs its released prompt compression, decode-step selection and attention after the same dense prefill, inside the paper's compiled gpt-fast harness. Its cells give released / in place, where in place reads the selected entries with the FlashInfer decode kernel instead of gathering them. Speedups over dense decoding come from one three-block sweep, the kernel cost is the step time beyond the byte time, and crossings are predicted from the sweep alone and measured in four fresh-process blocks, as in Table 2. Quality follows Table R2.
| Measure | Static selection 30% | RocketKV 30% | RocketKV 4,096 tokens |
|---|---|---|---|
| En.MC accuracy change at 64K | −0.44 [−1.31, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] |
| En.QA F1 change at 64K | 0.31 [−0.50, 1.21] | −0.19 [−1.07, 0.55] | −0.80 [−2.32, 0.57] |
| Speedup at 4K | 1.028 | 0.917 / 0.924 | 0.829 / 0.876 |
| Speedup at 16K | 1.091 | 0.845 / 0.888 | 0.869 / 0.903 |
| Speedup at 32K | 1.175 | 0.812 / 0.898 | 0.950 / 1.000 |
| Speedup at 64K | 1.332 | 0.730 / 0.863 | 1.119 / 1.168 |
| Kernel cost at 64K (ms) | 0.19 | 21.67 / 14.32 | 8.15 / 6.85 |
| Crossing with 50% projection keep, predicted / measured | 68.5K / 68.7K [68.5K, 68.9K] (Table 2) | no crossing up to 96K / no crossing from 68K to 96K, both variants | 80.5K / 85.1K [84.9K, 85.2K] released, 76.4K / 73.2K [72.9K, 73.5K] in place |
In this harness the static selection is faster than RocketKV at both budgets at every measured length from 4K to 64K. At 64K it spends 0.19 ms beyond its byte time, since it scores once after prefill, while RocketKV repeats its selection, gather and attention at every step, which costs 21.67 ms beyond its bytes at 30% keep. At 4,096 tokens the speedup of RocketKV over dense decoding grows with the context and is 1.400x at 96K, or 1.474x in place. For this budget the byte crossover alone gives 56.0K, while with the kernel cost timed in the sweep the account predicts 80.5K against 85.1K measured. Near the predicted and the measured crossing the two step times differ by 0.23 ms at 80K. At 84K they differ by 0.37 ms. Both differences are at most 1.3% of the 29 ms step. On InfiniteBench at 64K every interval against dense decoding contains zero for the selection and for RocketKV.
Response to Reviewer Uyxo (1/15)
We thank the reviewer for finding the problem interesting and for valuing the timing protocol, in which dense and sparse modes share one split-K FlashAttention kernel after a real-text prefill.
Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.
Table and section numbers follow the submitted PDF unless marked revised. In the revised PDF, Tables R2, R4, R5, R6, R7 and R10 are Tables 27, 15, 21, 23, 6 and 11. Table R3 is Table 26, and Table R8 is Table 31. Tables R12, R13, R14, R15, R16, R17, R18 and R19 are Tables 16, 24, 34, 36, 28, 37, 29 and 30. The submitted Tables 7, 8, 11, 13, 14, 15, 16 and 17 are Tables 8, 9, 17, 19, 22, 25, 32 and 33.
W1. Predictive account and controlled timing.
Answer. The paper uses the established TEAL projection branch and a SnapKV-style KV selection. It does not claim new branch algorithms, but predicts unmeasured crossings, controls dense timing and chooses configurations from single-branch measurements.
Equation 5 has no fitted parameter, the kernel costs come from separate sweeps, and Section 4.3 states that "no cell of the crossing campaigns enters any prediction". In Figure 4 the "lines are predicted without any crossing cell", and the paper places all six crossings it measures, for three keep-ratio pairs, two models and two GPUs, within 4.1K tokens of them (Table 2). To our knowledge, no prior work measures projection-side and KV-cache sparsity against each other along the context axis (Section 2), let alone predicts where they cross.
The same account, unchanged, also places crossings outside the submitted campaigns. It places both crossings of LLaMA-2-7B-32K, a model with 32 KV heads, within 0.2K tokens and predicts the crossing of Qwen2.5-7B-Instruct at 95.7K against 90.4K measured (Table R5, under W6). It places the crossing of a KV method that selects again at every decode step within 2.8K tokens (Table R4). For RocketKV with its released code at its default budget of 4,096 tokens it predicts 80.5K against 85.1K measured (Table R12, under W2).
The branches are the published methods because the question the paper answers is which of two established families a deployment should run. The paper also contributes a timing protocol that corrects how KV speedups are reported, since the attention path of the dense baseline alone moves the speedup of the same KV policy at 128K from 1.29x to 6.92x (Section 4.4, Figure 5, Table 11).