- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper benchmarks nine KV-cache compression methods jointly on task score, memory and latency, and finds that none of them dominates across all three axes. Based on this analysis, it proposes KVRoute, a pre-inference router with a KIVI-4bit fallback that reduces severe degradation.
Critical strengths
A controlled benchmark of nine KV-cache compression methods that jointly measures task score, peak memory, decode-phase memory and latency on matched prompts.
An empirical analysis showing that no method dominates across all axes, and that switching from peak to decode-phase memory changes which methods are Pareto-optimal.
KVRoute, a pre-inference router built on tokenizer features with a KIVI-4bit fallback, which achieves a lower severe-degradation rate than any fixed compressor.
Critical weaknesses
Almost all of KVRoute's gain over FullKV comes from LongBench-v2, where the 3B model performs below chance; on the other two cohorts it is roughly on par with FullKV.
The claimed risk advantage depends mainly on a single cohort (LongBench 8B), and the paper does not compare against a simple random mix of KIVI-4bit and FullKV, which appears competitive on RULER and LongBench 8B.
All experiments use batch size 1, a 3B model and reference implementations, so the memory and latency conclusions may not hold in real serving setups, and some results (e.g., Q-Filters at 11.2) look like implementation issues.
The routing utility (Eq. 5) differs from the one used in the diagnosis (Eq. 4) and picks a different winner, and the router is evaluated only under peak memory even though the paper shows that decode memory changes the frontier.
Questions for authors
See the above weaknesses
3 comments
Response to Reviewer Uj2x (2/2)
W3: Realism (batch-1, 3B model, reference implementations). The revision adds native 7B (Mistral), 14B (Qwen2.5) and hybrid 35B-A3B (Qwen3.5) models with inputs of up to 128k tokens, and three studies beyond batch-1 (App. G.11, G.12, G.15): static batching (SnapKV doubles the feasible batch at 16k; throughput), a streaming service under load (SnapKV: p95 response time -77 to -92%, peak memory -67 to -71%), and a paged serving engine (AsyncLLMEngine) where KV-Compress shrinks the live cache by 71-72% but raises p95 response time by 13-17%. Results describe a method together with its implementation, and these serving studies show why that matters. An equal-output-length control confirms that SnapKV's speedup is not caused by shorter generations (-43.9%; App. G.9).
Q-Filters at 11.2. The 11.2 score is from the diagnosis operating point that removes 75% of the cache, whereas the routing catalog uses 25% removal; both are now labeled in Table 1. A fresh study with the unmodified upstream implementation (kvpress 0.2.4, pinned filters, Llama-3.2-3B, 64 prompts 3 repeats; App. G.13) confirms the strong budget sensitivity: -0.40 points [-1.35, 0.38] at 25% removal but -9.58 [-16.97, -3.12] at 75%. At 75% removal, Q-Filters also has the largest peak-memory saving in the study (20.7%) and sits on the peak-memory Pareto front: an extreme point of the trade-off, not a broken configuration.
W4: Eq. 4 vs. Eq. 5; routing under decode memory. Eq. 4 (task-wise min–max normalization) serves the retrospective diagnosis, where heterogeneous task scores must be comparable; Eq. 5 (native task score with cohort resource bounds) serves per-request routing within a cohort. Because the normalization alone can change which fixed method wins, without any measurement changing, we compare methods within one objective; this sensitivity is itself our central point. Re-scored under decode-phase memory, the frozen router's gain over FullKV increases on all three cohorts (App. G.6):
| Cohort | Peak memory | Decode memory |
|---|---|---|
| LongBench 8B | +0.10 | +1.53 |
| RULER 3B | -0.22 | +0.07 |
| LongBench-v2 3B | +8.59 | +13.07 |
Under decode memory the best fixed method changes from KIVI to PyramidKV/SnapKV; a source-trained FullKV/SnapKV gate for the decode objective lowers the LongBench 8B tail (31/3,200 severe), exactly the objective-dependence documented in Sec. 3.4.
We hope that the Qwen2.5-14B results, the random-mixture control and the revised positioning address your concerns. We would appreciate it if you would consider updating your score, and we are happy to discuss any further questions.
Dear Reviewer Uj2x,
Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are happy to answer any remaining questions or clarify further points.
Response to Reviewer Uj2x (1/2)
Thank you for the careful analysis. The controls you asked for made the paper noticeably sharper.
W1: Gain driven by LongBench-v2 with a below-chance 3B model. We agree that the 3B LongBench-v2 cohort cannot show quality-preserving inference. We now treat it as a resource-stress cohort and report per-cohort results next to the average (Table 2). Excluding it, KVRoute is at parity with FullKV (-0.06 points) at ≤1.4% observed severe loss. To address competence directly, we ran a model that actually solves the task on the full benchmark (Sec. 6.3, App. G.1): Qwen2.5-14B-Instruct, native 128k context, all 503 questions.
The paired quality difference is -0.20 points [-0.96, +0.41]. SnapKV saves 39.3% latency, 27.8% peak memory and 93.5% retained KV. In the evaluation utility of the paper (Eq. 5), this is a gain of +20.94 points [19.73, 22.16] over FullKV, more than twice the +8.59 of the archived 3B cohort. So on a capable model, ultra-long LongBench-v2 is exactly where compression pays off: large resource savings at unchanged accuracy.
W1 (cont.): Risk advantage relies on one cohort. The tail reduction is significant on two cohorts (versus always-KIVI: -3.09 percentage points [-3.73, -2.50] on LongBench 8B and -1.32 [-1.60, -1.05] on RULER) and is reproduced natively on unseen Mistral contexts (0/300 severe vs. 5/300 and 32/300 for always-KIVI/SnapKV; App. G.7) and on hybrid Qwen3.5 (0/300 vs. 2/300 and 4/300; App. G.14).
W2: Random KIVI/FullKV mixture (App. G.3).
Most of the gain comes from adapting how many requests are compressed to the workload (4% of RULER vs. 91% of LongBench-v2), which a global mixture cannot do. A random policy that is told KVRoute's compression rate for each cohort reaches similar tails on LongBench 8B and LongBench-v2. But such a policy cannot be deployed: that rate is exactly what KVRoute infers from the prompt before inference, and every random mixture that must use one global rate gains less at the same risk. We agree that discrimination within a workload is modest (out-of-fold AUC 0.64); the main benefit of KVRoute is choosing the compression rate per workload, plus the tail control of the risk check. Sec. 6.2 now describes the contribution in exactly these terms.