acceptodds
Under review as a conference paper at ICLR 2027

One Compressor Does Not Fit All

Abstract

The key-value (KV) cache dominates the memory cost of long-context inference, and many methods compress it. They are usually compared on one metric: how much task accuracy survives at a given compression ratio. Because compression is done for system efficiency, this comparison is underdetermined. Holding the model, GPU, prompts and execution stack fixed, we measure nine KV-cache configurations jointly on task score, GPU memory and latency across three benchmarks and two model scales. No configuration dominates, and the disagreement reaches the memory axis itself: attention-based eviction cuts decode-phase memory by 25% while raising whole-inference peak memory by 4.5%. Every LongBench task admits five to seven Pareto-optimal configurations, and a hindsight per-prompt oracle raises balanced utility from 0.668 to 0.714, with 87.7% of this gap left below task type. We then measure what a selector can capture from the prompt alone. Cross-fitted estimators show that the available request descriptions saturate at task granularity and that most per-prompt preference is tied to the executing model, while about 60% of the headroom concentrates in a detectable top decile. KVRoute is built on that structure: a detector over 46 tokenizer-derived features decides whether to route, a guarded ranker decides how, and refusal returns the best fixed compressor. Across three pre-specified transfer cohorts, it improves evaluation utility over FullKV by 2.82 points on average, with up to 38.2% less peak memory. This mean is dominated by ultra-long-context LongBench-v2, whose absolute 3B accuracy is below chance and which we therefore interpret as a resource-stress cohort rather than evidence of model competence. KVRoute limits severe degradation to 1.6% of requests in its worst cohort, below every fixed compressor (4.5-72.4%), and decides in 14-30ms on RULER and LongBench. Under this FullKV-relative utility and risk definition, dynamic selection combines compression efficiency with a low severe-loss rate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache compression, Large Language Models, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Uj2xOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper benchmarks nine KV-cache compression methods jointly on task score, memory and latency, and finds that none of them dominates across all three axes. Based on this analysis, it proposes KVRoute, a pre-inference router with a KIVI-4bit fallback that reduces severe degradation.

Critical strengths

A controlled benchmark of nine KV-cache compression methods that jointly measures task score, peak memory, decode-phase memory and latency on matched prompts.

An empirical analysis showing that no method dominates across all axes, and that switching from peak to decode-phase memory changes which methods are Pareto-optimal.

KVRoute, a pre-inference router built on tokenizer features with a KIVI-4bit fallback, which achieves a lower severe-degradation rate than any fixed compressor.

Critical weaknesses

Almost all of KVRoute's gain over FullKV comes from LongBench-v2, where the 3B model performs below chance; on the other two cohorts it is roughly on par with FullKV.

The claimed risk advantage depends mainly on a single cohort (LongBench 8B), and the paper does not compare against a simple random mix of KIVI-4bit and FullKV, which appears competitive on RULER and LongBench 8B.

All experiments use batch size 1, a 3B model and reference implementations, so the memory and latency conclusions may not hold in real serving setups, and some results (e.g., Q-Filters at 11.2) look like implementation issues.

The routing utility (Eq. 5) differs from the one used in the diagnosis (Eq. 4) and picks a different winner, and the router is evaluated only under peak memory even though the paper shows that decode memory changes the frontier.

Questions for authors

See the above weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer Uj2x (1/2)

Thank you for the careful analysis. The controls you asked for made the paper noticeably sharper.

W1: Gain driven by LongBench-v2 with a below-chance 3B model. We agree that the 3B LongBench-v2 cohort cannot show quality-preserving inference. We now treat it as a resource-stress cohort and report per-cohort results next to the average (Table 2). Excluding it, KVRoute is at parity with FullKV (-0.06 points) at ≤1.4\le 1.4% observed severe loss. To address competence directly, we ran a model that actually solves the task on the full benchmark (Sec. 6.3, App. G.1): Qwen2.5-14B-Instruct, native 128k context, all 503 questions.

CorrectAccuracy
FullKV178/50335.4% (chance: 25%)
SnapKV177/50335.2%

The paired quality difference is -0.20 points [-0.96, +0.41]. SnapKV saves 39.3% latency, 27.8% peak memory and 93.5% retained KV. In the evaluation utility of the paper (Eq. 5), this is a gain of +20.94 points [19.73, 22.16] over FullKV, more than twice the +8.59 of the archived 3B cohort. So on a capable model, ultra-long LongBench-v2 is exactly where compression pays off: large resource savings at unchanged accuracy.

W1 (cont.): Risk advantage relies on one cohort. The tail reduction is significant on two cohorts (versus always-KIVI: -3.09 percentage points [-3.73, -2.50] on LongBench 8B and -1.32 [-1.60, -1.05] on RULER) and is reproduced natively on unseen Mistral contexts (0/300 severe vs. 5/300 and 32/300 for always-KIVI/SnapKV; App. G.7) and on hybrid Qwen3.5 (0/300 vs. 2/300 and 4/300; App. G.14).

W2: Random KIVI/FullKV mixture (App. G.3).

  • A random KIVI/FullKV mixture whose KIVI share (15%) is calibrated on the source data gains +0.91 points at 0.78% worst-cohort severe loss.
  • At the same worst-cohort risk ($\approx$1.6%), the best global mixture (35% KIVI) gains +1.96 points, vs. +2.82 for KVRoute and +3.14 for the same-feature binary gate, i.e. 44-60% more utility at equal risk.

Most of the gain comes from adapting how many requests are compressed to the workload (4% of RULER vs. 91% of LongBench-v2), which a global mixture cannot do. A random policy that is told KVRoute's compression rate for each cohort reaches similar tails on LongBench 8B and LongBench-v2. But such a policy cannot be deployed: that rate is exactly what KVRoute infers from the prompt before inference, and every random mixture that must use one global rate gains less at the same risk. We agree that discrimination within a workload is modest (out-of-fold AUC 0.64); the main benefit of KVRoute is choosing the compression rate per workload, plus the tail control of the risk check. Sec. 6.2 now describes the contribution in exactly these terms.

AuthorsOpenReview ↗

Response to Reviewer Uj2x (2/2)

W3: Realism (batch-1, 3B model, reference implementations). The revision adds native 7B (Mistral), 14B (Qwen2.5) and hybrid 35B-A3B (Qwen3.5) models with inputs of up to 128k tokens, and three studies beyond batch-1 (App. G.11, G.12, G.15): static batching (SnapKV doubles the feasible batch at 16k; 2.76×2.76\times throughput), a streaming service under load (SnapKV: p95 response time -77 to -92%, peak memory -67 to -71%), and a paged serving engine (AsyncLLMEngine) where KV-Compress shrinks the live cache by 71-72% but raises p95 response time by 13-17%. Results describe a method together with its implementation, and these serving studies show why that matters. An equal-output-length control confirms that SnapKV's speedup is not caused by shorter generations (-43.9%; App. G.9).

Q-Filters at 11.2. The 11.2 score is from the diagnosis operating point that removes 75% of the cache, whereas the routing catalog uses 25% removal; both are now labeled in Table 1. A fresh study with the unmodified upstream implementation (kvpress 0.2.4, pinned filters, Llama-3.2-3B, 64 prompts ×\times 3 repeats; App. G.13) confirms the strong budget sensitivity: -0.40 points [-1.35, 0.38] at 25% removal but -9.58 [-16.97, -3.12] at 75%. At 75% removal, Q-Filters also has the largest peak-memory saving in the study (20.7%) and sits on the peak-memory Pareto front: an extreme point of the trade-off, not a broken configuration.

W4: Eq. 4 vs. Eq. 5; routing under decode memory. Eq. 4 (task-wise min–max normalization) serves the retrospective diagnosis, where heterogeneous task scores must be comparable; Eq. 5 (native task score with cohort resource bounds) serves per-request routing within a cohort. Because the normalization alone can change which fixed method wins, without any measurement changing, we compare methods within one objective; this sensitivity is itself our central point. Re-scored under decode-phase memory, the frozen router's gain over FullKV increases on all three cohorts (App. G.6):

CohortPeak memoryDecode memory
LongBench 8B+0.10+1.53
RULER 3B-0.22+0.07
LongBench-v2 3B+8.59+13.07

Under decode memory the best fixed method changes from KIVI to PyramidKV/SnapKV; a source-trained FullKV/SnapKV gate for the decode objective lowers the LongBench 8B tail (31/3,200 severe), exactly the objective-dependence documented in Sec. 3.4.

We hope that the Qwen2.5-14B results, the random-mixture control and the revised positioning address your concerns. We would appreciate it if you would consider updating your score, and we are happy to discuss any further questions.

Dear Reviewer Uj2x,

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are happy to answer any remaining questions or clarify further points.

Reviewer_5d6vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main theme of the paper revolves around KV cache compression. The main argument seems to be that it is difficult to evaluate different KV cache compression methods using a single metric because compression is tightly coupled to the system efficiency. As such, the paper seems to suggest that joint evaluation of task score, GPU memory (whole inference peak and decode-phase), and latency is a must.

The paper also seems to provide some data points with "Empirical Diagnosis": no single compressor dominates across all three axes, whole-inference peak memory is prefill-dominated, a lot of the potential gains or improvements are missed by using just one fixed setting for everything.

The paper proposes "KVRoute" which is a system that is designed to capture the instance-level efficiency headroom safely. This is a two-stage pre-inference selector: Stage 1 that uses 46 tokenizer-derived features via gradient-boosted classifier to decide whether a request has enough opportunity or risk to warrant routing; Stage 2 that employs a multi-output regressor to predict utility gains and quality drops and check safety thresholds before routing to a specific compressor.

Critical strengths
  • The paper seems to base its findings on a large set of experiments that provide interesting observations.
  • The overall effort to control tail-end in routing seems reasonable.
  • The paper makes an effort to use quite rigorous validation techniques.
Critical weaknesses
  • Generalization of the KVRoute's risk-check seems to be limited.
  • Limited experimentation that focuses on batch-1. It leaves opens on continuous batching, scheduling, and overall serving complexities.
Questions for authors
  • Given that KVRoute's default-risk check suffered a severe transfer failure on RULER (triggering FullKV on 96% of requests due to over-predicted quality loss), this may suggest that it might require a significant manual adjusting of the safety thresholds for every single new dataset or task type. How can this be automated?
  • In production environments characterized by continuous batching, variable queue lengths, and memory-bound prefill/decode interference, how do the reported latency and peak memory trade-offs hold up?
  • How do multi-axis compression trade-offs and KVRoute's dynamic routing generalize to Prefill-Decode Disaggregation, where prefill and decode tasks run on physically separate node pools?
  • Would KVRoute's per-request dynamic switching between heterogeneous compressors introduce execution stalls or hardware underutilization on non-GPU platforms, given their strict tensor shape and irregular memory access constraints?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5d6v (1/2)

Thank you for the deployment-oriented questions. We added four experiments to answer them.

W1/Q1: Risk transfer and automated calibration. On RULER the policy is conservative, not unsafe: FullKV on 96% of requests, 1 severe failure in 6,500 (0.02%). Thresholds are selected automatically, on source (LongBench) data only: grouped out-of-fold predictions, then the threshold maximizing utility subject to <1% severe loss and a mean-quality floor. For a new workload, the same procedure simply runs on a few labeled target requests. We tested this on RULER, re-selecting only the threshold over 200 group-disjoint splits and evaluating on held-out requests (App. G.4):

LabelsCompressionNet utility vs. FullKVSevere loss
0 (frozen)4.2%-0.220.02%
10049.2%+2.620.62%
1,00070.6%+4.190.83%

On LongBench 8B, where deadline fallbacks rather than thresholds drive most decisions, recalibration keeps observed severe loss near 1% (binary gate: 1.04% at 1,000 labels). In practice, the risk budget is a per-deployment setting, refreshed by occasional recalibration when the workload drifts. Without any recalibration, the frozen policy also stays safe on two fresh cohorts: on 300 unseen Mistral contexts it compresses 18% of requests with zero severe losses (always-KIVI/SnapKV: 5 and 32; App. G.7), and on BABILong with hybrid Qwen3.5 it falls back to FullKV under the shift, which costs almost nothing there because no compressor lowers peak memory by more than 0.1% (App. G.14). We do not claim a deployment guarantee from these finite test sets; for new workloads, we recommend the risk budget plus a small labeled recalibration set.

W2/Q2: Batching, queueing, prefill/decode interference. We kept the core matrix at batch size 1 on purpose, so that effects can be attributed to the compressor rather than the scheduler. Beyond it, we added three native measurements (32 real 16k/30k-token prompts, Mistral-7B in half precision):

  • Static batching (App. G.11). At 16k, SnapKV completes batch 16 while FullKV runs out of memory above 8. At batch 8, SnapKV's token throughput is 2.76×2.76\times FullKV at both lengths, with 32/42% lower peak.
  • Streaming service under load (App. G.12): random (Poisson) arrivals at three rates, a first-come-first-served queue, at most four active requests, and new requests joining the running decode batch at every step (576 requests, three repeated traces):
Rate (req./s)p95 response FullKVSnapKVp95 time to first token FullKVSnapKV
0.05130.29 s6.87 s2.41 s2.33 s
0.09186.92 s8.21 s41.25 s2.70 s
0.122120.08 s9.20 s76.58 s3.16 s

Peak memory falls by 67-71%. Under queueing, the cache savings turn into much shorter waiting times. The third study, with a paged serving engine, follows in (2/2).

AuthorsOpenReview ↗

Response to Reviewer 5d6v (2/2)

W2/Q2 (cont.): Native paged engine (App. G.15). We also ran KV-Compress inside a serving engine with paged memory and continuous batching (AsyncLLMEngine; same 32 prompts, three request rates, three repeats, 576 requests, no preemptions). The live cache shrinks by 71-72%, yet p95 response time rises by 13-17% and peak memory by about 1%. Next to the streaming SnapKV result above, this is the paper's main point showing up in serving: the same kind of cache saving can shorten or lengthen response times, depending on which method runs on which backend. Serving decisions should therefore rest on measured latency and memory, not on the compression ratio. Evaluations by cache size or accuracy alone cannot reveal this, and our routing framework uses exactly these measured costs as input.

Policy-homogeneous serving. The policy is chosen once, before prefill, so requests can be dispatched to policy-homogeneous batches or worker pools (as in our serving studies). No batch mixes cache layouts, and no running sequence switches compressor.

Q3: Prefill/decode disaggregation. Disaggregation makes our memory finding even more relevant. A prefill node is bounded by the construction-time peak, a decode node by persistent cache residency, the interconnect by the transferred cache size, and these quantities rank compressors differently. On Mistral, KIVI shrinks the stored cache by 68.6% but peak memory by only 7.7% (App. G.7); on Qwen/LongBench-v2, SnapKV shrinks them by 93.5% and 27.8%. A disaggregated deployment can therefore pick compressors per phase, e.g. methods that minimize cache size when transfer bandwidth is the bottleneck. Our utility (Eq. 5) expresses this by substituting the phase-specific cost (App. F).

Q4: Switching stalls on non-GPU hardware. Selection happens once per request before prefill, so a live cache is never converted between compressors. The switching cost is a dispatch decision plus any backend kernel-loading cost. The router decision takes a median of 14-30 ms on RULER/LongBench; on ultra-long inputs feature extraction is slower (median 0.28 s on LongBench-v2), and exceeding the deadline returns the default policy. On accelerators with static compilation (TPUs, NPUs), heterogeneous policies are best served by pre-compiled per-policy executables or dedicated pools, avoiding recompilation stalls (App. F). Compressor latency rankings are implementation-specific (our Mistral, Llama and paged stacks differ), which is exactly why the framework measures them rather than assuming them.

A fixed-output-length control (exactly 128 tokens, Qwen2.5-14B) further shows that SnapKV's speedup does not stem from shorter generations (-43.9% CUDA time; App. G.9).

We hope that the batching, serving and recalibration experiments, together with the revised text, address your deployment concerns. We would appreciate it if you would consider updating your score, and we are glad to clarify anything that remains open.

Dear Reviewer 5d6v,

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are happy to answer any remaining questions or clarify further points.

Reviewer_8DBDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies per-request selection among existing KV-cache compression methods. Through a joint evaluation of task quality, memory consumption, and latency, the authors show that no single configuration consistently dominates. They propose KVRoute, a pre-inference router trained on LongBench v1 using Llama-3.2-3B-Instruct. It combines a lightweight classifier, a multi-output regressor, and calibrated decision rules to select a compression configuration or FullKV. Its main empirical benefit is a lower severe-degradation rate than fixed compressors, although its mean utility remains below that of fixed KIVI-4bit.

Critical strengths
  1. Extensive empirical evaluation. The paper compares multiple compression families under a common setup and measures quality, memory, and latency jointly.

  2. Useful distinction between memory metrics. Separating whole-inference peak memory from decode-phase memory reveals trade-offs that cache-size measurements alone would miss.

  3. Attention to per-request failures. Reporting severe quality degradation alongside aggregate utility helps expose risks hidden by average scores.

  4. Transparent reporting of limitations. The authors acknowledge weak transfer, underperformance against the best fixed compressor, and the limited interpretation of LongBench-v2 results with a below-chance backbone.

Critical weaknesses
  1. The learned detector shows weak transfer. Its AUC falls to 0.503 on LongBench 8B and 0.574 on RULER, suggesting that the learned signal does not transfer reliably across models or workloads. Moreover, the reported source AUC of 0.960 is an in-sample result, which limits its value as evidence of generalization. These findings raise concerns about applicability to more distinct workloads, such as agentic tasks and mathematical reasoning.

  2. The incremental benefit over fixed KIVI is limited. KVRoute has lower mean utility than fixed KIVI on all three transfer cohorts. On LongBench-v2, it reduces severe degradation from 1.8% to 1.6%, while utility gains over FullKV decrease from 10.51 to 8.59 points. The paper supports a utility–risk trade-off, but provides weaker evidence that accurate per-request compressor selection delivers substantial additional value.

  3. The decisions are dominated by two configurations. KVRoute selects FullKV for approximately 96% of RULER requests and KIVI for approximately 91% of LongBench-v2 requests. This raises the question of whether most of the benefit comes from choosing between KIVI and FullKV, rather than routing among the broader compressor catalog. The sharply different behavior across these long-context benchmarks also warrants further analysis.

  4. Architectural and deployment coverage is narrow. The evaluation uses two Llama models with dense attention, and the system measurements are conducted on a single GPU type under batch-1 execution. Applicability to newer model families, hybrid attention architectures, and production serving settings remains unclear.

  5. The relationship to dynamic sparse attention needs stronger justification. As efficient inference increasingly incorporates dynamic sparse attention and architecture-specific cache management, the paper should clarify where request-level selection among these compressors remains useful, and whether it complements those approaches.

Questions for authors
  1. What is the first-stage detector’s grouped out-of-fold AUC on LongBench 3B? How much of the gap from 0.960 to the transfer results reflects source overfitting versus distribution shift?

  2. How does KVRoute compare with a simpler KIVI-versus-FullKV policy using the same features and quality guard? Does the larger compressor catalog provide a measurable benefit?

  3. What happens if the first-stage detector is removed while retaining the regressor, guards, and default action? This would help isolate the detector’s contribution.

  4. What input characteristics or prediction errors explain the near-universal selection of FullKV on RULER and KIVI on LongBench-v2? Can this be quantified by task and context length?

  5. Is the reduction from 1.8% to 1.6% severe degradation on LongBench-v2 statistically meaningful? Please provide uncertainty estimates for the utility and risk differences.

  6. How well does the method transfer to another model family or to hybrid attention? How would its candidate pool and decision rules interact with dynamic sparse attention?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8DBD (1/2)

Thank you for the precise and constructive questions. We ran every analysis you asked for.

W1/Q1: Grouped out-of-fold AUC; overfitting vs. shift. On the 3,200 LongBench-3B source requests, grouped by exact context (2,881 groups, three folds), the out-of-fold AUC of Stage 1 is 0.641 [0.609, 0.674], with average precision 0.214 against a prevalence of 0.132. The 0.960 in the submission was an in-sample statistic and is replaced by this value (Sec. 6.2, App. G.2). So overfitting on the source explains the drop from in-sample to out-of-fold, and distribution shift explains the further drop on the transfer cohorts (0.574 RULER, 0.503 LongBench 8B). Safety does not depend on Stage 1: the explicit risk check controls the tail (Q3) and keeps the severe-loss rate at or below 1.6% on every transfer cohort. On a truly new workload and architecture (BABILong reasoning on hybrid Qwen3.5; App. G.14), the frozen policy falls back to FullKV and keeps baseline accuracy with zero severe losses, while always-KIVI/SnapKV cause 2 and 4. To be clear, it falls back because of the shift, not because it predicts that compression saves nothing; on this model the fallback simply costs almost nothing, since no compressor lowers peak memory by more than 0.1%. For agentic or math workloads, the threshold can be recalibrated automatically with a few labels (reply to 5d6v; 100 labels raise RULER compression from 4.2% to 49.2% at 0.62% severe loss).

W2/Q2: KIVI-vs-FullKV control with the same features and risk check.

  • Restricting the frozen router to {KIVI, FullKV} leaves all RULER and LongBench-v2 decisions unchanged; on LongBench 8B one harmful Q-Filters choice disappears (44→4344 \to 43 severe failures).
  • An independently trained binary gate on the same 46 features (same risk check, fallback and router cost) reaches +3.14 points over FullKV vs. +2.82 for KVRoute, at the same 1.59% worst-cohort severe loss (paired difference 0.32 [0.26, 0.40]; Table 2).

The binary gate is not an external baseline: it is KVRoute's own design (same 46 features, same risk check) with two actions. That it does at least as well tells us the gain comes from the new part, request-level risk control, and not from the size of the action pool. It also makes the method simpler to deploy, so we now recommend it as the default and keep the multi-action ranker as the general formulation. Always-KIVI has higher mean utility but a much heavier tail: 143 vs. 44 severe failures on LongBench 8B (-3.09 percentage points [-3.73, -2.50]) and 87 vs. 1 on RULER (-1.32 [-1.60, -1.05]). KVRoute is built for the risk-constrained objective.

W3/Q3: Removing Stage 1 (ranker, risk checks, default, timeouts and router cost kept; App. G.2):

CohortKVRoutew/o Stage 1Severe (KVRoute / w/o)
RULER 3B-0.22-0.121 / 0
LongBench-v2 3B+8.59+8.598 / 8
LongBench 8B+0.10-0.0444 / 23

Stage 1 acts as an opportunity filter (+0.14 points on LongBench 8B) and does not drive the LongBench-v2 result. Removing the risk check, in contrast, raises severe failures from 44 to 149 (LongBench 8B) and from 1 to 85 (RULER): the risk check is the component that matters.

AuthorsOpenReview ↗

Response to Reviewer 8DBD (2/2)

W3/Q4: FullKV on RULER, KIVI on LongBench-v2, by task and length (App. G.5). RULER: 6,227/6,500 FullKV choices come from the default-risk check, whose source-calibrated threshold requires a predicted KIVI gain of +1.53 points. KIVI's actual mean loss on RULER is only 0.25 points, so the price of this caution is missed savings, not quality (1 severe failure in 6,500). Compression concentrates where predicted headroom is largest: niah_multikey_3 (36.6%), niah_multiquery (6.8%), niah_multivalue (6.0%), ≤3.2\le 3.2% elsewhere. LongBench-v2: all 457 KIVI choices are deadline fallbacks, with KIVI on 108/108 long, 215/215 medium and 134/180 short inputs; all 46 FullKV choices are short. Beyond the deadline the policy returns its default, KIVI-4bit, the compressor closest to FullKV quality (42.38 vs. 42.52). On ultra-long inputs, where memory pressure is highest, this is exactly what we want.

Q5: Significance. You are right that 1.8% →\to 1.6% on LongBench-v2 is not significant (9 vs. 8 of 503; -0.20 percentage points [-0.62, 0.00]); we no longer claim it. Where risk control matters, the reductions are large and significant (paired bootstrap over context groups, 95% CIs):

CohortUtility vs. FullKVUtility vs. KIVISevere risk vs. KIVI
LB 8B+0.10 [-0.03, 0.21]-0.45 [-0.54, -0.35]-3.09 [-3.73, -2.50]
RULER-0.22 [-0.25, -0.18]-5.92 [-6.08, -5.75]-1.32 [-1.60, -1.05]
LB-v2+8.59 [7.93, 9.22]-1.93 [-2.37, -1.55]-0.20 [-0.62, 0.00]

Significant here (CI excludes 0) are the tail reductions against always-KIVI on LongBench 8B (-3.09 percentage points, 143 vs. 44 severe failures) and RULER (-1.32, 87 vs. 1), and the utility gain over FullKV on LongBench-v2 (+8.59); the lower mean utility against KIVI is the price of this tail control.

W4/Q6: Other families, hybrid attention, serving (Sec. 6.3, App. G).

  • Two new dense families. Mistral-7B (2,700 measurements) and Qwen2.5-14B (all 503 LongBench-v2 questions, up to 128k tokens) show the same central finding: several Pareto-optimal configurations, and averages that hide per-request failures. On 300 unseen Mistral contexts the frozen router compresses 18% with 0/300 severe losses (+0.65 [0.43, 0.88]), while always-KIVI/SnapKV incur 5/300 and 32/300.
  • Hybrid attention. On Qwen3.5-35B-A3B (10 full-attention and 30 linear-attention layers; BABILong, 2,700 generations; App. G.14), the frozen policy keeps baseline accuracy with zero severe losses. This model makes our memory point very concrete: KIVI shrinks the attention cache by 68% but peak memory by only 0.1% (the recurrent state stays uncompressed), and SnapKV even enlarges the cache by 44% because it expands the key-value heads to all query heads. KVRoute falls back to FullKV under this shift, which costs almost nothing here and avoids the 2 and 4 severe losses of always applying KIVI or SnapKV.
  • Beyond batch-1 and toward production serving. You noted that the system measurements were batch-1 only. We added three serving studies (Mistral-7B, 32 real 16k/30k-token prompts):
    • Static batching (App. G.11): at 16k tokens, SnapKV runs batch size 16 where FullKV runs out of memory above 8, and it generates 2.76×2.76\times as many tokens per second at batch size 8.
    • Streaming service under load (App. G.12): with random arrivals, a waiting queue and continuous decode batching (576 requests), SnapKV cuts the 95th-percentile (p95) response time by 77-92% and peak memory by 67-71%.
    • Paged serving engine (AsyncLLMEngine; App. G.15): KV-Compress shrinks the live cache by 71-72% but raises the p95 response time by 13-17%.

So the trade-offs we measure at batch size 1 carry over to serving, and the serving benefit depends on which method runs on which backend. This is exactly why our framework measures latency and memory directly instead of inferring them from cache size. Full details are in our reply to Reviewer 5d6v (W2/Q2).

W5: Dynamic sparse attention. The two approaches complement each other. QUEST or DuoAttention decide within the forward pass which tokens or heads to attend to; KVRoute decides before prefill which cache policy, including none, a request should use, so a sparse-attention method is one more catalog action. We measured QUEST with its official evaluator on Mistral-7B (App. G.13): +2.48/+1.14 points at 1,024/2,048 tokens, with the gain concentrated in code tasks; the task-dependent preference that motivates routing. H2O (App. G.8) shows the resource side: a 90% smaller cache gave only 6% lower peak memory. Related work now covers these methods (Sec. 2).

We hope that the new analyses and the revised text address your six questions. If so, we would appreciate it if you would consider updating your score, and we are happy to answer any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: what is new, new evidence, and revised manuscript

We thank all reviewers for their careful reading and were glad to see the value of measuring task quality, latency and memory together recognized. We ran every requested control plus eleven new native experiments on three more models (Mistral-7B, Qwen2.5-14B, hybrid Qwen3.5-35B-A3B) and two serving backends. All changes in the revised PDF are marked in blue (new Sec. 6.3 and App. G.1–G.15).

What is new compared to prior work. Earlier audits compare KV-cache compressors on aggregate accuracy, memory or latency, often one axis at a time (e.g., Gao et al., 2025; KVPress). We measure all three axes together, prompt by prompt (433,857 matched measurements, nine configurations), and find what amounts to an empirical no free lunch for KV-cache efficiency:

  1. No compressor wins on all three axes. Every LongBench task has five to seven Pareto-optimal configurations, and gains on one axis are paid on another. Accuracy is the axis that separates compressors least: the top two differ significantly on only 1 of 16 tasks, while memory and latency differences are clear at p<10−20p<10^{-20}. Ranking compressors by accuracy alone uses the least informative axis.
  2. Cache size does not predict system cost. 93.5% less cache gives 27.8% lower peak memory on Qwen2.5-14B; H2O's 90.3% gives 6.3%; KIVI's 68% on hybrid Qwen3.5 gives 0.1%. In serving, SnapKV in a streaming service cuts the 95th-percentile (p95) response time by 77-92%, whereas KV-Compress in a paged engine shrinks the live cache by 71-72% but raises p95 response time by 13–17% (App. G.1, G.8, G.12, G.14, G.15).
  3. Mean accuracy hides tail failures. Compressors within one point of FullKV on average still fail severely on 4.7% (KIVI) and 13.0% (SnapKV) of Mistral requests, and H2O on 21% of Llama requests (App. G.7, G.8).

KVRoute builds on this: for each request, before inference, it decides whether and how to compress, guarded by an explicit risk check. At equal tail risk, it gains 44-60% more utility than the best random KIVI/FullKV mixture (Sec. 6.2), and on 300 unseen Mistral contexts it compresses 18% of requests with zero severe losses (+0.65 [0.43, 0.88]). On hybrid Qwen3.5, under a distribution shift, it almost always falls back to FullKV; there, compressing would have saved at most 0.1% peak memory but caused 2–4 severe losses.

Further new evidence. On all 503 LongBench-v2 questions, Qwen2.5-14B keeps its FullKV accuracy under SnapKV (178 vs. 177 correct; -0.20 [-0.96, 0.41]) at 39.3% lower latency (G.1). With static batching, SnapKV runs batch size 16 at 16k tokens where FullKV runs out of memory above 8, at 2.76×2.76\times throughput (G.11). The risk check is what makes the method safe (without it: 44→14944 \to 149 severe failures on LongBench 8B), and a two-action gate with the same features reaches +3.14 at the same risk (G.2). With 100 target labels, recalibration raises RULER compression from 4.2% to 49.2% at 0.62% severe loss (G.4). QUEST and Q-Filters show task- and budget-dependent quality (G.13).

Revisions. (i) The measurement finding now comes first. (ii) Sec. 6.3 and App. G add all new experiments. (iii) Table 2 adds the binary gate and the random mixture; Sec. 6.2 reports ablations, CIs and recalibration. (iv) The in-sample detector AUC is replaced by the out-of-fold value. (v) Related work positions H2O, Scissorhands, QUEST and DuoAttention, and contrasts our joint per-request measurement with aggregate audits such as Gao et al. (2025). (vi) App. F discusses serving, disaggregation and non-GPU hardware.

Dear Reviewer 8DBD,

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are happy to answer any remaining questions or clarify further points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.