- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DuetKV, a query-agnostic KV-cache compression framework that combines sparse exact KV retention with progressively constructed latent traces under a fixed cache budget. A learned constructor synthesizes episodic traces from complete pre-eviction hidden states during prefill, while the backbone and base eviction policy remain frozen; the traces are projected into the model’s native KV space and jointly attended with retained exact states. The constructor is trained by self-distillation from a full-cache teacher without downstream query information. The method is evaluated across four long-context backbones, three eviction policies, cache budgets from 10% to 75%, and 12 prefill-intensive tasks, with additional decoding-intensive evaluations on AIME24 and MATH-500. DuetKV improves its base policies in 17 of 18 policy–budget settings, recovers near full-cache performance under aggressive compression, and provides large gains on retrieval and reasoning tasks with modest prefill overhead. Ablations comparing random traces, additional exact capacity, trace cardinality, and calibration support the claim that the gains mainly come from the learned latent trace content.
Critical strengths
DuetKV presents a clear and well-motivated way to combine exact token retention with learned latent representations. Constructing traces progressively from pre-eviction hidden states, without depending on the current eviction outcome, gives the method a distinct advantage over approaches that add synthetic states only after full prefill or derive them from the retained subset.
The empirical validation is broad and supports the practical value of the method. DuetKV is tested across multiple backbones, eviction policies, compression budgets, prefill-intensive tasks, and decoding-intensive reasoning tasks. It improves the base policies in 17 of 18 policy–budget settings, with large gains under aggressive compression and only modest prefill overhead. The random-trace and extra-capacity controls further indicate that the improvement comes from the learned trace content rather than simply reallocating cache capacity.
Critical weaknesses
The evaluation is broad in task count but still concentrated on the Qwen family and on constructors trained separately for particular backbones. Evidence for transfer across genuinely different architectures, model scales, and unseen context distributions is limited; the reported Qwen3-to-Qwen3-FP8 transfer is a relatively narrow test. This makes it difficult to determine whether DuetKV is a generally applicable compression principle or a method that requires substantial model-specific training.
The practical efficiency benefit is not fully established for end-to-end serving. The reported overhead focuses mainly on prefill latency and peak KV memory, while decoding experiments use a particular continuous-compression protocol and limited reasoning benchmarks. The constructor, segment-wise cross-attention, and repeated trace updates may have different costs under batching, longer generations, or varying numbers of reused queries, so the headline accuracy gains under compression do not yet demonstrate a consistent real-world throughput or latency advantage.
Questions for authors
-
This paper reports improvements across 70 task–budget cells and claims gains in 17 of 18 policy–budget settings. Could the authors provide confidence intervals or results across multiple random seeds, especially for the smaller gains and for the decoding benchmarks?
-
DuetKV traces inserted at earlier segments can affect later hidden states and therefore the base eviction policy’s subsequent scores. How is the comparison with the original eviction policy controlled so that improvements are attributable to latent traces rather than changes in the trajectory of the evictor itself?
-
Do the reported memory and latency measurements include the constructor’s parameters, temporary activations, cross-attention computation, and trace-update overhead? Please report end-to-end throughput and latency for different batch sizes and numbers of reused queries.
-
A separate constructor appears to be trained for each backbone, while training is performed only at and evaluated at several other budgets. How much training data and compute are required per model, and how well does the method transfer to unseen architectures, scales, and context distributions without retraining?
-
The abstract reports a 170K-token Retr.KV improvement from 31.4 to 80.2 on Qwen2.5-14B, whereas the main text reports 32.6 to 77.2 on Qwen2.5-7B. Are these results from different experimental settings? Please reconcile the numbers and specify the exact model, protocol, and baseline for each headline result.
5 comments
Author Response to Reviewer R3-GNyq (Part 2/3)
W2 + Q3: End-to-end serving efficiency
1. Memory and latency accounting.
All latency measurements include latent trace construction, segment-wise cross-attention, projection into native KV states, and trace updates whenever invoked. The paper's peak KV memory metric covers retained exact KV states, excluding episodic latent traces, constructor parameters, and temporary activations. Appendix B.3 already reports the 198 MiB constructor parameter footprint, while Table R3.3 further reports trace storage and peak GPU allocation.
Table R3.3 (new). Prefill latency (median of five runs) and memory accounting for Fast KVzip with and without DuetKV on Qwen2.5-7B-Instruct-1M (Fig. 6), at 170K tokens and .
| Metric | Fast KVzip | + DuetKV |
|---|---|---|
| Prefill latency | 8.239 s | 8.392 s (+1.9%) |
| Peak exact KV memory | 1.655 GiB | 1.655 GiB |
| Episodic latent trace storage | 0 | 2.41 MiB |
| Resident constructor parameters | 0 | 198 MiB |
| Peak allocated-memory increment | 3.862 GiB | 3.861 GiB |
Peak allocated-memory increment denotes the increase in GPU allocation relative to the prefill baseline, including temporary activations and workspace. The additional latent traces require only 2.41 MiB (approximately 0.14% of the reported exact-KV memory), while the measured peak GPU allocation remains essentially unchanged. Together with the 1.9% prefill latency overhead, these results indicate that DuetKV introduces negligible additional peak memory overhead and minimal runtime latency overhead. For Qwen2.5-14B-Instruct-1M (Sec. 4.4), trace storage is 8.25 MiB and constructor parameters occupy 485 MiB, while peak exact KV memory remains unchanged.
2.Shared-context reuse.
We evaluate three 170K-token contexts, each shared across sequential queries with 128 generated tokens per query. Episodic latent traces are constructed during shared prefill and reused without modification. End-to-end latency includes shared prefill, compression, query processing, and generation.
As shown in Table R3.4, DuetKV introduces only 0.7%–1.9% additional end-to-end latency over Fast KVzip across all 15 context-matched settings, without a consistent increase in relative overhead as the number of reused queries grows. At (N=16), DuetKV achieves a 1.3$\times$ end-to-end speedup over Full-KV, while maintaining nearly the same latency as Fast KVzip.
Table R3.4 (new). End-to-end latency, amortized per-query latency, and sequential throughput for queries sharing one context, with 128 generated tokens per query. Values are medians across three contexts, while relative overhead is computed from context-matched comparisons.
| Reused queries | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| Full-KV, end-to-end (s) | 25.26 | 29.23 | 37.14 | 52.95 | 84.58 |
| Fast KVzip, end-to-end (s) | 11.73 | 15.18 | 22.12 | 35.98 | 63.72 |
| + DuetKV, end-to-end (s) | 11.93 | 15.46 | 22.53 | 36.68 | 64.96 |
| + DuetKV, amortized (s/query) | 11.93 | 7.73 | 5.63 | 4.59 | 4.06 |
| + DuetKV, throughput (queries/s) | 0.084 | 0.129 | 0.178 | 0.218 | 0.246 |
3.Generation-length sensitivity.
We examine fixed traces in the prefill-intensive regime and periodic trace construction in the decoding-intensive regime. In the latter, Fast KVzip compresses the cache every 128 tokens, while DuetKV constructs traces every 1,024 tokens. We use greedy decoding from a synthetic 190-token prompt, matching the median AIME24 prompt length, with a 32,768-token generation budget and a 4,096-token KV budget.
Table R3.5 (new). Generation-length sensitivity. Prefill-intensive results report context-matched per-token latency changes for three 160K-token contexts on Qwen2.5-7B-Instruct-1M (). Decoding-intensive results report cumulative latency changes on Qwen3-8B.
| Prefill-intensive, generated tokens | 128 | 512 | 1024 | 2048 |
|---|---|---|---|---|
| Context A, per-token change | −1.6% | +0.3% | +1.6% | +1.6% |
| Context B, per-token change | +4.0% | +1.6% | +1.6% | +1.1% |
| Context C, per-token change | −4.8% | −0.5% | +2.0% | +2.3% |
| Decoding-intensive, generated tokens | 512 | 2048 | 8192 | 32768 |
| Cumulative latency change | +1.9% | +2.2% | +1.3% | +0.9% |
The results show modest per-token latency changes in the prefill-intensive regime and no sustained increase in cumulative overhead during decoding-intensive generation, with only +0.9% overhead at 32,768 tokens.
4. Batching.
Our current implementation inherits the batch-size-one limitation of the underlying eviction framework, whose variable-length attention layout is indexed by KV head rather than batch. Therefore, the reported results measure single-stream latency and sequential throughput rather than concurrent serving performance.
These additional efficiency results are reported in Appendix B.3 of the revised manuscript.
Author Response to Reviewer R3-GNyq (Part 3/3)
Q1: Confidence intervals and random seeds
We have added paired bootstrap 95% confidence intervals for all 70 prefill task and budget configurations, with the full results reported in Appendix C.7 of the revised manuscript. For decoding, Tables R3.6 and R3.7 report the 16-seed AIME24 results and the expanded two-seed MATH-500 results, respectively. All confidence intervals are computed using 10,000 bootstrap resamples and the percentile method, resampling evaluation examples for prefill and problems for decoding while preserving all seeds associated with each problem. We have also updated Figure 5 and the corresponding results in Sec. 4.2 of the revised manuscript.
Table R3.6. AIME24 results on Qwen3-8B with trace construction interval (Appendix B.5). Accuracy is reported as mean ± SD across the 16 seeds.
| KV budget | Fast KVzip | DuetKV | Gain | 95% CI |
|---|---|---|---|---|
| 512 | 15.00 ± 6.32 | 27.50 ± 4.79 | +12.50 | [+6.25, +19.58] |
| 1024 | 42.71 ± 6.23 | 49.79 ± 6.38 | +7.08 | [+2.50, +12.29] |
| 2048 | 62.50 ± 4.47 | 66.46 ± 6.02 | +3.96 | [+0.42, +8.12] |
| 3072 | 69.79 ± 5.23 | 72.71 ± 5.74 | +2.92 | [−0.42, +6.25] |
| 4096 | 72.71 ± 5.61 | 71.25 ± 5.69 | −1.46 | [−5.21, +1.88] |
| 6144 | 74.38 ± 5.40 | 73.54 ± 3.33 | −0.83 | [−3.75, +2.29] |
Table R3.7 (new). MATH-500 results on Qwen3-8B with trace construction interval (Appendix B.5).
| KV budget | Fast KVzip (seed 0 / 1) | DuetKV (seed 0 / 1) | Gain | 95% CI |
|---|---|---|---|---|
| 512 | 79.4 / 79.4 | 85.8 / 87.0 | +7.00 | [+4.90, +9.20] |
| 1024 | 90.8 / 90.4 | 92.0 / 91.8 | +1.30 | [−0.20, +2.80] |
| 2048 | 93.6 / 94.6 | 94.4 / 94.4 | +0.30 | [−0.70, +1.40] |
| 3072 | 94.2 / 94.4 | 94.0 / 93.2 | −0.70 | [−1.70, +0.30] |
| 4096 | 93.8 / 94.0 | 93.6 / 94.2 | +0.00 | [−1.00, +1.00] |
Q2: Controlling the evictor's trajectory
Admitted traces can alter subsequent importance scores and retention decisions of the base eviction policy. To examine whether the performance gains mainly arise from these changes, we conduct two ablations under matched KV budgets.
-
DuetKV$_{\mathrm{defer}}$ delays trace admission until after prefill, preserving Fast KVzip's original prefill retention trajectory while still constructing learned traces from each segment. It follows the same importance estimator, admission rule, and allocation of exact and latent capacity as DuetKV.
-
Random$_{\mathrm{post}}$ instead preserves DuetKV's prefill trajectory but replaces the learned traces with norm-matched Gaussian vectors after prefill. For comparison, the original Random ablation in Table 1 substitutes random traces during prefill, allowing them to influence subsequent retention decisions.
Table R3.8. Effects of trace content and prefill retention trajectories under matched KV budgets. Scores are averaged across task and budget configurations within each group and across all 70 configurations overall.
| Method | Prefill trajectory | Retrieval | Contextual QA | Redundancy | Overall |
|---|---|---|---|---|---|
| Fast KVzip | Fast KVzip | 43.31 | 67.03 | 37.12 | 49.07 |
| DuetKV | DuetKV | ||||
| DuetKV$_{\mathrm{defer}}$ | Fast KVzip | ||||
| Random | Noise-influenced | ||||
| Random | DuetKV |
Results. retains 95.6% of DuetKV's overall gain of +3.77, achieving +3.61 despite restoring Fast KVzip's original retention decisions. In contrast, underperforms Fast KVzip by 0.89 points despite preserving DuetKV's prefill retention trajectory. These results indicate that learned latent representations contribute substantially to DuetKV's improvement, beyond the changes in eviction decisions. We have included these ablations in Sec. 4.3 and Appendix C.5 of the revised manuscript.
Q5: Reconciliation of headline results
The two results use the same evaluation protocol but different model backbones, each with its own Fast KVzip baseline. At on Retr.KV, DuetKV improves accuracy from 31.4 to 80.2 on Qwen2.5-14B-Instruct-1M (abstract) and from 32.6 to 77.2 on Qwen2.5-7B-Instruct-1M (introduction). The evaluation setup is detailed in Sec. 4.1 and Appendix B.2.
Author Response to Reviewer R3-GNyq (Part 1/3)
We thank the reviewer for their careful review and constructive feedback. We address each comment below. All changes in the revised manuscript are highlighted in blue.
W1 + Q4: Generalization, Transferability, and Training Cost
1. Cross-architecture generalization. To evaluate DuetKV beyond the Qwen family, we additionally conduct experiments on Llama-3.1-8B-Instruct, using Fast KVzip as the base evictor. We follow the same training and evaluation protocols and hyperparameters as Qwen2.5-7B-Instruct-1M (Appendix B.2), without Llama-specific tuning. As shown in Table R3.1, DuetKV improves Fast KVzip by 11.62 points on average across 70 task–budget configurations. At KV budget ratios of 0.2–0.5, DuetKV recovers approximately 99–102% of Full-KV performance. These results support DuetKV's applicability across backbone architectures, with normalized per-ratio curves in Appendix C.1 and additional Llama-3.1-8B quantization results in §4.5 of the revised manuscript.
Table R3.1 (new). Llama-3.1-8B-Instruct with Fast KVzip as the base evictor. Scores are averaged over 12 valid tasks at ρ≥0.2 and 10 at ρ=0.1, totaling 70 task–budget pairs. The last two rows normalize each task to its full-cache score before averaging, as in Figure 4. Subscript gains are computed from unrounded scores.
2.Training data and cost. The original submission already details the training data and optimization settings (§4.1, Appendix B.1–B.2). We reuse the same source corpus (FineWeb-Edu, PG-19, NarrativeQA, and GitHub code) across backbones. Backbone-specific tokenization and eligibility filtering yield 1,590 eligible contexts for the Qwen backbones and 1,589 for Llama, each paired with five queries. Full-cache teacher continuations are generated separately for each backbone. Only Tθ is optimized, with the backbone and base evictor frozen. Each constructor is trained once at ρ=0.1 for 300 steps (400 for Qwen3-8B) and evaluated at all six cache ratios without retraining. Training times are reported in Table R3.2, with per-step times and peak GPU memory in the revised Appendix B.2.
Table R3.2 (new). Training time across 4 backbones on 8 × H100 GPUs.
3. Transferability and unseen distributions. We distinguish transfer across backbones from generalization across budgets and data distributions. The constructor uses backbone-specific hidden representations and frozen K/V projections, and its parameters depend on the backbone architecture. Therefore, like Fast KVzip and RestoreKV, DuetKV requires backbone-specific training of its learned components rather than direct checkpoint transfer. Nevertheless, the same training procedure and hyperparameters apply to both Qwen and Llama without architecture-specific tuning. The original submission also demonstrates transferring the Qwen3-8B constructor to its FP8-weight version without retraining.
For unseen data distributions, we evaluate the same constructors without task-specific retraining on synthetic retrieval tasks that differ substantially from the training corpus, with a source-document-level train-evaluation split and contamination checks documented in the original submission (§4.1, Appendix B.1). On Llama-3.1-8B-Instruct, DuetKV improves Fast KVzip by 50.9, 25.7, and 24.7 points on Retr.KV, Retr.MultiHop, and Retr.PrefSuf, respectively, averaged across six cache ratios. These results show that backbone-specific constructors generalize across KV budgets and unseen evaluation tasks without additional training.