acceptodds
Under review as a conference paper at ICLR 2027

DuetKV: Complementing Exact KV Retention with Progressive Episodic Latent Traces

Abstract

Query-agnostic key–value (KV) cache compression reduces the cache once for reuse across future queries, yet accuracy degrades sharply under tight budgets. Selective eviction preserves the original KV states but retains only a sparse token-aligned subset, while KV compaction constructs new representations at the cost of precision. Recent complementary approaches combine both, yet derive the synthetic component from the retention outcome or only after full context prefill. This leaves open how to construct complementary representations progressively from complete pre-eviction states while preserving a fixed KV budget. We introduce DuetKV, which pairs sparse exact retention with progressively constructed episodic latent traces under a fixed KV budget. For each context segment, a learned constructor synthesizes traces from its complete pre-eviction hidden states without conditioning on the current retention outcome. Frozen backbone projections map these traces into native KV states that participate alongside retained exact states in standard attention. The constructor is trained offline by self-distillation from a full-cache teacher, while the backbone and base eviction policy remain frozen. We evaluate DuetKV across 4 long-context backbones, 3 base eviction policies, and 12 prefill-intensive tasks. Across policy-budget settings, DuetKV improves the task-averaged score in 17 of 18 cases and recovers Full-KV performance with Fast KVzip at 5X compression. On 170K-token Retr.KV with Qwen2.5-14B-Instruct-1M, it raises Fast KVzip from 31.4 to 80.2 at the same 5.7 GiB peak KV memory with only 1.7% additional prefill latency. The same constructor also improves decoding-intensive reasoning under continuous KV compression without decoding-specific retraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, KV cache eviction, long-context inference, large language models, query-agnostic compression, latent memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GNyqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DuetKV, a query-agnostic KV-cache compression framework that combines sparse exact KV retention with progressively constructed latent traces under a fixed cache budget. A learned constructor synthesizes episodic traces from complete pre-eviction hidden states during prefill, while the backbone and base eviction policy remain frozen; the traces are projected into the model’s native KV space and jointly attended with retained exact states. The constructor is trained by self-distillation from a full-cache teacher without downstream query information. The method is evaluated across four long-context backbones, three eviction policies, cache budgets from 10% to 75%, and 12 prefill-intensive tasks, with additional decoding-intensive evaluations on AIME24 and MATH-500. DuetKV improves its base policies in 17 of 18 policy–budget settings, recovers near full-cache performance under aggressive compression, and provides large gains on retrieval and reasoning tasks with modest prefill overhead. Ablations comparing random traces, additional exact capacity, trace cardinality, and calibration support the claim that the gains mainly come from the learned latent trace content.

Critical strengths

DuetKV presents a clear and well-motivated way to combine exact token retention with learned latent representations. Constructing traces progressively from pre-eviction hidden states, without depending on the current eviction outcome, gives the method a distinct advantage over approaches that add synthetic states only after full prefill or derive them from the retained subset.

The empirical validation is broad and supports the practical value of the method. DuetKV is tested across multiple backbones, eviction policies, compression budgets, prefill-intensive tasks, and decoding-intensive reasoning tasks. It improves the base policies in 17 of 18 policy–budget settings, with large gains under aggressive compression and only modest prefill overhead. The random-trace and extra-capacity controls further indicate that the improvement comes from the learned trace content rather than simply reallocating cache capacity.

Critical weaknesses

The evaluation is broad in task count but still concentrated on the Qwen family and on constructors trained separately for particular backbones. Evidence for transfer across genuinely different architectures, model scales, and unseen context distributions is limited; the reported Qwen3-to-Qwen3-FP8 transfer is a relatively narrow test. This makes it difficult to determine whether DuetKV is a generally applicable compression principle or a method that requires substantial model-specific training.

The practical efficiency benefit is not fully established for end-to-end serving. The reported overhead focuses mainly on prefill latency and peak KV memory, while decoding experiments use a particular continuous-compression protocol and limited reasoning benchmarks. The constructor, segment-wise cross-attention, and repeated trace updates may have different costs under batching, longer generations, or varying numbers of reused queries, so the headline accuracy gains under compression do not yet demonstrate a consistent real-world throughput or latency advantage.

Questions for authors
  1. This paper reports improvements across 70 task–budget cells and claims gains in 17 of 18 policy–budget settings. Could the authors provide confidence intervals or results across multiple random seeds, especially for the smaller gains and for the decoding benchmarks?

  2. DuetKV traces inserted at earlier segments can affect later hidden states and therefore the base eviction policy’s subsequent scores. How is the comparison with the original eviction policy controlled so that improvements are attributable to latent traces rather than changes in the trajectory of the evictor itself?

  3. Do the reported memory and latency measurements include the constructor’s parameters, temporary activations, cross-attention computation, and trace-update overhead? Please report end-to-end throughput and latency for different batch sizes and numbers of reused queries.

  4. A separate constructor appears to be trained for each backbone, while training is performed only at ρ=0.1\rho=0.1 and evaluated at several other budgets. How much training data and compute are required per model, and how well does the method transfer to unseen architectures, scales, and context distributions without retraining?

  5. The abstract reports a 170K-token Retr.KV improvement from 31.4 to 80.2 on Qwen2.5-14B, whereas the main text reports 32.6 to 77.2 on Qwen2.5-7B. Are these results from different experimental settings? Please reconcile the numbers and specify the exact model, protocol, and baseline for each headline result.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer R3-GNyq (Part 1/3)

We thank the reviewer for their careful review and constructive feedback. We address each comment below. All changes in the revised manuscript are highlighted in blue.


W1 + Q4: Generalization, Transferability, and Training Cost

1. Cross-architecture generalization. To evaluate DuetKV beyond the Qwen family, we additionally conduct experiments on Llama-3.1-8B-Instruct, using Fast KVzip as the base evictor. We follow the same training and evaluation protocols and hyperparameters as Qwen2.5-7B-Instruct-1M (Appendix B.2), without Llama-specific tuning. As shown in Table R3.1, DuetKV improves Fast KVzip by 11.62 points on average across 70 task–budget configurations. At KV budget ratios of 0.2–0.5, DuetKV recovers approximately 99–102% of Full-KV performance. These results support DuetKV's applicability across backbone architectures, with normalized per-ratio curves in Appendix C.1 and additional Llama-3.1-8B quantization results in §4.5 of the revised manuscript.

Table R3.1 (new). Llama-3.1-8B-Instruct with Fast KVzip as the base evictor. Scores are averaged over 12 valid tasks at ρ≥0.2\rho\geq0.2 and 10 at ρ=0.1\rho=0.1, totaling 70 task–budget pairs. The last two rows normalize each task to its full-cache score before averaging, as in Figure 4. Subscript gains are computed from unrounded scores.

Cache ratio 0.75  0.5  0.4  0.3  0.2  0.1 
Full-KV46.8846.8846.8846.8846.8846.17
Fast KVzip47.3334.3333.4633.1932.6922.54
+ DuetKV48.76+1.4348.76_{+1.43}47.74+13.4147.74_{+13.41}47.38+13.9347.38_{+13.93}47.62+14.4347.62_{+14.43}45.71+13.0145.71_{+13.01}36.41+13.8636.41_{+13.86}
Fast KVzip, % of Full-KV102.566.563.465.765.248.4
+ DuetKV, % of Full-KV106.198.899.5101.999.379.5

2.Training data and cost. The original submission already details the training data and optimization settings (§4.1, Appendix B.1–B.2). We reuse the same source corpus (FineWeb-Edu, PG-19, NarrativeQA, and GitHub code) across backbones. Backbone-specific tokenization and eligibility filtering yield 1,590 eligible contexts for the Qwen backbones and 1,589 for Llama, each paired with five queries. Full-cache teacher continuations are generated separately for each backbone. Only Tθ\mathcal T_\theta is optimized, with the backbone and base evictor frozen. Each constructor is trained once at ρ=0.1\rho=0.1 for 300 steps (400 for Qwen3-8B) and evaluated at all six cache ratios without retraining. Training times are reported in Table R3.2, with per-step times and peak GPU memory in the revised Appendix B.2.

Table R3.2 (new). Training time across 4 backbones on 8 ×\times H100 GPUs.

Backbone Steps  Wall-clock time 
Qwen2.5-7B-Instruct-1M3003.43 h
Llama-3.1-8B-Instruct3004.24 h
Qwen3-8B4006.18 h
Qwen2.5-14B-Instruct-1M3009.23 h

3. Transferability and unseen distributions. We distinguish transfer across backbones from generalization across budgets and data distributions. The constructor uses backbone-specific hidden representations and frozen K/V projections, and its parameters depend on the backbone architecture. Therefore, like Fast KVzip and RestoreKV, DuetKV requires backbone-specific training of its learned components rather than direct checkpoint transfer. Nevertheless, the same training procedure and hyperparameters apply to both Qwen and Llama without architecture-specific tuning. The original submission also demonstrates transferring the Qwen3-8B constructor to its FP8-weight version without retraining.

For unseen data distributions, we evaluate the same constructors without task-specific retraining on synthetic retrieval tasks that differ substantially from the training corpus, with a source-document-level train-evaluation split and contamination checks documented in the original submission (§4.1, Appendix B.1). On Llama-3.1-8B-Instruct, DuetKV improves Fast KVzip by 50.9, 25.7, and 24.7 points on Retr.KV, Retr.MultiHop, and Retr.PrefSuf, respectively, averaged across six cache ratios. These results show that backbone-specific constructors generalize across KV budgets and unseen evaluation tasks without additional training.

AuthorsOpenReview ↗

Author Response to Reviewer R3-GNyq (Part 2/3)

W2 + Q3: End-to-end serving efficiency

1. Memory and latency accounting.

All latency measurements include latent trace construction, segment-wise cross-attention, projection into native KV states, and trace updates whenever invoked. The paper's peak KV memory metric covers retained exact KV states, excluding episodic latent traces, constructor parameters, and temporary activations. Appendix B.3 already reports the 198 MiB constructor parameter footprint, while Table R3.3 further reports trace storage and peak GPU allocation.

Table R3.3 (new). Prefill latency (median of five runs) and memory accounting for Fast KVzip with and without DuetKV on Qwen2.5-7B-Instruct-1M (Fig. 6), at 170K tokens and ρ=0.1\rho=0.1.

Metric Fast KVzip  + DuetKV 
Prefill latency8.239 s8.392 s (+1.9%)
Peak exact KV memory1.655 GiB1.655 GiB
Episodic latent trace storage02.41 MiB
Resident constructor parameters0198 MiB
Peak allocated-memory increment3.862 GiB3.861 GiB

Peak allocated-memory increment denotes the increase in GPU allocation relative to the prefill baseline, including temporary activations and workspace. The additional latent traces require only 2.41 MiB (approximately 0.14% of the reported exact-KV memory), while the measured peak GPU allocation remains essentially unchanged. Together with the 1.9% prefill latency overhead, these results indicate that DuetKV introduces negligible additional peak memory overhead and minimal runtime latency overhead. For Qwen2.5-14B-Instruct-1M (Sec. 4.4), trace storage is 8.25 MiB and constructor parameters occupy 485 MiB, while peak exact KV memory remains unchanged.

2.Shared-context reuse.

We evaluate three 170K-token contexts, each shared across N∈{1,2,4,8,16}N\in\{1,2,4,8,16\} sequential queries with 128 generated tokens per query. Episodic latent traces are constructed during shared prefill and reused without modification. End-to-end latency includes shared prefill, compression, query processing, and generation.

As shown in Table R3.4, DuetKV introduces only 0.7%–1.9% additional end-to-end latency over Fast KVzip across all 15 context-matched settings, without a consistent increase in relative overhead as the number of reused queries grows. At (N=16), DuetKV achieves a 1.3$\times$ end-to-end speedup over Full-KV, while maintaining nearly the same latency as Fast KVzip.

Table R3.4 (new). End-to-end latency, amortized per-query latency, and sequential throughput for NN queries sharing one context, with 128 generated tokens per query. Values are medians across three contexts, while relative overhead is computed from context-matched comparisons.

Reused queries NN 1  2  4  8  16 
Full-KV, end-to-end (s)25.2629.2337.1452.9584.58
Fast KVzip, end-to-end (s)11.7315.1822.1235.9863.72
+ DuetKV, end-to-end (s)11.9315.4622.5336.6864.96
+ DuetKV, amortized (s/query)11.937.735.634.594.06
+ DuetKV, throughput (queries/s)0.0840.1290.1780.2180.246

3.Generation-length sensitivity.

We examine fixed traces in the prefill-intensive regime and periodic trace construction in the decoding-intensive regime. In the latter, Fast KVzip compresses the cache every 128 tokens, while DuetKV constructs traces every 1,024 tokens. We use greedy decoding from a synthetic 190-token prompt, matching the median AIME24 prompt length, with a 32,768-token generation budget and a 4,096-token KV budget.

Table R3.5 (new). Generation-length sensitivity. Prefill-intensive results report context-matched per-token latency changes for three 160K-token contexts on Qwen2.5-7B-Instruct-1M (ρ=0.1\rho=0.1). Decoding-intensive results report cumulative latency changes on Qwen3-8B.

Prefill-intensive, generated tokens 128  512  1024  2048 
Context A, per-token change−1.6%+0.3%+1.6%+1.6%
Context B, per-token change+4.0%+1.6%+1.6%+1.1%
Context C, per-token change−4.8%−0.5%+2.0%+2.3%
Decoding-intensive, generated tokens5122048819232768
Cumulative latency change+1.9%+2.2%+1.3%+0.9%

The results show modest per-token latency changes in the prefill-intensive regime and no sustained increase in cumulative overhead during decoding-intensive generation, with only +0.9% overhead at 32,768 tokens.

4. Batching.

Our current implementation inherits the batch-size-one limitation of the underlying eviction framework, whose variable-length attention layout is indexed by KV head rather than batch. Therefore, the reported results measure single-stream latency and sequential throughput rather than concurrent serving performance.

These additional efficiency results are reported in Appendix B.3 of the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer R3-GNyq (Part 3/3)

Q1: Confidence intervals and random seeds

We have added paired bootstrap 95% confidence intervals for all 70 prefill task and budget configurations, with the full results reported in Appendix C.7 of the revised manuscript. For decoding, Tables R3.6 and R3.7 report the 16-seed AIME24 results and the expanded two-seed MATH-500 results, respectively. All confidence intervals are computed using 10,000 bootstrap resamples and the percentile method, resampling evaluation examples for prefill and problems for decoding while preserving all seeds associated with each problem. We have also updated Figure 5 and the corresponding results in Sec. 4.2 of the revised manuscript.

Table R3.6. AIME24 results on Qwen3-8B with trace construction interval I=1024I=1024 (Appendix B.5). Accuracy is reported as mean ± SD across the 16 seeds.

 KV budget  Fast KVzip  DuetKV  Gain  95% CI 
51215.00 ± 6.3227.50 ± 4.79+12.50[+6.25, +19.58]
102442.71 ± 6.2349.79 ± 6.38+7.08[+2.50, +12.29]
204862.50 ± 4.4766.46 ± 6.02+3.96[+0.42, +8.12]
307269.79 ± 5.2372.71 ± 5.74+2.92[−0.42, +6.25]
409672.71 ± 5.6171.25 ± 5.69−1.46[−5.21, +1.88]
614474.38 ± 5.4073.54 ± 3.33−0.83[−3.75, +2.29]

Table R3.7 (new). MATH-500 results on Qwen3-8B with trace construction interval I=512I=512 (Appendix B.5).

 KV budget  Fast KVzip (seed 0 / 1)  DuetKV (seed 0 / 1)  Gain  95% CI 
51279.4 / 79.485.8 / 87.0+7.00[+4.90, +9.20]
102490.8 / 90.492.0 / 91.8+1.30[−0.20, +2.80]
204893.6 / 94.694.4 / 94.4+0.30[−0.70, +1.40]
307294.2 / 94.494.0 / 93.2−0.70[−1.70, +0.30]
409693.8 / 94.093.6 / 94.2+0.00[−1.00, +1.00]

Q2: Controlling the evictor's trajectory

Admitted traces can alter subsequent importance scores and retention decisions of the base eviction policy. To examine whether the performance gains mainly arise from these changes, we conduct two ablations under matched KV budgets.

  • DuetKV$_{\mathrm{defer}}$ delays trace admission until after prefill, preserving Fast KVzip's original prefill retention trajectory while still constructing learned traces from each segment. It follows the same importance estimator, admission rule, and allocation of exact and latent capacity as DuetKV.

  • Random$_{\mathrm{post}}$ instead preserves DuetKV's prefill trajectory but replaces the learned traces with norm-matched Gaussian vectors after prefill. For comparison, the original Random ablation in Table 1 substitutes random traces during prefill, allowing them to influence subsequent retention decisions.

Table R3.8. Effects of trace content and prefill retention trajectories under matched KV budgets. Scores are averaged across task and budget configurations within each group and across all 70 configurations overall.

MethodPrefill trajectory Retrieval  Contextual QA  Redundancy  Overall 
Fast KVzipFast KVzip43.3167.0337.1249.07
DuetKVDuetKV51.96+8.6551.96_{+8.65}68.59+1.5568.59_{+1.55}38.03+0.9138.03_{+0.91}52.85+3.7752.85_{+3.77}
DuetKV$_{\mathrm{defer}}$Fast KVzip51.60+8.2951.60_{+8.29}67.98+0.9567.98_{+0.95}38.52+1.3938.52_{+1.39}52.69+3.6152.69_{+3.61}
RandomNoise-influenced42.68−0.6342.68_{-0.63}65.06−1.9765.06_{-1.97}37.23+0.1137.23_{+0.11}48.24−0.8348.24_{-0.83}
Randompost_{\mathrm{post}}DuetKV42.38−0.9342.38_{-0.93}66.12−0.9166.12_{-0.91}36.31−0.8236.31_{-0.82}48.19−0.8948.19_{-0.89}

Results. DuetKVdefer\mathrm{DuetKV}_{\mathrm{defer}} retains 95.6% of DuetKV's overall gain of +3.77, achieving +3.61 despite restoring Fast KVzip's original retention decisions. In contrast, Randompost\mathrm{Random}_{\mathrm{post}} underperforms Fast KVzip by 0.89 points despite preserving DuetKV's prefill retention trajectory. These results indicate that learned latent representations contribute substantially to DuetKV's improvement, beyond the changes in eviction decisions. We have included these ablations in Sec. 4.3 and Appendix C.5 of the revised manuscript.


Q5: Reconciliation of headline results

The two results use the same evaluation protocol but different model backbones, each with its own Fast KVzip baseline. At ρ=0.1\rho=0.1 on Retr.KV, DuetKV improves accuracy from 31.4 to 80.2 on Qwen2.5-14B-Instruct-1M (abstract) and from 32.6 to 77.2 on Qwen2.5-7B-Instruct-1M (introduction). The evaluation setup is detailed in Sec. 4.1 and Appendix B.2.

Reviewer_RrgZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DuetKV is a query-agnostic compression method that utilizes both the original KV cache—retained via existing eviction policies—and "episodic latent traces" generated from the hidden states of each segment prior to eviction. These traces are constructed incrementally, independent of the current KV selection decisions, and replace a portion of the original KV cache to maintain a fixed total KV budget. The constructor is trained using a full-cache teacher model, with the backbone and the eviction policy held constant. The authors evaluate the method across four model configurations and three eviction policies, and demonstrate its applicability to continuous compression during decoding without decoding-specific retraining.

Critical strengths
  1. The paper demonstrates the effectiveness of learned traces under a matched KV budget. In Table 1, the average score for Qwen2.5-7B-Instruct-1M improves from 49.1 with Fast KVzip to 52.9 with DuetKV. Replacing the traces with random vectors—while maintaining the same capacity and norm—causes the score to drop to 48.2. This serves as evidence that the content of the learned traces is more critical to performance improvement than the mere allocation of cache space.

  2. It demonstrates the practicality of incremental construction and its transition to the decoding phase. Instead of constructing the entire original context cache upfront, the method compresses the context in segments; under the 10% KV budget condition shown in Figure 6, it incurs an additional prefill latency of only 1.3–2.5% compared to Fast KVzip. Furthermore, the constructor trained offline for prefill compression can be applied to the decoding phase without the need for retraining. This leads to performance improvements on the Qwen3-8B model—with a 512-token budget—increasing the AIME24 score from 12.1 to 25.4 and the MATH-500 score from 79.4 to 85.8.

Critical weaknesses
  1. There is a lack of direct comparison with existing complementary KV representation methods. While §2 discusses related approaches that combine original KV data with complementary representations—such as ResKV and IndexMem, RestoreKV—the primary experimental comparisons focus on pruning-based methods, including KVzip, Fast KVzip, Expected Attention, and DuoAttention. Consequently, although the effectiveness of incorporating "traces" into pruning policies is demonstrated, the advantages of the proposed design—which is both incremental and independent of selection outcomes—over existing complementary representation methods have not been sufficiently evaluated in terms of accuracy and computational cost.

  2. The contribution of information from specific segments has not been disentangled from the effects of learning components that are shared across multiple segments. As shown in Appendix C.2, the average cosine similarity of trace values ​​between segments is 0.993. While this does not necessarily imply that the traces are redundant, a random ablation approach—which alters both the shared components and the segment-specific components simultaneously—fails to distinguish which of these is responsible for performance improvements. Although the authors provide evidence of query-dependent routing toward relevant segments, the causal contribution of segment-specific trace content to downstream accuracy remains unclear.

Questions for authors
  1. Regarding Weakness 1: Could you select a representative, comparable method from the existing complementary KV-caching techniques discussed in §2, and then compare its accuracy and cache construction cost against your approach, using the same total KV budget?

  2. Regarding Weakness 2: Is it possible to conduct an ablation study—while keeping the cache budget, positional assignments, and trace norms fixed—in which the traces for specific segments are replaced with shared traces or traces from a different context, in order to verify whether the content of those specific segments is actually necessary for performance improvement?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer R2-RrgZ (Part 1/2)

We appreciate the reviewer's thoughtful comments and constructive suggestions. We have addressed the concerns through additional experiments and analyses, as detailed below. All revisions to the manuscript are highlighted in blue.


W1 ≡ Q1: Direct comparison with a complementary KV representation

Selection. We select RestoreKV for direct comparison due to its related design, released code, and shared backbone, and additionally include published results from IndexMem. Figure 1(a) of the original submission already includes RestoreKV+ in the KVPress performance envelope at 8x and 16x, which we now report separately. Since KVzip+ relies on full-context prefill followed by eviction, which is incompatible with DuetKV's progressive training pipeline, we transfer the Fast KVzip-trained constructor to KVzip+ without retraining.

Table R2.1. Average accuracy (%) across 13 tasks on RULER-4K and RULER-16K with Qwen3-8B. All methods follow the KVPress protocol except IndexMem, whose results are from the original paper.

Method4K16K
 16×  10×  8×  4×  2×  16×  10×  8×  4×  2× 
Uncompressed94.7794.7794.7794.7794.7792.9492.9492.9492.9492.94
Fast KVzip45.3479.4987.8394.5694.8946.8976.3283.6892.0392.82
 + DuetKV58.1085.3290.0794.2294.8259.8482.0088.4392.2092.42
KVzip+76.1691.3993.8294.8994.9666.6384.3487.4192.4593.12
 + DuetKV81.3891.9193.5694.5694.9077.4789.9490.7892.1592.51
 + RestoreKV+85.6793.5494.1894.8694.9479.7388.6790.2092.4992.72
IndexMem—46.2—72.978.9—56.0—72.280.0

Accuracy. On the same KVzip+ base policy, DuetKV approaches RestoreKV+ at lower compression ratios, and the performance gap under aggressive compression narrows at longer contexts, with DuetKV outperforming RestoreKV+ by +1.27 and +0.58 points at 10×10\times and 8×8\times on RULER-16K, respectively. This comparison may be affected by cross-evictor transfer, which RestoreKV reports can degrade performance (their Appendix A.3)

Table R2.2 (new). Prefill and construction latency on an H100 GPU with Qwen3-8B, measured using CUDA events on concatenated RULER-4K contexts. Construction times are measured directly within each run.

Base policyStage4K8K16K32K64K
Fast KVzipprefill (s)0.200.390.852.146.11
 + DuetKV construction (ms)3229293754
KVzip+prefill (s)2.513.756.7415.1442.07
 + DuetKV construction (ms)2427303962
 + RestoreKV+ construction (ms)565662105192

Efficiency. Both methods place their constructed representations in the KV cache at a matched total KV budget and add no query-time computation, so construction is the only stage at which their costs differ. On the same KVzip+ base policy DuetKV constructs in 24 to 62 ms, which is 2.0 to 3.1 times faster than RestoreKV+ across context lengths. We have added both comparisons to Appendix C.2 of the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer R2-RrgZ (Part 2/2)

W2 ≡ Q2: Segment-specific versus shared trace content

Controlled substitution. We perform post-prefill substitutions of episodic latent traces before query processing, preserving cache occupancy, positional anchors, and per-slot norms. The substituted traces are mapped to keys and values through the frozen backbone projections. This leaves the original prefill computation and eviction decisions unchanged.

Table R2.3 (new). Mean score on Qwen2.5-7B-Instruct-1M with the gain over Fast KVzip as a subscript, averaged by task group and across the same 70 task–budget configurations as Table 1. Shared uses slot-wise means across the segments of the same document. Cross-document substitutes same-index traces from unrelated documents, produced by the same constructor under the same budget and cycling through donor segments as needed. Constant uses a document-wide mean direction with per-slot norms preserved. Randompost_{\mathrm{post}} substitutes norm-matched Gaussian traces after prefill. Zero-value (γv=0\gamma_v{=}0) retains the learned trace keys but zeros their projected values.

Intervention Retr.  Contextual QA  Redund.  All 70 
Fast KVzip43.3167.0337.1249.07
DuetKV51.96+8.6551.96_{+8.65}68.59+1.5568.59_{+1.55}38.03+0.9138.03_{+0.91}52.85+3.77\mathbf{52.85_{+3.77}}
Trace content
 Shared51.85+8.5451.85_{+8.54}68.28+1.2568.28_{+1.25}38.07+0.9538.07_{+0.95}52.72+3.6552.72_{+3.65}
 Cross-document52.20+8.8952.20_{+8.89}68.65+1.6268.65_{+1.62}37.67+0.5537.67_{+0.55}52.82+3.7552.82_{+3.75}
 Constant51.69+8.3851.69_{+8.38}68.14+1.1168.14_{+1.11}37.66+0.5437.66_{+0.54}52.49+3.4252.49_{+3.42}
 Randompost_{\mathrm{post}}42.38−0.9342.38_{-0.93}66.12−0.9166.12_{-0.91}36.31−0.8236.31_{-0.82}48.19−0.89\mathbf{48.19_{-0.89}}
Readout pathway
 Zero-value (γv=0\gamma_v{=}0)41.74−1.5741.74_{-1.57}66.43−0.6066.43_{-0.60}36.47−0.6636.47_{-0.66}48.12−0.95\mathbf{48.12_{-0.95}}

Mechanistic interpretation. We added complementary ablations to disentangle learned value contributions from source-segment specificity. Zeroing the projected trace values while retaining their keys yields an average score 0.95 points below Fast KVzip. This preserves the trace keys and their participation in attention normalization while removing their value contributions, leaving the attention-weighted readout over retained exact entries without renormalization. Similarly, replacing learned traces with norm-matched noise after prefill yields a comparable deficit (−0.89 points), indicating that neither attention redistribution alone nor arbitrary representations explain the gains.

Conversely, Shared, Cross-document, and Constant preserve gains of +3.65, +3.75, and +3.42 points, respectively, compared with +3.77 for DuetKV. These results suggest that the gains depend on learned trace content without requiring strict correspondence to the source segments, consistent with the high cross-segment similarity of synthesized values (mean cosine 0.993, versus 0.987 for the corresponding source-segment means). The modest Redundancy loss under Cross-document, but not Shared, and the lower gain under Constant suggest possible contributions from document-level information and slot-wise directional variation. These observations are consistent with our predictive self-distillation objective, which targets the full-cache model's predictive distribution without imposing segment-wise KV reconstruction targets, encouraging predictive information to remain accessible to the frozen attention stack (Appendix A.3). The new ablations are reported in Appendix C.4.

Reviewer_jNZVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

DuetKV is a query-agnostic KV-cache compression method that complements token eviction with learned context compaction. The eviction side mainly reuses an existing algorithm (e.g., Fast KVzip) unmodified; the paper's actual contribution is the second side — how a trace constructor is trained, when it fires, and how it shares a fixed total KV budget with the base evictor.

During prefill, a constructor synthesizes a small number of compact latent traces — summaries of the segment, via cross-attention over its full pre-eviction hidden states. These summaries replace part of the KV entries the frozen base eviction policy would otherwise retain, based on that policy's own importance scores. The two components differ in temporal scope: once admitted, a trace becomes part of the cache the evictor scores against going forward, so the evictor's decisions are progressive and history-dependent, while each new trace is built only from its own segment's pre-eviction hidden states — never from prior traces or the evictor's choices — making trace construction episodic and segment-local. This progressive-but-independent construction is what distinguishes DuetKV from prior complementary methods. The constructor is trained offline via self-distillation from a frozen full-cache teacher, while the backbone and base evictor stay frozen throughout. During decoding, the constructor is not retrained and fires at a fixed multiple of the interval at which the evictor triggers.

In the experimental results, across 4 backbones, 3 base evictors, and 12 prefill-intensive tasks plus 2 decoding-intensive reasoning benchmarks, DuetKV improves task-averaged scores in 17/18 policy-budget settings (Figure 3) and generalizes to decoding without decoding-specific retraining (Figure 5), at ~1.3–2.5% added prefill latency and no extra peak memory. Compared with the default Fast KVzip alone: DuetKV closes a 4.4-point gap at ρ=0.2 (5× compression) and roughly half of a 21.2-point gap at ρ=0.1 (10× compression) (Section 4.2, Figure 3).

Critical strengths
  1. Broad validation. Results span 3 evictors × 4 backbones × 14 tasks, backed by concrete memory/latency numbers against a real baseline (5.7 GiB / +1.7% prefill latency vs. KVzip's 31.0 GiB / 165.6s at 170K tokens), not just relative accuracy curves.
  2. Rigorous ablation studies. Isolate the effect cleanly from both sides of the trade-off: replacing traces with norm-matched Gaussian noise (Random) collapses the gain from +21.9 to +1.9 points at ρ=0.1, while restoring the evictor's full exact allocation on top of the traces (DuetKV+, so traces no longer cost anything) improves the result by only +0.3 points — together showing the displaced low-value tokens weren't worth much, and the gain comes specifically from the traces' learned content. Traces also receive 16.3% of attention while occupying only 0.34% of cache budget, confirming they carry real information rather than just extra capacity or architecture.
  3. Decent improvement over the unmodified baseline. DuetKV closes a 4.4-point gap to Full-KV at ρ=0.2 (5× compression) and roughly half of a 21.2-point gap at ρ=0.1 (10× compression) over Fast KVzip alone.
Critical weaknesses
  1. All compression ratios are measured in slots retained, not bytes stored, leaving the comparison against quantization untested. Every reported budget (ρ\rho) is a ratio over token/trace slots, and it's unclear whether DuetKV composes with well-established quantization methods. The KV cache's precision in this paper seems to be BF16, and FP8 is usually considered near-lossless for KV storage, not to mention many INT4 and INT2 works have reported encouraging results. If DuetKV's slot-level gains at, say, ρ=0.8\rho=0.8 also hold when the cache is additionally cast to FP8 (so the same accuracy is reached at half the bytes of an unquantized ρ=0.8\rho=0.8 cache), that would meaningfully extend the "no-degradation" region in Figure 3 beyond what's currently shown. The paper provides no such experiment or even a discussion of whether this composability should be expected to hold.

  2. The decoding-time eviction rule ignores an importance signal the paper's own pipeline already computes for free. When the trace population exceeds the fixed cap B_\max, DuetKV discards the trace group with the smallest merged temporal gap (Eq. 83) — a purely time-based criterion. Since the evictor has already computed fresh per-token importance scores every Δ\Delta tokens, logging a running aggregate of those scores as metadata, rather than discarding them, would give the constructor a content-based importance proxy at zero extra cost — especially since trace utilization (attention received, §4.4) varies by two orders of magnitude across layers. Neither this alternative nor its absence is discussed anywhere in §3.4 or Appendix B.5, where the temporal-only rule is specified.

  3. The 40% latent-budget cap looks overcautious relative to the paper's own evidence, with no justification offered. The admission rule (Eq. 11) gates trace insertion in part on a global cap \eta_\max = 0.40 on the fraction of the native budget that can be reassigned to traces, stated once with no motivating argument. Appendix B.2 reports that this cap "never binds during training or on nine of the twelve evaluation tasks," and binds on at most 6.35% of segments in every reported setting except GSM8K (where short contexts push it to 99.83% of segments at ρ=0.1) — i.e., the paper's own data suggests the cap is almost never the active constraint, leaving it unclear why 0.40 was chosen over a substantially lower value, or what would happen if it were.

Questions for authors

please see Weaknesses above.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer R1-jNZV (Part 1/3)

We thank the reviewer for recognizing the contributions of our work and for providing constructive feedback. We address the questions and concerns below. All changes in the revised manuscript are highlighted in blue.

W1: Byte-level budgets and composability with KV quantization

1.Byte accounting. DuetKV measures cache capacity in KV entries indexed by layer, KV head, and position, rather than input tokens. Episodic latent traces are projected through the frozen backbone's WKW_K and WVW_V and stored at the cache precision, each occupying one KV entry with the same per-entry storage cost as a retained exact KV state. Under BF16, the target cache ratio ρ\rho therefore corresponds to the normalized KV storage budget:

BDuetKVBF16BFullKVBF16=NDuetKVNFullKV=ρ.\frac{B_{\mathrm{DuetKV}}^{\mathrm{BF16}}}{B_{\mathrm{FullKV}}^{\mathrm{BF16}}}=\frac{N_{\mathrm{DuetKV}}}{N_{\mathrm{FullKV}}}=\rho.

where NN counts KV entries over the same cache scope. Runtime byte accounting agrees with this nominal relationship within 0.15% across six cache ratios at a 169,428-token context. Quantization changes the per-entry storage cost without altering KV-entry accounting, making the two mechanisms structurally compatible while leaving accuracy preservation to empirical evaluation.

2.Quantization composability. We evaluate KIVI-style group-wise asymmetric quantization (group size 64) on both retained exact KV states and episodic latent traces, on Qwen2.5-7B-Instruct-1M and Llama-3.1-8B-Instruct, across the same 70 task–budget configurations as Table 1. Quantization is simulated through quantization and dequantization, with scale and zero-point metadata included in the byte estimates. The reported storage savings are therefore estimates rather than measured GPU memory or latency reductions.

Table R1.1 (new). KV quantization results averaged over the same 70 task–budget configurations as Table 1, for each backbone. Byte ratios denote nominal per-entry storage relative to BF16, including quantization metadata.

 Model  KV precision  Bytes / BF16  Fast KVzip  DuetKV  Gain 
Qwen2.5-7BBF161.00×49.0752.85+3.77
Qwen2.5-7BINT80.53×48.9952.80+3.82
Qwen2.5-7BINT40.28×48.1650.83+2.67
Llama-3.1-8BBF161.00×34.2545.87+11.62
Llama-3.1-8BINT40.28×33.5143.67+10.17

At matched KV-entry budgets, INT8 nearly preserves BF16 accuracy and preserves DuetKV's gain over Fast KVzip at 0.53× the per-entry storage. INT4 incurs greater accuracy degradation under entry-matched comparisons that do not account for its lower storage cost, and it still retains 70.7% of the BF16 gain on Qwen2.5-7B and 87.5% on Llama-3.1-8B.

3.Accuracy–storage trade-off. Table R1.2 compares the available KV precisions at the same KV-entry budget ρ=0.4\rho=0.4 on the common ten-task subset, which is where DuetKV reaches its highest average accuracy under every precision on Qwen2.5-7B.

Table R1.2 (new). Accuracy and estimated KV storage at ρ=0.4\rho=0.4, normalized to each model's full BF16 KV cache.

 Model  Bytes / full KV  KV precision  ρ\rho  Fast KVzip  DuetKV 
Qwen2.5-7B40.0%BF160.453.3056.24
Qwen2.5-7B21.3%INT80.453.0356.18
Qwen2.5-7B11.3%INT40.452.6054.05
Llama-3.1-8B40.0%BF160.429.4246.68
Llama-3.1-8B11.3%INT40.429.5043.68

On Qwen2.5-7B, INT8 nearly matches BF16 accuracy with 1.88× lower estimated KV storage, while INT4 reduces storage by 3.56× with a 2.19-point accuracy loss. At comparable storage budgets near 10% of the full BF16 cache, INT4 outperforms INT8 and BF16 (54.05 versus 53.33 and 41.49), reversing the entry-matched ranking. Quantization narrows DuetKV's gain over Fast KVzip on Qwen2.5-7B, from +9.98 to +4.78 with INT8 and +1.45 with INT4. On Llama-3.1-8B, INT4 similarly outperforms BF16 at comparable storage (43.68 at 11.3% versus 36.41 at 10.0%), while DuetKV's gain remains substantial (+14.18 versus +13.86 with BF16). These results demonstrate that quantization improves accuracy at fixed storage budgets, although its effect on DuetKV's relative gain depends on the backbone.

Complete accuracy–storage curves for both Qwen2.5-7B and Llama-3.1-8B are provided in the new Section 4.5. For clarity, Qwen3-8B-FP8 in Figure 4 uses FP8-quantized weights but retains a BF16 KV cache.

AuthorsOpenReview ↗

Author Response to Reviewer R1-jNZV (Part 2/3)

W2 — Temporal vs. Importance-Based Trace Eviction

To address this concern, we evaluate an importance-aware alternative to strided retention using the token importance scores already computed by the base eviction policy. For each construction event jj, we average these scores over its II-token interval. When a new construction event would exceed the latent capacity Bmax⁡B_{\max}, we replace the temporal criterion in Eq. (83) with

\qquad j_{\ell h}^{\star}=\arg\min_{2\le j\le K}u_j^{\ell h}, \qquad K=\frac{B_{\max}}{M},$$ that is, we discard the interior event of lowest average importance. Here, $T_j$ denotes the token interval associated with event $j$, $s_t^{\pi,\ell h}$ is the base policy's importance score for token $t$ at layer $\ell$ and KV head $h$, and $j$ indexes the $K+1$ candidate events in temporal order within each pair. As in the original strided retention rule, the variant preserves the earliest and latest construction events and discards all $M$ latent entries of the selected interior event. Unlike the shared temporal criterion, importance scores are layer- and head-specific, allowing different pairs to discard different events. **Table R1.3 (new).** Strided versus importance-aware latent retention on MATH-500 (Qwen3-8B, two seeds). Δ is the seed-averaged accuracy difference, importance-aware minus strided, with 95% CIs from problem-level paired bootstrap. | KV budget |  Strided (seed 0 / 1)  |  Discard lowest (seed 0 / 1)  |  Δ [95% CI]  | |:---:|:---:|:---:|:---:| | 512 | 85.8 / 87.0 | 87.0 / 86.0 | +0.1 [−1.4, +1.6] | | 1024 | 92.0 / 91.8 | 92.0 / 92.4 | +0.3 [−1.0, +1.6] | | 2048 | 94.4 / 94.4 | 94.4 / 93.4 | −0.5 [−1.5, +0.4] | | 3072 | 94.0 / 93.2 | 93.0 / 93.2 | −0.5 [−2.8, +0.6] | The importance-aware rule does not separate from strided retention at any budget: every interval contains zero and the largest absolute difference is 0.5 points. This ablation is reported in **Appendix B.5** of the revised manuscript.
AuthorsOpenReview ↗

Author Response to Reviewer R1-jNZV (Part 3/3)

W3: Rationale and Sensitivity of the Latent-Budget Cap ηmax⁡\eta_{\max}

The 40% cap is mainly introduced to avoid allocating too much KV capacity to latent traces, especially for short contexts. Since DuetKV constructs a fixed number of traces per segment, their relative cost is typically small for long contexts but can become substantial for short ones. This also explains why the cap is rarely triggered in most of our experiments.

1.Effect of the cap. The cap becomes important for short contexts, where a fixed number of traces can occupy a large fraction of the available KV budget. Table R1.4 shows two short-context tasks where the cap becomes relevant. On GSM8K, removing the cap leads to a 21.0-point drop relative to Fast KVzip, while the default cap prevents this degradation. On SQuAD, the cap improves the gain from +2.55 to +3.67 points, despite being triggered on only 6.35% of segments.

Table R1.4 (new). Cap activation and its effect at ρ=0.1\rho=0.1 and ηmax⁡=0.40\eta_{\max}=0.40.

Task Median context nn  M/(nρ)M/(n\rho) at median  Segments refused  No cap →\to ηmax⁡=0.40\eta_{\max}=0.40 
SQuAD178 tokens22.5%6.35%+2.55 → +3.67
GSM8K83 tokens48.2%99.83%−21.00 → +0.00

2.Sensitivity to ηmax⁡\eta_{\max}. We further evaluate the effect of different cap values, ranging from 0.10 to 1.00 (uncapped). We report results under two evaluation sets. The 70-cell setting follows the original evaluation, covering 12 tasks and six KV budgets, with two configurations excluded at (ρ=0.1\rho=0.1). We additionally include GSM8K at (ρ=0.1\rho=0.1), where the protected window is adjusted for short contexts and trace admission is affected by the cap, resulting in a 71-cell setting. The remaining excluded configuration, ICL.ManyShot at (ρ=0.1\rho=0.1), involves no eviction and is therefore unaffected by the cap.

Table R1.5 (new). Sensitivity of the reported results to the latent-budget cap ηmax⁡\eta_{\max}.

ηmax⁡\eta_{\max} 70-cell mean score  Δ\Delta  Cells changed  71-cell mean score  Δ\Delta 
Fast KVzip (reference)49.07——48.85—
1.00 (uncapped)52.83+3.757—52.26+3.408
0.40 (in the paper)52.85+3.773452.57+3.720
0.3552.85+3.772552.57+3.719
0.3052.86+3.781552.58+3.728
0.2552.85+3.773552.57+3.720
0.2052.78+3.708752.50+3.656
0.1052.60+3.5231252.32+3.473

Mean scores are averaged over the corresponding task–budget configurations. Δ\Delta denotes the improvement over Fast KVzip, and Cells changed counts configurations whose scores differ from the uncapped setting. The analysis combines directly measured results with values derived from the admission rule for selected configurations. Details are provided in the new Appendix C.6.

Table R1.5 shows that performance remains stable across (ηmax⁡∈[0.25,0.40]\eta_{\max}\in[0.25,0.40]), with less than 0.01-point variation. Lower thresholds slightly reduce performance by suppressing useful traces. Compared with the uncapped setting, the default cap improves the 70-cell mean by only 0.016 points, but improves the 71-cell mean by 0.311 points, mainly by preventing degradation on short contexts. Together with Table R1.5, these results show that the cap has little effect on overall performance but can prevent substantial degradation in short-context settings. These results support using 0.40 as a relatively permissive threshold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.