- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RDKV, a rate-distortion framework for jointly optimizing token/channel eviction and quantization of the KV cache. The paper argues that eviction and quantization should not be treated as two independent compression operations. Instead, they can be viewed as different points on a single bit-allocation curve, where zero bits correspond to eviction and positive bit-widths correspond to quantized retention.
For the V cache, RDKV derives token importance from the cumulative attention received by each token. For the K cache, it derives channel importance from the perturbation of the attention logits caused by removing a channel. These importance weights are combined with empirical quantization-distortion estimates to formulate a discrete rate-distortion allocation problem over the bit-width set {0, 2, 4, 8, 16}. The resulting multiple-choice allocation problem is approximately solved using Lagrangian relaxation and one-dimensional bisection. The paper also introduces a TriZone packed layout to execute mixed-bit decoding without first expanding the compressed cache back to FP16.
The method is evaluated on LongBench, Needle-in-a-Haystack, RULER, and InfiniteBench using LLaMA-3.1-8B-Instruct as the primary model, with additional experiments on Mistral-7B, Qwen3-4B, LLaMA-2-13B, and Qwen2.5-72B. On LLaMA-3.1-8B-Instruct, RDKV achieves the highest LongBench average among the evaluated compression baselines at all reported budgets. At a budget of 1024L, it obtains 49.47 compared with 49.82 for FullKV, while at a more aggressive budget of 64L it obtains 45.97 compared with 43.23 for SnapKV plus ZipCache. At 128K context length, the paper reports approximately 4.5x lower decoding latency and a reduction from 58.1 GB to 30.5 GB peak memory relative to FullKV FlashAttention-2 decoding.
Critical strengths
-
The paper provides a clean and meaningful unification of KV-cache eviction and quantization. The rate-distortion formulation explains why binary eviction can be too coarse and why quantizing every token can waste bits on low-impact cache units. The action-space ablation supports this motivation: at the 128L budget, the reported LongBench averages are 39.06 for quantization-only, 42.53 for eviction-only, 46.91 for the tri-state configuration, and 47.75 for the full RDKV action space.
-
The paper connects the algorithmic formulation to an executable inference system. The TriZone layout is not merely a theoretical storage scheme: the paper reports approximately 18 ms/token with TriZone versus approximately 36 ms/token without it, and compares RDKV against FullKV and AdaKV on actual A100 hardware. This makes the work more practically relevant than a method evaluated only through reconstructed-cache accuracy.
Critical weaknesses
-
The effective compression budget is not fully comparable to physical memory usage. In the implementation details, the matched budgets are computed from nominal bit-widths while quantization metadata such as scales and zero-points are explicitly excluded for all methods. At the very low budgets emphasized by the paper, these metadata and layout costs may be non-negligible. The paper should report the actual bytes transferred and stored, including all scales, zero-points, masks, padding, indices, and other TriZone metadata.
-
The paper’s default allocation is static over the prefetched context, while the KV cache continues to grow during generation. The dynamic variant is evaluated only as an additional experiment and increases decoding latency from 18.70 to 24.73 ms/token while reducing peak memory only from 30.36 to 30.32 GB. Therefore, the strongest system claims primarily concern long-prefix decoding rather than long-input-and-long-output serving. The limitations of static allocation for long generated sequences should be made more explicit, and end-to-end results with realistic output lengths would strengthen the paper.
Questions for authors
-
In Appendix B, the paper states that scales and zero-points are excluded from the matched-budget accounting. What are the effective physical compression ratios after including all quantization metadata, per-token/per-channel information, padding, masks, and TriZone packing overhead?
-
The additive objective in Eq. (6) treats the distortion caused by each cache unit as separable. The validation in Appendix I uses six held-out 8K prefixes and 41 matched-budget allocations, obtaining Spearman correlations of 0.762 and 0.769 with end-to-end vocabulary KL. Can the authors provide a larger validation across models, context lengths, and task distributions, and report whether the surrogate also predicts downstream task accuracy?
-
How are the 32 calibration sequences drawn from LongBench prefixes separated from the LongBench evaluation examples? Please clarify whether the calibration data are disjoint at the document, task, and test-example levels.
-
The paper uses an equal K/V budget split by default and reports a small sweep over ratios 0.4, 0.5, and 0.6. How sensitive is the allocation to this choice on the other model families, especially LLaMA-2-13B and Qwen2.5-72B?
-
The paper reports that RDKV outperforms the best evaluated baseline by 9.1% on average. Please provide the exact aggregation procedure for this number, including the benchmarks, budgets, models, and whether the improvement is absolute or relative.
-
How much additional computation is required to calculate the probe attention weights, empirical distortion tables, per-head allocation, and TriZone packing during prefill? The paper reports a 6% prefill overhead at 128K, but a breakdown for shorter contexts and different batch sizes would clarify the deployment trade-off.
-
Why is HqeKV included only in an additional RULER comparison rather than in the main LongBench and system comparisons? Please discuss whether the baseline implementation or budget definition prevents a direct comparison.
3 comments
Response to Reviewer 3uQu [Part 2]
Q2 — Additive Surrogate Validation
Following your suggestion, we extend the additive surrogate validation across additional models, context lengths, and task distributions. We also examine its relationship with downstream task accuracy.
Extended Validation. We extend the evaluation in Appendix I to LLaMA-3.1-8B and Qwen3-4B across 8K–32K contexts, two cache budgets (128L and 512L), and five task categories from LongBench and RULER. Following the same 41-allocation validation protocol, the surrogate maintains Spearman correlations of approximately 0.75 with end-to-end KL, consistent with the original results (0.762/0.769).
| Model | Prefixes | ρ (128L) | CV R² (128L) | ρ (512L) | CV R² (512L) |
|---|---|---|---|---|---|
| LLaMA-3.1-8B | 72 | 0.747 | 0.967 | 0.747 | 0.917 |
| Qwen3-4B | 72 | 0.755 | 0.946 | 0.754 | 0.901 |
| Combined | 144 | 0.751 | 0.953 | 0.753 | 0.909 |
Surrogate–downstream accuracy validation. We further evaluate LLaMA-3.1-8B on all 11 RULER tasks at 8K, 16K, and 32K under two cache budgets, using 100 samples per task. We compare ten allocation variants and measure the Spearman correlation between surrogate distortion and downstream accuracy. The experimental results are shown below.
| Method | 8K / 128L | 8K / 512L | 16K / 128L | 16K / 512L | 32K / 128L | 32K / 512L |
|---|---|---|---|---|---|---|
| RDKV (Heuristic) | 71.7 | 90.5 | 69.7 | 85.2 | 66.4 | 79.5 |
| RDKV (MCKP) | 73.9 | 94.7 | 72.6 | 90.0 | 67.6 | 83.2 |
We report correlations across all ten allocations, eight allocations sharing the same K/V budget split, and six progressively perturbed allocations.
| Allocation Set | 8K / 128L | 8K / 512L | 16K / 128L | 16K / 512L | 32K / 128L | 32K / 512L | Mean |
|---|---|---|---|---|---|---|---|
| All 10 allocations | −0.81 | −0.70 | −0.79 | −0.76 | −0.84 | −0.79 | −0.78 |
| Same K/V split (8) | −0.90 | −0.83 | −0.90 | −0.83 | −0.90 | −0.90 | −0.88 |
| Graded perturbations (6) | −0.94 | −0.94 | −0.94 | −0.94 | −0.94 | −0.94 | −0.94 |
The consistently negative correlations indicate that lower surrogate distortion is associated with higher downstream accuracy across all six settings.
Q3 — Calibration Data Separation
Thank you for raising this question. We audited the calibration protocol and conducted additional experiments to verify the impact of calibration data separation.
The original LongBench calibration pipeline did not explicitly enforce disjointness from evaluation data. To address this, we constructed a new calibration set using 32 sequences of 16K tokens from WikiText-2 train. We verified that these sequences have no exact or near-duplicate matches with the 14,750 evaluation inputs under our overlap criteria. Calibration uses only KV reconstruction errors, without evaluation labels or scores.
We then repeated downstream evaluations, changing only the calibration table.
| Benchmark | Original | Independent | Δ (pp) |
|---|---|---|---|
| LongBench 128L | 47.82 | 47.71 | -0.12 |
| LongBench 512L | 49.21 | 49.26 | +0.05 |
| RULER 16K | 95.66 | 95.30 | -0.36 |
| RULER 64K | 80.06 | 79.76 | -0.30 |
The performance differences are within 0.36 percentage points across all four settings. These findings are consistent with the calibration robustness results in Appendix J, which further evaluates cross-corpus, cross-model, and calibration-free variants. We will clarify the calibration protocol and include the new data-isolation results in the revised manuscript.
Response to Reviewer 3uQu [Part 3]
Q4 — Cross-Model K/V Budget Split Sensitivity
Following your suggestion, we extend the K/V budget split ablation to LLaMA-2-13B-Chat and Qwen2.5-32B-Instruct across three cache budgets.
| Model | Budget | |||
|---|---|---|---|---|
| LLaMA-2-13B | 128L | 29.42 | 29.46 | 29.11 |
| LLaMA-2-13B | 256L | 29.81 | 29.71 | 29.71 |
| LLaMA-2-13B | 512L | 30.10 | 29.96 | 29.78 |
| Qwen2.5-32B | 128L | 45.87 | 46.37 | 46.02 |
| Qwen2.5-32B | 256L | 47.14 | 47.72 | 47.12 |
| Qwen2.5-32B | 512L | 48.03 | 48.45 | 48.18 |
The optimal split varies slightly across models. LLaMA-2-13B favors at larger budgets, while Qwen2.5-32B consistently favors . Overall, the equal split performs best in four of six settings and remains within 0.14 accuracy points (less than 0.5% relative) of the best tested ratio in the remaining two. We therefore recommend as a robust default without model-specific tuning.
Since the original GH200 environment is no longer accessible, all configurations, including , were rerun under identical settings on A100 GPUs to ensure consistent comparisons. Due to computational and rebuttal time constraints, we evaluate Qwen2.5-32B instead of Qwen2.5-72B and will include the 72B results in the revised manuscript.
Q5 — Aggregation of the 9.1% Improvement
The 9.1% is a relative improvement, averaged over 12 settings on LLaMA-3.1-8B-Instruct (LongBench at five budgets, RULER at six lengths, and InfiniteBench). The reference is ThinK, the baseline with the highest average score across these settings (61.31, vs. 60.73 AdaKV, 60.60 SnapKV, 59.34 SnapKV+ZipCache). We compute the relative gain over ThinK in each setting and take the mean. We will state this clarification in the revision.
Q6 — Prefill Computation and Overhead
Following your question, we extend the prefill overhead evaluation across context lengths, batch sizes, and storage-optimized implementations on LLaMA-3.1-8B-Instruct (A100-64GB, 1024L budget).
Context-length sensitivity. With repeated measurements (median of 3 runs, HF generate FullKV baseline as in Appendix H.1), RDKV adds 4.8% to TTFT at 128K and batch size 1, or 7.4% against the faster 8K-chunked FullKV prefill. The overhead increases to 12% at 64K and 31% at 32K. Table 12 reports 6.0% at 128K from an earlier single measurement.
Batch-size sensitivity. Measured against the FullKV forward pass in a separate sweep at 128K, the overhead decreases slightly from 5.2% to 4.5% as the batch size grows from 1 to 4. FullKV cannot accommodate batch size 4 on the same GPU.
We will update Table 12 with the repeated measurements and add the context-length and batch-size sweeps in the revision.
Storage optimization. We additionally evaluate Compacted RDKV against the submitted implementation under identical conditions. The compacted implementation reduces KV-cache storage from 208.4 MiB to 137.6 MiB (−33.97%), with only 0.5% additional TTFT and 1.02% additional decode latency. Both implementations produce identical generated outputs. The compacted RDKV implementation is included in the updated supplementary code submitted with this rebuttal.
Q7 — HqeKV Baseline Coverage
We previously compared RDKV with HqeKV only on RULER because its fixed context lengths allow budget matching. On LongBench, varying input lengths and discrete allocation granularity prevent HqeKV's fixed compression ratios from exactly matching RDKV's fixed bit budget across examples. Following your suggestion, we extend the comparison with HqeKV to system performance.
We evaluate both methods on the same A100-64GB using LLaMA-3.1-8B-Instruct, batch size 1, context lengths from 16K to 128K, and 1,024 generated tokens. Both methods use their optimized decoding implementations, with inference-time compression overhead included.
| Metric | RDKV | HqeKV |
|---|---|---|
| Decode latency | 18–20 ms/token | 49–62 ms/token |
| End-to-end latency | 21–49 s | 52–92 s |
| KV-cache storage | 137–138 MiB | 220–257 MiB |
Under nominally matched bit budgets, RDKV achieves 2.7–3.3× faster decoding and 1.9–2.7× end-to-end speedup, with comparable TTFT. HqeKV also uses 1.6–1.9× more physical prompt KV storage, mostly quantization metadata. Even when HqeKV's physical storage is reduced to match RDKV's at 16K–64K, its decoding latency remains 50–56 ms/token.
With these responses, we hope we have addressed your concerns. If so, we would be grateful if you would consider raising your score, and we are glad to clarify anything further. We will add the relevant clarifications and additional results to the revised manuscript, following your helpful comments. Thank you again for your time and thoughtful consideration.
Response to Reviewer 3uQu [Part 1]
We would like to sincerely thank Reviewer 3uQu for providing a detailed review and insightful suggestions.
Following your suggestion, we measured the effective compression ratios with overhead counted. Including scales, zero-points, indices, padding and allocation rounding, the effective compression ratio at 128K context and 1024L budget is 89.0× for the submitted implementation with mask removal. Our further compaction raises it to 119.1×, compared with 128× when metadata is ignored. The storage breakdown at 1024L (MiB, averaged over 100 RULER inputs, 50 each at 16K and 64K) is:
The original FP32 padding mask is redundant and has been removed without changing the output, so masks contribute no storage after the revision.
More details, including the benchmark comparison under matched KV-cache storage and the impact on decoding latency, are provided in the Details of Revision 1 under the Summary of Revisions and Global Response. The results of each task and the nominal-budget mapping are added in Appendix N of the revised PDF. The updated implementation of both versions is included in the revised supplementary material.
Following your suggestion, we extend our evaluation to long-output serving to assess the end-to-end efficiency and overhead of periodic re-budgeting. We evaluate Blockwise RDKV with a re-budgeting interval of M=256 against FullKV (FA2) on LLaMA-3.1-8B-Instruct, using a 120K-token input and generating 1K, 2K, and 4K tokens under Btotal=1024L.
Across all output lengths, Blockwise RDKV achieves 2.09–2.33× end-to-end speedup and 1.65–1.67× peak-memory reduction over FullKV. Periodic compression accounts for only 8–9% of total execution time, including all re-budgeting and packing operations.
As output length increases, TPOT rises from 23.11 to 33.02 ms due to periodic compression and the growing number of compressed blocks processed during decoding. However, RDKV maintains clear end-to-end gains at all tested length of output.
We will incorporate these results into the revised manuscript and clarify the limitations of long-output serving.