- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates reusable KV cache compression in scenarios where subsequent query conditions are unknown. The proposed method employs low-bit storage to retain a greater number of context positions within a fixed byte budget, and utilizes self-distillation to train low-rank adapters, enabling the model to adapt to a sparse, quantized cache. The authors conduct experiments using various selection criteria and implement fused attention kernels, along with integration into serving systems.
Critical strengths
- The authors distinguish between two sources of gain, which increases reserved slots and improving model adaptation, and further provide controlled experiments comparing actual byte-level matching against fixed reserved slots, making the results easier to analyze.
- The approach includes both the execution of compressed caching and a practical implementation within a serving system, demonstrating that storage savings can be translated into tangible gains in concurrent service capacity, rather than remaining merely a theoretical estimate.
Critical weaknesses
- Most of the performance gains shown in Table 1 stem from retaining more positions after reducing precision, while the contribution from the model adaptation mechanism is relatively minor. Given that existing research already explores the combination of quantization and token eviction, the specific contribution of the adaptation mechanism introduced in this paper requires clearer delineation.
- The adaptation mechanism suffers from significant issues regarding task transferability. As shown in Section 5.6, the LongBench score for Llama (at a 3% cache budget) drops from 41.74, achieved using a frozen 4-bit scheme, to 31.16. Furthermore, no consistent gains are observed on SCBench. This undermines the method’s credibility as a general-purpose, reusable caching solution.
- The evaluation of benefits for practical deployment remains insufficient. An improvement in task scores under a fixed cache budget does not automatically translate to enhanced end-to-end inference efficiency. It is recommended to provide a more comprehensive analysis by comparing metrics such as time-to-first-token (TTFT), decoding throughput, and adaptation costs, accounting for actual VRAM usage (including quantization metadata and adaptation overhead). This would clarify whether the observed performance gains can be effectively realized as tangible system-level benefits.
Questions for authors
- In the context of actual byte-level matching, what are the primary gains and performance regressions of the adaptation module compared to the frozen 4-bit baseline when evaluated on real-world question-answering tasks?
- Can the results from the existing tasks be used to explain the causes of these performance regressions? Furthermore, could you clarify for which specific tasks or data conditions the use of this adaptation module is recommended?
3 comments
Raised my score from 2 to 3
Thanks to the authors for their detailed response. My concerns have been adequately addressed, and I have thus raised my score from 2 to 3 accordingly.
Thank you for the discussion
Thank you for your updated assessment. Your feedback helped us clarify ByteKV’s contribution and strengthen the evaluation. We appreciate your time and engagement.
Thank you for connecting the transfer questions with practical deployment benefits and costs.
W1. What adaptation adds
The frozen 4-bit control already receives the coverage benefit of lower precision. Adaptation then improves accuracy while the retained positions and storage stay fixed (Table 8, KVzip). This shows that adaptation adds value beyond storing more positions. ByteKV trains a shared LoRA adapter to produce and use the sparse, quantized cache, with a frozen full-cache teacher supplying the targets.
W2, Q1, Q2. Transfer, equal storage, and use cases
The new adapters improve real QA scores at unchanged context-KV storage. ByteKV-KVzip uses RestoreKV's released LongAlpaca, PG-19, and FLAN mixture, with one seed per model. Both training data and selector differ from the original KeyDiff experiments, and some training text overlaps evaluation documents. Frozen and adapted 4-bit caches share retained positions, quantizer, and metadata. At a nominal 3% of full 16-bit KV payload, adaptation gains are:
At 3%, the gains extend beyond classification: Qasper, HotpotQA, and MuSiQue improve, and removing TREC leaves mean gains of 1.81 on Llama and 6.26 on Qwen3. Code completion (RepoBench-P) declines on both models. Both benchmark means also improve at 5%, but Llama's LongBench gain there depends on TREC: excluding it gives 41.24 versus frozen 4-bit's 41.35. At 10%, Llama's LongBench mean is 0.89 points below frozen 4-bit.
Document and book QA at tight budgets are the supported use cases. The original corpus trains identifier lookup, while the broader mixture includes semantic matching and answer synthesis. That closer task match offers a plausible explanation for the QA gains. The general response provides the full baseline comparison.
W3. Deployment benefits and costs
Section 5.8 evaluates KeyDiff serving with identical LoRA-merged weights for compressed and full-cache inference. Merging removes separate adapter operations at runtime. Timing fixes context length at 8K and output length at 1024 tokens; accuracy is measured separately. At concurrency 64:
The serving benefit is higher concurrent capacity. Smaller caches let more requests stay resident, reducing queueing before the first token. Inter-token latency rises, while single-request decoding is essentially unchanged. Peak throughput reaches 1.91 times the full-cache peak. KV storage includes metadata, page rounding, and uncompressed question and generated-token KV; weights and workspaces consume additional VRAM.
The new KVzip adapters cost about 12.3 GPU-hours on Llama and 45.2 on Qwen3 to train, excluding score precomputation. Each is trained once and reused across all tested tasks and budgets; no new training is required for each context or question.