- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether the quality degradation caused by low-bit KV-cache quantization can be recovered without modifying the quantizer itself. The proposed approach keeps both the base model and cache format fixed, and trains low-rank adapters on the attention Q/K/V projections through self-distillation from the floating-cache model.
Critical strengths
- The method itself seems to be working well. The experiment results look not bad on several benchmarks.
Critical weaknesses
-
The writing is very hard to understand; in the introduction, a lot of things are not defined. In the methodology section, the author uses techniques from KIVI and KVarN without giving any preliminaries.
-
The paper did not report metrics like latency and throughput. Even I understand this paper is focusing more on the effectiveness of the adapter itself, but I'm still curious about the inference overhead it introduces and also how long it takes to train this adapter. Is it really worth it?
-
Lack of motivation: it's like simply combining quantization and adapter together. Why should this adapter be designed specifically for this quantization scenario?
Questions for authors
- This setting seems to be working on the online updating scenarios. Which means if you see a new data sample, you can use this sample to update the LoRA adapter. This experiment is not required at the rebuttal, but I hope the author can give some discussion and it would be great to see some preliminary testing.
2 comments
Thank you for the effort on rewriting the paper, and also the additional overhead report. I encourage the author to keep refine the paper. I'll keep my original score.
Response to Reviewer aqte
Thank you for the review. The presentation concerns led to a restructured paper.
W1. Writing and preliminaries. We rewrote the paper. Section 2 now introduces the KV cache, the quantization scheme, KIVI and KVarN before they are used; each term is defined at first use, and the method has a schematic. Does the revised version read better?
W2. Overhead and training time; is it worth it?
W3. Motivation. In practice, a deployment rarely chooses its cache format for quality alone. The format has to match an attention kernel that is hardware-accelerated and maintained by the serving framework's community. vLLM, for example, ships a fixed FP8 cache with native kernels motivated by hardware support and reports small quality losses that persist for some models (https://vllm.ai/blog/2026-04-22-fp8-kvcache). Going below 8 bits saves more memory but costs more quality, so cheap recovery under a fixed format is one ingredient for moving this practice to low-bit caches. Adapting the model provides that recovery without adding cache memory. The query, key and value projections are the weights that write and read the cache, so the adapters go there and are trained through the quantized cache.
Q. Online updating. The setting allows it since training is label-free. In a preliminary test (60 label-free updates on three MMLU subjects, 581 held-out examples), no accuracy improvement was observed: 50.6% against 52.3% for the static adapter (−1.72 points, paired 95% interval [−4.48, +1.04]). A tuned study of online schedules is beyond the scope of this paper.