- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents iKVC, a training-free inference method intended to mitigate accuracy degradation in post-training-quantized LLMs during long-horizon generation. The method periodically uses a full-precision model to recompute KV representations for recently generated tokens, replaces the corresponding entries in the quantized model’s KV cache, and generates a bonus token from the full-precision model. The correction interval controls the trade-off between accuracy and throughput. The paper motivates the method with empirical measurements of KV-cache error and theoretical bounds relating query/key/value deviations to attention-output error. Experiments compare iKVC with PTQ alone and IntactKV for HQQ, GPTQ, and AutoRound, primarily on AIME2025, and report results for several Qwen model sizes. The paper also measures throughput on an RTX 3080 Ti under a CPU-offloading setup.
Critical strengths
- The paper examines an important failure mode that is not addressed by correcting only the prompt KV cache: quantization errors can also affect KV entries generated during decoding. The proposed periodic correction is a clear, intuitive extension of prompt-time KV correction to generated tokens.
- The evaluation considers multiple PTQ methods (HQQ, GPTQ, and AutoRound), several correction intervals, and more than one model size. The throughput analysis also makes the cost of invoking the full-precision model explicit rather than presenting accuracy recovery in isolation.
Critical weaknesses
- The theoretical analysis does not fully establish the claimed practical accumulation effect. Lemma 2 lower-bounds the expected squared Frobenius norm of the cache error by summing assumed per-token lower bounds; this follows directly from the definition of the Frobenius norm, but it does not show that the error grows in a way that is consequential for model outputs. Lemma 3 is an upper bound, and the stated comparison in Proposition 1 is restricted to correction interval K=1. The theoretical results therefore provide limited support for the general accuracy claims at larger intervals.
- The accuracy evaluation is narrow for a paper about long-horizon generation: the main reported benchmark is AIME2025, which has only 30 questions, averaged over three runs. Results on additional task types and longer generations would be needed to establish that the gains are robust rather than benchmark- or sampling-specific.
- iKVC requires access to and repeated execution of a full-precision model in addition to the quantized model. In the reported memory-constrained setup, this involves CPU offloading and incurs substantial overhead, especially at short correction intervals. The practical benefit should be compared against deployment-relevant alternatives under matched hardware, memory budgets, and quality targets; the current comparison to standalone PTQ does not fully establish when this additional model and transfer cost is worthwhile.
Questions for authors
- Can the authors report accuracy and throughput across a broader set of long-generation tasks, with more evaluation examples and confidence intervals, while sweeping the correction interval? This would help determine whether the AIME2025 gains generalize.
- For correction intervals K greater than 1, what theoretical guarantee, if any, can be established? In particular, can the authors quantify how much error remains in the uncorrected KV-cache window and connect that quantity to downstream task accuracy?
- Can the authors compare iKVC against strong deployment baselines under the same hardware and memory budget, including the cost of loading/offloading and running the full-precision model? Please clarify the settings in which iKVC improves the accuracy-throughput trade-off over using the quantized model alone or a different serving strategy.
- How does iKVC behave when the quantized model’s generated tokens diverge from those of the full-precision model? Does recomputing KV representations for the same token sequence correct the relevant errors even when the hidden states at intermediate layers differ between the two models?
4 comments
I thank the authors for a careful reply. The answer to Q4 is clear and, in my view, correct: recomputing KV representations for the realized sequence removes the cache-side discrepancy without claiming sequence-level equivalence, and stating that residual error from quantized FFN and LayerNorm remains is the right framing. The extension of Proposition 1 to K > 1 in Appendix B.1 is welcome, though as the authors themselves note it bounds attention-output error rather than downstream accuracy, so my soundness concern is now about scope rather than correctness. The resource accounting in Q3 is more informative than Section 5 of the submitted version, and I accept that a larger correction interval can offset the cost of the auxiliary full-precision pass. What decides my score is the evaluation. The central claim concerns long-horizon generation, and the evidence is still AIME2025 with 30 questions over three runs. The authors state that analogous long-horizon experiments are underway; until those exist, robustness of the gains across task types remains unestablished, and the comparison still lacks a serving alternative at matched memory and quality. I therefore keep my rating at 2 (weak rejection), while noting that a revision adding a second long-generation benchmark with confidence intervals would materially change my assessment.
We sincerely thank you for your thoughtful comments and positive assessment. We are now conducting accuracy and throughput measurements on the BIG-Bench Extra Hard benchmark (https://arxiv.org/abs/2502.19187). It may be just before the end of the discussion period that all the results become available, but we would appreciate your patience in the meantime.
Dear Reviewer utzM,
To further evaluate the generalizability of iKVC beyond AIME2025, we conducted additional experiments on BIG-Bench Extra Hard (BBEH) using three PTQ methods (HQQ, GPTQ, and AutoRound). Could you please see below? We apologize for the delay and hope this addresses your concern.
Sincerely,
Additional Results on BIG-Bench Extra Hard (BBEH)
The results demonstrate that iKVC controls the accuracy–throughput trade-off by adjusting the KV-cache correction interval. Although accuracy varies non-monotonically with , throughput consistently increases as grows.
Due to time constraints, accuracy evaluations for AutoRound at are still ongoing. We expect similar improvements over PTQ alone, as observed with HQQ and GPTQ, and plan to incorporate the remaining results into the revised manuscript once they are available.
Table: BBEH accuracy and decode throughput.
| PTQ method | Configuration | Accuracy (%) ↑ | Throughput (tokens/s) ↑ |
|---|---|---|---|
| — | Vanilla (BF16) | 24.00 | 0.621 |
| HQQ | PTQ only | 18.00 | 41.195 |
| HQQ | IntactKV w/ training | 20.67 | — |
| HQQ | IntactKV w/o training | 22.67 | — |
| HQQ | iKVC, K=1 | 23.33 | 0.412 |
| HQQ | iKVC, K=20 | 23.33 | 3.883 |
| HQQ | iKVC, K=40 | 25.33 | 6.871 |
| HQQ | iKVC, K=100 | 23.33 | 13.590 |
| HQQ | iKVC, K=200 | 20.00 | 20.211 |
| HQQ | iKVC, K=400 | 24.67 | 26.710 |
| GPTQ | PTQ only | 16.00 | 18.926 |
| GPTQ | IntactKV w/ training | 14.00 | — |
| GPTQ | IntactKV w/o training | 20.67 | — |
| GPTQ | iKVC, K=1 | 22.67 | 0.351 |
| GPTQ | iKVC, K=20 | 18.67 | 3.097 |
| GPTQ | iKVC, K=40 | 20.00 | 5.190 |
| GPTQ | iKVC, K=100 | 24.00 | 8.960 |
| GPTQ | iKVC, K=200 | 18.67 | 11.960 |
| GPTQ | iKVC, K=400 | 20.67 | 15.009 |
| AutoRound | PTQ only | 20.00 | 19.162 |
| AutoRound | IntactKV w/ training | 20.00 | — |
| AutoRound | IntactKV w/o training | 19.33 | — |
| AutoRound | iKVC, K=1 | 24.00 | 0.353 |
| AutoRound | iKVC, K=20 | 20.00 | 3.128 |
| AutoRound | iKVC, K=40 | 26.00 | 5.246 |
| AutoRound | iKVC, K=100 | Ongoing | 9.136 |
| AutoRound | iKVC, K=200 | Ongoing | 12.187 |
| AutoRound | iKVC, K=400 | Ongoing | 14.644 |
We sincerely thank you for recognizing the significance of our work in the field of quantization and our straightforward approach. We also appreciate your feedback, which has helped us strengthen the justification and presentation of our method.
Weakness 2, Question1: The evaluation scope is limited.
Short comment. We acknowledge the limitation of the evaluation scope, and analogous experiments on a long-horizon task beyond AIME are underway. Results will be shared once available.
For further detail, our AIME2025 evaluation used a maximum generation length of 32,768 tokens (the official maximum token length of Qwen3-8B) per question across multiple runs, allowing substantial generation lengths.
Weakness 1, Question 2: Lack of general guarantees if iKVC could suppress the upper bound of attention-output error for KV-cache correction intervals K>1.
Short comment. We have added Appendix B.1 to extend Proposition 1 to K≥1 in the revised manuscript.
For further detail, Appendix B.1 presents the generalized form of Proposition 1 for K≥1. Please note that this extension concerns upper bounds of attention-output errors rather than a guarantee of improved downstream task accuracy, which we have already assessed empirically in the submitted version.
Weakness 3, Question 3: The effects of additional full-precision model and CPU-GPU transfer may require justification through deployment-relevant comparisons under matched resource budgets and quality targets.
Short comment. Section 5 (“Experiments”) in the submitted version has already demonstrated that the throughput overhead of the additional full-precision model can be offset by increasing the KV-cache correction interval.
For further detail, the throughput measurements use the same RTX 3080 Ti system. The full-precision model’s parameter-placement limits are 11 GiB of VRAM and 12 GiB of CPU RAM for standalone full-precision inference, versus 3 GiB and 16 GiB, respectively, for its correction role in PTQ+iKVC, leaving GPU capacity for the quantized model. This means PTQ+iKVC throughput can be closer to that of the standalone PTQ model, even with tighter VRAM budgets than full-precision standalone inference. Our toy model (Section 4.2) also estimates the whole-throughput overhead well.
Question 4: Divergent token prediction or intermediate hidden states between PTQ-quantized and full-precision models may limit how much iKVC can recover accuracy.
Short comment. iKVC recomputes KV representations for the same realized token sequence used by the quantized model, never reversing previous token choices. The replaced KV cache does not inherit the quantized model’s intermediate hidden-state errors for those positions.
For further detail, both models are conditioned on the same evolving sequence, including tokens generated by the quantized model and full-precision bonus tokens. The full-precision model does not independently generate a competing sequence. If the realized sequence differs from what the full-precision model would have predicted, iKVC computes the full-precision KV representations conditional on that realized sequence. It does not directly recover the counterfactual sequence.
During KV-cache correction, the full-precision model computes its own intermediate hidden states using its full-precision weights and retained full-precision prompt cache. Consequently, the replaced KV cache does not inherit the quantized model’s intermediate hidden-state errors for those positions. However, subsequent quantized decoding continues to use its existing weights, including quantized FFN, LayerNorm, and query-projection weights where applicable. Errors arising from these computations can therefore remain in the current hidden states, and newly computed KV representations can introduce further errors before the next correction.
Thus, iKVC removes the discrepancy between the corrected KV cache and full-precision computation on the same sequence, rather than making the entire decoding computation equivalent to full precision.