acceptodds
Under review as a conference paper at ICLR 2027

iKVC: Inference-Time KV-Cache Correction for Quantized LLMs in Long-Horizon Tasks

Abstract

Post-training quantization (PTQ) reduces the memory footprint of large language models (LLMs) and can improve inference throughput. However, the resulting quantization errors can significantly degrade accuracy, creating a fundamental trade-off between the efficiency gained from quantization and the accuracy retained from the full-precision model, particularly in long-horizon generation. Prior work mitigates this degradation by preserving the KV cache of selected prompt tokens at full precision, thereby reducing errors introduced during prefill. This strategy, however, does not address errors introduced during decoding, as quantized weights continuously perturb the key and value representations of newly generated tokens, and these errors accumulate in the KV cache, altering subsequent predictions. We empirically observe that this error accumulation becomes non-negligible as a large number of tokens are generated, and we theoretically derive that the accumulated KV-cache errors can cause the attention output to deviate from the values computed by the original full-precision model. Motivated by this finding, we propose inference-time KV-cache Correction (iKVC), a training-free inference method that periodically corrects the KV cache for both prompt and generated tokens before decoding errors substantially accumulate. The correction interval is configurable, allowing iKVC to explicitly control the accuracy–throughput trade-off. Across experiments, iKVC consistently improves the trade-off over PTQ alone in long-horizon tasks. Our results identify KV-cache error accumulation as an important, previously underexplored factor in achieving accurate and efficient long-horizon tasks with quantized LLMs, suggesting that managing such errors during inference can complement PTQ techniques.

Primary area:
generative models
Keywords:
LLM, Quantization, KV cache

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_utzMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents iKVC, a training-free inference method intended to mitigate accuracy degradation in post-training-quantized LLMs during long-horizon generation. The method periodically uses a full-precision model to recompute KV representations for recently generated tokens, replaces the corresponding entries in the quantized model’s KV cache, and generates a bonus token from the full-precision model. The correction interval controls the trade-off between accuracy and throughput. The paper motivates the method with empirical measurements of KV-cache error and theoretical bounds relating query/key/value deviations to attention-output error. Experiments compare iKVC with PTQ alone and IntactKV for HQQ, GPTQ, and AutoRound, primarily on AIME2025, and report results for several Qwen model sizes. The paper also measures throughput on an RTX 3080 Ti under a CPU-offloading setup.

Critical strengths
  1. The paper examines an important failure mode that is not addressed by correcting only the prompt KV cache: quantization errors can also affect KV entries generated during decoding. The proposed periodic correction is a clear, intuitive extension of prompt-time KV correction to generated tokens.
  2. The evaluation considers multiple PTQ methods (HQQ, GPTQ, and AutoRound), several correction intervals, and more than one model size. The throughput analysis also makes the cost of invoking the full-precision model explicit rather than presenting accuracy recovery in isolation.
Critical weaknesses
  1. The theoretical analysis does not fully establish the claimed practical accumulation effect. Lemma 2 lower-bounds the expected squared Frobenius norm of the cache error by summing assumed per-token lower bounds; this follows directly from the definition of the Frobenius norm, but it does not show that the error grows in a way that is consequential for model outputs. Lemma 3 is an upper bound, and the stated comparison in Proposition 1 is restricted to correction interval K=1. The theoretical results therefore provide limited support for the general accuracy claims at larger intervals.
  2. The accuracy evaluation is narrow for a paper about long-horizon generation: the main reported benchmark is AIME2025, which has only 30 questions, averaged over three runs. Results on additional task types and longer generations would be needed to establish that the gains are robust rather than benchmark- or sampling-specific.
  3. iKVC requires access to and repeated execution of a full-precision model in addition to the quantized model. In the reported memory-constrained setup, this involves CPU offloading and incurs substantial overhead, especially at short correction intervals. The practical benefit should be compared against deployment-relevant alternatives under matched hardware, memory budgets, and quality targets; the current comparison to standalone PTQ does not fully establish when this additional model and transfer cost is worthwhile.
Questions for authors
  1. Can the authors report accuracy and throughput across a broader set of long-generation tasks, with more evaluation examples and confidence intervals, while sweeping the correction interval? This would help determine whether the AIME2025 gains generalize.
  2. For correction intervals K greater than 1, what theoretical guarantee, if any, can be established? In particular, can the authors quantify how much error remains in the uncorrected KV-cache window and connect that quantity to downstream task accuracy?
  3. Can the authors compare iKVC against strong deployment baselines under the same hardware and memory budget, including the cost of loading/offloading and running the full-precision model? Please clarify the settings in which iKVC improves the accuracy-throughput trade-off over using the quantized model alone or a different serving strategy.
  4. How does iKVC behave when the quantized model’s generated tokens diverge from those of the full-precision model? Does recomputing KV representations for the same token sequence correct the relevant errors even when the hidden states at intermediate layers differ between the two models?
4 comments

We sincerely thank you for recognizing the significance of our work in the field of quantization and our straightforward approach. We also appreciate your feedback, which has helped us strengthen the justification and presentation of our method.

Weakness 2, Question1: The evaluation scope is limited.

Short comment. We acknowledge the limitation of the evaluation scope, and analogous experiments on a long-horizon task beyond AIME are underway. Results will be shared once available.

For further detail, our AIME2025 evaluation used a maximum generation length of 32,768 tokens (the official maximum token length of Qwen3-8B) per question across multiple runs, allowing substantial generation lengths.

Weakness 1, Question 2: Lack of general guarantees if iKVC could suppress the upper bound of attention-output error for KV-cache correction intervals K>1K > 1.

Short comment. We have added Appendix B.1 to extend Proposition 1 to K≥1K \geq 1 in the revised manuscript.

For further detail, Appendix B.1 presents the generalized form of Proposition 1 for K≥1K \geq 1. Please note that this extension concerns upper bounds of attention-output errors rather than a guarantee of improved downstream task accuracy, which we have already assessed empirically in the submitted version.

Weakness 3, Question 3: The effects of additional full-precision model and CPU-GPU transfer may require justification through deployment-relevant comparisons under matched resource budgets and quality targets.

Short comment. Section 5 (“Experiments”) in the submitted version has already demonstrated that the throughput overhead of the additional full-precision model can be offset by increasing the KV-cache correction interval.

For further detail, the throughput measurements use the same RTX 3080 Ti system. The full-precision model’s parameter-placement limits are 11 GiB of VRAM and 12 GiB of CPU RAM for standalone full-precision inference, versus 3 GiB and 16 GiB, respectively, for its correction role in PTQ+iKVC, leaving GPU capacity for the quantized model. This means PTQ+iKVC throughput can be closer to that of the standalone PTQ model, even with tighter VRAM budgets than full-precision standalone inference. Our toy model (Section 4.2) also estimates the whole-throughput overhead well.

Question 4: Divergent token prediction or intermediate hidden states between PTQ-quantized and full-precision models may limit how much iKVC can recover accuracy.

Short comment. iKVC recomputes KV representations for the same realized token sequence used by the quantized model, never reversing previous token choices. The replaced KV cache does not inherit the quantized model’s intermediate hidden-state errors for those positions.

For further detail, both models are conditioned on the same evolving sequence, including tokens generated by the quantized model and full-precision bonus tokens. The full-precision model does not independently generate a competing sequence. If the realized sequence differs from what the full-precision model would have predicted, iKVC computes the full-precision KV representations conditional on that realized sequence. It does not directly recover the counterfactual sequence.

During KV-cache correction, the full-precision model computes its own intermediate hidden states using its full-precision weights and retained full-precision prompt cache. Consequently, the replaced KV cache does not inherit the quantized model’s intermediate hidden-state errors for those positions. However, subsequent quantized decoding continues to use its existing weights, including quantized FFN, LayerNorm, and query-projection weights where applicable. Errors arising from these computations can therefore remain in the current hidden states, and newly computed KV representations can introduce further errors before the next correction.

Thus, iKVC removes the discrepancy between the corrected KV cache and full-precision computation on the same sequence, rather than making the entire decoding computation equivalent to full precision.

I thank the authors for a careful reply. The answer to Q4 is clear and, in my view, correct: recomputing KV representations for the realized sequence removes the cache-side discrepancy without claiming sequence-level equivalence, and stating that residual error from quantized FFN and LayerNorm remains is the right framing. The extension of Proposition 1 to K > 1 in Appendix B.1 is welcome, though as the authors themselves note it bounds attention-output error rather than downstream accuracy, so my soundness concern is now about scope rather than correctness. The resource accounting in Q3 is more informative than Section 5 of the submitted version, and I accept that a larger correction interval can offset the cost of the auxiliary full-precision pass. What decides my score is the evaluation. The central claim concerns long-horizon generation, and the evidence is still AIME2025 with 30 questions over three runs. The authors state that analogous long-horizon experiments are underway; until those exist, robustness of the gains across task types remains unestablished, and the comparison still lacks a serving alternative at matched memory and quality. I therefore keep my rating at 2 (weak rejection), while noting that a revision adding a second long-generation benchmark with confidence intervals would materially change my assessment.

We sincerely thank you for your thoughtful comments and positive assessment. We are now conducting accuracy and throughput measurements on the BIG-Bench Extra Hard benchmark (https://arxiv.org/abs/2502.19187). It may be just before the end of the discussion period that all the results become available, but we would appreciate your patience in the meantime.

Dear Reviewer utzM,

To further evaluate the generalizability of iKVC beyond AIME2025, we conducted additional experiments on BIG-Bench Extra Hard (BBEH) using three PTQ methods (HQQ, GPTQ, and AutoRound). Could you please see below? We apologize for the delay and hope this addresses your concern.

Sincerely,


Additional Results on BIG-Bench Extra Hard (BBEH)

The results demonstrate that iKVC controls the accuracy–throughput trade-off by adjusting the KV-cache correction interval. Although accuracy varies non-monotonically with KK, throughput consistently increases as KK grows.

Due to time constraints, accuracy evaluations for AutoRound at K∈{100,200,400}K \in \{100,200,400\} are still ongoing. We expect similar improvements over PTQ alone, as observed with HQQ and GPTQ, and plan to incorporate the remaining results into the revised manuscript once they are available.

Table: BBEH accuracy and decode throughput.

PTQ methodConfigurationAccuracy (%) ↑Throughput (tokens/s) ↑
—Vanilla (BF16)24.000.621
HQQPTQ only18.0041.195
HQQIntactKV w/ training20.67—
HQQIntactKV w/o training22.67—
HQQiKVC, K=123.330.412
HQQiKVC, K=2023.333.883
HQQiKVC, K=4025.336.871
HQQiKVC, K=10023.3313.590
HQQiKVC, K=20020.0020.211
HQQiKVC, K=40024.6726.710
GPTQPTQ only16.0018.926
GPTQIntactKV w/ training14.00—
GPTQIntactKV w/o training20.67—
GPTQiKVC, K=122.670.351
GPTQiKVC, K=2018.673.097
GPTQiKVC, K=4020.005.190
GPTQiKVC, K=10024.008.960
GPTQiKVC, K=20018.6711.960
GPTQiKVC, K=40020.6715.009
AutoRoundPTQ only20.0019.162
AutoRoundIntactKV w/ training20.00—
AutoRoundIntactKV w/o training19.33—
AutoRoundiKVC, K=124.000.353
AutoRoundiKVC, K=2020.003.128
AutoRoundiKVC, K=4026.005.246
AutoRoundiKVC, K=100Ongoing9.136
AutoRoundiKVC, K=200Ongoing12.187
AutoRoundiKVC, K=400Ongoing14.644
Reviewer_K2wvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes periodically refreshing the KV cache of a quantized language model using its full-precision counterpart. After the quantized model generates a block of tokens, the BF16 model recomputes their cached representations and supplies a bonus token; BF16 is also used for prompt prefill. The method requires no additional training and trades correction frequency against inference cost. Experiments evaluate several Qwen model sizes and quantizers on AIME2025, with a separate consumer-GPU throughput study. The results show that this approach can recover accuracy in some configurations, but its practical advantage depends strongly on the correction interval and quantizer.

Critical strengths
  1. The method is easy to understand and provides a straightforward way to improve an already quantized model without retraining it. The correction interval is an interpretable control over the amount of full-precision computation, and the evaluation considers several quantization methods rather than one implementation.
  2. There are meaningful accuracy gains in some tested configurations, particularly with HQQ. The throughput evaluation also identifies a plausible use case in which full-precision weights must be offloaded and correction can still be faster than the measured offloaded full-precision baseline. These results deserve credit despite the uneven gains across configurations.
Critical weaknesses
  1. My main reservation is the significance of the contribution. Periodically using the full-precision model to refresh the quantized model's intermediate state is a fairly direct construction. Simplicity can be a strength, but here I do not find the experimental evidence sufficiently compelling to turn this narrow intervention into a substantial contribution. The paper would be stronger if it established more clearly what practically important capability or operating regime this design enables beyond spending additional full-precision computation to recover quality.
  2. The empirical support is limited and uneven. Accuracy is evaluated on AIME2025 with models from the Qwen family, and the gains vary considerably across model sizes and quantizers. This limits the evidence for broad relevance to long-horizon tasks. Correction also requires keeping the full-precision model available and can impose substantial throughput overhead. Larger correction intervals reduce that overhead, but the accuracy and throughput evaluations do not fully characterize the same operating points. Consequently, the results do not yet establish a convincing, broadly attractive quality–cost tradeoff.
Questions for authors
  1. Which deployment regime do the authors consider the strongest justification for iKVC, and what is the most informative matched accuracy–throughput comparison supporting that use case?
  2. Which results best support the intended scope beyond AIME-style reasoning, and how would the authors distinguish the main contribution from a straightforward allocation of additional full-precision inference compute?
4 comments

We sincerely thank you for recognizing the clear problem formulation, straightforward approach, and the significance of our work in the field of quantization. We also appreciate your constructive feedback, which has helped us strengthen the justification and presentation of our method.

Weakness1, Question1: The intended deployment regime and the matched accuracy–throughput comparison supporting it need clarification.

Short comment. We agree the submitted version lacks the explanations for the above. We have clarified them in Section 1 ("Introduction") and Section 5 ("Experiments") in the revised manuscript. We also aligned the accuracy and throughput evaluations (Figures 4 & 5) across correction intervals in the revised manuscript.

For further detail, iKVC aims to enable users to achieve high inference performance equivalent to an LLM, even in memory-constrained environments where an LLM cannot remain resident in GPU VRAM. We consider an RTX 3080 Ti as such a regime. A more practical deployment strategy is to save the full-precision model on a cloud GPU and keep the quantized model local. In this case, the full-precision model will be called only at KV-cache correction intervals, allowing it to perform KV-cache correction for other jobs during the waiting time. However, due to resource limitations, we cannot test this deployment regime.

To control the accuracy-throughput tradeoff, the most useful correction interval depends entirely on which aspect (accuracy or throughput) the user considers more important. We updated Figures 4 and 5 to use a common set of intervals, K∈1,20,40,100,200,400K \in {1, 20, 40, 100, 200, 400} under matching generation settings in the revised manuscript. Please check.

We have clarified these aspects above in Section 1 (“Introduction”) and Section 5 ("Experiments").

Weakness 2, Question2: Evidence beyond AIME-style reasoning and the contribution of KV-cache correction beyond bonus-token generation for accuracy recovery may need to be clearer.

Short comment. We acknowledge the limitation of the evaluation scope, and analogous experiments on a long-horizon task beyond AIME are underway. We also agree that the current results do not isolate the effects of bonus-token generation and KV cache correction, and an ablation without bonus-token generation is underway. Results will be shared once available.

For further detail, bonus-token generation is introduced primarily to improve efficiency by reusing the next-token distribution of the full-precision model that was already computed during KV-cache correction, rather than to recover task accuracy. The ablation will disable this generation step while retaining prompt cache initialization and periodic KV-cache correction. Comparing it with standard iKVC will assess the incremental accuracy-recovery effect of bonus-token generation and whether accuracy recovery persists without it.

Thank you for the clarifications and the additional Qwen3-14B results. The new experiment is helpful, but the gains appear sensitive to the choice of K, and my concerns about correction overhead and generalization beyond AIME remain. I will therefore maintain my current rating.

Dear Reviewer K2wv,

We sincerely thank you for taking the time to review our paper and appreciate your recognition of the additional results for Qwen3-14B. We also acknowledge that the effectiveness of our method is unclear due to the limited scope of the evaluation. Therefore, we are now conducting accuracy and throughput measurements on the BIG-Bench Extra Hard benchmark (https://arxiv.org/abs/2502.19187). It may be just before the end of the discussion period that all the results become available, but we would appreciate your patience in the meantime.

Sincerely,

Dear Reviewer K2wv,

To further evaluate the generalizability of iKVC beyond AIME2025, we conducted additional experiments on BIG-Bench Extra Hard (BBEH) using three PTQ methods (HQQ, GPTQ, and AutoRound). Could you please see below? We apologize for the delay and hope this addresses your concern.

Sincerely,


Additional Results on BIG-Bench Extra Hard (BBEH)

The results demonstrate that iKVC controls the accuracy–throughput trade-off by adjusting the KV-cache correction interval. Although accuracy varies non-monotonically with KK, throughput consistently increases as KK grows.

Due to time constraints, accuracy evaluations for AutoRound at K∈{100,200,400}K \in \{100,200,400\} are still ongoing. We expect similar improvements over PTQ alone, as observed with HQQ and GPTQ, and plan to incorporate the remaining results into the revised manuscript once they are available.

Table: BBEH accuracy and decode throughput.

PTQ methodConfigurationAccuracy (%) ↑Throughput (tokens/s) ↑
—Vanilla (BF16)24.000.621
HQQPTQ only18.0041.195
HQQIntactKV w/ training20.67—
HQQIntactKV w/o training22.67—
HQQiKVC, K=123.330.412
HQQiKVC, K=2023.333.883
HQQiKVC, K=4025.336.871
HQQiKVC, K=10023.3313.590
HQQiKVC, K=20020.0020.211
HQQiKVC, K=40024.6726.710
GPTQPTQ only16.0018.926
GPTQIntactKV w/ training14.00—
GPTQIntactKV w/o training20.67—
GPTQiKVC, K=122.670.351
GPTQiKVC, K=2018.673.097
GPTQiKVC, K=4020.005.190
GPTQiKVC, K=10024.008.960
GPTQiKVC, K=20018.6711.960
GPTQiKVC, K=40020.6715.009
AutoRoundPTQ only20.0019.162
AutoRoundIntactKV w/ training20.00—
AutoRoundIntactKV w/o training19.33—
AutoRoundiKVC, K=124.000.353
AutoRoundiKVC, K=2020.003.128
AutoRoundiKVC, K=4026.005.246
AutoRoundiKVC, K=100Ongoing9.136
AutoRoundiKVC, K=200Ongoing12.187
AutoRoundiKVC, K=400Ongoing14.644
Reviewer_uJY8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

iKVC tackles the fundamental issues where accuracy can degrade from the accumulation errors of quantized KVs and proposes a training-free inference method that periodically replaces KV-cache entries produced by a quantized LLM with those recomputed by its full-precision counterpart. The method initializes the prompt cache in full precision, corrects the cache for generated tokens, and generates a full-precision bonus token after each correction interval. By varying this interval, iKVC trades additional computation and parameter-transfer overhead for improved generation accuracy.

Critical strengths
  • The paper highlights KV-representation errors introduced by quantized weights during decoding. This is a useful perspective on accuracy degradation in quantized LLMs.
  • The proposed method is straightforward, training-free, and applicable to multiple PTQ techniques. The configurable correction interval provides a simple way to control the additional inference cost.
Critical weaknesses
  • Regarding the cache-correction benefits analysis: iKVC combines full-precision initialization of the entire prompt cache, periodic cache correction, and full-precision bonus-token generation. At K = 1, approximately half the output tokens are generated directly by the full-precision model, making it difficult to attribute the accuracy gains specifically to cache correction.
  • Regarding he theoretical justification: The growth of the Frobenius norm in Lemma 2 follows from appending additional erroneous cache entries and does not itself establish increasing attention-output error.
  • Limited evaluation scope and consistency: The accuracy evaluation is restricted to 30 AIME2025 questions, averaged over three runs, using models from the Qwen3 family. The improvements are also not consistent across configurations: for GPTQ-quantized Qwen3-14B, accuracy decreases from 64.44% to 63.33% at K = 100, 1,000, and 10,000 (Figure 9). Additional long-generation tasks and analyses across generation lengths would help establish when the method reliably improves accuracy.
  • Incomplete accuracy–throughput comparison: Accuracy and throughput are evaluated at largely different correction intervals, with only K = 1 and K = 100 shared between Figures 4 and 5. Moreover, K = 1 achieves approximately 0.60 tokens/s, below the offloaded full-precision baseline’s 0.852 tokens/s. Joint accuracy–throughput measurements under matching generation settings, including IntactKV and relevant speculative-decoding baselines with comparable resources, would better establish the practical advantage of periodically executing the full-precision model.
Questions for authors

Please address the concerns raised in weaknesses.

5 comments

We sincerely thank you for recognizing the clear problem formulation and straightforward approach. We also appreciate your constructive feedback, which has helped us strengthen the justification of our method.

Weakness 1: Bonus token generation by the full-precision model may confound the accuracy gains attributed to KV-cache correction.

Short comment. We agree that the current results do not isolate these effects, and will evaluate an ablation without bonus-token generation to clarify its contribution. The ablation results will be shared once available.

For further detail, bonus-token generation is introduced primarily to improve efficiency by reusing the next-token distribution of the full-precision model that was already computed during KV-cache correction, rather than to recover task accuracy. The ablation will disable this generation step while retaining prompt cache initialization and periodic KV-cache correction. Comparing it with standard iKVC will assess the incremental accuracy-recovery effect of bonus-token generation and whether accuracy recovery persists without it.

Weakness 2: Lemma 2 does not itself establish increasing attention-output error.

Short comment. Lemma 2 is a theoretical proof that KV-cache error accumulates in proportion to tokens, while Lemma 3 complements Lemma 2 by proving that attention-output error has an upper bound that depends on accumulated KV-cache error, motivating KV-cache correction.

For further detail, under its stated assumptions, Lemma 2 establishes a lower bound on the expected squared Frobenius norm of the KV-cache error that grows linearly with the number of tokens. It does not establish an exact proportional relationship or a corresponding increase in attention-output error. Lemma 3 complements this result by quantifying how query, key, and value perturbations enter an upper bound on attention-output deviation. Key-cache errors affect the attention scores, whereas value-cache errors directly contribute to output deviation. Reducing cache-error norms reduces the corresponding bound, providing a theoretical motivation for KV-cache correction.

These results do not imply that a larger bound entails a larger realized error, or that a smaller bound guarantees improved task accuracy. We therefore assessed practical benefits empirically in Section 5 ("Experiments") in the submitted version and have clarified this interpretation in Sections 3.4 (“Insight”) and 4.2 ("Theoretical guarantee") in the revised manuscript.

Weakness 3: The evaluation scope is limited.

Short comment. We acknowledge the limitations of the evaluation scope and will evaluate smaller correction intervals for GPTQ-quantized Qwen3-14B and a long-horizon task beyond AIME to strengthen iKVC's effectiveness.

For further detail, GPTQ-quantized Qwen3-8B and Qwen3-4B-Thinking-2507 (Figures 4 and 8) show higher mean accuracy than PTQ alone at the evaluated intervals up to K = 1,000. We therefore plan to test intermediate intervals below K = 100 to investigate whether more frequent correction improves accuracy in this configuration.

Additionally, evaluation on a long-horizon task beyond AIME will help assess whether the observed benefits extend beyond mathematical reasoning. We will share these results once available.

Weakness 4: Unmatched KV-cache correction intervals between accuracy and throughput evaluations make the practical advantage unclear.

Short comment. We aligned the accuracy and throughput evaluations across correction intervals. Please check Figures 4 & 5 in the revised manuscript. We also argue that comparisons with speculative-decoding methods would mislead readers, as their objectives differ from ours.

For further detail, K = 1 is just an ablation to ensure it is slower than the offloaded full-precision baseline in our measurements. In the revised manuscript, we updated Figures 4 and 5 to use a common set of intervals, K∈1,20,40,100,200,400K \in {1, 20, 40, 100, 200, 400}, under matching generation settings.

In Section 6 ("Related Work") in the revised manuscript, we have explained the difference in research objectives between speculative decoding and iKVC: "Although both speculative decoding and our method employ two language models during inference, their objectives fundamentally differ. Speculative decoding aims to accelerate LLM inference with memory-rich environments. In contrast, our method aims to enable users to achieve high inference performance comparable to an LLM, even in memory-constrained environments."

Additional results: AIME2025 with GPTQ-quantized Qwen3-14B at several tighter correction intervals

We conducted additional AIME2025 experiments for GPTQ-quantized Qwen3-14B using tighter correction intervals K∈{10,20,40,60,80}K\in\{10,20,40,60,80\}, beyond K∈{1,100,1000,10000}K\in\{1,100,1000,10000\}. This would answer your questions:

Effectiveness for GPTQ-quantized Qwen3-14B

  • An effective correction interval exists even for a Qwen3-14B model quantized using GPTQ.
  • The highest average accuracy was achieved with an interval of K = 10, representing an 8.89-point improvement over using GPTQ alone.

Consistency of accuracy recovery across configurations

  • The KV cache correction interval that yields the greatest improvement in accuracy varies by PTQ method.
  • This is partly because the "magnitude of propagated errors" and the "distribution of parameters containing errors" across layers and projections differ depending on the PTQ method used.
  • Such differences in error distribution affect both how KV cache errors influence attention calculations for subsequent token predictions and how a corrected cache affects other layers (e.g., FFN or LayerNorm) that remain quantized. Therefore, increasing the frequency of KV cache corrections does not guarantee a monotonic increase in accuracy; the optimal interval for maximizing accuracy gains can vary with the model and the PTQ method employed.
MethodCorrection interval KAIME2025 accuracy (%)
GPTQ only—64.44 ± 5.67
GPTQ + iKVC165.56 ± 1.57
GPTQ + iKVC1073.33 ± 4.71
GPTQ + iKVC2066.67 ± 2.72
GPTQ + iKVC4063.33 ± 4.71
GPTQ + iKVC6070.00 ± 4.71
GPTQ + iKVC8067.78 ± 1.57
GPTQ + iKVC10063.33 ± 2.72
Vanilla (just reference)—72.22 ± 1.57

*These results are scheduled to be included in a future revised version.

Reviewer_uJY8OpenReview ↗

Response to Authors' Rebuttal

Thank you for the clarifications and additional experiments. I appreciate the clearer interpretation of the theory and the alignment of correction intervals between accuracy and throughput evaluations.

My main concern remains whether periodic KV-cache correction improves accuracy independently of full-precision prompt initialization and bonus-token generation. Ablations separating these effects are still needed.

The additional results are encouraging, but gains over PTQ-only remain inconsistent across configurations and vary substantially with K. This leaves concerns about robustness and interval selection. Evaluation beyond Qwen models and AIME would also strengthen the evidence.

Finally, speculative decoding methods such as SpecExec explicitly target consumer GPUs with offloading. The memory-rich versus memory-constrained distinction therefore does not resolve whether KV correction offers a competitive accuracy–throughput trade-off against token verification under comparable resources.

I appreciate the improvements, but will maintain my current score until these concerns are more clearly addressed.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.