acceptodds
Withdrawn from ICLR 2027

RECAP: Calibrating Low-Bit Reasoning Models on the History They Read

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-bit quantization reduces the cost of serving reasoning models but can substantially degrade long chains of thought. Each step reads cached keys and values spanning thousands of tokens, allowing quantization errors to accumulate along the trajectory. Standard calibration operates on short segments that omit the earlier context needed by later reasoning steps. Even when this context is retained, cache reconstruction error alone does not capture how quantization errors affect subsequent attention. We present RECAP, a post-training quantization method that calibrates reasoning models on the history they read. Dynamic-prefix calibration separates long historical context from short-block reconstruction. It regenerates prefix keys and values with the current quantizer and rotates the active block to supervise every position along the trajectory. Prefix read geometry further supervises historical cache errors according to their effects on attention outputs, weighting key and value errors asymmetrically. With both components applied only during calibration, RECAP outperforms the baseline across multiple reasoning models and quantization settings. Under W4A4KV4, RECAP improves accuracy over the baseline by 13.3 percentage points on MATH500 with DeepSeek-R1-Distill-Qwen-1.5B and 15.8 points on AIME-120 with Qwen3-4B. These results highlight the importance of preserving historical dependencies during calibration for low-bit reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, reasoning language models, long-context calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Jo2YOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RECAP, the integer-based W4A4KV4 post-training quantization (PTQ) framework for reasoning models. This paper observes that the existing PTQ methods fail to capture the historical KV cache’s information while calibrating the later tokens in long-reasoning trajectories. To address this, RECAP proposes dynamic prefix calibration, which regenerates long KV caches with the current quantizer while reconstructing only a short recent blocks. Also, RECAP employs Prefix Read Geometry Loss which weights historical key and value errors differently. Experiments on DeepSeek-R1-Distill-Qwen 1.5 and 7B, Qwen3-4B, the proposed method shows clear improvement over the FlatQuant.

Critical strengths
  • Beyond the Microscaled FP4 (e.g., NVFP4), developing integer-quantized W4A4KV4 reasoning model is an important direction for the edge devices. I encourage authors to include this key motivation why exploring integer-based W4A4KV4 is important.
  • The proposed method shows clear improvement over FlatQuant, the main PTQ baseline.
  • The paper is easy to follow, and the figure and diagrams are intuitive.
  • Ablation studies including the impact of the Read Geometry Penalty helps to understand of the individual role of each calibration objective terms.
  • While the proposed method using calibration with NuminaMath, it improves the other domain performances such as GPQA or LCB.
Critical weaknesses
  • The critical weakness is the fairness of the comparison with FlatQuant. This paper simply adopt the accuracy from other paper. The calibration dataset, hyperparameters, and other settings are whole different. To isolate the gain from the proposed calibration objective, we highly recommend to re-run the baseline and fill in the step-by-step gap. If the main advantages are come from the calibration dataset (e.g., simple Wikitext-2 vs. FP model’s reasoning trajectory), the novelty will largely disappears.
  • I think that the proposed observation and method are not limited to long-decoding-based reasoning models. There can be similar issue in long-context task such as long document QA or summarization. Adding results on long-context benchmark such as LongBench would be beneficial.
  • Calibration overhead over the PTQ baseline should be discussed. It seems that the proposed method requires longer tensor during training, so it would be helpful to report the peak memory and total calibration time, compared to FlatQuant.
  • For generalization, adding the recent reasoning models (e.g., Qwen3.5) and larger models would be beneficial.
  • The proposed calibration objective can be extended to other general PTQ methods. Is there any specific reason to employ this object to FlatQuant. If not, can you add the results on the other method, such as GPTQ in weight-only quantization?
  • It would be helpful to add ablation study on W4A4KV16 and W16A16KV4.
Questions for authors

Please see weaknesses.

Reviewer_bHEUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAP to improve low-bit quantization of reasoning models. It reconstructs short blocks while retaining the preceding quantized KV cache, rotating the active block to supervise the full trajectory. It also introduces an auxiliary cache loss based on attention sensitivity. Experiments across three models, two quantization settings, and five benchmarks show consistent improvements over FlatQuant.

Critical strengths
  • The calibration technique is reasonable and practically motivated. Allowing a short reconstruction block to access a long quantized history addresses a clear limitation of short-window calibration.

  • The reported improvements are encouraging. Gains are consistent across the evaluated settings and substantial on several reasoning benchmarks.

Critical weaknesses
  1. The paper does not clearly distinguish a reasoning-specific problem from a general long-context calibration problem. Missing historical context, incomplete supervision, and unequal sensitivity to KV errors can all arise outside CoT reasoning. Neither component explicitly uses reasoning structure, and the experiments do not separate the effect of sequence length from that of reasoning. The larger gains on reasoning-heavy tasks are suggestive, but do not establish this distinction. A general calibration method can certainly be valuable for reasoning models; the motivation should simply be clearer about what the evidence supports.

  2. The baseline comparison is limited. Only QuaRot and FlatQuant are evaluated, and QuaRot largely collapses, leaving FlatQuant as the main meaningful comparator there. More strong baselines, such as SpinQuant should be added. Since RECAP uses self-generated reasoning traces, the paper should also clarify whether FlatQuant receives the same calibration data and budget.

  3. The motivation analysis has some evidential gaps. The position-wise KL results provide useful evidence that short-window and fixed-tail calibration fail in different regions. However, the importance-transfer experiment uses only four trajectories from an earlier quantization configuration. The claim that equal-sized cache errors have different effects is supported by an illustration and a later sensitivity derivation, rather than a direct perturbation experiment. The contribution statement should reflect the limited scope of these diagnostics.

  4. The introduction is difficult to follow before reading the method. Terms such as “position coverage” and “reader positions” appear before their meaning is clear. The transition from dynamic prefixes to read geometry also feels abrupt: the introduction should explain what remains unresolved after adding dynamic prefixes and how the auxiliary loss addresses it. Table 2 supports combining the two components. My concern here is how their connection is presented, not whether the second component provides a benefit.

  5. The calibration savings need practical measurements. Avoiding full-sequence reconstruction should reduce computation, but prefix KV states are still regenerated at every update, and collecting geometry statistics adds overhead. The operation-count analysis is helpful, but actual calibration time and peak memory are realistic to assess the overall savings.

Questions for authors
  1. Can you provide a length-matched comparison on reasoning and non-reasoning sequences? This would help determine whether the benefits are specific to reasoning or primarily reflect better long-context calibration.

  2. Table 2 shows gains from adding geometry loss to dynamic-prefix calibration. Could you also compare against a plain KV-MSE auxiliary loss under identical settings? This would clarify how much of the gain comes from geometry-aware weighting rather than simply adding direct cache supervision.

Reviewer_NJm8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The author believes that low-precision kv cache will accumulate errors along the inference trajectory, so dynamic calibration is required. Through Dynamic-prefix calibration and Prefix read geometry, the performance of DeepSeek-R1-Distill-Qwen-1.5B/7B and Qwen3-4B on the mathematical evaluation set has been improved.

Critical strengths
  • The method is relatively novel and seems to be mathematically effective
Critical weaknesses
  • What is the benchmark of AIME-120? It seems to have been written incorrectly and there is no citation
  • This method brings a lot of inference runtime overhead and has no engineering achievability
Questions for authors

No question

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.