acceptodds
Under review as a conference paper at ICLR 2027

QuantThink: Quantized Reasoning Models Can Learn When to Commit

Abstract

Large reasoning models (LRMs) achieve strong performance through extended chains of thought, but their long outputs impose substantial KV-cache and inference overhead. Although KV-cache quantization effectively reduces memory cost, we find that it can paradoxically prolong reasoning by inducing repeated post-solution verification and delaying answer generation even after a correct solution has already been reached. To address this issue, we introduce QuantThink, which learns when reasoning is sufficient to terminate. QuantThink combines (i) a sufficiency-aware trajectory formulation that identifies checkpoints for verification continuation or reasoning termination, and identifies redundant post-solution verification, and (ii) a parameter-efficient reinforcement learning method Q-GRPO that optimizes termination decisions under quantized inference. Across DeepSeek-R1-Distill-Qwen, QwQ, and Phi-4 models, five reasoning benchmarks, and three KV-cache quantizers—KVQuant, KIVI, and RotateKV—QuantThink improves pass@1 by up to 4.32 percentage points while reducing generated tokens by up to 63.1% compared with the KIVI K2V2 baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning Model, Quantization, Chain-of-Thought, Reinforcement Learning, Early Stopping

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_4vGBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies chain-of-thought inflation under low-bit KV-cache quantization. Its central claim is that quantized reasoning models often continue checking a solution after a correct answer is already available. QuantThink identifies candidate verify-or-commit boundaries using an offline dictionary of verification cues. It then creates early-exit answer branches at those boundaries and uses Q-GRPO to favor earlier correct commitments and shorter complete responses. The backbone is frozen and LoRA adapters are trained on quantized rollouts; inference requires no external stopping controller. Experiments cover four reasoning models, five benchmarks, and three KV-cache quantizers. For DeepSeek-R1-Distill-Qwen-7B under KIVI K2V2, the five-benchmark macro-average pass@1 rises from 66.10% to 70.42%, while average generated tokens fall from 12,784 to 5,738 (Table 2).

Critical strengths
  1. Relevant deployment question. The work evaluates both answer quality and trajectory length, recognizing that longer generation can offset the intended efficiency gains from a smaller KV cache.

  2. Targeted training objective. The proposed checkpoints correspond to the behavior the paper seeks to change. Measuring the full response through EOS also addresses a plausible failure mode in which reasoning is moved from the thinking span into the final answer.

  3. Useful empirical comparisons. The paper compares against GRPO, DEER, JET, and S-GRPO, and Table 11 separates the effects of semantic boundary placement and the two reward components.

  4. Measured inference benefits. On 7B MATH-500 under KIVI K2V2, QuantThink improves pass@1 from 84.6% to 88.8%, reduces generated tokens from 7,846 to 3,226, and lowers measured mean latency from 20.9 to 10.2 seconds (Table 3).

Critical weaknesses
  1. Incremental algorithmic novelty. S-GRPO [1] already constructs multiple early-exit branches from one reasoning trajectory and assigns higher rewards to earlier correct exits. JET [2] also trains reasoning models to stop after sufficient information has accumulated. QuantThink's identifiable additions are the verification-cue boundaries, its reward formulation, and the KV-cache-quantized training setting. Table 11 supports a gain from combining these elements on one model and benchmark, but the paper should characterize Q-GRPO as an extension of earlier exit-training methods and substantiate which addition generalizes beyond that setting. Prior work has also connected quantization to overthinking [3], although its quantization configurations differ from the KV-only comparisons here.

  2. Unreported effects of training-sample filtering. Algorithm 1 skips a prompt when no candidate branch is correct or the first correct branch occurs only at the final checkpoint. A trajectory without a verification episode offers no informative serial group either. These choices focus training on cases where the model can already answer before its final exit, but the paper does not report what fraction of rollouts survive or how survival varies by difficulty, model, and bit width. The reported test-set gains are useful, yet the omission makes it difficult to judge whether the method mainly addresses relatively easy overthinking cases while leaving hard quantization failures untouched.

  3. Uncertain definition of answer readiness and redundancy. The earliest correct checkpoint is determined by one sampled answer per boundary. A correct sample can be lucky rather than evidence that the answer is reliably available; subsequent verification may increase or decrease the probability of a correct answer. Appendix Figure 15 reports ten continuations at selected checkpoints and equal aggregate accuracy at and after the chosen point, but does not establish stability per question. The dictionary-based detector has high reported boundary F1 on manually annotated MATH-500 trajectories, yet exact trajectory matches are only 0.64-0.67 and the annotation sample size is not stated (Table 9). In addition, RVC is set to zero when no correct checkpoint exists, so RVC alone is difficult to compare across systems with different success rates.

  4. The source of accuracy improvements is not isolated. Section 4.2 applies the policy loss to the inserted stop token and to all generated conclusion tokens. Consequently, higher pass@1 may reflect improvements in final-answer generation as well as better stopping decisions. Reward-component ablations do not separate these effects. A stop-token-only versus answer-token-only ablation would directly test the proposed mechanism.

  5. Limited uncertainty and cost accounting. Cross-quantizer and bit-width transfer is reported on MATH-500 for one 7B model (Table 1), while detailed latency and peak-memory comparisons cover one model-task configuration (Table 3). Table 12 averages six independent single-response evaluations, which does not measure variation across independent training runs. Gains narrow at larger scale: on QwQ-32B, QuantThink's mean pass@1 is 82.88%, slightly below DEER's 83.00% (Table 8). Finally, the paper compares inference outcomes with a training-free marker penalty (Table 7) without reporting the additional training compute needed by QuantThink. These limits should temper broad claims about robustness and deployment efficiency.

Questions for authors
  1. What percentage of training prompts are skipped by each condition in Algorithm 1, broken down by model and task? How do the retained and skipped prompts differ in difficulty and base-model accuracy?

  2. If each candidate checkpoint is continued multiple times, how stable is the earliest correct checkpoint? What fraction of questions gain, lose, or retain answer accuracy after further verification?

  3. What are the pass@1 and token-length results when the policy loss is applied only to the stop token, only to conclusion tokens, and to both?

  4. Can the authors provide the complete verification dictionary, the size and agreement statistics for the manual boundary annotations, independent training-seed results, and training GPU-hours?

Reviewer_YCHCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies longer reasoning responses under low-bit KV-cache quantization. QuantThink uses verification cues and Q-GRPO to train models to answer correctly earlier and generate shorter responses. Experiments on four models and five benchmarks show an improved accuracy–length trade-off, without an additional stopping controller at inference.

Critical strengths
  1. The paper identifies a practically important problem: longer reasoning traces can offset the expected efficiency benefits of KV-cache quantization.
  2. The training objective encourages earlier correct answers while measuring the complete response length, including the final answer segment. Inference requires no separate stopping controller.
  3. On the 7B model, QuantThink improves macro-average pass@1 over S-GRPO by 2.34 percentage points while generating 17.2% fewer tokens. The appendix provides useful boundary controls, component ablations, and repeated evaluations.
Critical weaknesses
  1. The quantization-specific contribution is not fully isolated. All main RL baselines, including S-GRPO, use KIVI K2V2 rollouts during training, and Table 11 provides useful component ablations against S-GRPO. However, the paper does not compare the same Q-GRPO method trained on FP16 versus quantized rollouts under matched settings, with both evaluated at the same cache precision. The current evidence demonstrates gains under quantized inference, but it remains unclear how much comes specifically from adapting training to quantization rather than from a general improvement in early-exit training.
  2. The interpretation of later verification as redundant needs more direct evidence. Ac is the earliest checkpoint at which a sampled branch produces a correct answer, so its selection may be sensitive to sampling noise. Figure 15 provides helpful evidence by sampling ten continuations per checkpoint and reporting 66% aggregate accuracy both at Ac and at a later checkpoint. However, equal aggregate accuracy can hide differences between questions, with further verification helping some while harming others. Using independent samples to select Ac and evaluate its reliability, together with per-question results, would better support the claim that the removed verification is unnecessary.
  3. Training cost and experimental variability require clearer reporting. Generating an answer at every selected checkpoint and skipping some prompts changes the computation used during training. The paper does not report total branch-generation costs, training GPU-hours, or enough information to determine how computational budgets are matched with S-GRPO. Table 12 reports six separate single-response evaluations, which help assess decoding variation but do not establish stability across independent training seeds. Table 3 provides actual latency measurements, but the definition and measurement procedure for mean per-question end-to-end latency at batch size 8 need clarification.
Questions for authors
  1. Under otherwise matched settings and training budgets, how does Q-GRPO trained on FP16 rollouts compare with Q-GRPO trained on KIVI K2V2 rollouts when both are evaluated at the same cache precision?
  2. Could you use separate samples to select Ac and evaluate its reliability, then report per-question correctness rates and identify which questions benefit from or are harmed by further verification?
  3. How many branch answers are generated per training prompt, what fraction of prompts is skipped, and what are the total generated training tokens and GPU-hours for QuantThink and S-GRPO? How are their computational budgets matched, and do Table 12’s six evaluations use a fixed trained checkpoint for each method?
  4. In Table 3, does mean per-question latency refer to the actual prompt-to-answer time experienced by each request, or to batch wall-clock time divided by eight? Please clarify the measurement procedure.
Reviewer_EJH7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reasoning-length inflation induced by KV-cache quantization, attributing much of the additional computation to verification after a correct answer can already be produced. QuantThink identifies candidate commitment boundaries and trains LoRA adapters using Q-GRPO, which rewards earlier correct commitment and shorter complete responses. Experiments cover multiple reasoning models, mathematical and coding benchmarks, and several cache quantizers.

Critical strengths
  1. A concrete and practically relevant motivation. The paper examines how quantization changes generated computation, complementing conventional evaluations of memory consumption and final-answer accuracy. Localizing the additional tokens to verification behavior makes the problem more specific than general reasoning compression.
  2. Substantial empirical validation. The study includes comparisons with relevant early-exit methods, segmentation and reward ablations, transfer across quantization settings, manual boundary annotations, and repeated evaluations. Actual inference measurements strengthen the practical contribution: on 7B MATH-500, accuracy increases from 84.6% to 88.8% while reported latency decreases from 20.9 to 10.2 seconds.
Critical weaknesses
  1. The sufficiency supervision is noisy and selectively constructed. A single successful branch does not establish that a checkpoint reliably supports correct commitment. Moreover, Algorithm 1 discards trajectories whose earliest correct exit is the final checkpoint, removing examples where continued verification may be necessary. This filtering could bias training toward early commitment and warrants a direct ablation.
  2. The reward can prefer earlier commitment even when it requires more total computation. An earlier correct exit may need a lengthy conclusion, while a later exit may produce a short answer. However, the verification reward favors the earliest correct exit by at least 0.5 over any later correct exit, whereas the length advantage contributes at most 0.1 under the default setting. For example, two correct branches with total lengths of 2,000 and 1,000 tokens can receive combined advantages of 0.25 and −0.15, respectively, if the longer branch exits earlier. Thus, the objective can reinforce a more expensive response while discouraging a cheaper correct one. This conflicts with interpreting earlier commitment as improved end-to-end efficiency.
  3. The sufficiency test conflates an already sufficient prefix with the ability to complete the solution after switching output phases. After inserting </think>, the model can still perform substantial reasoning in the conclusion. A correct branch therefore establishes that the prefix plus its newly generated continuation can solve the problem, rather than that reasoning was already sufficient at the checkpoint. Counting complete-response length addresses computational cost, but does not validate the interpretation of (A_c) as an answer-ready point.
Questions for authors

As shown in weaknesses.

Reviewer_97XFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how low-bit KV-cache quantization inflates chain-of-thought length in reasoning models. It diagnoses the cause as post-solution verification persistence — the model reaches a correct answer at a similar point as full precision but then keeps re-checking instead of committing. QuantThink trains a parameter-efficient RL objective that rewards early commitment at answer-ready checkpoints and penalises continued verification, removing the inflation without an inference-time stopping rule.

Critical strengths
  1. The paper identifies a specific failure mode rather than a generic cost: quantization roughly triples the redundant-verification count while raising token usage, and accuracy does not improve beyond the first answer-ready checkpoint.
  2. The CoT-C metric checks whether the conclusion still follows from the pre-</think> reasoning, so shorter outputs cannot be credited to merely relocating the reasoning.
  3. Gains are attributed component by component against S-GRPO, with six-run mean±std for two models.
Critical weaknesses
  1. The "quantization-aware" claim is not isolated: the only quantization-specific element is that rollouts are generated under 2-bit KV quantization, and at full precision the trade-off reverses in S-GRPO's favour (Table 1).
  2. The margin over the strongest baseline is within run-to-run variance (14B: 79.3±0.8 vs. 78.1±0.9, Table 12).
  3. Baseline fairness cannot be judged: no training hyperparameters, seeds or baseline search budgets appear in the main text or appendix, and λ/η were tuned on MATH-500, which is also a test benchmark.
Questions for authors
  1. If QuantThink trained on FP16 rollouts still clearly wins under quantization I would raise my score; if the two are equivalent, the "quantization-aware" claim is unsupported and I would lower it.
  2. If the baselines received the same configuration, data, steps and search budget I would raise my score; if S-GRPO used defaults while λ/η were tuned on a test benchmark, the Table 2 margin is not attributable to the method and I would lower it.
  3. Paired bootstrap CIs for QuantThink − S-GRPO on 7B and 14B, from the six runs in Table 12?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.