acceptodds
Under review as a conference paper at ICLR 2027

RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Models

10 views

Abstract

Post-training quantization (PTQ) has become a widely adopted technique for reducing the memory footprint and inference cost of large language models (LLMs). However, recent studies reveal that when applied to reasoning models, PTQ not only degrades reasoning performance but also exacerbates overthinking, leading to longer reasoning trajectories. These issues may offset the efficiency gains expected from lower-precision inference. Existing approaches to mitigating these problems mainly rely on complex optimization procedures. More recent lightweight inference strategies instead use predefined overthinking markers, limiting their adaptability across quantized models. To address these issues, we propose **R**easoning **A**nalysis and **T**oken-level **I**nference **O**ptimization (RATIO), a framework that identifies model-specific overthinking tokens and assigns each a tailored penalty. RATIO first introduces Quantization-aware Reasoning Behavior Analysis (QRBA) to identify overthinking tokens by analyzing discrepancies between full-precision and quantized models. It then adopts Token-Specific Penalty Determination (TSPD), which leverages full-precision guidance to derive token-specific penalties without additional training. Extensive experiments show that RATIO achieves a better accuracy-efficiency trade-off than existing token-level interventions. Specifically, RATIO achieves up to 9.8 points accuracy improvement and reduces chain-of-thought (CoT) length by up to 51.3% compared with quantized baselines. We will release all the code and implementation of RATIO.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Quantization, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_pfyzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses two potential issues arising from parameter quantization in reasoning models: reduced reasoning accuracy and overthinking (i.e., longer chains of thought). Building on existing marker-based strategies, RATIO introduces a more fine-grained approach. It first compares the reasoning behaviors of full-precision and quantized models, then uses QRBA to identify model-specific tokens associated with inefficient reasoning that should be suppressed. Finally, TSPD assigns token-specific penalties to reduce the likelihood of selecting these tokens during generation. The method operates entirely at the inference-time decoding stage and does not require model retraining.

The main novelty lies in the systematic discovery of model-specific target tokens and the assignment of differentiated penalties, rather than relying on predefined markers and uniform penalties.

One limitation is that, although QRBA considers reasoning context during token selection, the penalties applied to selected tokens remain fixed during inference and do not dynamically adapt to the current reasoning state. The authors also acknowledge this limitation in the Discussion section. Moreover, token-specific penalties do not consistently outperform uniform penalties. For example, on Qwen-1.5B GPTQ-W3, a uniform penalty (QRBA, λ=1.0) achieves slightly better average accuracy and shorter CoT length than the full RATIO method. Finally, the evaluation of inference optimization primarily relies on reasoning accuracy and CoT length. Whether these metrics adequately reflect actual latency, throughput, and offline calibration costs requires more direct validation.

Critical strengths
  1. Empirical evaluation covers five models (1.5B–14B), multiple model architectures, and several quantization settings.
  2. Training-free Token Discovery and Calibration achieve substantial reductions in Chain-of-Thought (CoT) length compared to unmodified quantized baselines across various models and quantization settings.
Critical weaknesses

1.The benefits of token-specific calibration are not consistent across all settings. In the ablation study on Qwen-1.5B GPTQ-W3, the uniform penalty (QRBA, λ=1.0) achieves an average accuracy of 38.68% and a CoT length of 21.91k tokens, slightly outperforming the full RATIO method (37.40% accuracy and 22.05k tokens) on both metrics.

2.System-level efficiency evaluation is insufficient. The paper primarily uses CoT length as a proxy for efficiency, without sufficiently evaluating actual latency, throughput, or offline calibration costs. Therefore, stronger empirical evidence is needed to support claims about practical deployment benefits.

Questions for authors
  1. In the ablation study on Qwen-1.5B GPTQ-W3, the uniform penalty (λ=1.0) slightly outperforms the full RATIO method in both average accuracy and CoT length. Could the authors explain this exception and discuss whether the benefits of token-specific penalties are consistent across different quantization methods?

  2. The largest model evaluated in the current experiments has 14B parameters. Could the authors further validate RATIO on larger quantized reasoning models or additional model families to provide stronger evidence of its generalizability?

9 comments

Response to Reviewer pfyz (denoted as R4)

Q4-1: The benefits of token-specific calibration are not consistent across settings.

A4-1: We thank the reviewer for pointing out this issue. To further investigate the contribution of TSPD and provide a more comprehensive evaluation, we conduct two additional ablation studies: a broader uniform-penalty sweep and shuffled token–penalty assignments.

(1) Broader Uniform-Penalty Sweep.

We evaluate a broader range of uniform penalties, λ ∈ {0.5, 1, 1.5, 2, 3, 4}, while keeping the selected token set fixed on Qwen-7B. The results show that RATIO achieves the highest average accuracy under both AWQ (68.00%) and GPTQ (64.81%), outperforming the best uniform-penalty configurations by 1.15 and 0.58 percentage points, respectively. Importantly, RATIO also achieves the shortest average CoT length under AWQ (7.41k tokens). Under GPTQ, its average CoT length (10.59k tokens) is only marginally longer than that of the shortest uniform-penalty configuration (10.55k tokens), while achieving 1.39 percentage points higher average accuracy. These results demonstrate that TSPD provides a better overall accuracy–efficiency trade-off than a broad range of uniform penalties.

ModelMethodGPQA (Acc/CoT)AIME (Acc/CoT)GSM8K (Acc/CoT)MATH500 (Acc/CoT)HumanEval (Acc/CoT)Avg. AccAvg. CoT (k tokens)
Qwen-7BAWQ Base47.30/9.7727.14/21.3589.69/2.3387.80/6.4076.83/8.3465.759.64
Uniform λ=0.547.30/8.5527.14/15.9689.76/2.1287.60/5.1878.05/6.2165.977.60
Uniform λ=142.57/10.0325.71/20.9388.86/2.0688.40/4.9280.49/7.0765.219.00
Uniform λ=1.545.27/7.9730.00/17.3989.08/1.8488.80/4.7781.10/6.3366.857.66
Uniform λ=244.59/9.7227.14/16.8190.07/1.7289.00/4.9876.22/4.7665.407.60
Uniform λ=346.62/8.0427.14/18.1088.86/2.0585.40/5.1479.27/5.1465.467.69
Uniform λ=445.27/9.6424.29/16.8189.16/2.0786.40/4.7877.44/4.2664.517.51
RATIO47.97/8.3932.86/16.8089.46/1.8288.60/5.0781.10/4.9868.007.41
GPTQ Base40.54/11.3131.43/29.4488.63/3.5586.80/11.0671.95/9.4963.8712.97
Uniform λ=0.542.57/10.6328.57/32.7189.01/2.3286.60/8.1074.39/6.8164.2312.11
Uniform λ=143.24/10.6724.29/27.0688.55/2.8587.60/6.7977.44/6.9064.2210.85
Uniform λ=1.541.22/9.0524.29/29.4488.02/2.3286.20/7.1577.44/7.1163.4311.01
Uniform λ=237.16/9.5230.00/28.2187.41/2.3987.60/6.2174.39/7.0663.3110.68
Uniform λ=341.22/9.0822.86/28.8088.40/2.4386.40/6.5275.00/7.3062.7810.83
Uniform λ=443.92/9.6124.29/26.9888.10/2.1285.20/6.2975.61/7.7363.4210.55
RATIO43.92/9.4827.14/27.4588.40/2.3789.60/6.3075.00/7.3664.8110.59

Acc (%) / CoT length (k tokens) are reported.

Continuation of A4-1

(2) Shuffled Token–Penalty Assignments.

To further examine the importance of token-specific penalty assignments, we keep the selected token sets fixed and randomly shuffle their corresponding penalties on Qwen-7B and LLaMA-8B under both AWQ and GPTQ. RATIO consistently achieves higher average accuracy than shuffled assignments in all four settings. Meanwhile, RATIO achieves shorter average CoT lengths in three of the four settings, with only a marginal difference in the remaining setting. This demonstrates that the specific correspondence between tokens and their assigned penalties matters.

ModelMethodAIMEMATHGSM8KGPQAHumanEvalAvg.ΔAccΔLen%
Qwen-7BAWQ27.14/21.3587.80/6.4089.69/2.3347.30/9.7776.83/8.3465.75/9.64––
+Shuffled27.14/22.0688.60/4.8890.30/1.4747.30/9.2978.66/8.2966.40/9.20+0.65-4.56
+RATIO32.86/16.8088.60/5.0789.46/1.8247.97/8.3981.10/4.9868.00/7.41+2.25-23.13
GPTQ31.43/29.4486.80/11.0688.63/3.5540.54/11.3171.95/9.4963.87/12.97––
+Shuffled25.71/28.3987.20/6.9888.55/2.1743.24/9.6279.27/6.2364.79/10.68+0.92-17.66
+RATIO27.14/27.4589.60/6.3088.40/2.3743.92/9.4875.00/7.3664.81/10.59+0.94-18.35
LLaMA-8BAWQ17.14/16.8578.00/5.3983.40/2.4333.78/8.2172.56/5.9956.98/7.77––
+Shuffled15.71/14.8575.60/5.4882.64/2.1734.46/7.9072.56/5.0356.19/7.09-0.79-8.75
+RATIO18.57/16.6076.80/4.9583.32/2.0635.14/7.9573.78/3.7757.52/7.07+0.54-9.01
GPTQ17.14/23.4673.60/8.6274.37/1.7335.14/11.2070.73/7.9954.20/10.60––
+Shuffled15.71/21.6375.00/6.9274.15/1.6139.86/10.2566.46/6.6754.24/9.42+0.04-11.13
+RATIO14.29/22.4071.40/6.3373.77/1.5241.22/10.1175.00/6.8055.14/9.43+0.94-11.04

For each benchmark, results are reported as Acc (%) / Len (k).

Overall, these additional experiments provide strong evidence for the effectiveness of TSPD. Across a broader range of uniform penalties, RATIO consistently achieves the highest average accuracy while maintaining highly competitive CoT lengths. Furthermore, the shuffled-assignment results confirm that assigning penalties according to individual token characteristics is more effective than randomly mapping penalties to the selected tokens. Together, these findings further support the contribution of token-specific penalty to RATIO.

Q4-2: System-level efficiency evaluation is insufficient.

A4-2: We thank the reviewer for this valuable suggestion. CoT length serves as a meaningful indicator of inference efficiency, but it does not fully capture wall-clock latency. To provide stronger system-level evidence, we additionally measure actual wall-clock inference time and throughput using the Qwen-1.5B model with real GPTQ W3 quantization.

The results are reported below:

ModelConfigurationThroughput (tokens/s)Total Inference Time
1.5BReal GPTQ W3 Base584.715:26:43
1.5BReal GPTQ W3 RATIO613.983:02:31

RATIO reduces the total inference time by 44.1% while increasing throughput by approximately 5.0%, demonstrating that its reduction in CoT length translates into practical inference acceleration.

We also report the offline calibration cost. The stage-wise computational costs are summarized below.

StageRuntimeComputational Resource
Candidate Identification (QSTI)5 hoursGPU
Token validation (RTV)20 hoursGPU
Penalty Calculation (TSPD)5~10 minutesCPU only
Total25 GPU-hoursNVIDIA A6000

The peak GPU memory usage is 47.46 GiB, and the external contextual review costs approximately $2.5. TSPD's penalty calculation requires no GPU resources and takes negligible time.

Importantly, the resulting token list and corresponding penalties can be shared across users and reused for subsequent inference requests without recalibration. Based on the measured inference time savings of approximately 2.4 hours per equivalent workload, the calibration overhead can be amortized after approximately 11 inference runs, assuming comparable single-GPU execution conditions.

Q4-3: The benefits of token-specific penalties are not consistent across settings.

A4-3: We thank the reviewer for raising this important question. We acknowledge that on Qwen-1.5B with GPTQ-W3, a uniform penalty of λ=1.0 slightly outperforms RATIO in both average accuracy and CoT length. This indicates that token-specific penalties do not necessarily outperform a well-chosen uniform penalty in every individual configuration. The effectiveness of penalty allocation can vary depending on the model and quantization method, and a particular global penalty may happen to provide a favorable trade-off for a specific configuration.

To further investigate this issue and examine whether the benefits of token-specific penalties extend across different quantization methods, we conduct additional experiments on Qwen-7B under AWQ and GPTQ, comparing RATIO against six different uniform penalty values (λ ∈ {0.5, 1, 1.5, 2, 3, 4}) while keeping the selected token set fixed.

The results show that RATIO achieves the highest average accuracy under both AWQ (68.00%) and GPTQ (64.81%), outperforming the best uniform-penalty configurations by 1.15 and 0.58 percentage points, respectively. Importantly, RATIO also achieves the shortest average CoT length under AWQ (7.41k tokens). Under GPTQ, its average CoT length (10.59k tokens) is only marginally longer than that of the shortest uniform-penalty configuration (10.55k tokens), while achieving 1.39 percentage points higher average accuracy. Overall, these results demonstrate the effectiveness of TSPD, showing that token-specific penalty allocation provides a better overall accuracy–efficiency trade-off than applying a single global suppression strength, supporting the effectiveness of TSPD across different quantization methods.

ModelMethodGPQA (Acc/CoT)AIME (Acc/CoT)GSM8K (Acc/CoT)MATH500 (Acc/CoT)HumanEval (Acc/CoT)Avg. AccAvg. CoT (k tokens)
Qwen-7BAWQ Base47.30/9.7727.14/21.3589.69/2.3387.80/6.4076.83/8.3465.759.64
Uniform λ=0.547.30/8.5527.14/15.9689.76/2.1287.60/5.1878.05/6.2165.977.60
Uniform λ=142.57/10.0325.71/20.9388.86/2.0688.40/4.9280.49/7.0765.219.00
Uniform λ=1.545.27/7.9730.00/17.3989.08/1.8488.80/4.7781.10/6.3366.857.66
Uniform λ=244.59/9.7227.14/16.8190.07/1.7289.00/4.9876.22/4.7665.407.60
Uniform λ=346.62/8.0427.14/18.1088.86/2.0585.40/5.1479.27/5.1465.467.69
Uniform λ=445.27/9.6424.29/16.8189.16/2.0786.40/4.7877.44/4.2664.517.51
AWQ RATIO47.97/8.3932.86/16.8089.46/1.8288.60/5.0781.10/4.9868.007.41
GPTQ Base40.54/11.3131.43/29.4488.63/3.5586.80/11.0671.95/9.4963.8712.97
Uniform λ=0.542.57/10.6328.57/32.7189.01/2.3286.60/8.1074.39/6.8164.2312.11
Uniform λ=143.24/10.6724.29/27.0688.55/2.8587.60/6.7977.44/6.9064.2210.85
Uniform λ=1.541.22/9.0524.29/29.4488.02/2.3286.20/7.1577.44/7.1163.4311.01
Uniform λ=237.16/9.5230.00/28.2187.41/2.3987.60/6.2174.39/7.0663.3110.68
Uniform λ=341.22/9.0822.86/28.8088.40/2.4386.40/6.5275.00/7.3062.7810.83
Uniform λ=443.92/9.6124.29/26.9888.10/2.1285.20/6.2975.61/7.7363.4210.55
GPTQ RATIO43.92/9.4827.14/27.4588.40/2.3789.60/6.3075.00/7.3664.8110.59

For each benchmark, results are reported as Acc (%) / Len (k).

Reviewer_rbq3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve quantized reasoning models during inference. The authors propose RATIO. It compares a full-precision model with AWQ and GPTQ models to find tokens whose probabilities increase after quantization. It then checks whether these tokens are related to incorrect or repetitive reasoning, and gives each selected token a different logit penalty.

The experiments cover five models and five benchmarks. RATIO reduces the reported average CoT length in all ten main model–quantizer settings. Its largest reported gain is on the 1.5B model with AWQ-W3: average accuracy rises by 9.78 percentage points, while average CoT length falls by 51.34%.

Critical strengths
  1. The problem is relevant. Quantization may change both answer accuracy and the number of reasoning tokens, so measuring only accuracy is insufficient.

  2. RATIO needs no model retraining during deployment. The final intervention is a small set of token logit penalties.

  3. The paper tests several model sizes, two weight quantizers, and multiple tasks. The ablation on the 1.5B model helps show the effects of token selection and penalty assignment.

Critical weaknesses
  1. The separation between analysis and evaluation is unclear. Section 4.1 says that the authors use 50 questions each from AIME and GPQA-Diamond to validate tokens. These benchmarks are also used for final evaluation. The paper explains that its MATH-500 and GSM8K analysis questions come from training splits, but does not give the same clarification for AIME and GPQA. Since token validation uses correctness-related evidence, possible overlap matters. Please state the exact question IDs or provide a fully held-out evaluation.

  2. Accuracy improvement is not consistent. In Table 1, RATIO slightly reduces average accuracy for Qwen-14B with AWQ-W3, and gives only a 0.06-point gain for Llama-8B with AWQ-W3. On Qwen-14B with GPTQ-W3, its average accuracy is also below the fixed-penalty baseline. The paper should discuss these cases more directly. The “up to 9.78 points” result mainly describes one setting.

  3. There is no uncertainty analysis for the main accuracy results. Some differences are small, especially on AIME and GPQA. Multiple generation seeds, confidence intervals, or paired significance tests are needed before concluding that one method improves accuracy over another.

  4. Shorter CoT is not yet a complete inference-cost result. The paper reports generated CoT tokens, but does not report end-to-end latency, throughput, or the offline cost of producing the token list with full-precision and quantized models. These measurements would make the efficiency claim more convincing.

  5. Contextual token validation is hard to reproduce from the current description. The appendix gives detailed numerical gates, but a human-style judgment is still used to decide whether a token represents inefficient deliberation or useful self-correction. Please describe who made these decisions, what instructions they followed, and how stable the final token sets are.

Questions for authors

See weaknesses.

  • How much time and compute are needed to obtain the token list and penalties for a new model?
  • Could the authors test whether the same token list works when task type or prompt format changes?
9 comments

Response to Reviewer rbq3 (denoted as R3)

Q3-1: The separation between analysis and evaluation is unclear.

A3-1: We thank the reviewer for bringing this issue to our attention, and we sincerely apologize for this oversight. In the original submission, we mistakenly included the 50 AIME and 50 GPQA-Diamond questions used during QRBA when computing the reported evaluation results. To correct this, we removed these questions from the existing evaluation outputs and recomputed the results using the remaining 70 AIME questions and 148 GPQA-Diamond questions, without rerunning model inference. The corrected results have been updated in the revised PDF, and the same correction has been applied to all corresponding results reported in this rebuttal.

Importantly, this correction does not alter our original conclusions: RATIO continues to achieve the best overall accuracy and the shortest average CoT length across the reported comparisons. We also confirm that none of the questions used during QRBA are included in the corresponding reported AIME and GPQA-Diamond evaluation results.

Q3-2: Accuracy improvements are inconsistent across models and quantization settings.

A3-2: We thank the reviewer for this helpful suggestion. We agree that accuracy improvements vary across different models and quantization methods, and that the gains are not equally substantial in all settings. To clarify this point, we have added the following statement to the Overall Accuracy subsection:

"Accuracy gains vary across models and quantizers, but RATIO consistently improves average accuracy over the corresponding quantized baselines across all evaluated configurations."

The limitation is that RATIO does not always outperform stronger fixed-penalty baselines in accuracy. Nevertheless, RATIO consistently achieves higher average accuracy than the corresponding uncalibrated quantized baselines. Furthermore, we also clarify that RATIO is designed to achieve a favorable accuracy–efficiency trade-off, rather than maximizing accuracy alone. For example, on Qwen-14B with GPTQ-W3, RATIO achieves an average accuracy of 73.53%, slightly below the fixed-penalty baseline (73.63%). However, it reduces the average CoT length from 5.95k to 4.86k tokens, an additional reduction of approximately 18.3%. This illustrates that even when its accuracy is marginally lower than that of the fixed-penalty baseline, RATIO can provide substantially greater inference efficiency.

Q3-3: There is no uncertainty analysis for the main accuracy results.

A3-3: We thank the reviewer for this valuable suggestion. To examine the sensitivity of our results to generation randomness, we conduct additional experiments using two different random seeds (43 and 44), while keeping the other experimental settings unchanged. The original experiments used seed 42. We first repeat the Qwen-1.5B experiments under both AWQ and GPTQ and report the results below.

(1) Multiple-Seed Evaluation on Qwen-1.5B.

SeedQuantizationMethodGPQAAIMEGSM8KMATH500HumanEvalAvg. AccAvg. CoT (k tokens)
43AWQBase27.708.5757.1643.0017.0730.7041.03
43AWQFixed λ=421.628.5770.2058.0029.8837.6525.38
43AWQRATIO29.057.1473.8465.2037.8042.6121.06
44AWQBase22.975.7159.7443.8018.2930.1039.42
44AWQFixed λ=425.0010.0069.2962.2034.1540.1325.02
44AWQRATIO19.595.7175.2163.8036.5940.1818.86
43GPTQBase20.274.2970.2050.4012.2031.4729.81
43GPTQFixed λ=430.415.7172.1856.6016.4636.2723.68
43GPTQRATIO26.352.8673.2461.2022.5637.2422.47
44GPTQBase22.307.1470.0554.6012.8033.3828.77
44GPTQFixed λ=425.685.7171.4256.4017.0735.2623.75
44GPTQRATIO33.111.4372.3360.4021.3437.7220.97

Accuracy (%) and CoT length (k tokens) are reported.

Across both seeds, RATIO consistently achieves the highest average accuracy and the shortest average CoT length among the three methods under both AWQ and GPTQ. These results indicate that its overall accuracy and efficiency advantages persist across different generation seeds, rather than depending on a particular random seed.

(2) Additional Evaluation on LLaMA-8B.

To further examine whether these observations extend beyond Qwen-1.5B, we conduct additional experiments on LLaMA-8B using another generation seed (43). The results are summarized below.

QuantizationMethodAvg. Acc (%)Avg. CoT (k tokens)
AWQBase57.837.56
AWQFixed λ=355.476.47
AWQRATIO58.036.72
GPTQBase53.6811.28
GPTQFixed λ=353.5710.22
GPTQRATIO53.6510.04

For each benchmark, results are reported as Acc (%) / Len (k).

On LLaMA-8B, RATIO also demonstrates favorable results under both quantization methods, achieving higher average accuracy under AWQ and shorter CoT lengths under both AWQ and GPTQ. This provides further evidence that the observed benefits are not restricted to a single model.

Overall, these additional experiments demonstrate the robustness of RATIO across different generation seeds and model architectures, with consistent improvements in overall accuracy and reasoning efficiency.

Q3-4: CoT length alone is insufficient to establish practical inference efficiency.

A3-4: We thank the reviewer for this helpful suggestion. CoT length provides a meaningful indication of inference cost, as autoregressive generation incurs computational costs for each generated token. However, it does not fully capture actual inference efficiency. To address this limitation, we conducted additional experiments using real GPTQ W3 quantized models and measured both inference throughput and total wall-clock time.

The results on Qwen-1.5B are reported below.

ModelConfigurationThroughput (tokens/s)Total Time (hours)
Qwen-1.5BGPTQ W3 Base584.715.45
Qwen-1.5BGPTQ W3 + RATIO613.983.04

RATIO increases inference throughput from 584.71 to 613.98 tokens/s and reduces the total inference time by 44.1%, demonstrating that the reduction in CoT length translates into substantial practical efficiency gains.

Regarding offline calibration overhead, RATIO requires about 25 GPU-hours for one-time calibration per model configuration. Importantly, the resulting token list and corresponding penalties can be shared across users and reused for subsequent inference requests without recalibration. Based on the measured inference time savings of approximately 2.4 hours per equivalent workload, the calibration overhead can be amortized after approximately 11 inference runs, assuming comparable single-GPU execution conditions.

Q3-5: Reproducibility and stability of contextual token validation.

A3-5: We thank the reviewer for raising this important concern. Contextual review is performed by GPT-5.6 using a fixed prompt. We (the authors) subsequently inspect the LLM's recommendations and determine the final retention decisions. The prompt instructs the LLM to examine the functional role of each candidate across distinct reasoning trajectories, considering both supporting and opposing examples. In particular, it distinguishes unproductive repetition, repeated reconsideration without new evidence, and unresolved speculation from useful verification and effective self-correction. A token is recommended for retention only when multiple distinct trajectories support its recurrent association with unproductive reasoning. We also provide the complete prompt in the supplementary material.

For reproducibility, we have included the full contextual-review prompt in Appendix E.2 of the revised PDF.

To evaluate the consistency of contextual review, we repeated the procedure twice on Qwen-1.5B using the same prompt and sampling setup. Among the eight reviewed tokens, seven received consistent retain/exclude recommendations, with only one differing between the two runs. Furthermore, only a small number of candidates need contextual review under the specified numerical conditions. Therefore, uncertainty in the LLM's retain/exclude decisions for these borderline candidates has a limited influence on the composition of the final token set.

Q3-6: How much time and compute are needed to obtain the token list and penalties for a new model?

A3-6: We thank the reviewer for this question. Obtaining the token list for a new model requires approximately 25 GPU-hours on an NVIDIA A6000. The subsequent penalty computation only requires CPU resources and takes approximately 5 to 10 minutes, introducing negligible additional overhead. The API cost for GPT-5.6 contextual review is approximately $2.5.

Importantly, RATIO requires only one-time calibration for each model configuration, while the resulting token list and penalties can be reused across subsequent inference runs and shared among multiple users without recalibration. For example, on MATH-500, a single inference run saves over 2 hours compared with the uncalibrated quantized model. Consequently, the initial calibration cost of 25 GPU-hours can be fully amortized after approximately 11 repeated inference runs.

Q3-7: Generalization of the selected token sets to different task types or prompt formats.

A3-7: We thank the reviewer for this helpful suggestion. To evaluate cross-task generalization, we additionally test RATIO on LiveCodeBench for code generation. We directly apply the model-specific token sets and corresponding penalties obtained by RATIO, without task-specific token reselection.

The results are presented below.

ModelMethodLiveCodeBench Acc (%)CoT Length (k tokens)
Qwen-7BBF16 Base34.3312.59
AWQ W3 Base27.2419.45
+ Lotfi et al. (2026)25.7517.44
+ RATIO27.6116.21
GPTQ W3 Base19.0324.85
+ Lotfi et al. (2026)20.9027.30
+ RATIO23.1323.26
Qwen-1.5BBF16 Base15.3020.25
AWQ W3 Base3.7350.44
+ Lotfi et al. (2026)4.4848.28
+ RATIO5.2242.13
GPTQ W3 Base2.2447.93
+ Lotfi et al. (2026)2.9947.78
+ RATIO3.3643.75

For each benchmark, results are reported as Acc (%) / Len (k).

RATIO consistently improves accuracy and reduces CoT length across both models and quantization methods on LiveCodeBench, outperforming both the uncalibrated quantized baselines and fixed-token-and-penalty method[1]. These results demonstrate that RATIO can also be effectively applied to other evaluation benchmarks.

[1] Quantized reasoning models think they need to think longer, but they do not. In NeurIPS, 2026.

Reviewer_3ANaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RATIO, a training-free decoding calibration framework for quantized reasoning models. Its main additions are model-specific token selection and token-specific logit penalties. Quantization-aware Reasoning Behavior Analysis combines probability-shift measurements under shared reference prefixes with validation on freely generated reasoning trajectories to identify candidate overthinking tokens. Token-Specific Penalty Determination then estimates static penalties using full-precision probabilities, aggregates evidence across AWQ and GPTQ, and applies the resulting biases during decoding.

The evaluation covers five reasoning models and five benchmarks, primarily using 3-bit weight-only AWQ/GPTQ quantization, with an additional FlatQuant W4A4KV4 experiment on Qwen-1.5B. The results demonstrate reductions in average CoT length, generally accompanied by preserved or improved accuracy relative to uncalibrated quantized models. The largest reported improvements are 9.78 percentage points in average accuracy and a 51.34% reduction in average CoT length.

Critical strengths
  1. Useful empirical improvements with a simple deployment mechanism. Once calibrated, RATIO only requires static logit adjustments, without additional model forward passes during decoding. The substantial improvements on Qwen-1.5B demonstrate practical potential for mitigating inefficient reasoning under aggressive quantization.

  2. Evaluation across multiple models and tasks. The experiments cover several model sizes, two architecture families, and mathematical, scientific, and coding benchmarks. The additional FlatQuant experiment provides preliminary evidence that the approach can extend beyond weight-only quantization.

Critical weaknesses
  1. The incremental contribution over prior token-penalty methods is not sufficiently isolated or established. Lotfi et al. already identify quantization-amplified overthinking and propose suppressing overthinking markers through logit penalties. RATIO primarily adds model-specific token selection and token-specific penalty estimation. However, the evidence for the latter is mixed, and the component ablation is limited to Qwen-1.5B. In Table 2, applying a uniform penalty of λ=1\lambda=1 to the QRBA-selected tokens achieves both higher accuracy and shorter CoT than full RATIO under GPTQ-W3: 38.68% versus 37.40% accuracy, and 21.91k versus 22.05k tokens. The paper acknowledges this result, but its broader claim of more reliable token-specific calibration is not convincingly established. Comparisons against uniformly penalized selected tokens across more models, and shuffled per-token penalties with the same penalty distribution, would help determine whether the specific token-to-penalty assignment contributes beyond token selection and overall suppression strength.

  2. The calibration pipeline is insufficiently specified and costed, making its reproducibility and practical advantage difficult to assess. Although Appendix C provides numerical gates, it does not sufficiently operationalize the lexical recovery and contextual review steps that determine which candidates are retained. The review agent, decision protocol, context sampling procedure, and consistency controls are unclear. Furthermore, calibration requires full-precision scoring, rollouts from two quantized variants, and independent full-precision generations, but no stage-wise runtime, memory, or review cost is reported. The efficiency evaluation measures CoT length rather than wall-clock latency and does not compare the additional calibration cost with the underlying GPTQ/AWQ quantization cost. Consequently, the deployment benefits are promising, but the end-to-end cost and reproducibility of obtaining them remain unresolved.

  3. The evaluation also concentrates on aggressive low-bit settings, with substantial residual accuracy degradation in several configurations even after applying RATIO. This leaves an important deployment question unanswered: whether RATIO offers meaningful efficiency gains when quantization already preserves near-full-precision accuracy. Higher-precision experiments would help establish whether the method provides value in capability-preserving deployment settings, beyond partially recovering heavily degraded models.

Questions for authors
  1. Calibration cost and practical efficiency. What are the runtime, GPU-hours, peak memory, and any human or external-model costs of QSTI, RTV, and TSPD? Which computations are reused between stages? Please compare the additional calibration cost with GPTQ/AWQ quantization and report measured inference latency or a break-even estimate for amortizing calibration.

  2. Shared versus quantizer-specific calibration. What motivates using the same token set and penalty values for AWQ and GPTQ, particularly the minimum of their median correction magnitudes? How does this compare with estimating penalties separately for each quantizer while holding the selected token set fixed?

  3. RTV reproducibility and evaluation separation. How are lexical recovery and contextual review implemented: deterministic rules, human annotation, or an LLM? Please provide the applicable rules, prompts, context sampling procedure, and retention criteria. Are all questions used for token selection, correctness-conditioned validation, and parameter selection excluded from the reported evaluation, particularly for AIME and GPQA?

  4. Utility at higher precision. The experiments only cover ≤ 4-bit quantization, and several configurations suffer substantial accuracy loss even with RATIO. Does the method still meaningfully reduce CoT length while preserving accuracy under higher-precision, near-lossless quantization, such as 8-bit? This would clarify its value in more practical deployment settings.

12 comments

Response to Reviewer 3ANa (denoted as R2)

Q2-1: What are RATIO's incremental contributions over prior token-penalty methods, and how can the effectiveness of token-specific penalty determination be further validated?

A2-1: We thank the reviewer for the question. Building on Lotfi et al., RATIO adds model-specific token selection through QRBA and token-specific penalties through TSPD, replacing predefined markers and uniform penalties. We further test TSPD using shuffled assignments and a broader uniform-penalty sweep.

(1) Shuffled Token–Penalty Assignments.

We randomly shuffle token–penalty assignments while preserving the selected token sets and penalty distributions on Qwen-7B and LLaMA-8B under AWQ and GPTQ. RATIO achieves higher average accuracy in all four settings, with superior or comparable CoT efficiency.

ModelMethodAIMEMATHGSM8KGPQAHumanEvalAvg.ΔAccΔLen%
Qwen-7BAWQ27.14/21.3587.80/6.4089.69/2.3347.30/9.7776.83/8.3465.75/9.64––
+Shuffled27.14/22.0688.60/4.8890.30/1.4747.30/9.2978.66/8.2966.40/9.20+0.65-4.56
+RATIO32.86/16.8088.60/5.0789.46/1.8247.97/8.3981.10/4.9868.00/7.41+2.25-23.13
GPTQ31.43/29.4486.80/11.0688.63/3.5540.54/11.3171.95/9.4963.87/12.97––
+Shuffled25.71/28.3987.20/6.9888.55/2.1743.24/9.6279.27/6.2364.79/10.68+0.92-17.66
+RATIO27.14/27.4589.60/6.3088.40/2.3743.92/9.4875.00/7.3664.81/10.59+0.94-18.35
LLaMA-8BAWQ17.14/16.8578.00/5.3983.40/2.4333.78/8.2172.56/5.9956.98/7.77––
+Shuffled15.71/14.8575.60/5.4882.64/2.1734.46/7.9072.56/5.0356.19/7.09-0.79-8.75
+RATIO18.57/16.6076.80/4.9583.32/2.0635.14/7.9573.78/3.7757.52/7.07+0.54-9.01
GPTQ17.14/23.4673.60/8.6274.37/1.7335.14/11.2070.73/7.9954.20/10.60––
+Shuffled15.71/21.6375.00/6.9274.15/1.6139.86/10.2566.46/6.6754.24/9.42+0.04-11.13
+RATIO14.29/22.4071.40/6.3373.77/1.5241.22/10.1175.00/6.8055.14/9.43+0.94-11.04

(2) Broader Uniform-Penalty Sweep.

We evaluate uniform penalties λ∈{0.5,1,1.5,2,3,4}\lambda\in\lbrace 0.5,1,1.5,2,3,4\rbrace on the same 17 selected Qwen-7B tokens. RATIO achieves the highest average accuracy under AWQ (68.00%) and GPTQ (64.81%) with competitive CoT lengths, supporting TSPD beyond Qwen-1.5B.

ModelMethodAIMEMATHGSM8KGPQAHumanEvalAvg.ΔAccΔLen%
Qwen-7BAWQ27.14/21.3587.80/6.4089.69/2.3347.30/9.7776.83/8.3465.75/9.64––
+Uniform λ=0.527.14/15.9687.60/5.1889.76/2.1247.30/8.5578.05/6.2165.97/7.60+0.22-21.16
+Uniform λ=125.71/20.9388.40/4.9288.86/2.0642.57/10.0380.49/7.0765.21/9.00-0.54-6.64
+Uniform λ=1.530.00/17.3988.80/4.7789.08/1.8445.27/7.9781.10/6.3366.85/7.66+1.10-20.54
+Uniform λ=227.14/16.8189.00/4.9890.07/1.7244.59/9.7276.22/4.7665.40/7.60-0.35-21.16
+Uniform λ=327.14/18.1085.40/5.1488.86/2.0546.62/8.0479.27/5.1465.46/7.69-0.29-20.23
+Uniform λ=424.29/16.8186.40/4.7889.16/2.0745.27/9.6477.44/4.2664.51/7.51-1.24-22.10
+RATIO32.86/16.8088.60/5.0789.46/1.8247.97/8.3981.10/4.9868.00/7.41+2.25-23.13
GPTQ31.43/29.4486.80/11.0688.63/3.5540.54/11.3171.95/9.4963.87/12.97––
+Uniform λ=0.528.57/32.7186.60/8.1089.01/2.3242.57/10.6374.39/6.8164.23/12.11+0.36-6.63
+Uniform λ=124.29/27.0687.60/6.7988.55/2.8543.24/10.6777.44/6.9064.22/10.85+0.35-16.35
+Uniform λ=1.524.29/29.4486.20/7.1588.02/2.3241.22/9.0577.44/7.1163.43/11.01-0.44-15.11
+Uniform λ=230.00/28.2187.60/6.2187.41/2.3937.16/9.5274.39/7.0663.31/10.68-0.56-17.66
+Uniform λ=322.86/28.8086.40/6.5288.40/2.4341.22/9.0875.00/7.3062.78/10.83-1.09-16.50
+Uniform λ=424.29/26.9885.20/6.2988.10/2.1243.92/9.6175.61/7.7363.42/10.55-0.45-18.66
+RATIO27.14/27.4589.60/6.3088.40/2.3743.92/9.4875.00/7.3664.81/10.59+0.94-18.35

For each benchmark, results are reported as Acc (%) / Len (k). ΔAcc and ΔLen% are measured relative to the corresponding quantized baseline.

Q2-2: How can the calibration pipeline be reproduced, and what are its computational costs and practical inference benefits?

A2-2:

(1) Calibration Pipeline and Reproducibility.

We thank the reviewer for raising these concerns. To clarify the calibration pipeline, lexical recovery uses a fixed vocabulary of 63 words, matching decoded token strings after removing leading/trailing whitespace and ignoring capitalization, without substring matching or stemming.

The contextual review is performed offline using GPT-5.6 with subsequent human verification, guided by a predefined prompt. The LLM evaluates each candidate token by distinguishing useful verification and self-correction from unproductive repetition and backtracking, recommending retention only when multiple distinct reasoning trajectories support its association with unproductive reasoning. Human reviewers then check these recommendations and determine the final token set. For context sampling, we use random seed 42 and extract 100 tokens before and after each sampled occurrence from the original AWQ and GPTQ reasoning trajectories. To ensure procedural consistency, we apply unified decision criteria, require consideration of both supporting and opposing evidence, and maintain traceable decisions through context IDs and explicit rationales, without using downstream evaluation results to guide selection.

For reproducibility, we have included the complete list of 63 candidate words and the full contextual-review prompt in Appendix E of the revised PDF.

(2) Calibration Cost Analysis.

We further report the computational cost of the calibration pipeline and compare it with the underlying quantization process. All measurements are conducted on an NVIDIA A6000 GPU.

StageGPU Time
Candidate Identification5 h
Token Validation20 h
Total Calibration25 h
GPTQ/AWQ Quantization~10 min

The calibration pipeline has a peak GPU memory usage of 47.46 GiB, with negligible CPU processing overhead. The three rounds of LLM-based contextual review incur an estimated API cost of approximately $2.5. Although calibration requires additional computation compared with quantization itself, it is performed only once per model configuration, and the resulting token-specific penalties can be reused across subsequent inference requests without repeated calibration.

(3) Inference Latency and Cost Amortization.

CoT length serves as a meaningful indicator of inference efficiency, as autoregressive generation incurs computational costs for each generated token, but it does not fully capture wall-clock latency. To further validate the practical benefits of RATIO, we additionally measure the actual wall-clock inference time on Qwen-1.5B under GPTQ-W3 real quantization.

ModelGPTQ-W3 BaseGPTQ-W3 + RATIO
Qwen-1.5B5:26:433:02:31

RATIO reduces the total inference time by 44.1%, demonstrating that the reduction in CoT length translates into substantial practical acceleration. Moreover, since RATIO requires only one-time calibration, its overhead can be amortized across repeated inference workloads. Under comparable single-GPU execution conditions, the 25 GPU-hour calibration cost can be offset after approximately 11 equivalent inference runs, while subsequent requests continue to benefit from reduced inference time.

Q2-3: Effectiveness at higher precision.

A2-3: We thank the reviewer for this valuable suggestion. We agree that evaluating RATIO only under aggressive low-bit quantization is insufficient to fully establish its practical utility. In particular, it is important to determine whether RATIO can still provide meaningful inference-efficiency improvements when quantization already preserves relatively high accuracy. To investigate this, we conduct additional experiments on Qwen-1.5B using GPTQ W4 and RTN W8, with the BF16 model serving as a full-precision reference.

The results are presented below:

MethodGPQA (Acc/CoT)AIME (Acc/CoT)GSM8K (Acc/CoT)MATH500 (Acc/CoT)HumanEval (Acc/CoT)Avg. AccAvg. CoT (k tokens)
BF16 Base37.16/9.5724.29/22.3484.69/2.8185.60/6.0673.17/6.5060.989.46
GPTQ W4 Base29.73/14.0518.57/21.7684.15/3.4881.80/7.6759.15/8.3154.6811.05
GPTQ W4 RATIO28.38/11.1721.43/15.2383.70/1.5383.20/4.9763.41/5.5456.027.69
RTN W8 Base27.03/9.8825.71/21.2284.53/2.5985.80/5.9564.63/7.2057.549.37
RTN W8 RATIO33.11/8.5725.71/14.7883.93/1.4985.40/4.0463.41/5.2858.316.83

For each benchmark, results are reported as Acc (%) / Len (k).

The results demonstrate that RATIO remains effective under higher-precision quantization. Under GPTQ W4, RATIO improves average accuracy by 1.34 percentage points while reducing average CoT length by 30.41%. More importantly, under RTN W8, RATIO further improves average accuracy from 57.54% to 58.31% and reduces average CoT length by 27.11% (9.37k to 6.83k tokens). Compared with the BF16 reference, RTN W8 RATIO achieves a comparable average accuracy of 58.31% versus 60.98%, while reducing average CoT length by 27.80%. These results demonstrate that RATIO provides substantial additional inference-efficiency gains even under higher-precision quantization, extending its benefits beyond aggressive low-bit settings.

Q2-4: What are the stage-wise calibration costs, how are computations reused, and how do these costs compare with quantization and inference savings?

A2-4: We thank the reviewer for this question. We provide a detailed breakdown of the computational costs of each calibration stage, clarify the reuse of intermediate data, and report the measured inference latency to assess the practical benefits of RATIO.

(1) Stage-wise Calibration Cost.

We report the computational costs of the three calibration stages, with all GPU measurements conducted on an NVIDIA A6000.

StageComputational Cost
Candidate Identification (QSTI)5 GPU-hours
Token Validation (RTV)20 GPU-hours
TSPD5~10 min (CPU)
Total Calibration25 GPU-hours
GPTQ/AWQ Quantization~10 min

The peak GPU memory usage of the calibration pipeline is 47.46 GiB, with negligible additional CPU processing overhead. The three rounds of GPT-5.6-based contextual review incur an estimated API cost of approximately $2.5.

(2) Computation Reuse.

The intermediate data collected during Candidate Identification (QSTI) are directly reused by TSPD for token-specific penalty estimation, avoiding redundant data generation and model inference. Consequently, TSPD does not require additional GPU computation, and its processing time is approximately 5~10 minutes.

(3) Practical Inference Efficiency and Cost Amortization.

We additionally measure the actual wall-clock inference time using the Qwen-1.5B model with real GPTQ W3 quantization to evaluate the practical efficiency gains of RATIO.

ModelGPTQ-W3 BaseGPTQ-W3 + RATIO
Qwen-1.5B5:26:433:02:31

RATIO reduces the total inference time by 44.1%. Assuming comparable single-GPU execution conditions, the 25 GPU-hour calibration cost can be offset after approximately 11 equivalent inference runs. Importantly, RATIO requires calibration only once per model configuration, and the resulting token set and corresponding penalties can be shared across users and reused for subsequent inference requests without recalibration.

Reviewer_kujVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RATIO, a training-free framework for addressing accuracy degradation and excessive reasoning length in quantized reasoning models. It identifies model-specific overthinking tokens through comparisons with full-precision models and assigns token-specific logit penalties during decoding. Across multiple models and quantization settings, the method generally improves the accuracy–efficiency trade-off, with reported gains of up to 9.8 percentage points in accuracy and reductions of up to 51.3% in CoT length.

Critical strengths
  • Interesting problem and a lightweight approach. RATIO addresses quantization-induced overthinking with training-free token-level calibration, making it practically appealing.

  • Strong results across diverse settings. Experiments across multiple model sizes, families, and quantization methods show substantial CoT reductions while generally preserving or improving accuracy.

Critical weaknesses
  • The ablations do not sufficiently isolate the proposed contributions. Table 2 only evaluates Qwen-1.5B, and a uniform penalty of 1 already outperforms TSPD under GPTQ in both accuracy and CoT length. I believe the following additional abaltions would make the contribution clearer: (1) keep the selected tokens fixed and compare TSPD with shuffled token–penalty assignments and a broader uniform-penalty sweep; (2) separately evaluate probability-shift selection, numerical trajectory validation, and contextual review, while controlling for token-set size; and (3) compare model-specific token sets with sets transferred from another compatible model under the same penalty rule. These tests would establish whether the gains come from the proposed selection and calibration mechanisms rather than simply suppressing a suitable group of tokens.

  • The connection between the selected tokens and overthinking remains insufficiently established. Higher token probabilities and associations with failed trajectories do not necessarily indicate overthinking; these tokens may also support useful verification or error correction. Despite repetition statistics and contextual review, it remains unclear whether RATIO selectively suppresses redundant reasoning. Shorter CoT and higher accuracy alone do not establish this distinction.

Questions for authors
  • How is the contextual review in RTV performed? Is it done by humans, an LLM, or predefined rules? How do you distinguish unnecessary repetition from useful verification or self-correction?

  • How much do the selected token sets overlap across model sizes and families? After accounting for tokenizer differences, are there common tokens or reasoning behaviors across models?

  • Could you use more intuitive names or fewer abbreviations? QRBA, RTV, and TSPD are difficult to remember, and I often needed to look back to recall their functions.

5 comments

Response to Reviewer kujV (denoted as R1)

Q1-1: The ablations do not sufficiently isolate the proposed contributions.

A1-1: We thank the reviewer for these constructive suggestions. To better isolate the contributions of RATIO's token selection and token-specific penalty determination, we conduct four additional ablation studies covering shuffled penalty assignments, a broader uniform-penalty sweep, component-wise ablations, and cross-model transfer. The detailed results are presented below.

(1) Shuffled Token–Penalty Assignments.

We keep the selected token sets fixed and randomly shuffle their corresponding penalties on Qwen-7B and LLaMA-8B under both AWQ and GPTQ. RATIO achieves higher average accuracy in all four settings and shorter CoT length, supporting the effectiveness of token-specific penalty assignments.

ModelMethodAIMEMATHGSM8KGPQAHumanEvalAvg.ΔAccΔLen%
Qwen-7BAWQ27.14/21.3587.80/6.4089.69/2.3347.30/9.7776.83/8.3465.75/9.64––
+Shuffled27.14/22.0688.60/4.8890.30/1.4747.30/9.2978.66/8.2966.40/9.20+0.65-4.56
+RATIO32.86/16.8088.60/5.0789.46/1.8247.97/8.3981.10/4.9868.00/7.41+2.25-23.13
GPTQ31.43/29.4486.80/11.0688.63/3.5540.54/11.3171.95/9.4963.87/12.97––
+Shuffled25.71/28.3987.20/6.9888.55/2.1743.24/9.6279.27/6.2364.79/10.68+0.92-17.66
+RATIO27.14/27.4589.60/6.3088.40/2.3743.92/9.4875.00/7.3664.81/10.59+0.94-18.35
LLaMA-8BAWQ17.14/16.8578.00/5.3983.40/2.4333.78/8.2172.56/5.9956.98/7.77––
+Shuffled15.71/14.8575.60/5.4882.64/2.1734.46/7.9072.56/5.0356.19/7.09-0.79-8.75
+RATIO18.57/16.6076.80/4.9583.32/2.0635.14/7.9573.78/3.7757.52/7.07+0.54-9.01
GPTQ17.14/23.4673.60/8.6274.37/1.7335.14/11.2070.73/7.9954.20/10.60––
+Shuffled15.71/21.6375.00/6.9274.15/1.6139.86/10.2566.46/6.6754.24/9.42+0.04-11.13
+RATIO14.29/22.4071.40/6.3373.77/1.5241.22/10.1175.00/6.8055.14/9.43+0.94-11.04

(2) Broader Uniform-Penalty Sweep.

We evaluate uniform penalties λ∈{0.5,1,1.5,2,3,4}\lambda\in\lbrace 0.5,1,1.5,2,3,4\rbrace on the same 17 selected tokens of Qwen-7B. RATIO achieves the highest average accuracy under both AWQ (68.00%) and GPTQ (64.81%), together with competitive CoT lengths, further demonstrating the advantages of token-specific calibration beyond the Qwen-1.5B setting examined in the original ablation study.

ModelMethodAIMEMATHGSM8KGPQAHumanEvalAvg.ΔAccΔLen%
Qwen-7BAWQ27.14/21.3587.80/6.4089.69/2.3347.30/9.7776.83/8.3465.75/9.64––
+Uniform λ=0.527.14/15.9687.60/5.1889.76/2.1247.30/8.5578.05/6.2165.97/7.60+0.22-21.16
+Uniform λ=125.71/20.9388.40/4.9288.86/2.0642.57/10.0380.49/7.0765.21/9.00-0.54-6.64
+Uniform λ=1.530.00/17.3988.80/4.7789.08/1.8445.27/7.9781.10/6.3366.85/7.66+1.10-20.54
+Uniform λ=227.14/16.8189.00/4.9890.07/1.7244.59/9.7276.22/4.7665.40/7.60-0.35-21.16
+Uniform λ=327.14/18.1085.40/5.1488.86/2.0546.62/8.0479.27/5.1465.46/7.69-0.29-20.23
+Uniform λ=424.29/16.8186.40/4.7889.16/2.0745.27/9.6477.44/4.2664.51/7.51-1.24-22.10
+RATIO32.86/16.8088.60/5.0789.46/1.8247.97/8.3981.10/4.9868.00/7.41+2.25-23.13
GPTQ31.43/29.4486.80/11.0688.63/3.5540.54/11.3171.95/9.4963.87/12.97––
+Uniform λ=0.528.57/32.7186.60/8.1089.01/2.3242.57/10.6374.39/6.8164.23/12.11+0.36-6.63
+Uniform λ=124.29/27.0687.60/6.7988.55/2.8543.24/10.6777.44/6.9064.22/10.85+0.35-16.35
+Uniform λ=1.524.29/29.4486.20/7.1588.02/2.3241.22/9.0577.44/7.1163.43/11.01-0.44-15.11
+Uniform λ=230.00/28.2187.60/6.2187.41/2.3937.16/9.5274.39/7.0663.31/10.68-0.56-17.66
+Uniform λ=322.86/28.8086.40/6.5288.40/2.4341.22/9.0875.00/7.3062.78/10.83-1.09-16.50
+Uniform λ=424.29/26.9885.20/6.2988.10/2.1243.92/9.6175.61/7.7363.42/10.55-0.45-18.66
+RATIO27.14/27.4589.60/6.3088.40/2.3743.92/9.4875.00/7.3664.81/10.59+0.94-18.35

Continuation of A1-1

(3) Component-wise Ablation Study.

We separately evaluate probability-shift selection, numerical trajectory validation, contextual review, and their combinations on Qwen-7B, keeping the token-set size fixed at 17 across all variants. The complete RATIO pipeline achieves the highest average accuracy and shortest CoT length under both AWQ and GPTQ, highlighting the complementary contributions of these different components.

ModelMethodAIMEMATHGSM8KGPQAHumanEvalAvg.ΔAccΔLen%
Qwen-7BAWQ27.14/21.3587.80/6.4089.69/2.3347.30/9.7776.83/8.3465.75/9.64––
+Probability-Shift Selection31.43/19.9988.40/5.4089.92/2.1339.86/9.3278.05/7.1865.53/8.80-0.22-8.71
+Numerical Trajectory Validation22.86/20.0490.80/5.4389.99/2.2645.95/9.3980.49/7.2666.02/8.88+0.27-7.88
+Contextual Review28.57/16.4589.80/4.8390.60/2.2648.65/8.3380.49/5.2467.62/7.42+1.87-23.03
+Numerical without Contextual Review25.71/21.2089.00/5.8689.99/2.4154.05/9.8481.10/7.6467.97/9.39+2.22-2.59
+RATIO32.86/16.8088.60/5.0789.46/1.8247.97/8.3981.10/4.9868.00/7.41+2.25-23.13
GPTQ31.43/29.4486.80/11.0688.63/3.5540.54/11.3171.95/9.4963.87/12.97––
+Probability-Shift Selection20.00/31.3486.80/7.5388.93/2.4543.24/11.2573.78/9.1862.55/12.35-1.32-4.78
+Numerical Trajectory Validation24.29/33.2884.40/10.9989.46/3.3141.89/11.4378.05/7.9063.62/13.38-0.25+3.16
+Contextual Review28.57/27.9387.40/6.6588.93/2.0043.24/9.8775.00/6.8964.63/10.67+0.76-17.73
+Numerical without Contextual Review28.57/38.0385.60/10.9987.41/3.7540.54/11.9276.22/8.2863.67/14.59-0.20+12.49
+RATIO27.14/27.4589.60/6.3088.40/2.3743.92/9.4875.00/7.3664.81/10.59+0.94-18.35

(4) Cross-model Token–Penalty Transfer.

We examine bidirectional transfer between Qwen-1.5B and Qwen-7B, transferring the selected tokens together with their corresponding penalties. Transferred configurations consistently improve over the quantized baselines, while model-specific RATIO achieves higher accuracy and shorter CoT length across all four settings, indicating partial cross-model transferability as well as the benefits of model-specific calibration.

ModelMethodAIMEMATHGSM8KGPQAHumanEvalAvg.ΔAccΔLen%
Qwen-1.5BAWQ5.71/56.7144.80/36.1361.18/22.1622.97/36.2817.68/39.8130.47/38.22––
+Transferred (7B)5.71/35.6760.60/14.9069.29/7.8724.32/20.5940.24/20.0540.03/19.82+9.56-48.14
+RATIO7.14/26.4562.80/13.8672.18/3.6325.68/19.7035.37/22.6840.63/17.26+10.16-54.84
GPTQ4.29/51.1454.00/24.2768.76/9.5429.05/23.4210.37/39.3133.29/29.54––
+Transferred (7B)4.29/42.0160.20/16.8270.58/5.8027.03/25.3123.78/28.1637.18/23.62+3.89-20.04
+RATIO4.29/42.9161.40/14.0671.49/3.1131.08/20.7318.29/29.9037.31/22.14+4.02-25.05
Qwen-7BAWQ27.14/21.3587.80/6.4089.69/2.3347.30/9.7776.83/8.3465.75/9.64––
+Transferred (1.5B)30.00/17.0089.60/4.7589.61/1.5343.92/8.6479.27/5.9566.48/7.57+0.73-21.47
+RATIO32.86/16.8088.60/5.0789.46/1.8247.97/8.3981.10/4.9868.00/7.41+2.25-23.13
GPTQ31.43/29.4486.80/11.0688.63/3.5540.54/11.3171.95/9.4963.87/12.97––
+Transferred (1.5B)27.14/28.1688.00/6.9488.25/1.7642.57/11.1076.83/7.6164.56/11.11+0.69-14.34
+RATIO27.14/27.4589.60/6.3088.40/2.3743.92/9.4875.00/7.3664.81/10.59+0.94-18.35

For each benchmark, results are reported as Acc (%) / Len (k).

ΔAcc and ΔLen% are measured relative to the corresponding quantized baseline.

Q1-2: The connection between the selected tokens and overthinking remains insufficiently established.

A1-2: We thank the reviewer for this insightful comment. We acknowledge that RATIO applies token-level penalties and cannot dynamically distinguish beneficial verification from redundant reasoning based on the context of each token occurrence. Nevertheless, our extensive experiments across multiple models, quantization settings, and benchmarks demonstrate that RATIO effectively reduces CoT length while maintaining or improving reasoning accuracy. Furthermore, as illustrated by the example in our appendix, RATIO can effectively mitigate redundant reasoning and repeated verification that may otherwise introduce errors.

Response
PromptCompute 992+99+199^2+99+1 in your head.
GPTQ-W3First, 992=(100−1)2=10,000−200+1=9,80199^2=(100-1)^2=10,000-200+1=9,801. Adding 99 gives 9,9009,900, and adding 1 gives 9,9019,901. ... Let me verify each step again. 99299^2 is 9,801, then 9,801+99=9,9009,801+99=9,900, and adding 1 gives 9,901. ... Alternatively, maybe I can factor it as 99×(99+1)+199\times(99+1)+1. That gives 99×100+1=9,90199\times100+1=9,901. Alternatively, if I think of 992+99+199^2+99+1 as (99+1)2−1(99+1)^2-1, that is 1002−1=9,999100^2-1=9,999. Wait, no, that is incorrect. ... So, 9999. ... 1099999999999999999999999999999999999999.
RATIOFirst, 992=(100−1)2=10,000−200+1=9,80199^2=(100-1)^2=10,000-200+1=9,801. Next, 9,801+99=9,9009,801+99=9,900, and 9,900+1=9,9019,900+1=9,901. Thus, the final answer is 9901\boxed{9901}.

In this example, GPTQ-W3 initially reaches the correct answer but continues unnecessary verification, eventually introducing errors. In contrast, RATIO produces the correct answer without redundant reasoning, illustrating its practical effectiveness in mitigating overthinking.

Q1-3: How is the contextual review in RTV performed? How to distinguish unnecessary repetition from useful verification or self-correction?

A1-3: We thank the reviewer for this question. The contextual review in RTV is performed offline using GPT-5.6 with subsequent human verification, rather than relying solely on predefined rules. After numerical screening, we provide the LLM with the original reasoning contexts surrounding each candidate token from both AWQ and GPTQ, requiring it to examine supporting and opposing evidence across distinct reasoning trajectories. Human reviewers then check the LLM's recommendations and make the final decisions on whether to retain each token.

Specifically, our prompt distinguishes useful verification, which examines concrete constraints, assumptions, or potential errors, and useful self-correction, which identifies and resolves specific mistakes, from unproductive reasoning, which repeatedly revisits established conclusions or cycles through alternatives without meaningful progress. The LLM recommends including a token in the penalty set only when multiple distinct reasoning trajectories provide evidence of its recurrent association with unproductive reasoning, while also accounting for its potential contributions to useful verification or self-correction. The complete prompt is provided in Appendix E.2 in the revised PDF.

Q1-4: Are there common selected tokens and overthinking behaviors across different model sizes and families?

A1-4: We thank the reviewer for this question. Comparing the selected tokens at the decoded-text level to account for tokenizer differences, we observe meaningful overlap across model sizes and families. After removing leading and trailing whitespace, lowercasing, and merging duplicate decoded forms, the pairwise Jaccard overlap between the five model-specific sets ranges from 20.8% to 45.0%. In particular, but, wait, and something appear in all five models, while tokens such as question, hmm, and back are also shared by multiple models.

Beyond token-level overlap, we identify common patterns of unproductive reasoning across models, including restarting or backtracking without introducing new evidence, repeatedly checking an already established result. For example, expressions such as "Wait, but maybe I'm missing something. Let me think again" and "Let me check ... again" can signal repeated reconsideration or verification without meaningful progress. These observations suggest that, despite model-specific differences in token selection, certain linguistic markers and associated overthinking behaviors are shared across different models.

Q1-5: Could the method terminology and abbreviations be simplified to improve readability?

A1-5: We thank the reviewer for this helpful suggestion. To improve readability, we have removed the abbreviations QSTI and RTV, replacing them with the more intuitive terms Candidate Identification and Token Validation, respectively. These changes have already been made in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.