- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses two potential issues arising from parameter quantization in reasoning models: reduced reasoning accuracy and overthinking (i.e., longer chains of thought). Building on existing marker-based strategies, RATIO introduces a more fine-grained approach. It first compares the reasoning behaviors of full-precision and quantized models, then uses QRBA to identify model-specific tokens associated with inefficient reasoning that should be suppressed. Finally, TSPD assigns token-specific penalties to reduce the likelihood of selecting these tokens during generation. The method operates entirely at the inference-time decoding stage and does not require model retraining.
The main novelty lies in the systematic discovery of model-specific target tokens and the assignment of differentiated penalties, rather than relying on predefined markers and uniform penalties.
One limitation is that, although QRBA considers reasoning context during token selection, the penalties applied to selected tokens remain fixed during inference and do not dynamically adapt to the current reasoning state. The authors also acknowledge this limitation in the Discussion section. Moreover, token-specific penalties do not consistently outperform uniform penalties. For example, on Qwen-1.5B GPTQ-W3, a uniform penalty (QRBA, λ=1.0) achieves slightly better average accuracy and shorter CoT length than the full RATIO method. Finally, the evaluation of inference optimization primarily relies on reasoning accuracy and CoT length. Whether these metrics adequately reflect actual latency, throughput, and offline calibration costs requires more direct validation.
Critical strengths
- Empirical evaluation covers five models (1.5B–14B), multiple model architectures, and several quantization settings.
- Training-free Token Discovery and Calibration achieve substantial reductions in Chain-of-Thought (CoT) length compared to unmodified quantized baselines across various models and quantization settings.
Critical weaknesses
1.The benefits of token-specific calibration are not consistent across all settings. In the ablation study on Qwen-1.5B GPTQ-W3, the uniform penalty (QRBA, λ=1.0) achieves an average accuracy of 38.68% and a CoT length of 21.91k tokens, slightly outperforming the full RATIO method (37.40% accuracy and 22.05k tokens) on both metrics.
2.System-level efficiency evaluation is insufficient. The paper primarily uses CoT length as a proxy for efficiency, without sufficiently evaluating actual latency, throughput, or offline calibration costs. Therefore, stronger empirical evidence is needed to support claims about practical deployment benefits.
Questions for authors
-
In the ablation study on Qwen-1.5B GPTQ-W3, the uniform penalty (λ=1.0) slightly outperforms the full RATIO method in both average accuracy and CoT length. Could the authors explain this exception and discuss whether the benefits of token-specific penalties are consistent across different quantization methods?
-
The largest model evaluated in the current experiments has 14B parameters. Could the authors further validate RATIO on larger quantized reasoning models or additional model families to provide stronger evidence of its generalizability?
9 comments
Continuation of A4-1
(2) Shuffled Token–Penalty Assignments.
To further examine the importance of token-specific penalty assignments, we keep the selected token sets fixed and randomly shuffle their corresponding penalties on Qwen-7B and LLaMA-8B under both AWQ and GPTQ. RATIO consistently achieves higher average accuracy than shuffled assignments in all four settings. Meanwhile, RATIO achieves shorter average CoT lengths in three of the four settings, with only a marginal difference in the remaining setting. This demonstrates that the specific correspondence between tokens and their assigned penalties matters.
| Model | Method | AIME | MATH | GSM8K | GPQA | HumanEval | Avg. | ΔAcc | ΔLen% |
|---|---|---|---|---|---|---|---|---|---|
| Qwen-7B | AWQ | 27.14/21.35 | 87.80/6.40 | 89.69/2.33 | 47.30/9.77 | 76.83/8.34 | 65.75/9.64 | – | – |
| +Shuffled | 27.14/22.06 | 88.60/4.88 | 90.30/1.47 | 47.30/9.29 | 78.66/8.29 | 66.40/9.20 | +0.65 | -4.56 | |
| +RATIO | 32.86/16.80 | 88.60/5.07 | 89.46/1.82 | 47.97/8.39 | 81.10/4.98 | 68.00/7.41 | +2.25 | -23.13 | |
| GPTQ | 31.43/29.44 | 86.80/11.06 | 88.63/3.55 | 40.54/11.31 | 71.95/9.49 | 63.87/12.97 | – | – | |
| +Shuffled | 25.71/28.39 | 87.20/6.98 | 88.55/2.17 | 43.24/9.62 | 79.27/6.23 | 64.79/10.68 | +0.92 | -17.66 | |
| +RATIO | 27.14/27.45 | 89.60/6.30 | 88.40/2.37 | 43.92/9.48 | 75.00/7.36 | 64.81/10.59 | +0.94 | -18.35 | |
| LLaMA-8B | AWQ | 17.14/16.85 | 78.00/5.39 | 83.40/2.43 | 33.78/8.21 | 72.56/5.99 | 56.98/7.77 | – | – |
| +Shuffled | 15.71/14.85 | 75.60/5.48 | 82.64/2.17 | 34.46/7.90 | 72.56/5.03 | 56.19/7.09 | -0.79 | -8.75 | |
| +RATIO | 18.57/16.60 | 76.80/4.95 | 83.32/2.06 | 35.14/7.95 | 73.78/3.77 | 57.52/7.07 | +0.54 | -9.01 | |
| GPTQ | 17.14/23.46 | 73.60/8.62 | 74.37/1.73 | 35.14/11.20 | 70.73/7.99 | 54.20/10.60 | – | – | |
| +Shuffled | 15.71/21.63 | 75.00/6.92 | 74.15/1.61 | 39.86/10.25 | 66.46/6.67 | 54.24/9.42 | +0.04 | -11.13 | |
| +RATIO | 14.29/22.40 | 71.40/6.33 | 73.77/1.52 | 41.22/10.11 | 75.00/6.80 | 55.14/9.43 | +0.94 | -11.04 |
For each benchmark, results are reported as Acc (%) / Len (k).
Overall, these additional experiments provide strong evidence for the effectiveness of TSPD. Across a broader range of uniform penalties, RATIO consistently achieves the highest average accuracy while maintaining highly competitive CoT lengths. Furthermore, the shuffled-assignment results confirm that assigning penalties according to individual token characteristics is more effective than randomly mapping penalties to the selected tokens. Together, these findings further support the contribution of token-specific penalty to RATIO.
Q4-2: System-level efficiency evaluation is insufficient.
A4-2: We thank the reviewer for this valuable suggestion. CoT length serves as a meaningful indicator of inference efficiency, but it does not fully capture wall-clock latency. To provide stronger system-level evidence, we additionally measure actual wall-clock inference time and throughput using the Qwen-1.5B model with real GPTQ W3 quantization.
The results are reported below:
| Model | Configuration | Throughput (tokens/s) | Total Inference Time |
|---|---|---|---|
| 1.5B | Real GPTQ W3 Base | 584.71 | 5:26:43 |
| 1.5B | Real GPTQ W3 RATIO | 613.98 | 3:02:31 |
RATIO reduces the total inference time by 44.1% while increasing throughput by approximately 5.0%, demonstrating that its reduction in CoT length translates into practical inference acceleration.
We also report the offline calibration cost. The stage-wise computational costs are summarized below.
| Stage | Runtime | Computational Resource |
|---|---|---|
| Candidate Identification (QSTI) | 5 hours | GPU |
| Token validation (RTV) | 20 hours | GPU |
| Penalty Calculation (TSPD) | 5~10 minutes | CPU only |
| Total | 25 GPU-hours | NVIDIA A6000 |
The peak GPU memory usage is 47.46 GiB, and the external contextual review costs approximately $2.5. TSPD's penalty calculation requires no GPU resources and takes negligible time.
Importantly, the resulting token list and corresponding penalties can be shared across users and reused for subsequent inference requests without recalibration. Based on the measured inference time savings of approximately 2.4 hours per equivalent workload, the calibration overhead can be amortized after approximately 11 inference runs, assuming comparable single-GPU execution conditions.
Q4-3: The benefits of token-specific penalties are not consistent across settings.
A4-3: We thank the reviewer for raising this important question. We acknowledge that on Qwen-1.5B with GPTQ-W3, a uniform penalty of λ=1.0 slightly outperforms RATIO in both average accuracy and CoT length. This indicates that token-specific penalties do not necessarily outperform a well-chosen uniform penalty in every individual configuration. The effectiveness of penalty allocation can vary depending on the model and quantization method, and a particular global penalty may happen to provide a favorable trade-off for a specific configuration.
To further investigate this issue and examine whether the benefits of token-specific penalties extend across different quantization methods, we conduct additional experiments on Qwen-7B under AWQ and GPTQ, comparing RATIO against six different uniform penalty values (λ ∈ {0.5, 1, 1.5, 2, 3, 4}) while keeping the selected token set fixed.
The results show that RATIO achieves the highest average accuracy under both AWQ (68.00%) and GPTQ (64.81%), outperforming the best uniform-penalty configurations by 1.15 and 0.58 percentage points, respectively. Importantly, RATIO also achieves the shortest average CoT length under AWQ (7.41k tokens). Under GPTQ, its average CoT length (10.59k tokens) is only marginally longer than that of the shortest uniform-penalty configuration (10.55k tokens), while achieving 1.39 percentage points higher average accuracy. Overall, these results demonstrate the effectiveness of TSPD, showing that token-specific penalty allocation provides a better overall accuracy–efficiency trade-off than applying a single global suppression strength, supporting the effectiveness of TSPD across different quantization methods.
| Model | Method | GPQA (Acc/CoT) | AIME (Acc/CoT) | GSM8K (Acc/CoT) | MATH500 (Acc/CoT) | HumanEval (Acc/CoT) | Avg. Acc | Avg. CoT (k tokens) |
|---|---|---|---|---|---|---|---|---|
| Qwen-7B | AWQ Base | 47.30/9.77 | 27.14/21.35 | 89.69/2.33 | 87.80/6.40 | 76.83/8.34 | 65.75 | 9.64 |
| Uniform λ=0.5 | 47.30/8.55 | 27.14/15.96 | 89.76/2.12 | 87.60/5.18 | 78.05/6.21 | 65.97 | 7.60 | |
| Uniform λ=1 | 42.57/10.03 | 25.71/20.93 | 88.86/2.06 | 88.40/4.92 | 80.49/7.07 | 65.21 | 9.00 | |
| Uniform λ=1.5 | 45.27/7.97 | 30.00/17.39 | 89.08/1.84 | 88.80/4.77 | 81.10/6.33 | 66.85 | 7.66 | |
| Uniform λ=2 | 44.59/9.72 | 27.14/16.81 | 90.07/1.72 | 89.00/4.98 | 76.22/4.76 | 65.40 | 7.60 | |
| Uniform λ=3 | 46.62/8.04 | 27.14/18.10 | 88.86/2.05 | 85.40/5.14 | 79.27/5.14 | 65.46 | 7.69 | |
| Uniform λ=4 | 45.27/9.64 | 24.29/16.81 | 89.16/2.07 | 86.40/4.78 | 77.44/4.26 | 64.51 | 7.51 | |
| AWQ RATIO | 47.97/8.39 | 32.86/16.80 | 89.46/1.82 | 88.60/5.07 | 81.10/4.98 | 68.00 | 7.41 | |
| GPTQ Base | 40.54/11.31 | 31.43/29.44 | 88.63/3.55 | 86.80/11.06 | 71.95/9.49 | 63.87 | 12.97 | |
| Uniform λ=0.5 | 42.57/10.63 | 28.57/32.71 | 89.01/2.32 | 86.60/8.10 | 74.39/6.81 | 64.23 | 12.11 | |
| Uniform λ=1 | 43.24/10.67 | 24.29/27.06 | 88.55/2.85 | 87.60/6.79 | 77.44/6.90 | 64.22 | 10.85 | |
| Uniform λ=1.5 | 41.22/9.05 | 24.29/29.44 | 88.02/2.32 | 86.20/7.15 | 77.44/7.11 | 63.43 | 11.01 | |
| Uniform λ=2 | 37.16/9.52 | 30.00/28.21 | 87.41/2.39 | 87.60/6.21 | 74.39/7.06 | 63.31 | 10.68 | |
| Uniform λ=3 | 41.22/9.08 | 22.86/28.80 | 88.40/2.43 | 86.40/6.52 | 75.00/7.30 | 62.78 | 10.83 | |
| Uniform λ=4 | 43.92/9.61 | 24.29/26.98 | 88.10/2.12 | 85.20/6.29 | 75.61/7.73 | 63.42 | 10.55 | |
| GPTQ RATIO | 43.92/9.48 | 27.14/27.45 | 88.40/2.37 | 89.60/6.30 | 75.00/7.36 | 64.81 | 10.59 |
For each benchmark, results are reported as Acc (%) / Len (k).
Response to Reviewer pfyz (denoted as R4)
A4-1: We thank the reviewer for pointing out this issue. To further investigate the contribution of TSPD and provide a more comprehensive evaluation, we conduct two additional ablation studies: a broader uniform-penalty sweep and shuffled token–penalty assignments.
(1) Broader Uniform-Penalty Sweep.
We evaluate a broader range of uniform penalties, λ ∈ {0.5, 1, 1.5, 2, 3, 4}, while keeping the selected token set fixed on Qwen-7B. The results show that RATIO achieves the highest average accuracy under both AWQ (68.00%) and GPTQ (64.81%), outperforming the best uniform-penalty configurations by 1.15 and 0.58 percentage points, respectively. Importantly, RATIO also achieves the shortest average CoT length under AWQ (7.41k tokens). Under GPTQ, its average CoT length (10.59k tokens) is only marginally longer than that of the shortest uniform-penalty configuration (10.55k tokens), while achieving 1.39 percentage points higher average accuracy. These results demonstrate that TSPD provides a better overall accuracy–efficiency trade-off than a broad range of uniform penalties.
Acc (%) / CoT length (k tokens) are reported.