- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LLM-QRP, a quantization method that assigns different levels of precision to model layers according to their estimated importance for reasoning. Its main contribution is to use functional reasoning behavior to guide quantization, rather than relying on weight-level reconstruction errors or sensitivity metrics. Layers that contribute less to reasoning are quantized more aggressively, whereas layers with a substantial impact on reasoning are preserved at full precision. The authors evaluate the method on four models across GSM8K, TruthfulQA, and MMLU, comparing the performance of models quantized with LLM-QRP against those produced by other quantization methods. They also conduct component-selection ablation experiments to assess the relative importance of different components.
Critical strengths
A strength of the paper is its detailed mathematical formulation of the proposed method. It explicitly defines the logit-evolution and entropy-based profiling signals, their PCA-based aggregation, and the budget-constrained precision-allocation objective. These equations provide a concrete account of how reasoning-related signals are intended to guide quantization, allowing readers to scrutinize the assumptions, objective, and constraints. This makes the proposed approach easier to analyze and compare with other mixed-precision quantization methods.
Critical weaknesses
-
The ablation study does not establish the benefit of incorporating SLED. On SmolLM2-135M, quantizing the least-active components selected using entropy change alone achieves higher retention than using either SLED alone or the fused score R = PCA(SLED, delta H). The reported Qwen results likewise show no clear advantage for the fusion. Thus, a simpler entropy-only criterion performs at least as well in this ablation, and the additional SLED signal and PCA aggregation are not sufficiently justified by the evidence. This weakens the case for the proposed combined profiling method.
-
The evaluation does not directly demonstrate preservation of actual problem-solving performance. The reported comparisons rely on exp(-NLL), which measures the likelihood of reference answers under teacher forcing rather than the correctness of solutions generated by the quantized model. Retaining a similar answer-likelihood score does not necessarily imply that the model retains its ability to solve problems independently. Without generated-answer accuracy or other task-specific measures of practical performance, this proxy alone is insufficient to substantiate the central claim of reasoning preservation.
Questions for authors
The appendix results show that LLM-QRP does not consistently outperform existing baselines across models and datasets. Could you clarify the scope of the claimed advantages? Under matched memory budgets and consistent evaluation settings, what is the main benefit of LLM-QRP over the strongest baseline, and does the reported evidence establish that this benefit is robust to evaluation variability?
Since I don't see any rebuttal by authors, my score stay put.
Best regards, Reviewer ke83