acceptodds
Under review as a conference paper at ICLR 2027

Quantization for Reasoning Preservation in Small LLMs

Abstract

Large Language Models (LLMs) have demonstrated exceptional capabilities in textual inference, but their practical deployment is constrained by high computation and memory overhead. To address this, we propose LLM-QRP (Quantization for Reasoning Preservation), a layer-aware, mixed-precision quantization framework designed to reduce model memory footprint while preserving multi-step reasoning capabilities. Unlike standard uniform post-training quantization methods that treat all parameters equally, LLM-QRP identifies and protects critical "thinking" sub-components by analyzing layer-wise Self-Logits Evolution Decoding (SLED) and entropy dynamics across the residual stream. These signals are unified via Principal Component Analysis (PCA) into a data-driven reasoning importance score. We then formulate precision allocation as a 0-1 Multiple-Choice Knapsack Problem solved via Integer Linear Programming (ILP) to assign higher bit-widths to high-sensitivity components under strict VRAM targets. Experiments on sub-500M parameter models across GSM8K, TruthfulQA, and MMLU demonstrate that LLM-QRP achieves an optimal Pareto frontier between model compression and task accuracy, maintaining near-baseline reasoning performance while outperforming uniform 8-bit baselines and alternative quantization methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization, mixed-precision quantization, large language models, model compression, reasoning preservation, chain-of-thought, post-training quantization, edge deployment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ke83OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LLM-QRP, a quantization method that assigns different levels of precision to model layers according to their estimated importance for reasoning. Its main contribution is to use functional reasoning behavior to guide quantization, rather than relying on weight-level reconstruction errors or sensitivity metrics. Layers that contribute less to reasoning are quantized more aggressively, whereas layers with a substantial impact on reasoning are preserved at full precision. The authors evaluate the method on four models across GSM8K, TruthfulQA, and MMLU, comparing the performance of models quantized with LLM-QRP against those produced by other quantization methods. They also conduct component-selection ablation experiments to assess the relative importance of different components.

Critical strengths

A strength of the paper is its detailed mathematical formulation of the proposed method. It explicitly defines the logit-evolution and entropy-based profiling signals, their PCA-based aggregation, and the budget-constrained precision-allocation objective. These equations provide a concrete account of how reasoning-related signals are intended to guide quantization, allowing readers to scrutinize the assumptions, objective, and constraints. This makes the proposed approach easier to analyze and compare with other mixed-precision quantization methods.

Critical weaknesses
  1. The ablation study does not establish the benefit of incorporating SLED. On SmolLM2-135M, quantizing the least-active components selected using entropy change alone achieves higher retention than using either SLED alone or the fused score R = PCA(SLED, delta H). The reported Qwen results likewise show no clear advantage for the fusion. Thus, a simpler entropy-only criterion performs at least as well in this ablation, and the additional SLED signal and PCA aggregation are not sufficiently justified by the evidence. This weakens the case for the proposed combined profiling method.

  2. The evaluation does not directly demonstrate preservation of actual problem-solving performance. The reported comparisons rely on exp(-NLL), which measures the likelihood of reference answers under teacher forcing rather than the correctness of solutions generated by the quantized model. Retaining a similar answer-likelihood score does not necessarily imply that the model retains its ability to solve problems independently. Without generated-answer accuracy or other task-specific measures of practical performance, this proxy alone is insufficient to substantiate the central claim of reasoning preservation.

Questions for authors

The appendix results show that LLM-QRP does not consistently outperform existing baselines across models and datasets. Could you clarify the scope of the claimed advantages? Under matched memory budgets and consistent evaluation settings, what is the main benefit of LLM-QRP over the strongest baseline, and does the reported evidence establish that this benefit is robust to evaluation variability?

1 comment

Since I don't see any rebuttal by authors, my score stay put.

Best regards, Reviewer ke83

Reviewer_vN96OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reasoning aware mixed precision quantization framework named LLM-QRP. The main idea is to identify reasoning critical LLM components and allocate higher precision to them while applying more aggressive quantization to other components. Specifically, the authors apply Self-Logits Evolution Decoding (SLED) and entropy dynamics to measure component importance, and use PCA to aggregate these signals into a unified sensitivity score. Based on these scores, the authors use Integer Linear Programming (ILP) under memory constraints to solve the bit-width allocation problem. This method is evaluated on SmolLM2-135M and Qwen2.5-0.5B across GSM8K, TruthfulQA, and MMLU, with comparisons against uniform quantization and several existing quantization methods. The results suggest that LLM-QRP can reduce model storage by approximately 50% while maintaining near-BF16 performance measured by an exp(-NLL) retention metric.

Critical strengths
  1. Using Logits Evolution and Entropy Dynamics to analyze the roles of different components in the Cot generation process, rather than merely focusing on the quantitative errors of the final output. This motivation is quite good, and it was also observed in the experiments that these signals are helpful in identifying the components that are relatively sensitive to quantization.

  2. The paper proposes a relatively complete hybrid precision quantization framework for reasoning and perception called LLM-QRP, which combines SLED, entropy change analysis, PCA and integer linear programming. This combined approach is interesting and seems practical.

Critical weaknesses
  1. Although the paper compares with classic quantization methods such as GPTQ, AWQ, SpQR, and SmoothQuant, it lacks a comparison with more recent and competitive mixed precision quantization methods.

  2. The core objective of the paper is to retain the inference capability of small language models, but the experiments mainly used exp(-NLL) as the evaluation metric, rather than the accuracy of the final answer. The retention of the reference answer's predicted probability does not necessarily mean that the model can maintain correct multi-step inference ability during actual generation. Its effectiveness in real inference scenarios still needs to be verified.

  3. In the main experiments of the paper, LLM-QRP had an average quantization precision close to 8-bit, but there was a lack of sufficient assessment of the 3–4 bit quantization budget discussed in the method section. It has not yet been proven that this method still has a significant advantage in more aggressive low-bit quantization scenarios.

Questions for authors
  1. The author mentioned that the evaluated small models achieved an accuracy rate close to zero on the GSM8K dataset, which makes it difficult to assess the retention of reasoning ability by relying on the precise matching accuracy rate. Could the author consider conducting the evaluation on a simpler reasoning benchmark so that these models can obtain a meaningful non-zero accuracy rate? This would allow for a more direct determination of whether the proposed method truly retains the reasoning ability, rather than relying solely on exp(-NLL) as a proxy indicator.

  2. The paper measures the importance of components using SLED and Entropy Dynamics, and refers to this as the "Reasoning Sensitivity Score". However, there is a problem here: the Logits Evolution is large, which might merely indicate that a certain layer has a greater impact on the output distribution. The Entropy change is large, which might merely suggest that this component alters the uncertainty of the model's prediction. These phenomena do not necessarily mean that this component is specifically responsible for multi-step logical reasoning; it might just be a common quantitative sensitive component.

  3. Does quantification sensitivity depend on the Calibration Dataset? If we use a different set of calibration data, will the key components identified in the end remain the same?

1 comment
Reviewer_vN96OpenReview ↗

No response from authors

Since I don't see any rebuttal by authors, my score stay put.

Reviewer_Hxy8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method for post-training quantization. Specifically, the authors argue that different components of a network contribute to reasoning differently, and therefore it is important to quantize them differently. Overall, they show a mix-precision quantization at a module granularity work well with small LLMs.

Critical strengths
  1. The authors intuition on quantizing components differently is intuitive.
Critical weaknesses
  1. The authors restrict their method to small LLMs, which is counter-intuitive. Quantization is specifically useful for large LLMs, and quantization performance on sub-1B models will hardly scale to any modern sizes.

  2. Fair comparisons. Table 1 shows decent performance of the proposed method, but the compared methods seem to all have different budgets.

Questions for authors

Have you tried quantizing larger models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.