- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper examines how low-bit quantization changes specific model decisions. The authors measure a decision using the logit margin between two candidate first tokens before and after quantization, and define a "flip" when the sign of this margin changes. They evaluate 16 models from 8 families across several post-training quantization methods and decision types including tool use, safety/refusal, knowledge, and social bias.
The main finding is that the quantized margin can often be approximated as a regression , where captures shrinkage of the original margin and represents a systematic shift. The authors also find that different decision types can degrade at different rates; for example, under some 3-bit settings, whether to call a tool is substantially more sensitive than which tool to call.
Finally, the authors fit the slope, offset, and residual variance on half of the decisions in each condition and use them to predict the flip rate on the other half, reporting a median absolute error of 1.0 percentage point across 1,154 conditions.
Critical strengths
-
The empirical evaluation is broad, with 16 models from 8 families, several quantization methods and bit widths, and a range of decision types including tool use, refusal, knowledge, and social bias.
-
The experiments are generally careful and include a large number of controls. Specifically, the authors test alternative quantizers, activation/KV-cache quantization, alternative token pairs, and several checks on whether the measured margin corresponds to actual model behavior.
-
Some of the specific empirical observations are interesting. For example, the paper shows that whether-to-call decisions can degrade substantially before which-tool decisions under certain low-bit quantization settings, which is counterintuitive.
Critical weaknesses
-
My main concern is that I am not convinced the proposed affine relationship provides much actionable insight. It is a simple descriptive model of the relationship between pre- and post-quantization margins, but does not explain the underlying mechanism, and the fitted parameters need to be calibrated separately for the model, quantizer, bit width, and decision type being studied.
-
I am also unclear on the practical value of the forecasting result. If one already has access to the target quantized model and a representative validation set, one can directly measure the downstream behavior or flip rate of interest. The proposed model improves somewhat over simply reusing the calibration-set flip rate, but the improvement is modest on the harder mid-range conditions (4.3 pp median error versus 6.0 pp). More broadly, the paper identifies where quantization causes failures but provides relatively little guidance on what to do about them. The tested repair strategies do not yield a strong general mitigation, which limits the practical significance of the analysis in my view.
My overall judgment is that the work is extensive but does not provide enough new or actionable insight to meet the bar for ICLR.
Questions for authors
Main question: What practical use does the proposed forecasting framework provide beyond directly evaluating the quantized model on a representative validation set? Since the fitted parameters must be estimated separately for the target model, quantizer, bit width, and decision type, it seems that one could instead directly measure the downstream behavior or flip rate of interest. Is there an important setting where the proposed model substantially reduces the amount of quantized-model evaluation required, enables reliable prediction on new decisions, or otherwise changes what a practitioner would actually do?