acceptodds
Withdrawn from ICLR 2027

Which Decisions Low-Bit Quantization Breaks, and How to Predict Them

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Quantization saves memory by storing model weights with fewer bits. It can also change model decisions, such as whether to call a tool or which option to choose from a finite set. We study these decision changes in 16 language models from 8 families at 4, 3 and 2 bits, across several post-training quantization settings. Our evaluation covers tool use, safety, general knowledge and social bias, using BFCL, XSTest, MMLU, BoolQ, BBQ and synthetic tasks. The decision margin is the score difference between two possible first tokens, measured before and after quantization. Its size shows how strongly the model prefers one option, and a change in sign means the preference has flipped. Writing the margin before quantization as and the margin after quantization as , we find an approximately linear relationship across decisions: . The slope is usually below one and becomes smaller as precision falls, so quantization progressively shrinks decision margins. The offset is the same for every decision of one kind. Quantization therefore does not simply add random noise, and even a strong preference at full precision can flip. Quantization also affects different kinds of decisions to different degrees. Within tool use, whether to call a tool is often more sensitive than which tool to call: on 400 BFCL tasks, three of five models lose more completed calls than correct tool selections at 3-bit round-to-nearest. Under GPTQ and GGUF far fewer whether-to-call decisions flip than under plain rounding, so there is no single 3-bit failure point. The same relationship predicts how often decisions flip. Across 1,154 combinations of models, quantization settings, bit-widths and decision types drawn from our evaluation, we fit the slope, the offset and the spread around the fitted line on half of the decisions and predict the flip rate on the other half. The predicted flip rate differs from the observed flip rate by a median of 1.0 percentage point. Together, these results show a common pattern in how quantization changes decisions and provide a way to predict when those changes will occur.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization, tool use, LLM agents, post-training quantization, decision margins, safety refusal, evaluation, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_s2ngOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines how low-bit quantization changes specific model decisions. The authors measure a decision using the logit margin between two candidate first tokens before and after quantization, and define a "flip" when the sign of this margin changes. They evaluate 16 models from 8 families across several post-training quantization methods and decision types including tool use, safety/refusal, knowledge, and social bias.

The main finding is that the quantized margin can often be approximated as a regression m′≈cm+b+ϵm' \approx cm+b+\epsilon, where cc captures shrinkage of the original margin and bb represents a systematic shift. The authors also find that different decision types can degrade at different rates; for example, under some 3-bit settings, whether to call a tool is substantially more sensitive than which tool to call.

Finally, the authors fit the slope, offset, and residual variance on half of the decisions in each condition and use them to predict the flip rate on the other half, reporting a median absolute error of 1.0 percentage point across 1,154 conditions.

Critical strengths
  1. The empirical evaluation is broad, with 16 models from 8 families, several quantization methods and bit widths, and a range of decision types including tool use, refusal, knowledge, and social bias.

  2. The experiments are generally careful and include a large number of controls. Specifically, the authors test alternative quantizers, activation/KV-cache quantization, alternative token pairs, and several checks on whether the measured margin corresponds to actual model behavior.

  3. Some of the specific empirical observations are interesting. For example, the paper shows that whether-to-call decisions can degrade substantially before which-tool decisions under certain low-bit quantization settings, which is counterintuitive.

Critical weaknesses
  1. My main concern is that I am not convinced the proposed affine relationship m′≈cm+b+ϵm' \approx cm+b+\epsilon provides much actionable insight. It is a simple descriptive model of the relationship between pre- and post-quantization margins, but does not explain the underlying mechanism, and the fitted parameters need to be calibrated separately for the model, quantizer, bit width, and decision type being studied.

  2. I am also unclear on the practical value of the forecasting result. If one already has access to the target quantized model and a representative validation set, one can directly measure the downstream behavior or flip rate of interest. The proposed model improves somewhat over simply reusing the calibration-set flip rate, but the improvement is modest on the harder mid-range conditions (4.3 pp median error versus 6.0 pp). More broadly, the paper identifies where quantization causes failures but provides relatively little guidance on what to do about them. The tested repair strategies do not yield a strong general mitigation, which limits the practical significance of the analysis in my view.


My overall judgment is that the work is extensive but does not provide enough new or actionable insight to meet the bar for ICLR.

Questions for authors

Main question: What practical use does the proposed forecasting framework provide beyond directly evaluating the quantized model on a representative validation set? Since the fitted c,b,σc,b,\sigma parameters must be estimated separately for the target model, quantizer, bit width, and decision type, it seems that one could instead directly measure the downstream behavior or flip rate of interest. Is there an important setting where the proposed model substantially reduces the amount of quantized-model evaluation required, enables reliable prediction on new decisions, or otherwise changes what a practitioner would actually do?

Reviewer_QcRxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how using fewer bits to store model weights changes the decisions of language models. Across 16 models, the authors find that some decisions are more easily affected than others. For example, a model may still choose the right tool when asked to select one, but fail to start a tool call on its own. Quantization can both weaken the model’s original preference and push it toward another choice. The authors use a simple model to predict how often these preferences change, with good results on new test items from the same setting. However, changes in token preferences do not always match changes in actual responses, so predicting real tool-call failures remains difficult.

Critical strengths
  1. The paper studies how quantization affects tool calling through experiments across multiple models and quantization methods. It also compares the effects on different types of decisions.
  2. The authors develop a simple model to predict how often token preferences change after quantization and test it on held-out samples.
Critical weaknesses
  1. The analysis is detailed, but it mainly describes patterns rather than explaining why they occur. The technical novelty of the prediction model is also limited. Although the experiments cover several tasks beyond tool calling, the evidence that the method can predict actual generation failures remains limited. Further validation on freely generated responses would strengthen its practical value.
Questions for authors
  1. Beyond the reported weight- and layer-level checks, can attribution methods or controlled changes to specific parameters help explain why some decisions are more affected than others?
  2. How could the prediction model guide practical choices, such as selecting a quantization method, assigning different bit widths to layers, or deciding which parts of a model to keep at higher precision?
Reviewer_YKwROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper is heavily AI generated, so its hard to understand. Here's my attempt at understanding. The goal of this work seems to be trying to figure out when and for what types of inputs the degree of quantization affects the output generation. They define a decision margin that's the difference between the most probable and the second most probable token and find that this margin shrinks linearly with increased quantization. They find that in tool-calling the tool call itself changes very little with quantization but whether to call a tool at all does.

Critical strengths
  1. Looking at settings where quantization has consistent differences over full precision is an important problem to study, if this hasn't already been exhaustively studied before.
  2. Seems like the studies cover a wide spread of models and tasks.
Critical weaknesses
  1. My major concern here is that the prose is heavily AI generated. As a result, it's very difficult to follow the narrative flow or the arguments made.
  2. Its not clear to me that the specific margin gap over the two most probable tokens (if that's indeed what the decision margin is) is the right thing to measure and fit.
  3. It's very hard to follow, but it seems like some conclusions are made but there aren't clear takeaways or explanations/investigations for why things are the way they are (gap decreases with increased quantization).
Questions for authors
  1. Why not go further and try to find underlying reasons for these observations esp. gap decreases w/ increased quantization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.