acceptodds
Under review as a conference paper at ICLR 2027

Quantization-Induced Decision Changes Have Regime-Dependent Intervention Spectra

Abstract

Quantization can preserve aggregate task performance while replacing individual predictions. However, neither score changes nor disagreement rates reveal how the associated behavioral effects are organized across representation directions. To address this gap, we introduce intervention-spectrum analysis, which tracks paired-decision transfer along ordered shared directions and normalizes aggregate prefix transfer by the complete task-supported transfer rate. Using directions fitted to other folds' full-precision states, we project each held-out paired-state difference onto successive prefixes. On disagreeing inputs, we add these projections at their native magnitude to the quantized residual state, testing agreement with the original full-precision prediction regardless of correctness. Across five checkpoints and three quantizers, final-site task-supported interventions outperform dimension- and norm-matched ambient controls in fixed-choice answer selection and teacher-forced full-vocabulary prediction. Despite this shared directional advantage, quantizer-level summaries show that the first 16 directions attain 91.8-92.1% of the complete task-supported one-way transfer rate in fixed choice, versus 12.6-15.1% in full vocabulary. This contrast persists under plain pooled-PCA, while candidate restriction within the full-vocabulary computation does not reproduce compact accumulation. These findings distinguish directional advantage from concentration: effective transfer can remain broadly distributed along a shared representation ordering. An anonymous code repository is available at https://anonymous.4open.science/r/quantization-behavioral-drift-CD71/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, LLMs, model compression, representation intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zPbEOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper starts from the observation that quantization can preserve aggregate accuracy while still changing substantial fraction of individual predictions. Based on this observation, the authors propose intervention-spectrum analysis. It projects the residual-state difference between a fp model and its quantized version onto successive prefix of a cross-fitted PCA basis. Each projection is then patched into one model, and the analysis measures how much of the full-basis decision transfer a given number of leading direction recovers. In fixed-choice tasks, the top 16 directions recover of the full-basis transfer, whereas in full-vocabulary prediction they recover only about 15%, and transfer accumulates over many directions. Based on this contrast, the authors argue that directional advantage and concentration are distinct properties.

Critical strengths
  1. The motivation is intuitive and convincing.
  2. The authors comprehensively evaluate their method across various tasks.
  3. The paper is clearly written and easy to follow.
Critical weaknesses
  1. The novelty of this paper is somewhat limited.
  • The components are established methods, and the observation that similar accuracy can hide prediction changes was previously studied [1]
  • It is unclear what the analysis reveals about quantization. Specifically, the method itself is agnostic to quantization. It applies to any pair of models with a state difference, and the results are nearly identical across GPTQ, AWQ, and RTN.
  1. The comparison with quantization methods is insufficient.
  • The models used by the authors for their analysis are relatively outdated. By failing to include more recently cited papers—such as QuaRot [2], SpinQuant [3], and KVQuant [4]—in their analysis, the authors have not demonstrated whether their proposed method applies to state-of-the-art models; consequently, their analytical approach lacks sufficient validation.

[1] Dutta et al., "Accuracy is Not All You Need," 2024.

[2] Ashkboos et al., "QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs," 2024.

[3] Liu et al., "SpinQuant: LLM quantization with learned rotations," 2024.

[4] Hooper et al., "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization," 2024.

Questions for authors

The directions are ordered by PCA on full-precision states. In Section 4, Fisher-based directions transfer noticeably more than PCA directions at the same rank. Does this mean the “broadly distributed” result could change with a different ordering? Why is PCA the right choice for measuring concentration?

Reviewer_xbR5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at what happens inside an LLM when (weight) quantization changes its answer: for each input where the FP16 and quant disagree, the authors take the difference between the hidden states at the input to the last transformer block. They then fit a PCA basis to this. The question is: if you add back only the difference that lies in the first r directions, how often does the quantized model switch back to the full-precision answer? Plotting this rate against r gives the “intervention spectrum”.

The main finding of the paper is that the answer depends on decision type:

  • For multiple-choice questions, the first 16 directions already give a lot of the achievable recovery.
  • For free next-token prediction over the whole vocabulary (e.g. GSM8K), 16 directions give only ~13–15%, and thousands are needed.

This holds across many models and quantizers.

Critical strengths
  1. Good experimental design
  2. The "accuracy-doesn't-change, but many answers do" observation (flips) is already known, but well-analyzed here
  3. Good coverage of models, quantizers etc
  4. Repro package

Overall, I think the measurements are solid and the findings are real. My doubts are about interpretation.

Critical weaknesses
  1. My main concern is that a simpler explanation exists, i.e. that the answer format is the confounder. (see below)
  2. Both the task and the answer format change in the experiments, so it's a bit hard to distinguish one's impact from the other.
  3. Contribution mostly empirical, no key practical takeaway.
  4. A hard read.
Questions for authors

Q1: Wouldn't the main result be explained just by the multiple-choice format? In that setting, every question is answered by comparing the same few answer tokens. So it would seem natural that a few shared directions control the answer. Looking at the numbers, coverage at 16 directions drops as the number of options grows, and the quantizer itself doesn't seem to matter. You could test this directly by running a next-token task where all answers come from the same small token set (e.g. binary), or score the same multiple-choice questions over the full vocab.

Q2: Is this specific to quantization? If you replace quantization with e.g. 10-20% sparsity do you get the same?

Q3. What happens earlier in the network?

Reviewer_evi6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors study individual prediction flips between full-precision and 4-bit post-training quantization. They propose "intervention-spectrum analysis" to study whether the hidden-state differences responsible for these flips are concentrated in a few shared directions or spread across many. They conclude that concentration depends on the decision format.

Critical strengths
  • The work studies an interesting phenomenon wherein post-training quantization leads to disagreement with the full-precision model beyond the usually reported aggregated accuracy metrics.
Critical weaknesses
  • The paper is very hard to parse. The title is not clear, the abstract is dense and works with concepts before they are introduced, and the conclusion is very difficult to parse. As it stands, the presentation alone is enough to recommend rejection.

  • Can the authors provide a baseline for individual prediction changes for the same quantization, e.g. you report differences between calibration datasets in the appendix, what is the---using Table 1's notation---Changed for fixed GPTQ under different calibration datasets? Not FP versus Q, but Q1 versus Q2 which are generated from different calibration sets. Also it would be valuable to understand what is the expected Changed for FP under different devices and inference settings? It is known that the numerics of inference change with batch size, kernel backend, hardware and others. Such a number would ground what a changed of 12.78 means.

  • The analysis is grounded on directions computed from a variance estimator of the data. How do the results change with a another basis? Additionally, the work is limited to the study of single token argmax predictions. Analyzing greedy decoded answers from FP and Q on GSM8K or TriviaQA would strengthen the paper considerably and clarify whether the wrong-to-correct actually exist in practice.

  • What is the consequence of this finding? How can this be used to improve quantization methods?

  • Finally, the motivation for this work, that "Quantization can preserve aggregate task performance while replacing individual predictions" is documented in dutta which should be cited.

Questions for authors

Above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.