acceptodds
Under review as a conference paper at ICLR 2027

From Hidden States to Probability Measures: Measuring Quantization-Induced Deformation in Large Language Models

Abstract

Attention rows and next-token output distributions are probability measures. Prior work evaluates compressed models with accuracy, output KL, and answer flips; we extend this output-level evaluation to attention routing and treat both distributions as a measure-valued diagnostic layer. We compare discrepancies with and without ground geometry to characterize quantization-induced redistribution over positions and vocabulary outcomes. Using same-prefix, teacher-forced comparisons isolates local deformation from free-running history divergence. In the tested, comparatively mild INT8-versus-FP8 weights-only comparison, attention KL and \(W_1\) distinguish all six model-domain pairs with a consistent direction; perplexity is significant in only \(3/6\) and disagrees in direction even among significant pairs. Output KL has one directional exception. Under stronger INT4 weights-only quantization, perplexity also detects degradation. Adding activation quantization reverses the INT8/FP8 ordering in every tested pair, with model- and diagnostic-dependent magnitude and architecture-dependent layerwise shape. These diagnostics measure local redistribution and are not downstream-accuracy surrogates or full path-law comparisons.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization, transformers, attention, Wasserstein distance, KL divergence, language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5z9COpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed local diagnostics comparing attention rows over key positions and predictive distributions over the vocabulary to measure the effect of quantisation with respect to full precession model. Attention metrics detect consistent INT8–FP8 differences that perplexity sometimes misses. Adding activation quantisation reverses their relative ordering.

Critical strengths
  1. Their proposed measurements capture the local effects of quantisation on attention and output probability distributions, might be useful for revealing changes in model behaviour that output-only metrics miss.
Critical weaknesses
  1. The paper is not well written, so I find it difficult to follow. Sometimes information provided are too dense and not clarified clearly for instance they introduce the notation W1 in the abstract without any explanation. I had to read through to understand only later they meant Wasserstein distance.

  2. I understand the author studied the impact and diagnostic locally of quantised model, but it is not well motivated e.g. what is the problem of existing diagnostic the authors are trying to address. So, as a reader, I find it really hard what is the clear takeaway from the paper.

  3. It is also unclear what is the significance and usefulness of these diagnostics. The strongest improvement would be to state what attention diagnostics reveal that output diagnostics miss, and what evidence makes that finding meaningful as downstream predictors. For reference check this [1] paper.

[1] Lotfi, Sanae, et al. "Quantized reasoning models think they need to think longer, but they do not." arXiv preprint arXiv:2606.00206 (2026).

Questions for authors

Check the weaknesses.

Reviewer_cvnAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how quantization changes the internal attention distributions and next-token output distributions of large language models. It compares each quantized model with its full-precision counterpart while conditioning both on identical ground-truth prefixes. This design separates quantization-induced changes at a given prefix from differences caused by models generating different histories. The evaluation combines attention KL divergence, position-based attention Wasserstein-1 distance, output KL divergence, and an embedding-based output transport diagnostic, alongside perplexity and hidden-state L2 distance. Experiments cover Qwen3-4B, Mistral-7B, and Llama-3.2-3B using INT8, FP8, and INT4 weights-only quantization, plus INT8 and FP8 weights-and-activations quantization. The contribution is primarily an empirical diagnostic study, rather than a new quantization algorithm or divergence.

Critical strengths
  1. This paper points out a meaningful limitation of quantization evaluation, where perplexity or accuracy doesn't imply that a quantized model preserves the original model's distributions.
  2. Experiment analysis has been performed with three model families, covering a good range of models.
  3. The evaluation carefully controls the input context. Both the full-precision and quantized models receive the same ground-truth prefix at each step. This helps isolate quantization-induced changes from differences that arise after the models generate different continuations.
Critical weaknesses
  1. The contribution is not sufficiently distinguished from prior work on attention preservation under quantization. For example, “Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders” studies attention recovery and introduces attention-map and attention-output losses.
  2. The activation-quantization reversal may be strongly affected by the absence of calibration. Section 4.1 uses default Quanto quantization without a calibration dataset. This is consequential because Quanto’s official documentation states that activation quantization uses static scales and that calibration is needed to determine suitable activation scales.
  3. The significance of the additional differences revealed by attention diagnostics remains unclear. The paper shows where attention changes after quantization, but does not establish which changes are consequential for model behavior. Demonstrating that these diagnostics help identify harmful quantization choices or guide an effective intervention would strengthen their practical value.
Questions for authors

N/A

Reviewer_LD4EOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes to compare full-precision and their quantized checkpoint by treating the attention rows and next-token distributions as probability measures. The set-up is teacher-forced and covers two corpora IFBench and WikiText. The headline of the empirical study is that although they manage to separate INT8 and FP8, in the weight-space case, this does not track downstream accuracy.

Critical strengths
  • The study of diagnostics to better diagnose quantization performance is interesting and relevant
Critical weaknesses
  • The title is broader than what the paper studies, the abstract is hard to parse---e.g. ". Output KL has one directional exception."---and the conclusion is similarly difficult to follow. As it is, the presentation of the paper is enough to recommend rejection.

  • The empirical setting is narrow. Quantization methods are data-free, instead of the popular GPTQ, AWQ, SmoothQuant. Would the analysis generalize to these? Additionally, 8-bit quantization is a particularly uninteresting setting. How do these results translate across bit-levels?

  • The study of how to detect performance differences between quantization methods with statistical significance cheaply is very important and interesting. The current paper does not make this case. However, can the proposed diagnostics be used for this purpose?

Questions for authors

Above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.