- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed local diagnostics comparing attention rows over key positions and predictive distributions over the vocabulary to measure the effect of quantisation with respect to full precession model. Attention metrics detect consistent INT8–FP8 differences that perplexity sometimes misses. Adding activation quantisation reverses their relative ordering.
Critical strengths
- Their proposed measurements capture the local effects of quantisation on attention and output probability distributions, might be useful for revealing changes in model behaviour that output-only metrics miss.
Critical weaknesses
-
The paper is not well written, so I find it difficult to follow. Sometimes information provided are too dense and not clarified clearly for instance they introduce the notation W1 in the abstract without any explanation. I had to read through to understand only later they meant Wasserstein distance.
-
I understand the author studied the impact and diagnostic locally of quantised model, but it is not well motivated e.g. what is the problem of existing diagnostic the authors are trying to address. So, as a reader, I find it really hard what is the clear takeaway from the paper.
-
It is also unclear what is the significance and usefulness of these diagnostics. The strongest improvement would be to state what attention diagnostics reveal that output diagnostics miss, and what evidence makes that finding meaningful as downstream predictors. For reference check this [1] paper.
[1] Lotfi, Sanae, et al. "Quantized reasoning models think they need to think longer, but they do not." arXiv preprint arXiv:2606.00206 (2026).
Questions for authors
Check the weaknesses.