- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether quantization can preserve generation accuracy while degrading verification ability. It uses a frozen-candidate protocol, where full-precision and quantized models score the same fixed candidate solutions. Verification is assessed using AUROC, problem ranking, and the accuracy of selecting the highest-scoring candidate.
Critical strengths
- The paper tackles a critical problem: how does quantization affect verification, even if the generation accuracy is close to the full-precision base model? The authors evaluate full-precision and quantized verifiers on identical frozen candidate solutions, isolating changes in verification from changes in the generated candidates.
- The authors provide empirical comparisons across various quantization methods and model sizes (3B to 32B of the Qwen2.5 model) to study where verification degradation occurs. The main finding of the paper is that HQQ-8 substantially harms 7B verification while leaving 3B, 14B, and 32B approximately unchanged.
Critical weaknesses
-
W1: I am concerned about the novelty and contribution of the paper. Prior work already establishes that preserved task performance under quantization does not imply preserved model behavior. This has been investigated across various reliability and trustworthiness dimensions [1,2,3,4]. The authors extend this to frozen-candidate verification, but do not explain what additional insights this setting provides beyond the existing observations.
-
W2: The evaluation scope is very limited. The experiments cover only MATH problems and a single Yes/No verification prompt. Most experiments focus on the Qwen model, and the verification degradation is mainly observed for the 7B Qwen model. Additional benchmarks, model families, model sizes, and alternative verification methods need to be examined to establish robust findings and strengthen the authors' empirical claims.
[1] Rababah, Baha, et al. "The illusion of equivalency: Statistical characterization of quantization effects in LLMs." arXiv preprint arXiv:2607.08734 (2026).
[2] Ayadi, Sirine, et al. "Reliability Scaling Laws for Quantized Large Language Models." arXiv preprint arXiv:2607.10855 (2026).
[3] Hong, Junyuan, et al. "Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression." arXiv preprint arXiv:2403.15447 (2024).
[4] Dutta, Abhinav, et al. "Accuracy is not all you need." Advances in Neural Information Processing Systems 37 (2024): 124347-124390.
Questions for authors
- Q1: It seems to me that the observed verification degradation is model-dependent and is evaluated within a narrow setting. How robust is this finding across other benchmarks and verification settings? Also, what novel insights does the paper provide beyond prior work that studies behavioral shifts under model compression?
- Q2: Could the authors explain how to assess verification for open-ended text generation tasks?
- Q3: Could the authors explain what motivates using a single-token Yes/No verification interface?