acceptodds
Under review as a conference paper at ICLR 2027

When Quantization Preserves Generation but Degrades Verification: A Frozen-Candidate Study across Model Scale

Abstract

Quantized language models are commonly evaluated as generators, yet the same models are increasingly used to judge, rerank, and select candidate answers. These roles need not respond identically to reduced precision. We study this distinc- tion with a frozen-candidate protocol: for each model, eight candidates are gen- erated once in FP16 and then held byte-identical while the verifier is evaluated at FP16 and several quantized precisions. Correctness is assigned by determinis- tic semantic checking, so verifier comparisons are not confounded by a changing candidate pool. On 300 MATH-500 problems, Qwen2.5-7B-Instruct provides a clear example. HQQ 8-bit changes greedy generation accuracy from 0.467 to 0.473 (∆ = +0.006), while global verifier AUROC falls from 0.825 to 0.544 and within-problem ranking from 0.698 to 0.411. Best-of-eight selection falls from 0.500 to 0.430, below the exact random-selection rate of 0.455. The same 7B degradation is observed with bitsandbytes int8 and NF4. In contrast, the HQQ 8- bit result does not show a comparable verification drop at Qwen2.5-3B, -14B, or -32B; the 3B result is additionally sensitive to the quantizer, with NF4 producing a separate degradation. A Llama-3.1-8B experiment is not a strong cross-family test because its FP16 verifier is already near chance (AUROC = 0.471). Output- level analysis of the 7B model shows an approximately 8×reduction in correct- versus-incorrect margin separation and a higher decision-flip rate among correct candidates than incorrect candidates. The evidence therefore supports a narrower claim than a universal “quantization breaks verification” hypothesis: generation accuracy alone does not certify preservation of verifier ranking, and verification should be evaluated directly for the model, precision, interface, and candidate set used downstream.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models (LLMs), Model Quantization, Low-Bit Inference, Verification, Verifier Models, Correctness Verification, Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_acWjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether quantization can preserve generation accuracy while degrading verification ability. It uses a frozen-candidate protocol, where full-precision and quantized models score the same fixed candidate solutions. Verification is assessed using AUROC, problem ranking, and the accuracy of selecting the highest-scoring candidate.

Critical strengths
  • The paper tackles a critical problem: how does quantization affect verification, even if the generation accuracy is close to the full-precision base model? The authors evaluate full-precision and quantized verifiers on identical frozen candidate solutions, isolating changes in verification from changes in the generated candidates.
  • The authors provide empirical comparisons across various quantization methods and model sizes (3B to 32B of the Qwen2.5 model) to study where verification degradation occurs. The main finding of the paper is that HQQ-8 substantially harms 7B verification while leaving 3B, 14B, and 32B approximately unchanged.
Critical weaknesses
  • W1: I am concerned about the novelty and contribution of the paper. Prior work already establishes that preserved task performance under quantization does not imply preserved model behavior. This has been investigated across various reliability and trustworthiness dimensions [1,2,3,4]. The authors extend this to frozen-candidate verification, but do not explain what additional insights this setting provides beyond the existing observations.

  • W2: The evaluation scope is very limited. The experiments cover only MATH problems and a single Yes/No verification prompt. Most experiments focus on the Qwen model, and the verification degradation is mainly observed for the 7B Qwen model. Additional benchmarks, model families, model sizes, and alternative verification methods need to be examined to establish robust findings and strengthen the authors' empirical claims.

[1] Rababah, Baha, et al. "The illusion of equivalency: Statistical characterization of quantization effects in LLMs." arXiv preprint arXiv:2607.08734 (2026).

[2] Ayadi, Sirine, et al. "Reliability Scaling Laws for Quantized Large Language Models." arXiv preprint arXiv:2607.10855 (2026).

[3] Hong, Junyuan, et al. "Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression." arXiv preprint arXiv:2403.15447 (2024).

[4] Dutta, Abhinav, et al. "Accuracy is not all you need." Advances in Neural Information Processing Systems 37 (2024): 124347-124390.

Questions for authors
  • Q1: It seems to me that the observed verification degradation is model-dependent and is evaluated within a narrow setting. How robust is this finding across other benchmarks and verification settings? Also, what novel insights does the paper provide beyond prior work that studies behavioral shifts under model compression?
  • Q2: Could the authors explain how to assess verification for open-ended text generation tasks?
  • Q3: Could the authors explain what motivates using a single-token Yes/No verification interface?
Reviewer_DHBDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether quantization in LLMs causes greater degradation in verification than in generation. It evaluates five models on 300 MATH-500 problems using a frozen-candidate protocol, where eight FP16-generated candidates per problem are evaluated by verifiers at different precisions. The authors find that Qwen2.5-7B exhibits substantial verification degradation under 8-bit quantization while preserving generation accuracy. However, this effect is not consistently observed across other models.

Critical strengths

None

Critical weaknesses
  • The key finding is primarily based on one model, Qwen2.5-7B. The authors do try other models, but the effect is not consistently observed. Some of these models already have weaker baseline verification performance. There are several newer, stronger models, as well as models of comparable size, that the authors could have evaluated.
  • I could not find an AI-use statement in the paper and publication is severely underprepared for a conference submission.
Questions for authors
  • What fundamentally motivates/explains the gap between generation and verification performance under quantization?
  • Does the finding hold for newer model families and stronger reasoning models? Could the authors provide additional experiments to establish its generality?
  • How sensitive are the results to the verification interface? Does the degradation persist when the model is allowed to reason before providing its judgment, rather than using only the next-token Yes/No logits?
Reviewer_j4LcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether quantization can preserve a language model's ability to generate answers while degrading its ability to verify them. To separate the two, the authors sample eight candidate solutions per problem once at FP16 and keep them fixed. The same model judges these candidates by comparing its "Yes" and "No" logits, once in FP16 and once in each quantized version. These are HQQ from 8-bit down to 2-bit, plus bitsandbytes int8 and NF4. The experiments use 300 MATH-500 problems, Qwen2.5-Instruct from 3B to 32B, and Llama-3.1-8B. Verification is measured by AUROC and within-problem ranking accuracy W. The main claim is that Qwen2.5-7B keeps its greedy accuracy at 8 bits while its AUROC falls from 0.825 to 0.544 and best-of-8 selection drops below random. The other sizes show no comparable 8-bit drop.

Critical strengths

The paper asks whether a quantized model can still serve as the verifier in a best-of-N pipeline when its generation accuracy looks unchanged. This is a useful question in test-time compute, because a quantized model is commonly evaluated only as a generator, even when it is also used to rerank answers.

The main strength of the paper is its experimental control. Keeping the candidates and labels fixed means every arm judges the same answers, which rules out the most obvious alternative explanation. I also appreciate that the paper is honest about where the effect does not appear.

Critical weaknesses

However, I do not find the evidence sufficient to support the paper's main claim that 8-bit quantization is what degrades verification in Qwen2.5-7B while generation is preserved. The 7B drop is real. W falls from 0.698 to 0.411, and the bootstrap interval for this change, −0.374 to −0.205, is far from zero. What the paper does not show is its cause.

My main concern is that no experiment tests numerical changes without also applying quantization. Every quantized arm replaces the attention and MLP projections with new layers. The reference uses FP16, although Qwen2.5 is released in BF16. Testing three quantizers makes a bug in one library unlikely, but it does not rule out a step that all quantized arms share, or an FP16 reference that is itself the odd one out. Since the same 8-bit setting leaves the other sizes almost unchanged, Qwen2.5-7B may be a genuinely fragile checkpoint, but the paper does not yet show what it is fragile to. This matters because the main result rests on this one checkpoint. A control that uses the same layer replacement but keeps the original weights would test that step, and a BF16 arm would test the reference. As written, the paper shows that the quantized 7B arms are much worse verifiers, but does not convincingly demonstrate that quantization itself is the cause.

My second concern is that the model-specific conclusion rests on comparisons across sizes that say less than the paper suggests. The 32B verifier is already weak at FP16, with an AUROC of 0.562 and a W of 0.385, so its stable 8-bit results say little about preserving a useful verifier. The paper sets Llama aside for the same reason. At 3B, the NF4 arm lowers AUROC from 0.670 to 0.452. The paper calls this a separate degradation, but it is about three quarters of the 7B drop, and its generation accuracy and W are not reported. The data show no 8-bit drop at 3B or 14B, but do not show that only 7B keeps its generation accuracy while its verification degrades. Reporting the full results for every arm already run would at least show whether 3B does the same.

Overall, I think the paper studies a useful question with a clean design, but it does not yet show that quantization itself causes the 7B result. The two 7B controls reuse the stored candidates and should fit in the discussion period. I would raise my score if both keep the FP16 7B ranking while the 8-bit arms still lose it. A stronger next version would show the effect in a second model family or locate it in specific layers.

References:

[1] Dutta et al. (2024). Accuracy is Not All You Need.

Questions for authors
  1. Could the authors add the two 7B controls? One that uses the same layer replacement but keeps the original weights, and one in BF16? If both keep the FP16 ranking, I would raise my soundness and overall scores. If the control with the original weights loses the ranking, the layer replacement itself is the likely cause. If the BF16 arm loses it, the FP16 reference is the odd one out.

  2. Could the authors report generation accuracy, W, and best-of-8 accuracy for every arm already run at the other sizes, as Table 2 does for 7B? If 3B keeps its generation accuracy but loses W under NF4, it shows the same pattern as 7B, and the claim that only 7B is affected would need to change.

  3. Did the pilot that made 7B the focus use the same 300 problems? If so, 7B was picked after seeing these problems, so the 7B result does not count as an independent test. Could the authors also say what was corrected in Table 1’s 7B pool, and whether all precision arms of each model used the same code version and checkpoint revision? Any difference between arms would be another possible cause of the 7B drop.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.