acceptodds
Under review as a conference paper at ICLR 2027

Accuracy Is Not Enough: Measuring Instruction-Following Stability Under Post-Training Quantization

Abstract

Post-training quantization is routinely evaluated by task accuracy alone, leaving open whether the instruction-following behavior instilled during instruction tuning survives aggressive compression. We study this question for INT4 weight-only quantization of two compact instruction-tuned models, Llama 3 8B Instruct and Phi-3 Mini 3.8B Instruct, across seven prompt structures and three decoding temperatures on GSM8K and the MMLU high-school mathematics subset (12,600 generations). We score every generation both for task accuracy and for strict compliance with an explicit output contract, and we show that the parse-rate measure used in prior practice is saturated above 0.99 and cannot detect the effects we report. Three findings emerge. First, task correctness and instruction-level compliance decouple under INT4, and the compliance effect is the larger one: Phi-3 contract compliance falls from 0.107 to 0.055 while its accuracy simultaneously rises from 0.344 to 0.373, so a purely accuracy-based evaluation would record this model as unaffected or improved. Second, the effect is strongly model-dependent: Llama 3 compliance is stable and marginally better under INT4, and the two models also re-rank prompt effectiveness differently on accuracy. Third, the compliance loss is largest in exactly the exemplar-based prompts that appear protective on accuracy, so few-shot prompting cannot be assumed to stabilize behavior under compression. We discuss a distributional hypothesis that is consistent with these behaviors but that we do not test directly, and we argue that evaluation of quantized models should treat instruction-following stability as a first-class metric alongside accuracy.

Primary area:
datasets and benchmarks
Keywords:
post-training quantization, INT4, model compression, instruction following, prompt engineering, evaluation, small language models, large language models, mathematical reasoning, empirical study

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_TVQeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the impact of post-training quantization on instruction-following behavior beyond task accuracy. The authors compare FP16 and INT4 versions of Llama 3 8B and Phi-3 Mini across different prompting strategies and decoding temperatures, measuring both answer correctness and strict output-format compliance. Results show that Phi-3 exhibits substantially reduced compliance despite improved accuracy, while Llama 3 remains largely unaffected.

Critical strengths
  • The paper considers instruction-following compliance as an additional evaluation metric alongside task accuracy.
  • The experiments cover different prompting strategies and decoding temperatures.
Critical weaknesses
  • Limited novelty. Previous studies, including Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression (Hong et al., 2024) and Exploring the Trade-offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models from Edge to Giant (Lee et al., 2025), have already investigated the impact of quantization on instruction following and other behavioral properties beyond task accuracy. Both studies are cited in the paper, making its incremental contribution unclear.
  • Insufficient validation of the evaluation setup. The reported baseline accuracy is substantially below published model performance, but the authors have not verified whether their implementation reproduces expected results under standard evaluation protocols. The choice of FP16 rather than the original BF16 precision is also not justified, leaving uncertainty about the reliability of the reference configuration.
  • Limited validity of the instruction-following evaluation. The study relies on a single output-format constraint that prohibits explanations on mathematical reasoning tasks, potentially conflating instruction following with response verbosity and reasoning behavior. Without evaluation on independent instruction-following benchmarks such as IFEval, it remains unclear whether the observed degradation extends beyond this particular constraint.
Questions for authors
  • Can the authors validate the evaluation setup against the official evaluation protocols and explain the choice of FP16 over BF16? Does the baseline model reproduce its expected performance when evaluated without the output contract?
  • Have the authors verified the fully rendered prompts for both models and precision settings? Does the observed compliance degradation persist when the output contract is placed directly in the user prompt?
  • Would the observed degradation persist under other instruction types or output contracts that permit intermediate reasoning? In particular, could the authors evaluate whether the findings generalize to established instruction-following benchmarks such as IFEval?
Reviewer_pswdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper discusses how quantized models may fail to follow instructions despite maintaining decent benchmark performance. The authors demonstrate this issue by evaluating output-format compliance before and after quantization, showing that accuracy alone is insufficient to capture such behavioral degradation.

Critical strengths

The paper provides a systematic empirical evaluation across different prompt strategies, decoding temperatures, and model precisions.

The experimental results clearly illustrate that benchmark accuracy and output-format compliance do not necessarily correlate.

Critical weaknesses

The experimental scope is too narrow to support the paper's broad claims. The authors evaluate only two models, one quantization configuration, and a limited set of mathematical benchmarks.

The claims in the title and abstract are overstated relative to the experimental evidence. Although the paper claims to investigate instruction-following stability, the evaluation focuses exclusively on a simple output-format constraint. This is insufficient to draw general conclusions about the impact of quantization on instruction-following capabilities.

Questions for authors

How can the authors conclude that quantization degrades a model's instruction-following ability when the quantized model can still successfully solve the benchmark tasks? The observed degradation is limited to output-format compliance, which does not necessarily reflect the model's broader instruction-following capabilities. How do the authors justify generalizing this finding to instruction following as a whole?

Did the authors consider additional fine-tuning to recover instruction-following performance after quantization? Even a small amount of fine-tuning might help the quantized models maintain both task accuracy and instruction-following ability. It would be interesting to investigate whether the observed degradation persists after such fine-tuning.

Reviewer_1hQeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper tries to draw attention to evaluation issues in existing instruction-following models after quantization. This is an important problem because quantization aims to reduce computation while preserving the behavior of the original model as much as possible, without affecting its intended use or the end user. An LLM response has two aspects to consider: the answer itself and the requested format, such as an explanation, no explanation, or a one-line answer. The paper argues that the effectiveness of a quantized model cannot be judged solely based on its answer accuracy on benchmarks and should also take into account the desired format of the response.

The paper studies this phenomenon for INT4 quantization only, using two model families (Llama 3 and Phi-3) across seven different prompt families on two benchmark datasets. The authors demonstrate experimentally that, after quantization to INT4, a model may still retain its accuracy but show a significant drop in its ability to follow the requested formatting instructions. This phenomenon is model dependent, with the two models showing different behavior: Phi-3 shows a substantial drop in format following, while Llama 3 does not. The results also depend on the prompt, not only on the model.

Following these findings, the paper suggests measuring instruction-following ability alongside answer accuracy. It also discusses a possible explanation at the token level: quantization may weaken the probability advantage of tokens that keep the response in the requested format, making unwanted continuations, such as extra explanations, more likely. This could explain why the answer remains correct while the formatting instructions are violated. However, the paper does not directly test whether this is what causes the observed changes.

Critical strengths
  1. The paper draws attention to an important problem, although prior work has already highlighted this issue (details on that provided in the weakness section as first bullet point).

  2. The finding that quantization affects instruction following differently across different models is interesting.

Critical weaknesses
  1. The paper does not cite or compare with the findings of Dutta et al. [1.], which already shows that quantization can preserve benchmark accuracy while degrading instruction following or response. This limits the novelty of the paper and the authors should discuss this overlap and clarify what it adds beyond the earlier study.

  2. In many places including the start of the paper, it is difficult to follow and requires considerable effort to understand. Several paper-specific terms are used before they are clearly defined, and some sentences express relatively simple ideas in unnecessarily complicated language. The terminology should be more clearly connected to familiar concepts in LLM instruction-following research. For example:

Line 023: "Phi-3 contract compliance falls from.." ==> This term is used repeatedly before its precise meaning is explained. Since readers cannot be expected to know the authors’ particular definition, it should be defined at first use, preferably in simpler words (For example: instruction format compliance) ? Line 60: "Quantization is not behavior-neutral." ==> This statement is too broad to communicate the intended point clearly. The authors should specify which behavior may change, such as answer accuracy or adherence to the requested output format.

Line 77:  "Instruction following is thus a known weak point of quantized models; what is new here is a strict-contract metric, a saturation argument against parse rate, and the prompt–precision–metric interaction." ==> Hard to understand sentence as this sentence combines several concepts without first explaining what they mean (and some are non-standard terms in the community).

Line 117: "A compliance-aware evaluation protocol. We operationalize instruction-following stability..." ==> What does “operationalize” mean here ? It seems it stands for quantifying or evaluating the instruction-following ability/stability. Please consider rewording it so that readers or LLM researchers can understand it better without going through the entire paper first to understand its meaning.

  1. The paper does not provide qualitative comparison (before and after quantization) of actual model responses in either the main text or the appendix. Although the Appendix includes prompt templates and examples, these do not show how the evaluated models provide outputs. Showing responses before and after quantization to the same questions that have correct answers, with the formatting violations highlighted, would help readers understand what changes after quantization.

  2. The paper proposes a hypothesis to explain degradation in instruction following after quantization, but provides no targeted experiments to support it. The explanation therefore remains speculative.

[1.] Dutta et al., Accuracy is Not All You Need (https://proceedings.neurips.cc/paper_files/paper/2024/hash/e0e956681b04ac126679e8c7dd706b2e-Abstract-Conference.html)

Questions for authors

Line 178:"a response is compliant if and only if it consists of exactly one non-empty line and that line begins with the marker Answer:." ==> But this can still lead to response format violation if there is some additional text following the answer ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.