- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the impact of post-training quantization on instruction-following behavior beyond task accuracy. The authors compare FP16 and INT4 versions of Llama 3 8B and Phi-3 Mini across different prompting strategies and decoding temperatures, measuring both answer correctness and strict output-format compliance. Results show that Phi-3 exhibits substantially reduced compliance despite improved accuracy, while Llama 3 remains largely unaffected.
Critical strengths
- The paper considers instruction-following compliance as an additional evaluation metric alongside task accuracy.
- The experiments cover different prompting strategies and decoding temperatures.
Critical weaknesses
- Limited novelty. Previous studies, including Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression (Hong et al., 2024) and Exploring the Trade-offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models from Edge to Giant (Lee et al., 2025), have already investigated the impact of quantization on instruction following and other behavioral properties beyond task accuracy. Both studies are cited in the paper, making its incremental contribution unclear.
- Insufficient validation of the evaluation setup. The reported baseline accuracy is substantially below published model performance, but the authors have not verified whether their implementation reproduces expected results under standard evaluation protocols. The choice of FP16 rather than the original BF16 precision is also not justified, leaving uncertainty about the reliability of the reference configuration.
- Limited validity of the instruction-following evaluation. The study relies on a single output-format constraint that prohibits explanations on mathematical reasoning tasks, potentially conflating instruction following with response verbosity and reasoning behavior. Without evaluation on independent instruction-following benchmarks such as IFEval, it remains unclear whether the observed degradation extends beyond this particular constraint.
Questions for authors
- Can the authors validate the evaluation setup against the official evaluation protocols and explain the choice of FP16 over BF16? Does the baseline model reproduce its expected performance when evaluated without the output contract?
- Have the authors verified the fully rendered prompts for both models and precision settings? Does the observed compliance degradation persist when the output contract is placed directly in the user prompt?
- Would the observed degradation persist under other instruction types or output contracts that permit intermediate reasoning? In particular, could the authors evaluate whether the findings generalize to established instruction-following benchmarks such as IFEval?