- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why quantization can cause a model to stop following instructions even when standard evaluation scores appear relatively normal. Under specific 3-bit GPTQ settings, the quantization procedure gives excessive importance to the first token because its internal activation is unusually large. Protecting this token, whose output has little influence on the model’s loss, can increase errors at chat-template positions that matter more. The failure is rare and protocol-dependent: it occurs on 2 of 18 tested models under the short-document calibration, and GPTQModel’s own default column ordering avoids it. The authors investigate the mechanism by replacing and transplanting weight matrices, showing that the resulting failures also depend on the surrounding network. They then adjust calibration token weights to account for both activation size and loss sensitivity. This correction resolves the four tested collapse cases, but can hurt models that did not originally collapse.
Critical strengths
-
The paper identifies a practically important failure that standard quantization evaluations can miss. Under specific 3-bit GPTQ settings, instruction following deteriorates sharply even when multiple-choice accuracy remains competitive and perplexity does not reflect the severity of the failure. Comparisons across calibration protocols and implementation settings help establish when the problem occurs. This finding shows why instruction-following evaluation matters when assessing quantized chat models.
-
The paper provides persuasive intervention evidence for its proposed failure mechanism. Replacing the identified matrix with its RTN version restores IFEval from 15.5% to 62.2% on Llama and from 42.6% to 76.8% on Qwen, while leaving the other matrices unchanged. Transplantation experiments further show that the effect of the damage depends on the surrounding network. Together with the analysis of GPTQ’s error compensation, these results explain the failure beyond an observed association between quantization error and performance loss.
Critical weaknesses
-
The repair and intervention results need verification under standard prompt formatting. Generation evaluations for Llama and Mistral use duplicated BOS tokens. I acknowledge that Qwen has no BOS token and is unaffected, and that the existing single-BOS control shows the collapse itself persists (0.150 to 0.158). However, that control uses a checkpoint from before the quantization-loop correction and covers only the fp16 and collapsed checkpoints, not the corrected repairs, transplants, or mechanism statistics. Since the proposed mechanism concerns the initial positions and chat-template tokens, and the chat template contributes one of the two BOS tokens, this leaves a gap in establishing the Llama and Mistral results under standard single-BOS formatting.
-
The benefits of the proposed repair remain dependent on the setting. The correction repairs four tested collapse configurations, but can reduce scores on healthy models by up to 5.6 percentage points, and simpler alternatives perform better in some settings. These results leave it unclear when the correction should be preferred over chat-formatted calibration, local RTN repair, or group-aware column ordering.
Questions for authors
-
Can you re-run the single-BOS control under the corrected quantization loop, and extend it beyond the fp16 and collapsed checkpoints to cover at least one repair arm and one key transplant?
-
Can you provide any positive demonstration of the inferred cancellation, for example by showing that under the healthy draw, the output contributions of the twelve neighboring matrices’ errors at the template positions oppose the lesion’s contribution?
-
You note that LLM Compressor’s default uses the same global column order as your loop but was not run end to end. Can you add one end-to-end run (Llama and Qwen, short documents)?
A brief note before the discussion closes
Dear Reviewers,
Thank you again for your careful reviews. As the discussion period closes today (AoE), we wanted to send a brief note in case it is helpful: our responses are posted under each review, and the revised PDF marks new and revised text in blue (see the "Summary of revisions" comment).
If any concern remains, or anything in our replies is unclear, we would be glad to clarify before the deadline.
Best regards, The authors