acceptodds
Under review as a conference paper at ICLR 2027

One Token Owns the Calibration Hessian: Why Layer-wise Quantization Can Silently Break Instruction Following

Abstract

Quantized instruction-tuned models are usually validated with perplexity and multiple-choice accuracy, which can miss a collapse of instruction following. At 3 bits, reference GPTQ with act-order, calibrated on short documents as widely used libraries sample them by default, lowers the IFEval score of Llama-3.1-8B-Instruct from 0.768 to 0.155 and that of Qwen2.5-14B-Instruct from 0.820 to 0.426, below round-to-nearest (0.565, 0.697), while MMLU stays in the ordinary 3-bit range; GPTQModel's own column order avoids the collapse. We trace the failure to a blind spot of the layer-wise objective. At the down-projection that forms the attention sink, the first token owns the calibration Hessian, yet the loss is almost insensitive to its output. GPTQ's compensation, which we derive in closed form, therefore protects that token and writes its rounding error along the sink pattern into one matrix: a lesion that the chat-template tokens read out. Excising the lesion restores both models. The blind spot is present in all twenty-six models whose first token dominates a down-projection, and of the seventeen of these that we quantized, fourteen carry the lesion, yet only two collapse. Transplants indicate that the other matrices of a healthy network cancel the lesion, which would explain why the failure is rare and depends on how the calibration text is cut and which documents are drawn. Weighting each calibration token by its output-side sensitivity divided by its input energy removes the blind spot and the four collapses it was run on, at a cost of at most 5.6 points on healthy models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, GPTQ, attention sink, instruction following, calibration data, layer-wise reconstruction, large language models, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_dkD9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why quantization can cause a model to stop following instructions even when standard evaluation scores appear relatively normal. Under specific 3-bit GPTQ settings, the quantization procedure gives excessive importance to the first token because its internal activation is unusually large. Protecting this token, whose output has little influence on the model’s loss, can increase errors at chat-template positions that matter more. The failure is rare and protocol-dependent: it occurs on 2 of 18 tested models under the short-document calibration, and GPTQModel’s own default column ordering avoids it. The authors investigate the mechanism by replacing and transplanting weight matrices, showing that the resulting failures also depend on the surrounding network. They then adjust calibration token weights to account for both activation size and loss sensitivity. This correction resolves the four tested collapse cases, but can hurt models that did not originally collapse.

Critical strengths
  1. The paper identifies a practically important failure that standard quantization evaluations can miss. Under specific 3-bit GPTQ settings, instruction following deteriorates sharply even when multiple-choice accuracy remains competitive and perplexity does not reflect the severity of the failure. Comparisons across calibration protocols and implementation settings help establish when the problem occurs. This finding shows why instruction-following evaluation matters when assessing quantized chat models.

  2. The paper provides persuasive intervention evidence for its proposed failure mechanism. Replacing the identified matrix with its RTN version restores IFEval from 15.5% to 62.2% on Llama and from 42.6% to 76.8% on Qwen, while leaving the other matrices unchanged. Transplantation experiments further show that the effect of the damage depends on the surrounding network. Together with the analysis of GPTQ’s error compensation, these results explain the failure beyond an observed association between quantization error and performance loss.

Critical weaknesses
  1. The repair and intervention results need verification under standard prompt formatting. Generation evaluations for Llama and Mistral use duplicated BOS tokens. I acknowledge that Qwen has no BOS token and is unaffected, and that the existing single-BOS control shows the collapse itself persists (0.150 to 0.158). However, that control uses a checkpoint from before the quantization-loop correction and covers only the fp16 and collapsed checkpoints, not the corrected repairs, transplants, or mechanism statistics. Since the proposed mechanism concerns the initial positions and chat-template tokens, and the chat template contributes one of the two BOS tokens, this leaves a gap in establishing the Llama and Mistral results under standard single-BOS formatting.

  2. The benefits of the proposed repair remain dependent on the setting. The correction repairs four tested collapse configurations, but can reduce scores on healthy models by up to 5.6 percentage points, and simpler alternatives perform better in some settings. These results leave it unclear when the correction should be preferred over chat-formatted calibration, local RTN repair, or group-aware column ordering.

Questions for authors
  1. Can you re-run the single-BOS control under the corrected quantization loop, and extend it beyond the fp16 and collapsed checkpoints to cover at least one repair arm and one key transplant?

  2. Can you provide any positive demonstration of the inferred cancellation, for example by showing that under the healthy draw, the output contributions of the twelve neighboring matrices’ errors at the template positions oppose the lesion’s contribution?

  3. You note that LLM Compressor’s default uses the same global column order as your loop but was not run end to end. Can you add one end-to-end run (Llama and Qwen, short documents)?

Reviewer_weRcOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that GPTQ shows significant accuracy degradation on IFEval (worse than RTN) and attribute this failure to a blind spot of the layer-wise objective. This blind spot is common across all the models they explored.

Critical strengths
  • The observation and the corresponding solution seems interesting and will be instructive in the quantization research domain.
  • The authors show that the proposed method can generally address 3-bit and 4-bit collapses.
Critical weaknesses
  • The authors focus mainly on GPTQ, which is a quite old method. It would be more beneficial if they investigate recent methods (e.g, QuIP#) as well.
Questions for authors
  • Does the blind spot also appear in VLMs?

Comments on the paper

1 comment
AuthorsOpenReview ↗

A brief note before the discussion closes

Dear Reviewers,

Thank you again for your careful reviews. As the discussion period closes today (AoE), we wanted to send a brief note in case it is helpful: our responses are posted under each review, and the revised PDF marks new and revised text in blue (see the "Summary of revisions" comment).

If any concern remains, or anything in our replies is unclear, we would be glad to clarify before the deadline.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.