- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a method to identify and protect layers in mixed-precision quantization settings. The authors claim that the Hessian of the layer-wise optimization problem used in GPTQ and ResQ is uninformative for ranking the sensitivity of the layers. As a result they present a functional importance metric in order to rank which layers most affect model performance.
The main contributions that the authors claim are:
- The local hessian (XX^T), the signal used in GPTQ and ResQ, is uninformative about the importance of the a layer.
- They propose a method to find important layers: They take the quantized model and restore one sub-layer at a time (MLP or Attention), then they evaluate the model on a calibration subset of the benchmarks and rank the sub-layers according to the restored accuracy.
- They achieve iso-accuracy with FP16 model using less memory.
- Functional importance is positional and shared across independently trained models.
Validation:
- They show the negative correlation between local hessian (XX^T) and restoration rank. This result is presented in Figure 3.
- The method presented in section 3.
- In Table 2 they show that with REST-6 the accuracy of ARC can be improved and reach close to iso-accuracy.
- They showed that for Mistral-7B which shares the same number of layers, that putting the same transformer blocks in FP16 that made the Llama-8B model robust also makes the Mistal-7B model more robust.
Critical strengths
None
Critical weaknesses
-
The method does not deliver: REST-6 puts six out of 32 transformer blocks in FP16. As a result, the accuracy would be expected to improve since less quantization error is added into the model. But the method improves only on one benchmark ARC and is still worse than GPTQ with all the layers in 4-bit. (Table1 and Table 2)
-
There is a presentation issue regarding the Hessian. GPTQ and ResQ use the Hessian of a layer-wise optimization objective. This local Hessian (2XX^T) and depends only on the activations not the weights. In both works the local Hessian is used within a layer and not to rank layers across the model. Furthermore, this signal seems to be informative within a layer as GPTQ achieves iso-accuracy with the FP16 model (Table 1).
-
The comparison of REST-6 and HESS-6 is unfair: REST-6 is task supervised where HESS-6 uses WikiText-2 for calibration.
Questions for authors
None
4 comments
Which Hessian, at what scope, and over which input
"There is a presentation issue regarding the Hessian. GPTQ and ResQ use the Hessian of a layer-wise optimization objective. This local Hessian (2XX^T) and depends only on the activations not the weights. In both works the local Hessian is used within a layer and not to rank layers across the model. Furthermore, this signal seems to be informative within a layer as GPTQ achieves iso-accuracy with the FP16 model (Table 1)."
We agree with you on every point. VnmZ raises the same objection, so the full answer, with the new HAWQ-V2 experiment, is in our comment there (W1). In short: is the Hessian of GPTQ's layer-wise reconstruction objective, it depends on activations only, and it is used within a layer for error compensation at a uniform bit-width, not to rank layers. Using it as a cross-layer allocation criterion and attributing that use to GPTQ and ResQ was our error, and the claim has been withdrawn. We implemented HAWQ-V2's actual cross-layer metric and report there that it could not be estimated reliably enough at 8B to rank sub-layers (seed-to-seed rank agreement 0.832 against a required 0.90, an asymptote of 0.841, top-6 overlapping across probe seeds at 3 of 6).
On your last sentence we agree and want to make the agreement precise, because it also explains how two things that look contradictory can both be true. GPTQ's within-layer use of is informative, since it does reach near-FP16 accuracy, and the restoration probe applied to the GPTQ model itself still finds the same sub-layers: under GPTQ quantization, L14 attention is the top attention sub-layer on GSM8K at 46.4% recovery, and L6 and L7 MLP are again among the top MLP sub-layers on ARC. Error compensation inside a layer and functional importance across layers are different questions, and a criterion can answer the first well while saying nothing about the second. That is the distinction the paper should have drawn explicitly, and the revision draws it.
One further precision, since it bears directly on scope and LTNt asks about it too (their
Q4). Our trace is accumulated by a forward hook on each self_attn and mlp module over its
input hidden_states, the post-RMSNorm residual stream entering the sub-layer (d = 4096 in
both cases), pooled over all 262,144 token positions of the 128 × 2048 WikiText-2 calibration
corpus, as . That is one number per sub-layer, covering the input to
q/k/v_proj and to gate_proj/up_proj. It does not cover o_proj, whose input is the
attention output, or down_proj, whose input is the SwiGLU activation. GPTQ builds a separate
H per projection matrix. Our per-sub-layer scalar is therefore a strictly coarser object than
what GPTQ computes, which is your point stated in our own notation, and the revision
says so where the metric is defined.
We ran the control. The asymmetry is real, and it is not what produces the gap.
"The comparison of REST-6 and HESS-6 is unfair: REST-6 is task supervised where HESS-6 uses WikiText-2 for calibration."
We agree with you on the facts, and we should have flagged the asymmetry ourselves. REST-6 is selected from recovery measured on the task benchmarks; HESS-6 is selected from accumulated over WikiText-2 test, 128 × 2048, following the GPTQ calibration protocol. The two arms did not see the same information.
So we removed the asymmetry. We recomputed the trace from activations collected on the task prompts themselves (the same 500 GSM8K, 500 ARC and 500 MATH prompts the evaluation uses), re-selected the top six at the same composition and FP16 budget, and re-ran the head-to-head. The pre-registration, committed before any compute, names the outcome in which your objection holds and the paper's comparison does not survive. The code selected the outcome from the thresholds, not us.
The positive control first, because nothing below means anything without it. Recomputing the trace on WikiText-2 with the new code reproduces the published ranking at rank agreement 1.0000, Spearman ρ −0.4969, and re-selects exactly the published HESS-6. (This needed float16 rather than bfloat16: the published attention cut is decided by a 0.607% margin.)
Task calibration does not weaken the inversion. It slightly strengthens it.
| Trace computed on | ρ vs restoration ↑ | ρ vs depth |
|---|---|---|
| WikiText-2 (published) | −0.4969 | +0.663 |
| GSM8K prompts | −0.4746 | +0.642 |
| ARC prompts | −0.5182 | +0.697 |
| MATH prompts | −0.5465 | +0.757 |
| Pooled over all three | −0.5145 | +0.699 |
The re-selected set, HESS-6-task = attn:25, attn:27, mlp:27–30, differs from the published
HESS-6 by a single sub-layer. Evaluated at 788,529,152 FP16 parameters in both arms, the
identical budget and composition, on the primary endpoint:
| ARC | FP16 | NF4 | HESS-6-task | REST-6 |
|---|---|---|---|---|
| accuracy | 0.788 | 0.744 | 0.736 | 0.774 |
| recovery of the 39 flipped | — | — | 2/39 = 5.1% [0.6, 17.3] | 21/39 = 53.8% [37.2, 69.9] |
Paired difference +48.7pp, bootstrap 95% CI [+30.8, +66.7], REST-6 ahead in all 10,000
resamples; McNemar rest_vs_hess p = 0.0017. HESS-6-task is significantly worse than FP16
(p = 0.0008), as the published HESS-6 was.
The second column is why no calibration corpus could have fixed this. The trace's correlation with depth goes up under task calibration, not down: +0.663 on WikiText-2, +0.699 pooled, +0.757 on MATH. here is largely a function of where a sub-layer sits in the residual stream, so changing what you feed the model moves it by about 4% of a correlation unit. This is the same object LTNt's Q1 is about, measured from a second direction.
The design, the positive control and these limits are Appendix O of the revision.
Four things we will not claim from this run, and state in the paper.
- GSM8K separates nothing:
rest_vs_hess p = 0.7794. We pre-registered ARC as primary precisely because GSM8K carries too little net quantization damage to resolve (forward/reverse flips 36/14 on ARC against 48/36 on GSM8K). We do not cite GSM8K as support. - HESS-6-task's attention half is not uniquely determined. Its cut margin is 0.099%, and
across corpora the attention picks permute among L22, L25 and L27 while the MLP half never
leaves
mlp:27–30. What holds is not those two layers but the cluster: every member of it has 0.0000 single-layer ARC recovery and a bottom-quartile restoration rank (1, 2, 6, 7, 12, 14 of 64) against REST-6's 55, 60, 61, 62, 63 and 64. - Collateral breakage persists. On ARC, REST-6 repairs 21 and breaks 7 (net +14); HESS-6-task repairs 2 and breaks 6 (net −4). The mechanistic claim holds; the practical claim remains ARC-only.
- This equalises the corpus, not the objective. Restoration measures exact-match recovery while the trace measures mean squared activation; the symmetric comparison needs a Hessian supervised by the same correctness signal. That is the task-loss Hessian of our answer to your W2, which landed on "instrument not validated" (seed-to-seed rank agreement 0.832 against a pre-registered 0.90, three sub-layers with non-positive average trace). This run neither inherits nor repairs that, and it does not touch your granularity objection.
A calibration-free criterion is precisely what makes Hessian-guided allocation attractive: the appeal of is that it needs no labels. Had the criterion ranked layers correctly only once given task supervision, that would itself be a result about the criterion. It does not.
Response to Authors
We thank the authors for the responses. The authors explain that the method is a diagnostic tool and not a deployable allocator. This diagnostic tool identifies which layers are more sensitive to quantization, but when they restore these layers with FP16 precision the improvement is only on one task, even though they select these layers using the same benchmarks. As a results, this diagnostic tool has not been show to be useful. Furthermore, the method does not have a strong baseline comparison. For these reasons we maintain our original score.
What the paper is for, and why accuracy barely moves on GSM8K and MATH
"The method does not deliver: REST-6 puts six out of 32 transformer blocks in FP16. As a result, the accuracy would be expected to improve since less quantization error is added into the model. But the method improves only on one benchmark ARC and is still worse than GPTQ with all the layers in 4-bit. (Table1 and Table 2)"
We should have been clearer about the claim, and the revision is. REST-6 is a diagnostic. The object that carries the paper's argument is the ranking, and REST-6 is the construction that tests whether the ranking means anything at a fixed budget. It is not proposed as a deployable allocator, it is not tuned to beat GPTQ, and we agree that it does not. Removing that impression from the framing is one of the changes in the revision.
On the specific comparison with GPTQ-INT4, one measurement in our own Table 1 argues against reading that row as a precision effect at all: GPTQ-INT4 scores 85.4 on GSM8K against FP16's 80.2, 5.2pp above the unquantized model. Reducing precision cannot add capability. The GPTQ row is a different artifact: a separately released checkpoint (
hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4), not our base model put through GPTQ. The gap most likely reflects how its calibration interacts with the chat template and answer extractor. Nothing in the paper's argument rests on that row, and the revision marks it as not budget-comparable to the NF4 arms. The internally controlled comparison is REST-6 against HESS-6: same base model, same quantizer, same 788.5M FP16 parameters, same 2-attention/4-MLP composition, differing only in which six positions are chosen.On the expectation that accuracy should improve: it does not, and the reason is measurable rather than a matter of interpretation. Quantization damage is bidirectional. Restoring precision repairs forward flips, but it also removes the reverse flips, the examples the FP16 model gets wrong and the 4-bit model happens to get right.
On GSM8K and MATH the two channels nearly cancel, so there is almost no net accuracy for any allocation to recover, whatever it protects. On ARC the reverse channel is small, net damage is 4.4pp, and repairing forward flips shows up in accuracy. That is why the accuracy-level result appears on ARC only, and it is a property of where quantization damage is visible rather than of REST-6. We state it in the revision as a limit on what a 4-bit accuracy benchmark can resolve, not as a result in our favour.