acceptodds
Under review as a conference paper at ICLR 2027

FROM ATTENTION SENSITIVITY TO LAYER ROLE: REVISITING MIXED-PRECISION QUANTIZATION OF TRANSFORMERS

Abstract

Most post-training quantization pipelines keep each quantized weight matrix close to its pretrained counterpart, one matrix at a time, and stop there. Whether that proxy tracks what the attention block actually computes, or how errors in the Q, K and V projections compound once they meet inside the softmax, is rarely checked. The obvious fix is to write the objective on the attention output itself, over all three projections at once, and to use that same objective wherever the pipeline needs a signal. That is what JAB does. It defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block's real causally-masked attention output, and uses it both to fit the quantized weights (GPTQ warm start, then STE with learnable scales) and to score the block for a multiple-choice knapsack allocation. On attention-only quantization of Mistral-7B this pays off: at 3 bits JAB recovers – of the gap between uniform GPTQ and full precision, and its sensitivity estimate tracks an oracle costing 73 forward passes to within a fraction of a perplexity point. Once MLP layers enter the allocation, though, JAB is beaten by a role-aware offset rule that needs no sensitivity estimate at all, both on GPT-2 with QKV and MLP quantized together and on the full Mistral-7B model. With a 3-bit floor, this rule quantizes of Mistral-7B's weights to 4.5 bits per parameter at perplexity — within of full precision () at compression, 14 GB to 3.93 GB — against for JAB at the same budget, and to 3.5 bits at (). Whichever matrix a weight belongs to inside its block matters more than any sensitivity estimate we managed to compute. Two things came out sideways. Block-local reconstruction turned out to be an unreliable proxy for end-to-end perplexity: in one run a improvement in a block's own objective came with a increase in perplexity, which is why every allocation in this paper is validated end-to-end rather than on local objectives alone. And on attention-only quantization, fine-tuning consistently moved weights farther from their pretrained values while pulling attention outputs closer, with net gains. Post-training seems to recover attention behavior, not weights.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixed-precision quantization, large language models, transformer, sensitivity estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tRUAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks how many bits each part of an LLM should get when the model is quantized after training. The authors define one loss, called JAB, on the real output of an attention block (using the Q, K and V weights together). They use it twice: to fine-tune GPTQ-quantized weights, and, through its curvature (the Hessian trace), to score each block and pick bit widths. On attention-only quantization of Mistral-7B at 3 bits, this closes 77–90% of the gap between uniform GPTQ and fp16. From what I understand, this method makes it cheaper to get a good GPTQ quantization without needing to have end-to-end oracle search.

Critical strengths
  1. The paper uses one objective (JAB) both to fit the quantized weights and to decide how many bits each block gets. Treating the bit choice as part of the objective, instead of a separate heuristic, is a good idea. The budget is also counted per parameter, which is the right choice when matrices differ in size by up to 14x.
Critical weaknesses

Weakness

  1. On Mistral-7B, the role-beats-curvature claim holds only at 4.5 bits, and there it only matches uniform 4-bit. Section 6.3 ("Layer role outperforms curvature") says role beats curvature on Mistral-7B, but the tables support this only at 4.5 bits (Type-offset 6.933 vs JAB 7.158, Table 6). At 3.5 bits JAB is ahead (7.514 vs 7.875, Table 6), which the Conclusion admits (Section 7). At 4.5 bits Type-offset (6.933) only matches plain uniform 4-bit (6.938, Table 6 caption) while using more bits (3.56x vs 4.00x compression; Table 4), and adaptive allocation at B=4.3 is worse than uniform 4-bit (7.211 vs 6.938, Table 6 caption).

  2. The role rule and the attention-only baseline are not tested carefully. The role rule is "transposed from the GPT-2 oracle rather than tuned" (Section 6.2), and no ablation of the offsets is reported (Section 6.3). In the attention-only setting (Table 1), uniform 3-bit goes from fp16 4.818 to 8.738 (+81%), while quantizing all seven matrices to 3 bits costs +37% in Table 4 (9.126 vs 6.643), even though QKV is about 11.5% of the weights (Section 6.2). This is not explained (the two studies use different fp16 references, so the comparison is only indicative), and "gap closed 77-90%" (Table 1) depends on it.

  3. The paper as a whole is difficult to follow, and I would encourage the authors to restructure it. I first read the motivation as making GPTQ better, then as making attention quantization better, and only later understood that it seems to be the "proxy gap" (the objective that fits the weights is not the criterion that sets the bit widths). That is only named in paragraph 5 of the introduction (Section 1) and is never tested directly. It is also not clear whether the paper proposes a method or offers an analysis. The conclusion says the method works only on attention and does not transfer, then lists general lessons that hold independently of it, and ends on a claim about larger models that is not tested ("we test only to 7B"; Section 7).

  4. The contribution of the paper is not clear. The contributions listed in Section 1 include the JAB objective and the finding about matrix role, but the results do not show that either helps on the full model. The full-model results in Table 6 are GPTQ-only, so JAB's fine-tuning is tested only in the attention-only setting (Table 1), and the Conclusion says the method does not transfer beyond attention (Section 7). The role finding, which Section 6.3 presents as the main result, holds only at 4.5 bits (Weakness 1).

Formatting and presentation issues

  1. The abstract is three paragraphs and repeats the introduction. It should be one short paragraph.
  2. Section 6.5 says "all four criteria" return the uniform assignment at B = 3.0, but it is unclear which four criteria this means, since Table 6 shows three (JAB, JAB-norm, Type-offset).
  3. Section 6.5 says q and k hold 19.2% of the weights. Section 6.2 says the MLP carries 80.8% of the weights under the budget (the seven linear modules per block, 96.4% of the model), which leaves 19.2% for all four attention matrices, and q and k alone are about 9.6% (from the sizes in Section 6.2). So the number looks wrong or is not explained.
  4. Typos: "lemphwhich" (Section 6.3). Figure 1 is too small to read.
  5. Figure 1 is hard to follow as a pipeline. The boxes carry small numbers in the corner, such as (6), (15) and (17,18), that are never explained. They look like equation numbers, but they do not match the equations in the text (for example, Eq. 6 in Section 2 is the scoring formula, not the loss), and there are no numbered steps (1, 2, 3) to show the order.
  6. Published papers are cited with their venue but link to the arXiv version instead of the published one (OmniQuant at ICLR 2024, QuIP# at ICML 2024)
Questions for authors
  1. How much faster is searching with JAB than the end-to-end oracle search?
1 comment

Thank you for a very careful reading; every point led to a concrete change.

W1 — role vs curvature. Section 6.3 now states the perplexity result per regime (Type-offset leads at 4.5 bits, JAB at 3.5), and Section 6.4 adds the downstream comparison, which is decisive: JAB's full-model allocations answer at chance at both budgets, while Type-offset retains 87%87\% (3.5 bits) and 95%95\% (4.5 bits) of fp16 zero-shot accuracy. At 4.5 bits Type-offset beats uniform 4-bit by 1.6 points zero-shot and 1.7 points on MMLU at equal perplexity. The adaptive B=4.3B=4.3 row you cite is from the unconstrained run without the bit floor; it is now in Appendix J.2.

W2a — the attention-only baseline. The two studies differed in evaluation protocol, and the earlier attention-only study used a float32 GPTQ solve. We reran it with the float64 pipeline in all four calibration→evaluation pairings and a common fp16 reference (Appendix K.1). Uniform 3-bit is now within 4%4\% of fp16 on perplexity, but downstream it answers at chance (37.1%37.1\% zero-shot, 24.4%24.4\% MMLU), while the JAB allocation retains 69.1%69.1\% and, with joint refinement, 49.5%49.5\% MMLU. The revised Section 4 reports this comparison in place of the previous gap-closed figure.

W2b — offset ablation. We perturbed each of the seven offsets by ±1\pm 1 and evaluated five random permutations (Appendix K.2, Table 19). At 4.5 bits all five permutations lie within 6.904–6.938, and 16 of the 19 variants beat both trace-based criteria. The rule therefore needs no tuning: what matters is spending the surplus outside qq and kk, which the role rule does and the curvature criterion does not.

W3, W4 — framing and contribution. The paper now states its question in the first paragraph and lists four contributions:

  1. perplexity is not a valid acceptance test for allocation;
  2. for attention-only quantization, the matched JAB objective allocates well;
  3. for the whole model, a role rule outperforms attention curvature;
  4. controlled evidence on joint refinement, block-local objectives and the low-bit floor.

Fine-tuning was run in the full-model setting with the floor (7.514→11.607.514 \rightarrow 11.60 at 3.5 bits, 7.158→12.167.158 \rightarrow 12.16 at 4.5), which is why Table 4 reports GPTQ-only allocations; Section 6.5 explains this with the block-local proxy result. The conclusion no longer refers to untested larger models.

Q1 — speed vs the oracle. The oracle quantizes and evaluates every (layer, width) pair end to end, about 73 full-model evaluations on GPT-2. JAB needs one scoring pass of 10 Hutchinson probes on two batches per layer; a complete JAB run on GPT-2 (scoring, quantization and full evaluation) takes 62 s. The search cost is thus reduced from about 73 model evaluations to one, and the Hutchinson criterion stays within 0.17 perplexity of the oracle on average at budgets ≥3.5\geq 3.5 bits.

Formatting.

  1. Abstract: one paragraph.
  2. "All four criteria" corrected to three.
  3. 19.2%19.2\% is the share of all four attention matrices; qq and kk hold 9.6%9.6\%. Corrected: the concentration is therefore 77%77\% of trace mass on 9.6%9.6\% of the weights.
  4. Typo fixed; Figure 1 enlarged.
  5. The corner labels in Figure 1 were stale equation numbers and are removed; stages are labelled (a) and (b) in order.
  6. OmniQuant (ICLR 2024) and QuIP# (ICML 2024) now cite the published versions.

Thank you again for the detailed feedback, which clearly improved the paper. If these changes address your concerns, we would sincerely appreciate your reconsidering the score.

Reviewer_bEwaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a mixed-precision PTQ method that allocates bit-width and quantizes weights to preserve attention outputs and attention maps.

Critical strengths

None

Critical weaknesses
  1. The most crucial concern is that the evaluation is very weak. Unlike prior works that demonstrate their effectiveness by measuring PPL, 0-shot common sense reasoning tasks (e.g., ARC, PIQA, HellaSwag etc), MMLU, IFEval, the authors only evaluated PPL and next token accuracy.

  2. The literature survey is out-dated. There are many PTQ approaches that quantize by considering cross-layer dependencies. For backpropation-free methods, the authors can refer to BoA, TurboBoA, YAQA. For training-based methods, the authors need to consider OmniQuant, FlexRound, Block-AP of EfficientQAT which jointly quantize all seven linear modules inside the Transformer block. Without any comparison, it is very difficult to validate the efficacy of the proposed method.

  3. The authors need to improve the presentation.

Questions for authors

See Weaknesses

1 comment

Thank you for the review. We have addressed each weakness with new experiments and a substantially revised paper.

W1 — evaluation. We added the evaluation used in recent PTQ work: ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande, BoolQ (zero-shot) and MMLU (5-shot) for ten Mistral-7B configurations, plus full-split WikiText-2 perplexity (table in the global response, per-task scores in Appendix K). Our fp16 scores reproduce published values. IFEval applies to instruction-tuned models; Mistral-7B-v0.1 is a base model, so the knowledge and reasoning suite above is the appropriate one.

The evaluation strengthens the paper considerably. It shows that:

  1. the JAB allocation keeps 91%91\% of zero-shot accuracy at 3-bit attention, where uniform GPTQ answers at chance;
  2. the Type-offset rule is the best quantized model on all accuracy metrics at 3.56×3.56\times compression;
  3. perplexity, the metric used to accept allocations in most mixed-precision work, is uncorrelated with downstream accuracy (Spearman ρ=−0.10\rho = -0.10) and misses every collapsed model.

We believe (3) is directly useful to the community: allocation methods should be validated on downstream tasks.

W2 — literature. The introduction and Appendix A now discuss backpropagation-free attention-aware quantization (BoA, TurboBoA), block-wise training (OmniQuant, FlexRound, EfficientQAT Block-AP) and end-loss-guided rounding (YAQA, GuidedQuant). These methods determine how a matrix is rounded at a given bit width; our work addresses which bit width each matrix receives, holding the quantizer fixed at GPTQ. The two are complementary: any allocation in Section 6 can be combined with any of these quantizers, and the closest allocation method, APTQ, is compared directly (Table 2). Our GPT-2 control also connects to the end-loss line: a task-loss trace is a strong allocation score, consistent with the motivation of YAQA and GuidedQuant.

W3 — presentation. One-paragraph abstract; the introduction now states the question in its first paragraph and ends with an explicit contribution list; Figure 1 is enlarged and its labels fixed; tables report both perplexity and accuracy.

Given the new downstream evaluation and the expanded related work, we hope the paper now meets your expectations, and we would be grateful if you would reflect this in your final score.

Reviewer_HAjgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reconstruction objectives and bit-width allocation criteria for mixed-precision training quantization of Transformers. The proposed JAB framework combines attention-output reconstruction error with attention-map KL divergence into a joint QKV objective. It uses GPTQ initialization followed by refinement with STE and learnable scales, and derives allocation costs from the Hessian trace of the same objective. Experiments cover attention-only, MLP-only, and combined quantization settings on GPT-2 small and Mistral-7B. JAB allocation substantially improves attention-only quantization on Mistral, whereas a simple matrix-role prior performs better at some budgets when heterogeneous modules are quantized together.

Critical strengths
  • The paper addresses a meaningful question and reports informative negative results. It examines allocation, local refinement, and end-to-end quality rather than reporting only favorable configurations.

  • Some controlled results demonstrate clear improvements. In attention-only quantization of Mistral-7B, adaptive allocation substantially outperforms uniform 3-bit GPTQ across all four calibration–evaluation corpus pairings. The full-model experiments use parameter-weighted budgets and include additional controls with per-unit traces and a minimum bit-width constraint, reflecting attention to fair comparisons across heterogeneous modules.

Critical weaknesses
  • Key theoretical and methodological claims are insufficiently justified. Equation 4 requires stronger assumptions than stated (eg. The variance formula in Equation 4 requires stronger fourth-moment conditions than uncorrelatedness.). Equation 6 appears to omit the parameter-dimension normalization used in HAWQ-V2, and the full-model loss is not fully defined. These issues could affect the allocation results and undermine the conclusion that matrix role is more informative than curvature.

  • Novelty and practical benefits are not convincingly established. APTQ already uses attention-output reconstruction and curvature-based allocation, yet no direct comparison is provided. The claimed benefit of joint QKV optimization also lacks a separate-optimization control. Moreover, the role prior loses to JAB at 3.5 bits and, at 4.5 bits, barely improves over uniform 4-bit quantization despite using more storage.

Questions for authors
  • What exact objective is differentiated and optimized in the full-model experiments? With the block input fixed, how do the attention output and attention map in Equation 3 depend on the current block’s MLP weights?

  • What precisely is the block-output term mentioned in Appendix J.7?

  • What are the precise assumptions and scope of Equation 4? Zero-mean, uncorrelated, equal-variance errors are insufficient to determine the variance of a quadratic form.

  • What is the independent contribution of joint optimization relative to the closest prior methods?

  • With bit allocation, calibration data, objective normalization, and optimization budget matched, how does joint QKV optimization compare with optimizing the matrices separately using the same loss?

1 comment

Thank you for the careful technical reading. We address every point.

Eq. 4 assumptions (W1, Q3). We now state the assumption precisely: the residual δ\delta has independent entries with zero mean, common variance σ2\sigma^2 and finite fourth moment μ4\mu_4, with ΔL=12δ⊤Hδ\Delta L = \frac{1}{2}\delta^\top H\delta. For a symmetric AA,

Var[δ⊤Aδ]=(μ4−3σ4)∑jAjj2+2σ4∥A∥F2\mathrm{Var}[\delta^\top A\delta] = (\mu_4 - 3\sigma^4)\sum_j A_{jj}^2 + 2\sigma^4\lVert A\rVert_F^2

Since HsepH_{\rm sep} is the block-diagonal part of HLH_L, the two share their diagonal and the fourth-moment terms cancel in the difference; their off-diagonal supports are disjoint, so ∥HL∥F2=∥Hsep∥F2+∥Γ∥F2\lVert H_L\rVert_F^2 = \lVert H_{\rm sep}\rVert_F^2 + \lVert\Gamma\rVert_F^2. Hence Var[ΔL]−Var[ΔLsep]=12σ4∥Γ∥F2\mathrm{Var}[\Delta L] - \mathrm{Var}[\Delta L_{\rm sep}] = \frac{1}{2}\sigma^4\lVert\Gamma\rVert_F^2 for any μ4\mu_4, with no Gaussianity needed. This is the standard noise model for round-to-nearest; for GPTQ the identity holds approximately. Full proof: Appendix B.3.

Eq. 6 normalization (W1). Eq. 6 uses the raw trace; HAWQ-V2 uses Tr(H)/n\mathrm{Tr}(H)/n. The two give identical allocations when units are equal-sized, which covers the attention-only and MLP-only studies. For the full model we ran the HAWQ-V2 form: 7.726 at B=3.5B=3.5 and 7.124 at B=4.5B=4.5 (raw: 7.514 and 7.158). At 4.5 bits both forms remain behind Type-offset (6.933), so the role-versus-curvature conclusion is unchanged (Section 6.3, Appendix K.2).

Full-model objective (Q1, Q2). With the block input fixed, Eq. 3 is indeed independent of the MLP weights, which is why the full-model study uses the block-level loss, now Eq. 8:

Lblk(w)=∥f(X;W)−f(X;w)∥F2+λKL⋅KL(P∥Q(w))L_{\rm blk}(w) = \lVert f(X;W) - f(X;w)\rVert_F^2 + \lambda_{\rm KL}\cdot\mathrm{KL}(P\Vert Q(w))

with ff the whole decoder block and ww the concatenation of all seven matrices. MLP weights enter through the first term (the "block-output term" of Appendix J.7), and qq, kk additionally through the KL term. Each unit is scored by the trace of its own diagonal block of this Hessian (Section 2.5). This also explains Section 6.3: qq and kk carry 77% of the trace mass because they are the only units that reach the attention map.

APTQ comparison (W2, Q4). We implemented an APTQ-style score (attention-output Hessian trace per matrix, normalized by parameter count) and a HAWQ-V2 task-loss trace inside our pipeline, with the same quantizer, solver, data and budgets (GPT-2, Table 2 right). JAB outperforms the APTQ-style score at 3.5 bits (28.52 vs 49.35); the task-loss trace is strongest on GPT-2 perplexity. On Mistral-7B, where downstream accuracy can be measured, the JAB allocation preserves 91% of zero-shot accuracy where uniform quantization preserves none. The independent contribution relative to APTQ is therefore (i) the joint Q/K/VQ/K/V objective, which outperforms per-matrix refinement (below), and (ii) using one objective for both refinement and allocation, which on Mistral-7B yields the attention-only allocation that keeps the model working.

Joint vs separate optimization, matched (Q5). Same allocation, calibration batches, GPTQ warm start, learning-rate units and write-back gate; only the loss and partition differ (Table 2 left). The separate variants receive 200 steps per matrix, three times the steps of joint refinement.

BitsGPTQ onlyJointBlockwise (same loss)Per-matrix loss
3.090.7742.9947.5765.61
3.528.5228.3627.9527.93
4.026.2325.7125.6825.74
4.526.1925.7325.7525.61

At 3.0 bits joint refinement is clearly best: 10% better than block-coordinate descent of the same loss and 34% better than per-matrix losses, so both the attention-space objective and jointness contribute. Above 3 bits the three reach the same quality, and joint refinement does so with a third of the steps and 2.7×2.7\times less wall-clock.

Role prior vs uniform 4-bit (W2). With downstream evaluation, Type-offset at 4.5 bits is the best quantized model on every accuracy metric: 72.3% zero-shot and 58.1% MMLU, against 70.7% and 56.4% for uniform 4-bit, at equal perplexity (5.471 vs 5.466). At 3.5 bits JAB has lower perplexity but answers at chance (37.4%), while Type-offset retains 66.3%; the comparison on accuracy favours the role rule at both budgets.

We believe the corrected assumptions, the full proof and the matched controls resolve the technical concerns, and we would appreciate it if you would reconsider your score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

We thank the reviewers for detailed and constructive reviews. We ran every experiment requested and revised the paper accordingly. The main additions:

1. Downstream evaluation. Ten Mistral-7B configurations evaluated with lm-evaluation-harness on ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande and BoolQ (zero-shot), MMLU (5-shot) and full-split WikiText-2 perplexity. Our fp16 scores reproduce published values (HellaSwag 81.2, ARC-C 54.3, MMLU 62.6).

ConfigurationBits/paramPPLZero-shotMMLU
fp16165.30475.962.6
Attention-only (QQ,KK,VV), average 3 bits
Uniform 3-bit GPTQ–5.45837.124.4
JAB allocation–6.36069.142.7
JAB allocation + joint refinement–5.96569.149.5
Full model, all seven modules
Uniform3.06.62537.324.3
JAB3.55.85437.424.4
Type-offset3.56.10366.329.8
Uniform4.05.46670.756.4
JAB4.55.61339.224.6
Type-offset4.55.47172.358.1

Chance: 37.5 (zero-shot mean), 25 (MMLU).

2. What the downstream results establish.

  • Attention-only: the JAB allocation with joint refinement keeps 91%91\% of fp16 zero-shot accuracy and 79%79\% of MMLU at an average of 3 bits; uniform GPTQ at the same budget answers at chance.
  • Full model: the Type-offset rule is the best quantized model on every accuracy measure, at 3.56×3.56\times compression and with perplexity equal to uniform 4-bit.
  • Perplexity is not a valid acceptance test for allocation: across the nine quantized models, Spearman correlation between perplexity and zero-shot accuracy is ρ=−0.10\rho = -0.10, and perplexity flagged none of the four models that answer at chance.

3. New controls. Joint vs block-coordinate vs per-matrix Q/K/VQ/K/V refinement under matched allocation, data, warm start and gate; APTQ-style and HAWQ-V2 allocation scores; HAWQ-V2 trace normalization on Mistral-7B; an ablation of every Type-offset offset plus random permutations; float64 rerun of the attention-only study in all four corpus pairings.

4. Technical corrections. Eq. 4 now states the correct assumption (independent residuals, finite fourth moment) with a full proof; Eq. 6 states the normalization; the full-model objective is defined (new Eq. 8).

5. Presentation. One-paragraph abstract, restructured introduction with an explicit contribution list, enlarged Figure 1, corrected numbers and citations.

We hope these additions and revisions address your concerns, and we would be grateful if you would consider raising your score in light of them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.