- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks how many bits each part of an LLM should get when the model is quantized after training. The authors define one loss, called JAB, on the real output of an attention block (using the Q, K and V weights together). They use it twice: to fine-tune GPTQ-quantized weights, and, through its curvature (the Hessian trace), to score each block and pick bit widths. On attention-only quantization of Mistral-7B at 3 bits, this closes 77–90% of the gap between uniform GPTQ and fp16. From what I understand, this method makes it cheaper to get a good GPTQ quantization without needing to have end-to-end oracle search.
Critical strengths
- The paper uses one objective (JAB) both to fit the quantized weights and to decide how many bits each block gets. Treating the bit choice as part of the objective, instead of a separate heuristic, is a good idea. The budget is also counted per parameter, which is the right choice when matrices differ in size by up to 14x.
Critical weaknesses
Weakness
-
On Mistral-7B, the role-beats-curvature claim holds only at 4.5 bits, and there it only matches uniform 4-bit. Section 6.3 ("Layer role outperforms curvature") says role beats curvature on Mistral-7B, but the tables support this only at 4.5 bits (Type-offset 6.933 vs JAB 7.158, Table 6). At 3.5 bits JAB is ahead (7.514 vs 7.875, Table 6), which the Conclusion admits (Section 7). At 4.5 bits Type-offset (6.933) only matches plain uniform 4-bit (6.938, Table 6 caption) while using more bits (3.56x vs 4.00x compression; Table 4), and adaptive allocation at B=4.3 is worse than uniform 4-bit (7.211 vs 6.938, Table 6 caption).
-
The role rule and the attention-only baseline are not tested carefully. The role rule is "transposed from the GPT-2 oracle rather than tuned" (Section 6.2), and no ablation of the offsets is reported (Section 6.3). In the attention-only setting (Table 1), uniform 3-bit goes from fp16 4.818 to 8.738 (+81%), while quantizing all seven matrices to 3 bits costs +37% in Table 4 (9.126 vs 6.643), even though QKV is about 11.5% of the weights (Section 6.2). This is not explained (the two studies use different fp16 references, so the comparison is only indicative), and "gap closed 77-90%" (Table 1) depends on it.
-
The paper as a whole is difficult to follow, and I would encourage the authors to restructure it. I first read the motivation as making GPTQ better, then as making attention quantization better, and only later understood that it seems to be the "proxy gap" (the objective that fits the weights is not the criterion that sets the bit widths). That is only named in paragraph 5 of the introduction (Section 1) and is never tested directly. It is also not clear whether the paper proposes a method or offers an analysis. The conclusion says the method works only on attention and does not transfer, then lists general lessons that hold independently of it, and ends on a claim about larger models that is not tested ("we test only to 7B"; Section 7).
-
The contribution of the paper is not clear. The contributions listed in Section 1 include the JAB objective and the finding about matrix role, but the results do not show that either helps on the full model. The full-model results in Table 6 are GPTQ-only, so JAB's fine-tuning is tested only in the attention-only setting (Table 1), and the Conclusion says the method does not transfer beyond attention (Section 7). The role finding, which Section 6.3 presents as the main result, holds only at 4.5 bits (Weakness 1).
Formatting and presentation issues
- The abstract is three paragraphs and repeats the introduction. It should be one short paragraph.
- Section 6.5 says "all four criteria" return the uniform assignment at B = 3.0, but it is unclear which four criteria this means, since Table 6 shows three (JAB, JAB-norm, Type-offset).
- Section 6.5 says q and k hold 19.2% of the weights. Section 6.2 says the MLP carries 80.8% of the weights under the budget (the seven linear modules per block, 96.4% of the model), which leaves 19.2% for all four attention matrices, and q and k alone are about 9.6% (from the sizes in Section 6.2). So the number looks wrong or is not explained.
- Typos: "lemphwhich" (Section 6.3). Figure 1 is too small to read.
- Figure 1 is hard to follow as a pipeline. The boxes carry small numbers in the corner, such as (6), (15) and (17,18), that are never explained. They look like equation numbers, but they do not match the equations in the text (for example, Eq. 6 in Section 2 is the scoring formula, not the loss), and there are no numbered steps (1, 2, 3) to show the order.
- Published papers are cited with their venue but link to the arXiv version instead of the published one (OmniQuant at ICLR 2024, QuIP# at ICML 2024)
Questions for authors
- How much faster is searching with JAB than the end-to-end oracle search?
Thank you for a very careful reading; every point led to a concrete change.
W1 — role vs curvature. Section 6.3 now states the perplexity result per regime (Type-offset leads at 4.5 bits, JAB at 3.5), and Section 6.4 adds the downstream comparison, which is decisive: JAB's full-model allocations answer at chance at both budgets, while Type-offset retains 87% (3.5 bits) and 95% (4.5 bits) of fp16 zero-shot accuracy. At 4.5 bits Type-offset beats uniform 4-bit by 1.6 points zero-shot and 1.7 points on MMLU at equal perplexity. The adaptive B=4.3 row you cite is from the unconstrained run without the bit floor; it is now in Appendix J.2.
W2a — the attention-only baseline. The two studies differed in evaluation protocol, and the earlier attention-only study used a float32 GPTQ solve. We reran it with the float64 pipeline in all four calibration→evaluation pairings and a common fp16 reference (Appendix K.1). Uniform 3-bit is now within 4% of fp16 on perplexity, but downstream it answers at chance (37.1% zero-shot, 24.4% MMLU), while the JAB allocation retains 69.1% and, with joint refinement, 49.5% MMLU. The revised Section 4 reports this comparison in place of the previous gap-closed figure.
W2b — offset ablation. We perturbed each of the seven offsets by ±1 and evaluated five random permutations (Appendix K.2, Table 19). At 4.5 bits all five permutations lie within 6.904–6.938, and 16 of the 19 variants beat both trace-based criteria. The rule therefore needs no tuning: what matters is spending the surplus outside q and k, which the role rule does and the curvature criterion does not.
W3, W4 — framing and contribution. The paper now states its question in the first paragraph and lists four contributions:
Fine-tuning was run in the full-model setting with the floor (7.514→11.60 at 3.5 bits, 7.158→12.16 at 4.5), which is why Table 4 reports GPTQ-only allocations; Section 6.5 explains this with the block-local proxy result. The conclusion no longer refers to untested larger models.
Q1 — speed vs the oracle. The oracle quantizes and evaluates every (layer, width) pair end to end, about 73 full-model evaluations on GPT-2. JAB needs one scoring pass of 10 Hutchinson probes on two batches per layer; a complete JAB run on GPT-2 (scoring, quantization and full evaluation) takes 62 s. The search cost is thus reduced from about 73 model evaluations to one, and the Hutchinson criterion stays within 0.17 perplexity of the oracle on average at budgets ≥3.5 bits.
Formatting.
Thank you again for the detailed feedback, which clearly improved the paper. If these changes address your concerns, we would sincerely appreciate your reconsidering the score.