- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The study treats quantization of the final softmax (LM head) layer as a rate-distortion problem under output KL divergence rather than weighted MSE. It shows that the second-order error weight is class-aware: feature covariance coupled with per-class softmax curvature. Overall Paper is well written and explores an interesting direction.
Critical strengths
-
The formulation is principled and simple.
-
Results are consistent, large, and well ablated. Gains hold across five models, three head rates, two body quantizers (GuidedQuant, VPTQ), and zero-shot tasks, where LAMBADA improves by 5–7 points.
Critical weaknesses
-
The only comparison is WaterSIC with its refinements removed. Missing are GPTQ/RTN/AWQ group-wise on the head, QTIP/QuIP# head quantization, LFQ, GuidedQuant applied to the head, and a YAQA-style Hessian-sketch head.
-
Entropy-coded, variable-rate rows and columns are not a standard inference format. There is no decode or inference latency, kernel support, or runtime memory analysis, and no comparison at fixed-rate hardware-friendly formats.
-
Calibration is mostly on WikiText-2 and evaluation is in-domain, and there is no variance across calibration seeds.
-
The second-order KL expansion is assumed valid at 2 bits, where KL increases are large (e.g., +34.9% on a 2-bit body), but expansion accuracy is never measured at those operating points.
-
The paper claims the method applies to vision and speech classifiers and MoE routers, but all are not tested.
Questions for authors
See Weakness.