- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper argues that rounding to a fixed grid introduces a fixed amount of error, so quantizers cannot reduce the error, only relocate it. It measures distortion via the weight (activation) "operator" that acts on the activation (weight) quantized tensor, and introduces OptQuant, a quantizer whose role is to push error in the direction the operator attenuates via a GPTQ-style sequential error feedback. Such feedback is applied independently to:
- activations, with no need of calibration data, using non-uniform Gaussian levels (via Lloyd-Max).
- weights, using GPTQ (needs calibration data), with block-Hadamard rotation and non-uniform Gaussian levels. K and V use a custom quantization strategy. In addition, learned rotations for the input activations of o_proj and down_proj is also proposed (ROTA-STE).
Results are reported on several models of the Llama family (1B-70B), with additional results on Qwen and EXAONE in the appendices. Evaluation covers perplexity (Wikitext / C4) and accuracy on several conventional zero-shot tasks. Additional measurements (more limited) on long-context and reasoning tasks are also reported.
OptQuant (typically inclusive of ROTA-STE) showing improved performance against various competing quantization methodologies at 4 bits (W4A4KV4). It also achieves large margins at W3A3KV3 and at 2-bit weight-only, and wins on LongBench with 3-bit KV caches.
Critical strengths
The main strength is providing a unified view of several quantization approaches (GPTQ, SmoothQuant/AWQ, rotations, ...), showing that error feedback (GPTQ-style) moves the error, while rotation shrinks it. The proposed OptQuant achieves remarkable performance improvement, especially under aggressive quantization regimes (3 bits).
Critical weaknesses
The activation quantization process requires a massive D x D FP32 (or FP16) matrix per linear module, wiping out any compression gain. The authors acknowledge this limitation in the appendix B.3, but the main text does not discuss it. In addition, activations and KV cache use non-uniform levels (from Lloyd–Max) which can't be accelerated in conventional INT4 HW. Efficiency measurements are limited to overhead ratios for the activation pre-processing relative to a layer GEMM (Table 13a, whose GEMM operations, precision, and layer are unspecified, and which does not appear to include the relocation matmul) and a fused-kernel study (Table 13b). There is no low-bit GEMM, no end-to-end latency or throughput measurement. Given these two drawbacks (per-token D x D compensation step and non-uniform activation levels), I expect the activation relocation scheme (arguably the key contribution of this paper) to be impractical on real HW as presented.
It should be also noted that all competing algorithms (to which OptQuant is compared) utilize evenly-spaced quantization levels for activations, hence the direct comparison (Table 1, for example) is not entirely fair.
Questions for authors
As per "Critical Weaknesses", I would ask the authors for measured decode/prefill latency and memory including the compensation factors. W4A4KV4 results with uniform INT4 activation/KV grids would also be helpful for a direct comparison against presented baselines.
Individual Response
Thank you for the constructive review. We answer the weaknesses (W) and questions (Q) in order.
W1. The revision moves a component-wise ablation into the main text (Q2), with 8B and MoE results in the appendix.
W2. Per linear layer: a block-Hadamard on the input (fast Walsh-Hadamard, inverse folded into the weights), one relocation pass, rounding, and the INT4 GEMM; the o/down transform exists only in the optional learned variant. We withdraw the 16.5% overhead in Table 13a (measured at d=4096 with 32k tokens; not representative). Measured relocation cost vs. the layer's FP16 GEMM on B200: 3.9–6.7× at 8k-token prefill and 9–30× at batch-1 decode (FLOP lower bound 0.5–1.75×); the compensation factor, stored losslessly in INT4 (128-wide groups), takes 1.1–1.5× the INT4 weight memory. These costs move into the main text.
W3. Moved into the main text. The temporal (Σ-Δ) gain on the V cache is bounded by the lag-1 autocorrelation of attention rows (Prop. 3); the measured ≈0.05 predicts near-zero headroom, as a direct implementation confirms. We add a second case: optimizing the transform to minimize the sum of relocation pivots reaches the AM-GM bound, yet raises kurtosis (3.0 → 3.9–5.1) and increases realized distortion by 1.3–2.7 dB.
W4, W6, W7, W9. Presentation. Section 3 is expanded by about one page (bit-ladder and long-context tables move to the appendix); each subsection opens with the error mode it targets and its intuition, referring to Figure 1; the lag-1 autocorrelation and operator metric are defined in Section 2 before use; the abstract is fixed.
W5. Agreed: the claim holds for a fixed grid, and the revision says so. Transforms, scales and companding set the grid and the residual's magnitude; relocation then decides its placement. Q2 quantifies both.
W8. We add analysis from the new experiments: which component helps at which bit-width (Q2), relocation versus finer scales (Q1), when companding matters (Q3), and where the method is costly (W2).
Q1. Per token, after the Hadamard, activations are processed in 128-channel blocks: each block is rounded and its error, propagated through a factor U precomputed offline from the weights (H = WᵀW), is subtracted from the channels not yet quantized. Llama-2-7B W4A4KV4, without → with relocation: per-token scales 6.06 → 5.82, 128-group 5.80 → 5.70 (8B: 7.93 → 7.12, 7.10 → 6.80). Without relocation, A5 gives 5.74 (per-token) / 5.65 (128-group) and A6 5.64 (per-token), so relocation is worth about 0.7 bit. Given its current cost (W2), a wider grid is cheaper where A5/A6 hardware exists; on INT4-only hardware relocation recovers part of that precision.
Q2. Llama-2-7B:
Weights (weight-only, group 128), W4 / W3 / W2: RTN 5.78 / 8.22 / 1.2e5 → +relocation (GPTQ) 5.63 / 7.53 / div. → +block-Hadamard 5.59 / 6.60 / 1.8e5 → +Lloyd-Max 5.56 / 5.86 / 8.46. Activations (W4A4KV4), without → with relocation: uniform per-token 6.06 → 5.82; uniform 128-group 5.80 → 5.70; Lloyd-Max 128-group 5.79 → 5.69. Relocation is the largest weight-path step at W4; Hadamard and Lloyd-Max become essential at W3/W2. On activations, relocation adds 0.10–0.24 PPL (8B: 0.22–0.81) and finer group scales a comparable amount, which our ablation row L6 had attributed to companding; we correct this. The learned transform is optional: the training-free configuration now matches it in PPL on all five models.
Q3. It is not specific to down_proj: Lloyd-Max levels are used for all weights. Quantization is offline; dequantization is a 16-entry lookup as in NF4/LUT-GEMM kernels, with no extra GEMM. Impact (7B weight-only): W4 5.59 → 5.56, W3 6.60 → 5.86, W2 1.8e5 → 8.46. For activations it is optional: with 128-group scales uniform INT4 is within 0.1 PPL at W4 (7B to 13B), while at 3 bits it still matters (8B W3A3KV3: 11.67 vs. 8.99).
Q4. The appendix already includes Qwen2.5-7B/14B and Qwen3-8B. New MoE results (WikiText-2 PPL; weight-only group 128, GPTQ / ours; W4A4 with FP16 KV, RTN / ours):
The ordering holds on C4 and zero-shot accuracy (Mixtral W3: 63.9 vs. 69.6; FP16 70.6). Quantizing only the routed experts to W2 (attention and shared experts kept at higher precision) gives 7.90 / 7.30 / 4.97 vs. GPTQ's 16.0 / 24.4 / 40.4.