- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AlignQuant is a post-training W4/W8 mixed-precision quantization framework using hardware-aligned weight tiles for sensitivity-based precision allocation. It combines gradient-weighted projection-output perturbations to select tile precision. The authors then report 2.50x generation speedup over BF16.
Critical strengths
- The paper presents a complete implementation, including calibration, packing, specialized GPU kernels, and runtime optimizations. The evaluation covers multiple models, GPUs, workloads, and benchmarks.
- Using a common tile structure for precision allocation, storage, and execution is, of course, a reasonable design.
Critical weaknesses
- Block-wise mixed precision and sensitivity-driven bit allocation have been extensively studied. For example, FGMP, SFMP, and ScaleBITS… (The authors also cited these papers). The distinction of AlignQuant from these methods is not clear. Also, the proposed sensitivity score appears to be a heuristic, yet the paper does not compare it against prefill-only, decode-only, or alternative score combinations. Therefore, it’s not clear what is the technical contribution in the current version.
- AlignQuant uses approximately 5.61 bits per weight (Table 16, W4 60%), compared with roughly 4-bit GPTQ/AWQ baselines. Then it seems that the comparisons are made without matched budgets. I’m also curious about why GPTQ/AWQ exhibit unusually large HellaSwag accuracy drops (e.g., 71.52% to 52.62%/50.41% on Llama-3.2-3B-Instruct) despite relatively small perplexity changes. Also, the large speedups appear dependent on the specialized runtime. Table 16 shows that, under the same runtime, W8-only and W4-only achieve 141.72 and 138.96 tok/s, compared with 132.28 tok/s for the proposed mixed configuration. Moreover, the W4-only baseline achieves only 47.33 tok/s in Table 1 (Is that because the authors used a different backend?). It remains unclear how much of the advantage comes from the proposed quantization scheme versus differences in kernel implementations and runtime optimization.
Questions for authors
- What are the concrete technical differences from FGMP, SFMP, and ScaleBITS? Can the authors ablate prefill-only, decode-only, and joint sensitivity scoring?
- Can the authors provide matched-bit-budget quality comparisons? Please also provide interpretation on the accuracy drops.
- Can the authors compare against optimized quantization backends under comparable runtime conditions and report quality-memory-throughput tradeoffs? How much of the speedup is specifically attributable to mixed-precision allocation?