- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AdaptQuant, which selects MXFP4/6/8 per 16x32 tile using a statistic based on amax/RMS. The thresholds are derived once on a reference model and reused across models, avoiding per-model calibration while targeting Blackwell's block-scaled MMA granularity. Weight format selection is performed offline once per model, while activation format selection is performed dynamically during inference. Experiments on 13 LLMs report comparable accuracy to MicroMix while using slightly fewer effective weight bits, approximately 2.9x lower model quantization time, and similar overall inference throughput. The reported geometric mean prefill throughput is 1.8x over BF16.
Critical strengths
- The 16x32 tile granularity matches Blackwell's block-scaled MMA operand shape, making the format-selection granularity well motivated.
- The method uses a simple per-tile statistic and reuses thresholds across models rather than recalibrating for each model.
- The paper details the hardware implementation including the storage, and conversion.
Critical weaknesses
- Activation format selection requires computing both amax and RMS for every tile on every forward pass, but the paper does not isolate the runtime cost of this operation from quantization and GEMM.
- AdaptQuant has limited throughput advantage over MicroMix. AdaptQuant's largest advantage is reducing one-time quantization time from 211.6s to 72.3s, while steady-state throughput is similar to MicroMix and sometimes lower. Given that quantization is typically a one-time offline cost, reducing quantization time from a few hundred seconds to roughly one minute is not the main bottleneck.
Questions for authors
- For Table 1, what tensors/models are used to compute the reported SQNR values? Are they weights or activations, and are the values averaged across layers or models? *Could you provide a runtime breakdown of activation statistic computation, activation quantization, weight conversion, and GEMM?
- Is my understanding correct that AdaptQuant stores weights in mixed FP4/FP6 formats but widens them to a uniform format (upcast MXFP4 to MXFP6) before the actual MMA? If so, could this distinction be made clearer? If the advantage is storage, a bigger LLM model, like MoE, would be a better use case.
- Could you show accuracy versus effective-bit curves for MXFP4, MXFP6, AdaptQuant, MicroMix, and other baselines to demonstrate whether AdaptQuant lies on the Pareto frontier?