- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes AMS-Quant, a weight-only floating-point quantization scheme in which groups of k weights share their least significant mantissa bit. Starting from channel-wise RTN quantization, the method chooses the shared bit by minimizing group-wise weight reconstruction error, yielding nominal rates such as 5.33 and 4.25 bits per weight. The authors also implement packed CUDA linear kernels. Experiments on five Llama/Qwen models report near-FP16 average accuracy at 5.33 bits and improved accuracy over RTN FP4 at 4.25 bits. Selected memory-bound linear layers obtain up to 2.77× and 3.30× speedups over FP16.
Critical strengths
- Sharing one mantissa bit gives a concrete mechanism for intermediate storage rates with a lightweight offline search and straightforward bit reconstruction.
- The packed representation and CUDA results provide evidence that fractional nominal rates can translate into useful linear-layer latency reductions, beyond merely reducing an abstract bit count
- Table 2 reports instruction following, mathematical reasoning, and knowledge accuracy separately. The 5.33-bit results are consistently close to the FP16 averages across the tested models.
Critical weaknesses
- The accuracy baselines are limited to RTN quantization. Comparisons with other floating-point PTQ methods, such as AFPQ and LLM-FP4, are needed.
- Table 2 covers too few downstream tasks. More challenging benchmarks, such as MATH, GPQA, and HumanEval, should be added.
- The kernel evaluation is limited and lacks end-to-end speed comparisons. The gains in Table 3 seem mainly due to reduced memory traffic at lower bit widths. They do not isolate the benefits of the kernel design.
Questions for authors
- Could you clarify the mathematical rationale and steps of Adaptive Searching?
- Could you compare FP4.25 with MXFP4 at a matched storage budget, including scale overhead?