- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper improves MXFP4 quantization through inter-block permutation and micro-block Givens rotation. It groups outliers with low-importance channels and adaptively selects how many channels receive their energy. Evaluation includes reconstruction error, perplexity, and zero-shot accuracy on LLaMA2/3 and Qwen3.
Critical strengths
The observation that smaller outliers do not necessarily yield lower MXFP4 quantization error is useful. The proposed adaptive rotation substantially reduces reconstruction error for massive outliers.
Critical weaknesses
-
Closely related methods are missing from the discussion and experiments. DuQuant++ uses outlier-aware block rotations for MXFP4; PeRQ permutes channels before block rotation to balance activation mass across blocks; and TORQ uses Givens rotations to balance block energy and improve MXFP4 codebook utilization. These methods directly overlap with the proposed motivation and components. Without comparisons, the incremental contribution remains unclear, particularly given the modest improvement over MR-GPTQ.
-
No latency, throughput, or calibration-time results are provided. The runtime cost of permutation and adaptive Givens rotations relative to block Hadamard remains unclear, preventing assessment of the practical accuracy–speed trade-off.
Questions for authors
- What are the key methodological differences from DuQuant++, PeRQ, and TORQ?
- What advantages does the proposed method offer over these methods under matched evaluation settings?
- Could you compare inference speed and latency against the relevant baselines? If time is limited, a comparison with DuQuant++ alone would be sufficient.
I would raise my recommendation to acceptance if the authors clearly establish these distinctions and advantages and provide convincing speed and latency results on gpus.
2 comments
Response to Reviewer TvXW (Part 2/2)
Q3: "Could you compare inference speed and latency against the relevant baselines? If time is limited, a comparison with DuQuant++ alone would be sufficient."
Thank you for this suggestion. To establish an appropriate evaluation protocol, we carefully reviewed the latency measurements and implementation availability of the relevant methods:
-
BRQ: We could not locate an official public implementation. Its latency evaluation uses PyTorch and
microsoft/microxcalingon NVIDIA A800 GPUs, which do not natively support FP4 inference. The reported overhead is relative to an MXFP4 baseline, without a full-precision baseline in the latency table. (Paper) -
DuQuant++: The quantization and evaluation code is public, but we did not find an optimized native MXFP4 inference kernel in the released implementation. The paper reports quantization accuracy, without measured inference latency or speedup results. (Code; Paper)
-
PeRQ: The implementation is available in Brevitas. Its permutations are calibrated offline and merged into model weights to avoid additional permutation overhead. However, the paper does not report a measured end-to-end inference latency or speedup benchmark for PeRQ itself. (Code; Paper)
-
TORQ: The paper reports decoding throughput of 220 tokens/s versus 63 tokens/s for BF16 on Qwen3-30B-A3B with a 2k context length, corresponding to approximately 3.5× speedup. However, no public inference implementation is linked in the paper, limiting our ability to reproduce this result under matched settings. (Paper)
-
MR-GPTQ: Public code and optimized QuTLASS kernels are available. QuTLASS supports runtime-supplied rotation matrices, but the MXFP4 fused path we inspected applies a shared matrix across blocks. Supporting our block-dependent transforms and duquant++ requires adapting this path, which we have not completed within the rebuttal time constraints. (Code; Paper)
Following BRQ's use of MXFP4 emulation, we conducted a synthetic operator microbenchmark on an RTX 4090 using BF16 inputs and the same microsoft/microxcaling CUDA quantizer. We compare ours: inter-block permutation + per-block matrix multiplication + MXFP4 quantization against DuQuant++-style: per-block matrix multiplication + MXFP4 quantization. All blocks are represented as 32×32 matrices; the transform parameters are fixed synthetic representatives, and their construction is excluded from timing.
We test batch sizes 1/8, sequence lengths 512/1024/2048/4096, and hidden dimensions 4096/11008, flattening the inputs to [batch × sequence, hidden]. We report median latency over 50 runs after 10 warmup runs, with CUDA synchronization. For batch size 1 and sequence length 2048:
Table C. Operator latency on RTX 4090 at batch size 1 and sequence length 2048.
| Hidden dimension | DuQuant++-style (ms) | Ours (ms) | Additional latency (ms) |
|---|---|---|---|
| 4096 | 0.124 | 0.152 | 0.028 |
| 11008 | 0.371 | 0.481 | 0.110 |
Table C reports the latency of our method and duquant++. The additional latency is 0.028 ms and 0.110 ms for hidden dimensions 4096 and 11008, respectively. Both methods use the same per-block 32×32 matrix multiplication routine, so our micro-block design does not increase the matrix-multiplication arithmetic in this implementation. The additional permutation primarily involves data movement. We expect an optimized fused kernel to keep the additional runtime overhead modest by incorporating permutation into input loading and combining the block transform with quantization.
We hope these clarifications address your concerns. We believe that our work provides new insights into MXFP4 quantization and can substantially advance the field.
Response to Reviewer TvXW (Part 1/2)
We thank the reviewer for the thoughtful feedback and for recognizing our work. Below, we address the reviewer’s concerns point by point.
The core of our method is the adaptive micro-block transform, motivated by an often-overlooked property of MXFP4: reducing an outlier's magnitude does not necessarily reduce its quantization error. Due to the power-of-two E8M0 shared scale and the discrete E2M1 codebook, spreading an outlier across all 32 channels may therefore produce smaller magnitudes but higher error than spreading it over a smaller, appropriately chosen micro-block. Permutation supports micro-block transform by grouping outliers with low-importance channels.
The table below summarizes the inter-block processing and intra-block transformation in related methods.
Table A. Methodological comparison between related methods.
Thank you for this question. Our key advantage is an expanded transform design space: we treat the transform size itself as an optimization variable. Fixed-size designs such as DuQuant++ tie the transform size to the MXFP4 shared-scale group size and optimize the matrix within that prescribed size. Our MXFP4 quantization-error analysis reveals that a larger transform size does not necessarily yield lower quantization error. This motivates us to treat the transform size and the corresponding matrix as joint optimization variables, with reconstruction error as the objective. This extends the search from matrices at a single fixed size to transforms across multiple sizes.
To provide a controlled comparison, we incorporated our micro-block design into the standard DuQuant++ pipeline. We compare three variants, keeping the number of search steps identical and the MXFP4 quantization group size fixed at 32:
We evaluate these variants on LLaMA-3-8B-Instruct across six zero-shot tasks, with results shown below. The micro-block variant achieves the highest average accuracy of 65.11%, improving over the original DuQuant++ (64.10%) by 1.01 percentage points and over the fixed-size MSE variant (64.77%) by 0.34 percentage points. The latter comparison isolates the benefit of the micro-block design under the same MSE objective, supporting its value beyond changing the search objective alone.
Table B. Controlled comparison within DuQuant++ on LLaMA-3-8B-Instruct: zero-shot accuracy (%).