acceptodds
Under review as a conference paper at ICLR 2027

Improve MXFP4 Quantization via Inter-block Permutation and Micro-block Givens Rotation

Abstract

Large language models (LLMs) have achieved remarkable performance across diverse tasks, but their growing computational and memory costs pose substantial challenges for efficient deployment. Post-training quantization (PTQ) has emerged as a practical solution, while recent hardware support for microscaling formats such as MXFP4 further enables efficient low-bit inference. However, MXFP4 remains highly sensitive to massive activation outliers due to its power-of-two E8M0 shared scales and low-precision E2M1 values. Existing block-wise Hadamard transformations mitigate outliers by uniformly spreading their energy across an entire quantization block, but such magnitude reduction is not necessarily optimal for MXFP4. We observe that MXFP4 contains a set of favorable, exactly or nearly representable magnitudes, termed **lossless quantization points**, and quantization error is strongly determined by the distance of transformed values to these points. Based on this observation, we propose a novel transform framework for MXFP4. First, we propose *Inter-block Permutation* to group massive outliers with low-importance channels, reducing their interference with important features under block-wise shared scaling. Second, we propose *Micro-block Givens Rotation* to adaptively control the number of participating channels and redistribute outlier energy toward favorable quantization points, instead of uniformly dispersing it across the entire block. Experiments on the LLaMA2/3 and Qwen3 model families demonstrate that our method better adapts to the quantization characteristics of MXFP4 than the Hadamard transform and achieves superior quantization performance across multiple LLM benchmarks, providing new insights into the design of quantization methods for next-generation microscaling low-precision formats.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Microscaling FP4, Transform, LLM Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_TvXWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper improves MXFP4 quantization through inter-block permutation and micro-block Givens rotation. It groups outliers with low-importance channels and adaptively selects how many channels receive their energy. Evaluation includes reconstruction error, perplexity, and zero-shot accuracy on LLaMA2/3 and Qwen3.

Critical strengths

The observation that smaller outliers do not necessarily yield lower MXFP4 quantization error is useful. The proposed adaptive rotation substantially reduces reconstruction error for massive outliers.

Critical weaknesses
  1. Closely related methods are missing from the discussion and experiments. DuQuant++ uses outlier-aware block rotations for MXFP4; PeRQ permutes channels before block rotation to balance activation mass across blocks; and TORQ uses Givens rotations to balance block energy and improve MXFP4 codebook utilization. These methods directly overlap with the proposed motivation and components. Without comparisons, the incremental contribution remains unclear, particularly given the modest improvement over MR-GPTQ.

  2. No latency, throughput, or calibration-time results are provided. The runtime cost of permutation and adaptive Givens rotations relative to block Hadamard remains unclear, preventing assessment of the practical accuracy–speed trade-off.

Questions for authors
  1. What are the key methodological differences from DuQuant++, PeRQ, and TORQ?
  2. What advantages does the proposed method offer over these methods under matched evaluation settings?
  3. Could you compare inference speed and latency against the relevant baselines? If time is limited, a comparison with DuQuant++ alone would be sufficient.

I would raise my recommendation to acceptance if the authors clearly establish these distinctions and advantages and provide convincing speed and latency results on gpus.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TvXW (Part 1/2)

We thank the reviewer for the thoughtful feedback and for recognizing our work. Below, we address the reviewer’s concerns point by point.

Q1: *"What are the key methodological differences from DuQuant++, PeRQ, and TORQ?"

The core of our method is the adaptive micro-block transform, motivated by an often-overlooked property of MXFP4: reducing an outlier's magnitude does not necessarily reduce its quantization error. Due to the power-of-two E8M0 shared scale and the discrete E2M1 codebook, spreading an outlier across all 32 channels may therefore produce smaller magnitudes but higher error than spreading it over a smaller, appropriately chosen micro-block. Permutation supports micro-block transform by grouping outliers with low-importance channels.

The table below summarizes the inter-block processing and intra-block transformation in related methods.

Table A. Methodological comparison between related methods.

MethodInter-block processingIntra-block processing
DuQuant++None.(1) Greedily search for an outlier-aware orthogonal transform with transform size 32; (2) Reduce peak activation magnitude.
PeRQ(1) Channel permutation; (2) Reduce the expected maximum blockwise ℓ1\ell_1 norm, balancing activation mass across blocks.(1) Hadamard transform with transform size 32; (2) Suppress intra-group outliers.
TORQ(1) Inter-block Givens rotations; (2) Equalize block variances so that a few high-energy blocks do not dominate quantization error.(1) Alternate scale updates and intra-block Givens-angle searches; (2) Minimize codeword occupancy imbalance and improve MXFP4 codebook utilization.
Ours(1) Channel permutation; (2) Reduce the impact of outliers on other elements within the same group.(1) Equal-magnitude transform within micro-blocks of adaptively selected size n≤B=32n\leq B=32, uniformly redistributing the largest-magnitude component over the selected micro-block; Givens rotations are one possible implementation of our adaptive micro-block transform ; (2) Target favorable quantization points and reduce MXFP4 reconstruction error.

Q2: "What advantages does the proposed method offer over these methods under matched evaluation settings?"

Thank you for this question. Our key advantage is an expanded transform design space: we treat the transform size itself as an optimization variable. Fixed-size designs such as DuQuant++ tie the transform size to the MXFP4 shared-scale group size and optimize the matrix within that prescribed size. Our MXFP4 quantization-error analysis reveals that a larger transform size does not necessarily yield lower quantization error. This motivates us to treat the transform size and the corresponding matrix as joint optimization variables, with reconstruction error as the objective. This extends the search from matrices at a single fixed size to transforms across multiple sizes.

To provide a controlled comparison, we incorporated our micro-block design into the standard DuQuant++ pipeline. We compare three variants, keeping the number of search steps identical and the MXFP4 quantization group size fixed at 32:

  1. DuQuant++: Search for orthogonal transforms with a fixed size of 32, using peak activation magnitude as the search objective;
  2. DuQuant++*: Keep the transform size fixed at 32, but replace the peak-based objective with activation reconstruction MSE after quantization;
  3. DuQuant++* + micro-block (ours): Retain the MSE objective and introduce our micro-block design, allowing the transform size n≤32n\leq 32 to be selected adaptively instead of fixing it at 32.

We evaluate these variants on LLaMA-3-8B-Instruct across six zero-shot tasks, with results shown below. The micro-block variant achieves the highest average accuracy of 65.11%, improving over the original DuQuant++ (64.10%) by 1.01 percentage points and over the fixed-size MSE variant (64.77%) by 0.34 percentage points. The latter comparison isolates the benefit of the micro-block design under the same MSE objective, supporting its value beyond changing the search objective alone.

Table B. Controlled comparison within DuQuant++ on LLaMA-3-8B-Instruct: zero-shot accuracy (%).

MethodARC-EasyARC-ChallengeWinoGrandeHellaSwagOpenBookQAPIQAAvg. ↑
DuQuant++74.8750.9468.8273.3540.6076.0164.10
DuQuant++*74.9250.6869.3873.3543.0077.2664.77
DuQuant++* + micro-block75.3452.9069.6973.8141.4077.5365.11
AuthorsOpenReview ↗

Response to Reviewer TvXW (Part 2/2)

Q3: "Could you compare inference speed and latency against the relevant baselines? If time is limited, a comparison with DuQuant++ alone would be sufficient."

Thank you for this suggestion. To establish an appropriate evaluation protocol, we carefully reviewed the latency measurements and implementation availability of the relevant methods:

  1. BRQ: We could not locate an official public implementation. Its latency evaluation uses PyTorch and microsoft/microxcaling on NVIDIA A800 GPUs, which do not natively support FP4 inference. The reported overhead is relative to an MXFP4 baseline, without a full-precision baseline in the latency table. (Paper)

  2. DuQuant++: The quantization and evaluation code is public, but we did not find an optimized native MXFP4 inference kernel in the released implementation. The paper reports quantization accuracy, without measured inference latency or speedup results. (Code; Paper)

  3. PeRQ: The implementation is available in Brevitas. Its permutations are calibrated offline and merged into model weights to avoid additional permutation overhead. However, the paper does not report a measured end-to-end inference latency or speedup benchmark for PeRQ itself. (Code; Paper)

  4. TORQ: The paper reports decoding throughput of 220 tokens/s versus 63 tokens/s for BF16 on Qwen3-30B-A3B with a 2k context length, corresponding to approximately 3.5× speedup. However, no public inference implementation is linked in the paper, limiting our ability to reproduce this result under matched settings. (Paper)

  5. MR-GPTQ: Public code and optimized QuTLASS kernels are available. QuTLASS supports runtime-supplied rotation matrices, but the MXFP4 fused path we inspected applies a shared matrix across blocks. Supporting our block-dependent transforms and duquant++ requires adapting this path, which we have not completed within the rebuttal time constraints. (Code; Paper)

Following BRQ's use of MXFP4 emulation, we conducted a synthetic operator microbenchmark on an RTX 4090 using BF16 inputs and the same microsoft/microxcaling CUDA quantizer. We compare ours: inter-block permutation + per-block matrix multiplication + MXFP4 quantization against DuQuant++-style: per-block matrix multiplication + MXFP4 quantization. All blocks are represented as 32×32 matrices; the transform parameters are fixed synthetic representatives, and their construction is excluded from timing.

We test batch sizes 1/8, sequence lengths 512/1024/2048/4096, and hidden dimensions 4096/11008, flattening the inputs to [batch × sequence, hidden]. We report median latency over 50 runs after 10 warmup runs, with CUDA synchronization. For batch size 1 and sequence length 2048:

Table C. Operator latency on RTX 4090 at batch size 1 and sequence length 2048.

Hidden dimensionDuQuant++-style (ms)Ours (ms)Additional latency (ms)
40960.1240.1520.028
110080.3710.4810.110

Table C reports the latency of our method and duquant++. The additional latency is 0.028 ms and 0.110 ms for hidden dimensions 4096 and 11008, respectively. Both methods use the same per-block 32×32 matrix multiplication routine, so our micro-block design does not increase the matrix-multiplication arithmetic in this implementation. The additional permutation primarily involves data movement. We expect an optimized fused kernel to keep the additional runtime overhead modest by incorporating permutation into input loading and combining the block transform with quantization.

We hope these clarifications address your concerns. We believe that our work provides new insights into MXFP4 quantization and can substantially advance the field.

Reviewer_rWG8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an improved transformation method for LLM post-training quantization (PTQ) in MXFP4, a microscaling quantization format. Based on the observation that MXFP4 contains a set of favorable magnitudes, termed lossless quantization points, the paper introduces a two stage workflow to reduce quantization error. (1) Inter-block permutation groups massive outliers with low-importance channels. (2) Micro-block givens rotation adaptively controls the number of participating channels and redistributes outlier energy toward favorable quantization points. Results on LLaMA2/3 and Qwen3 models show improvements over Hadamard transform under some quantization methods and benchmarks.

Critical strengths

The paper is easy to follow, with an interesting analysis of MXFP4 quantization error vs. block-wise maximum value. The method is clear.

Critical weaknesses
  • The paper targets PTQ in MXFP4 only, and the method seems specific for MXFP4 format. It does not discuss or conduct experiments to show how similar ideas could generalize to other quantization formats (e.g., NVFP4).
  • The experimental results does not support the claims. (1) “Ours” outperforms “Hadamard” and “w/o”, only in some cases, and the trend seems random. The improvements are marginal even in improved cases. (2) The ablation results (Table 3) contradict with the design choice to combine two components.
Questions for authors
  • Could you provide more experimental results? (1) Table 1 only contains LLaMA results and Table 2 only contains Qwen results. (2) Only Qwen results compare with different PTQ methods. (3) No inference efficiency tests, although the method modifies inference computation.
  • Could you justify your design given contradictory ablation results in Table 3?
3 comments
AuthorsOpenReview ↗

Response to Reviewer rWG8 (Part 1/3)

We thank the reviewer for the thoughtful feedback and for recognizing our work. Below, we address the reviewer’s concerns point by point.

Q1: "Could you provide more experimental results? (1) Table 1 only contains LLAMA results and Table 2 only contains Qwen results. (2) Only Qwen results compare with different PTQ methods. (3) No inference efficiency tests, although the method modifies inference computation."

Thank you for this suggestion. We have added experimental results to broaden the evaluation across model families and PTQ methods.

(1) Evaluation across model families. We have added comparisons of Identity, Hadamard, and our transform on Qwen3-4B, Qwen3-8B, and Qwen3-14B with both RTN and GPTQ. These results, shown in Table A, complement the LLAMA results in the original Table 1.

Table A. WikiText PPL (↓) and zero-shot accuracy (↑) for various Qwen3 models under W4A4 MXFP4 quantization.

ModelQuantizerTransformWikiText2 PPL ↓Zero-shot Average ↑
Qwen3-4BRTNIdentity17.5057.96
Hadamard17.5057.61
Ours15.8858.93
GPTQIdentity16.1358.54
Hadamard16.1358.94
Ours16.1359.58
Qwen3-8BRTNIdentity11.0762.90
Hadamard11.0361.70
Ours11.8163.20
GPTQIdentity10.9462.37
Hadamard10.7563.95
Ours10.7564.85
Qwen3-14BRTNIdentity9.7366.96
Hadamard9.7167.72
Ours9.8168.68
GPTQIdentity9.4067.23
Hadamard9.1168.53
Ours9.6369.29

(2) Comparisons with PTQ methods on LLAMA. We have also added comparisons with PTQ methods on the LLAMA family, as reported in Table B, complementing the Qwen comparisons in the original Table 2. Ours achieves the highest average zero-shot accuracy on LLAMA2-7B (63.40%) and LLAMA3-8B (66.04%), while also matching the best PPL on LLAMA2-7B (5.84). On both evaluated LLAMA models, Ours improves both PPL and average accuracy over RTN, QuaRot, SpinQuant, SmoothQuant, and GPTQ, supporting its effectiveness across model families.

Table B. Comparison with PTQ baselines on LLAMA models under MXFP4 W4A4 quantization.

ModelMethodWikiText2 PPL ↓Zero-shot Acc. ↑
LLAMA2-7BRTN6.2061.22
QuaRot6.1961.36
SpinQuant6.0659.00
SmoothQuant6.1562.37
GPTQ6.1361.42
BRQ5.8462.83
MR-GPTQ5.8462.77
Ours5.8463.40
LLAMA3-8BRTN7.8463.68
QuaRot8.4460.63
SpinQuant7.6061.99
SmoothQuant7.8563.92
GPTQ7.5963.03
BRQ7.1065.96
MR-GPTQ7.1265.28
Ours7.1666.04

(3) Runtime overhead relative to Hadamard. Both block-wise Hadamard and our transform can be implemented as per-block 32×32 matrix multiplication. Under this implementation, the two transforms have the same matrix-multiplication arithmetic cost; our only additional online operation is the inter-block permutation. This permutation is implemented as channel indexing and data movement, with linear complexity in the number of activation elements. All transform matrices and permutation indices are constructed offline.

Following the MXFP4 emulation approach used in BRQ's latency evaluation, we benchmark both pipelines on an RTX 4090 using BF16 inputs and the microsoft/microxcaling CUDA quantizer. Both use the same per-block 32×32 batched matrix-multiplication routine, followed by the same MXFP4 quantization routine. Hadamard uses the same normalized Hadamard matrix in every block; ours uses micro-block Givens rotations for the selected blocks and Hadamard for the remaining blocks, with online permutation included. The inputs and our transform parameters are synthetic, and offline construction is excluded from timing. We report median latency over 50 runs after 10 warmup runs, with CUDA synchronization and randomized method order. This comparison uses matrix multiplication for both transforms.

Table C. Operator latency on RTX 4090 at batch size 1 and sequence length 2048. All pipelines include MXFP4 quantization; the “Ours w/o permutation” column isolates our transform without online permutation.

Hidden dimensionHadamard (ms)Ours w/o permutation (ms)Ours with permutation (ms)
40960.1210.1210.152
110080.3720.3700.486

Under the same block-matrix implementation, our pipeline achieves nearly identical latency to Hadamard when permutation is excluded. The additional runtime overhead arises primarily from data movement during permutation.

AuthorsOpenReview ↗

Response to Reviewer rWG8 (Part 2/3)

Q2: "Could you justify your design given contradictory ablation results in Table 3?"

Thank you for raising this point. Our design is motivated by reducing layer-wise reconstruction error. The ablations in Appendix B.1, Table 5 show that combining Inter-block Permutation and Micro-block Givens Rotation achieves the lowest or tied-lowest reconstruction error across all evaluated activation types. Table 6 further illustrates their complementary effects: for QKV activations at the default outlier ratio of 10/4096, the error decreases from 5.53e-4 without either component to 4.74e-4 with permutation alone or 1.04e-4 with micro-block rotation alone, and further to 8.25e-5 with both. Thus, adding either component to the other further reduces the reconstruction error.

We acknowledge that the combination does not achieve the highest average accuracy in the original Table 3. Layer-wise reconstruction error is a local objective, and reducing it does not guarantee monotonic improvements in downstream accuracy. We have also added the downstream ablation results in Table D. The combination achieves the lowest C4 PPL of 16.38, compared with 17.50 for micro-block rotation alone and 17.13 for permutation alone. Its average zero-shot accuracy is higher than rotation alone but lower than permutation alone. These results provide downstream support for the combined design, and its benefits depend on the evaluation task.

Table D. Additional downstream ablation results on Qwen3-8B.

TransformC4 PPL ↓
RTN17.61
Micro-block Givens only17.50
Inter-block Permutation only17.13
Ours (both)16.38
AuthorsOpenReview ↗

Response to Reviewer rWG8 (Part 3/3)

W1: "The paper targets PTQ in MXFP4 only, and the method seems specific for MXFP4 format. It does not discuss or conduct experiments to show how similar ideas could generalize to other quantization formats (e.g., NVFP4)."

Thank you for this suggestion. Our method is primarily designed for MXFP4. Its E8M0 shared scales are restricted to integer powers of two, which gives rise to the lossless quantization points discussed in our paper. Micro-block Givens Rotation exploits this structure by adaptively redistributing outlier energy toward these favorable points. Inter-block Permutation complements this rotation by distributing outliers across blocks and pairing them with low-importance channels, reducing their adverse impact on other elements within the same quantization group.

NVFP4 instead uses E4M3 block-level scales, which provide finer scale resolution and are not restricted to powers of two. The specific lossless-point structure exploited by our MXFP4 micro-block rotation therefore does not directly carry over to NVFP4. Accordingly, we do not apply this rotation to NVFP4. However, Inter-block Permutation addresses the influence of outliers under shared block scaling, without relying on power-of-two scales, and can thus be applied independently to NVFP4.

To evaluate this transfer, Table E reports W4A4 NVFP4 results on LLAMA2-7B and Qwen3-8B using both RTN and GPTQ. Inter-block Permutation achieves the highest average zero-shot accuracy in all four settings, improving over Identity by 0.17–0.65 percentage points, and yields lower PPL than Hadamard throughout. These results support the applicability of our permutation component beyond MXFP4.

Table E. PPL (↓) and zero-shot accuracy (↑) for LLAMA2-7B and Qwen3-8B under W4A4 NVFP4 quantization.

ModelQuantizerTransformWikiText PPL ↓Zero-shot Average ↑
LLAMA2-7BRTNIdentity5.7763.74
Hadamard5.9262.87
Inter-block Permutation (ours)5.7863.91
GPTQIdentity5.7263.38
Hadamard5.7363.45
Inter-block Permutation (ours)5.7263.76
Qwen3-8BRTNIdentity9.9865.43
Hadamard10.7564.19
Inter-block Permutation (ours)10.1165.60
GPTQIdentity10.0065.73
Hadamard10.2064.96
Inter-block Permutation (ours)9.9566.38

W2: "The experimental results does not support the claims. (1) “Ours” outperforms “Hadamard” and “w/o”, only in some cases, and the trend seems random. The improvements are marginal even in improved cases. (2) The ablation results (Table 3) contradict with the design choice to combine two components.."

Thank you for raising these concerns.

(1) The additional Qwen3 results in Table A show a more consistent accuracy advantage: our method achieves the highest average zero-shot accuracy among Identity, Hadamard, and Ours in all six model and quantizer configurations.

(2) As detailed in our response to Q2, the two components progressively reduce layer-wise reconstruction error, although this local objective does not guarantee monotonic downstream accuracy gains. The additional downstream ablation also shows that combining both components achieves the lowest C4 PPL.


We hope these clarifications address your concerns. We believe that our work provides new insights into MXFP4 quantization and can substantially advance the field.

Reviewer_zWHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies W4A4 post-training quantization for LLMs under the MXFP4 format. The main idea is that reducing the outlier magnitude is not always enough, because the MXFP4 quantization error can be non-monotonic due to the discrete E8M0 power-of-two scale and the E2M1 codebook. The paper proposes two components. Inter-block Permutation groups outlier channels with low-importance channels, while Micro-block Givens Rotation adaptively selects how many channels are used to spread the outlier energy. Experiments are conducted on LLaMA2, LLaMA3, and Qwen3 models. The method can significantly reduce the reconstruction error, but the downstream accuracy improvements are not always consistent.

Critical strengths
  • The paper provides a clear motivation for MXFP4 quantization. It shows that simply reducing the outlier magnitude is not always optimal, because the MXFP4 quantization error can be non-monotonic.
  • The proposed Micro-block Givens Rotation is simple and well motivated. It adaptively selects the number of channels used to spread the outlier energy, and it significantly reduces the reconstruction error in some layers.
  • The paper includes experiments on several LLM families and different quantization settings. The proposed method achieves competitive results compared with Hadamard-based methods and other PTQ baselines.
Critical weaknesses
  • The downstream improvements are not consistent. In several settings, the proposed method performs similarly to or even worse than the Hadamard baseline, although the reconstruction error is much lower.

  • The two proposed components do not always work better when combined. In Table 3, combining Inter-block Permutation and Micro-block Givens Rotation gives lower average downstream accuracy than using either component alone.

  • The paper does not provide enough efficiency analysis. The method searches over different micro-block sizes during calibration, but the paper does not report the calibration runtime or end-to-end inference efficiency, such as latency or throughput.

Questions for authors
  1. Why does the large reduction in reconstruction error not always lead to better downstream accuracy? It would be helpful to provide more analysis on this gap.

  2. Why does combining Inter-block Permutation and Micro-block Givens Rotation give lower average accuracy than using either component alone in Table 3? Could the authors report more results for the Givens-only setting?

  3. What is the additional calibration cost of the proposed method compared with block-wise Hadamard? It would also be helpful to report end-to-end inference latency or throughput.

  4. The paper motivates the method using “lossless quantization points,” but the final algorithm selects the micro-block size by directly minimizing reconstruction error. Could the authors clarify the role of the lossless points in the actual optimization?

7 comments

Dear Authors,

As the discussion period is approaching its end, I noticed that I have not yet received a rebuttal response to my review.

I would be happy to engage in further discussion and welcome any clarifications or additional information you may wish to provide regarding my concerns.

I look forward to hearing from you.

Best regards,

Reviewer zWHP

AuthorsOpenReview ↗

Response to Reviewer zWHP (Part 1/2)

We thank the reviewer for the thoughtful feedback and for recognizing our work. Below, we address the reviewer’s concerns point by point.

Q1: "Why does the large reduction in reconstruction error not always lead to better downstream accuracy? It would be helpful to provide more analysis on this gap."

Thank you for raising this point. Reconstruction error measures local numerical fidelity, while downstream accuracy depends on the final predictions of the entire model. Errors in different channels and layers can have different effects on the output, and their impact may be amplified or offset as they propagate through the network. A smaller reconstruction error therefore does not necessarily imply a smaller change in the scores that determine the predicted answer.

Moreover, zero-shot accuracy depends on answer rankings. Reduced reconstruction error may leave many predictions unchanged, while changes near decision boundaries can help some examples and hurt others. Differences between calibration data and downstream tasks can further weaken this relationship. These factors can explain why a large relative reduction in reconstruction error yields only modest or inconsistent accuracy gains.

We therefore use reconstruction error as a calibration objective and assess its downstream benefits separately through perplexity and task accuracy. We will clarify this distinction and discuss the limitations of the local objective in the revised manuscript.

Q2: "Why does combining Inter-block Permutation and Micro-block Givens Rotation give lower average accuracy than using either component alone in Table 3? Could the authors report more results for the Givens-only setting?"

Thank you for this suggestion. Our calibration minimizes layer-wise reconstruction error, which does not guarantee monotonic improvements in downstream accuracy. The ablations in Appendix B.1 show that permutation and micro-block rotation complement each other in reducing reconstruction error, while Table 3 highlights that this benefit does not always translate into better task performance. We will clarify this limitation in the revised manuscript.

We have added Givens-only results with both RTN and GPTQ in Table B. On LLaMA2-7B and Qwen3-8B, Givens only improves average zero-shot accuracy over both w/o and Hadamard in all four settings, with gains of 0.30–1.68 percentage points over Hadamard. Its PPL improvements are less consistent, particularly for Qwen3-8B with RTN. These results support the usefulness of the rotation component while showing that its downstream benefits depend on the model, quantizer, and evaluation metric.

Table B. WikiText2 perplexity (↓) and average zero-shot accuracy (↑) under W4A4 MXFP4 quantization.

ModelMethodTransformWikiText2 PPL ↓Zero-shot mean ↑
LLaMA2-7BRTNw/o6.2061.22
Hadamard6.1761.05
Givens only6.1362.24
GPTQw/o6.1361.42
Hadamard5.9362.92
Givens only5.9363.22
Qwen3-8BRTNw/o11.0762.90
Hadamard11.0361.70
Givens only11.2963.38
GPTQw/o10.9462.37
Hadamard10.9463.95
Givens only10.9464.49
AuthorsOpenReview ↗

Response to Reviewer zWHP (Part 2/2)

Q3: "What is the additional calibration cost of the proposed method compared with block-wise Hadamard? It would also be helpful to report end-to-end inference latency or throughput."

Thank you for this suggestion. We address the one-time offline calibration cost and the inference-time overhead separately below.

(1) Calibration time. Block-wise Hadamard uses a fixed transform and requires no data-dependent search for its transform parameters. Our method introduces an offline calibration stage to collect activation statistics. Once calibrated, the permutation indices and block rotation matrices are fixed and reused during inference. Table C reports the offline calibration time for both quantizers.

Table C. Offline calibration time for Hadamard and our method on Qwen3-8B.

QuantizerHadamard (min)Ours (min)
RTN0.668.06
GPTQ10.3328.63

Although our method takes longer to calibrate than standard block-wise Hadamard, the total time is 8.06 minutes with RTN and 28.63 minutes with GPTQ on Qwen3-8B. We consider this overhead acceptable for PTQ because calibration is performed only once before deployment, and the resulting transforms are reused across subsequent inference requests without repeating the calibration process.

(2) Inference time. Both methods use the same per-block 32×32 matrix-multiplication routine, so our transform has the same matrix-multiplication cost as Hadamard; the only additional online operation is permutation.

Following the MXFP4 emulation approach in BRQ's latency evaluation (Section 5.3), we measure operator-level latency on an RTX 4090 using synthetic BF16 inputs and the microsoft/microxcaling CUDA quantizer. Table D reports median latency over 50 runs after 10 warmup runs, with CUDA synchronization and randomized method order. Without permutation, our pipeline has nearly identical latency to Hadamard. The additional overhead mainly comes from data movement and is small in absolute terms. Fusing permutation with the block transform and quantization could further reduce this overhead. End-to-end latency and throughput remain to be evaluated.

Table D. Operator latency on RTX 4090 at batch size 1 and sequence length 2048. All pipelines include MXFP4 quantization.

Hidden dimensionHadamard (ms)Ours w/o permutation (ms)Ours with permutation (ms)
40960.1210.1210.152
110080.3720.3700.486

Q4: "The paper motivates the method using “lossless quantization points,” but the final algorithm selects the micro-block size by directly minimizing reconstruction error. Could the authors clarify the role of the lossless points in the actual optimization?"

Thank you for your question. The lossless quantization points provide the motivation and guidance for our micro-block transform. While conventional outlier-smoothing methods generally favor larger transform blocks to spread outliers more widely, MXFP4’s lossless points imply that reconstruction error need not decrease monotonically with block size. A smaller micro-block may therefore yield lower error. This motivates searching over micro-block sizes using reconstruction error to select the best candidate. As future work, we aim to develop a method that directly exploits lossless quantization points to optimize the transform, without searching over candidate micro-block sizes.


We hope these clarifications address your concerns. Our key insight is that reducing activation magnitudes does not necessarily reduce MXFP4 quantization error. Building on this observation, we expand the transform design space by treating the transform size itself as an optimization variable. We believe this perspective provides new insights into MXFP4 quantization and can guide the design of more effective quantization methods.

Reviewer_TtueOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a format-aware orthogonal transformation for MXFP4 quantization. It observes that quantization errors are non-monotonic with respect to magnitude due to the power-of-two shared scaling and E2M1 representation. Based on this observation, the method introduces Inter-block Permutation to group outlier channels with less important channels and Micro-block Givens Rotation to distribute outliers across multiple channels. The method is evaluated through layer-wise activation reconstruction error, WikiText perplexity, and zero-shot accuracy on LLaMA2/3 and Qwen3-8B under W4A4 quantization.

Critical strengths

The paper identifies the non-monotonic relationship between quantization error and magnitude in MXFP4, suggesting that spreading outliers across an entire block may not always be optimal. This format-aware insight leads to a substantial reduction in reconstruction error for Down activations.

Critical weaknesses
  1. The downstream improvements over existing baselines are marginal. Across the six configurations in Table 1, the average accuracy improvement over Hadamard is only 0.03, and the improvement over MR-GPTQ is only 0.07. However, the conclusion still claims that the proposed method outperforms Hadamard, which is not sufficiently supported by the results.
  2. The claimed complementarity is not supported by downstream performance. In Table 3, combining permutation and Givens rotation achieves 63.05, which is lower than permutation alone (63.22) and Givens rotation alone (63.38). The authors should further investigate why the substantial reductions in reconstruction error do not translate into better downstream performance.
  3. Similar permutation and block-wise rotation techniques have already been proposed in DuQuant and further extended to MXFP4 quantization in DuQuant++. However, the paper neither discusses these closely related methods nor includes them in the experimental comparisons.
Questions for authors

1. How can the authors demonstrate that the reported improvements are significant? The gains are marginal: +0.03 average accuracy over Hadamard across the six configurations in Table 1 and +0.07 over MR-GPTQ on Qwen3-8B. Please consider repeating the experiments with different calibration samples and random seeds for the transformations, and report paired per-question confidence intervals against these baselines. 2. There appear to be inconsistencies between the equations and algorithms. Eq. (10) uses the product of activation and weight norms, while Algorithm 1 uses their ratio. Similarly, Eq. (18) and Algorithm 2 appear to swap sine and cosine terms, and the values in Algorithm 2 do not yield equal magnitudes under Eq. (13). Please clarify which formulations were used in the experiments. 3. I suggest an upper-bound experiment to better assess the potential of the proposed approach. Specifically, keep the identified 10 outlier blocks (out of 4096) in FP16 while leaving the remaining blocks unchanged, and compare the results with standard block Hadamard. A substantial improvement would support the motivation, while performance comparable to Hadamard would suggest limited potential gains from addressing these outliers.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ttue (Part 1/4)

We thank the reviewer for the thoughtful feedback and for recognizing our work. Below, we address the reviewer’s concerns point by point.

Q1: "How can the authors demonstrate that the reported improvements are significant? The gains are marginal: +0.03 average accuracy over Hadamard across the six configurations in Table 1 and +0.07 over MR-GPTQ on Qwen3-8B. Please consider repeating the experiments with different calibration samples and random seeds for the transformations, and report paired per-question confidence intervals against these baselines."

Thank you for this suggestion. We therefore distinguish variability across random seeds and calibration subsets from uncertainty in the paired accuracy difference on the evaluation questions.

Random seeds. Table A summarizes the additional experiments on Qwen3-8B with GPTQ under MXFP4 W4A4 quantization across different random seeds. Our method achieves an average zero-shot accuracy of 64.57%, compared with 64.37% for Hadamard, a gain of 0.20 percentage points. It also achieves lower C4 perplexity.

Calibration samples. We additionally evaluate five different calibration subsets on Qwen3-8B with GPTQ under MXFP4 W4A4 quantization, as summarized in Table B. Our method achieves an average zero-shot accuracy of 64.73%, compared with 64.70% for Hadamard, with smaller variation across calibration subsets (±0.07 versus ±0.43). It also achieves lower C4 perplexity (15.88 versus 15.94), although its WikiText2 perplexity is higher (10.94 versus 10.84). These results suggest that our method's average accuracy is relatively stable across the evaluated calibration subsets.

Paired per-question confidence intervals. Table C shows that, on Qwen3-8B with RTN, our method improves the six-task average over Hadamard by 1.11 percentage points (95% CI: [+0.21, +2.00]). The interval lies entirely above zero, demonstrating a statistically significant average accuracy improvement over Hadamard.

Table A. GPTQ results across five different seeds on Qwen3-8B under MXFP4 W4A4 quantization.

TransformWikiText2 PPL ↓C4 PPL ↓Six-task average accuracy (%) ↑
Identity10.93 ± 0.1116.38 ± 0.0062.74 ± 0.38
Hadamard10.83 ± 0.1015.98 ± 0.1364.37 ± 0.26
Ours10.90 ± 0.0815.82 ± 0.1164.57 ± 0.31

Table B. GPTQ results across five different calibration subsets on Qwen3-8B under MXFP4 W4A4 quantization.

TransformWikiText2 PPL ↓C4 PPL ↓Six-task average accuracy (%) ↑
Identity10.94 ± 0.0116.44 ± 0.1262.48 ± 0.27
Hadamard10.84 ± 0.1115.94 ± 0.1264.70 ± 0.43
Ours10.94 ± 0.0115.88 ± 0.0364.73 ± 0.07

Table C. Paired per-question 95% confidence intervals on Qwen3-8B. Δ = Ours − baseline, in percentage points; higher is better (↑). The six-task average gives equal weight to each task.

TaskQuestionsOurs − Identity: Δ [95% CI] (pp)Ours − Hadamard: Δ [95% CI] (pp)
WinoGrande1,267+4.03 [+1.03, +7.02]+2.21 [−0.71, +5.13]
OpenBookQA500+1.40 [−1.40, +4.20]+2.20 [−0.60, +5.00]
PIQA1,838+0.27 [−1.41, +2.01]+0.38 [−1.25, +2.01]
HellaSwag10,042−0.15 [−0.76, +0.47]+0.14 [−0.47, +0.75]
ARC-e2,376−2.53 [−4.00, −1.05]−0.76 [−2.31, +0.76]
ARC-c1,172−1.45 [−4.01, +1.11]+2.47 [−0.17, +5.12]
Six-task average17,195+0.26 [−0.66, +1.15]+1.11 [+0.21, +2.00]
AuthorsOpenReview ↗

Response to Reviewer Ttue (Part 2/4)

Q2: "There appear to be inconsistencies between the equations and algorithms. Eq. (10) uses the product of activation and weight norms, while Algorithm 1 uses their ratio. Similarly, Eq. (18) and Algorithm 2 appear to swap sine and cosine terms, and the values in Algorithm 2 do not yield equal magnitudes under Eq. (13). Please clarify which formulations were used in the experiments."

Thank you for your careful reading and for identifying these inconsistencies. In our experiments, we used the product of the activation and weight norms, following Wanda and Eq. (10). The ratio in Algorithm 1 was a typographical error. The assignments of cc and ss in Algorithm 2 were also reversed in the pseudocode. Our experiments used the values in Eq. (18): c=1/rc=1/\sqrt{r} and s=(r−1)/rs=\sqrt{(r-1)/r}. We have corrected both algorithms in the updated PDF,.

Q3: "I suggest an upper-bound experiment to better assess the potential of the proposed approach. Specifically, keep the identified 10 outlier blocks (out of 4096) in FP16 while leaving the remaining blocks unchanged, and compare the results with standard block Hadamard. A substantial improvement would support the motivation, while performance comparable to Hadamard would suggest limited potential gains from addressing these outliers."

Thank you for suggesting this upper-bound experiment. To assess the potential benefit of addressing outlier blocks, we consider LLaMA2-7B under RTN MXFP4 W4A4 quantization, using the same standard block Hadamard transform in all quantized settings. The experiment retains both the weights and activations of the selected blocks in FP16, while keeping the remaining blocks quantized. A random-block control retains exactly the same number of FP16 blocks in each layer.

To clarify the selection granularity, 10/4096 (approximately 0.24%) is the outlier-channel ratio. We identify outlier channels by their peak activations on the calibration set and retain the complete quantization blocks containing them. The actual fraction of blocks retained in FP16 ranges from 3.12% to 7.85% across layers. This additional experiment uses 128 WikiText2 training sequences of 2048 tokens for calibration (seed 0), with matching FP16 and Hadamard baselines reported below.

Retaining outlier blocks in FP16 improves the six-task average accuracy by 1.47 percentage points over standard Hadamard and 1.35 percentage points over the matched random-block control, while reducing WikiText2 PPL from 6.17 to 5.84. These results support the motivation for specifically addressing outlier blocks in this setting. This high-precision retention experiment indicates potential benefits, but does not guarantee that an MXFP4 transform can achieve the same gains.

Table D. Upper-bound comparison on LLaMA2-7B under RTN MXFP4 W4A4 quantization.

SettingWikiText2 PPL ↓Six-task average accuracy (%) ↑
FP16 reference5.4764.85
Standard block Hadamard6.1761.56
Hadamard + outlier blocks in FP16 (weights and activations)5.8463.03
Hadamard + matched random blocks in FP16 (weights and activations)6.1061.68
AuthorsOpenReview ↗

Response to Reviewer Ttue (Part 3/4)

W1: "The downstream improvements over existing baselines are marginal. Across the six configurations in Table 1, the average accuracy improvement over Hadamard is only 0.03, and the improvement over MR-GPTQ is only 0.07. However, the conclusion still claims that the proposed method outperforms Hadamard, which is not sufficiently supported by the results."

Thank you for raising this concern. As detailed in Q1, additional experiments on Qwen3-8B with GPTQ show higher mean accuracy and lower C4 perplexity than Hadamard across both random seeds and calibration subsets. With RTN, the six-task average accuracy improves by 1.11 percentage points (95% CI: [+0.21, +2.00]), supporting a statistically significant gain in this setting. We will incorporate these results into the revised manuscript to strengthen the downstream evaluation.

W2: "The claimed complementarity is not supported by downstream performance. In Table 3, combining permutation and Givens rotation achieves 63.05, which is lower than permutation alone (63.22) and Givens rotation alone (63.38). The authors should further investigate why the substantial reductions in reconstruction error do not translate into better downstream performance."

Thank you for raising this point. Our calibration minimizes layer-wise reconstruction error, which does not guarantee monotonic improvements in downstream accuracy. The ablations in Appendix B.1, Table 5 show that combining Inter-block Permutation and Micro-block Givens Rotation achieves the lowest or tied-lowest reconstruction error across all evaluated activation types. Table 6 further illustrates their complementary effects: for QKV activations at the default outlier ratio of 10/4096, the error decreases from 5.53e-4 without either component to 4.74e-4 with permutation alone or 1.04e-4 with micro-block rotation alone, and further to 8.25e-5 with both.

Reconstruction error measures local numerical fidelity, while downstream accuracy depends on the final predictions of the entire model. Errors in different channels and layers can have different effects on the output, and their impact may be amplified or offset as they propagate through the network. Moreover, zero-shot accuracy depends on answer rankings: reduced reconstruction error may leave many predictions unchanged, while changes near decision boundaries can help some examples and hurt others. Differences between calibration data and downstream tasks can further weaken this relationship. These factors can explain why a large relative reduction in reconstruction error yields only modest or inconsistent accuracy gains.

We have also added downstream ablation results in Table E. The combination achieves the lowest C4 PPL of 16.38, compared with 17.50 for micro-block rotation alone and 17.13 for permutation alone. These results support the usefulness of combining the two components for perplexity, while the original Table 3 shows that this benefit does not consistently extend to average zero-shot accuracy.

Table E. Additional downstream ablation results on Qwen3-8B with GPTQ under MXFP4 W4A4 quantization.

TransformC4 PPL ↓
Identity17.61
Micro-block Givens only17.50
Inter-block Permutation only17.13
Ours (both)16.38

We will clarify that the two components are complementary in reducing reconstruction error, and qualify the downstream claim accordingly. We use reconstruction error as a calibration objective and assess its benefits separately through perplexity and task accuracy.

AuthorsOpenReview ↗

Response to Reviewer Ttue (Part 4/4)

W3: "Similar permutation and block-wise rotation techniques have already been proposed in DuQuant and further extended to MXFP4 quantization in DuQuant++. However, the paper neither discusses these closely related methods nor includes them in the experimental comparisons."

Thank you for pointing out these closely related methods. We agree that DuQuant and DuQuant++ should be discussed explicitly, and we will add this discussion and the following comparison to the revised manuscript.

The core of our method is the adaptive micro-block transform, motivated by the observation that reducing an outlier's magnitude does not necessarily reduce its MXFP4 quantization error. Due to the power-of-two E8M0 shared scale and the discrete E2M1 codebook, spreading an outlier across all 32 channels may produce smaller magnitudes but higher error than spreading it over a smaller, appropriately chosen micro-block. Permutation supports this transform by grouping outliers with low-importance channels.

DuQuant++ searches for an outlier-aware orthogonal transform with a fixed transform size of 32, using peak activation magnitude as its objective. Our method treats the transform size itself as an optimization variable, selecting a micro-block size n≤32n\leq 32 using reconstruction error while retaining the standard MXFP4 quantization group size of 32. This expands the search from matrices at a single fixed size to transforms across multiple sizes.

To provide a controlled comparison, we incorporated our micro-block design into the standard DuQuant++ pipeline. We compare three variants, keeping the number of search steps identical and the MXFP4 quantization group size fixed at 32:

  1. DuQuant++: Search for orthogonal transforms with a fixed size of 32, using peak activation magnitude as the search objective;
  2. DuQuant++*: Keep the transform size fixed at 32, but replace the peak-based objective with activation reconstruction MSE after quantization;
  3. DuQuant++* + micro-block: Retain the MSE objective and introduce our micro-block design, allowing the transform size n≤32n\leq 32 to be selected adaptively instead of fixing it at 32.

We evaluate these variants on LLaMA3-8B-Instruct across six zero-shot tasks, with results shown in Table F. The micro-block variant achieves the highest average accuracy of 65.11%, improving over the original DuQuant++ (64.10%) by 1.01 percentage points and over the fixed-size MSE variant (64.77%) by 0.34 percentage points. The latter comparison isolates the benefit of the micro-block design under the same MSE objective, supporting its value beyond changing the search objective alone.

Table F. Controlled comparison within DuQuant++ on LLaMA3-8B-Instruct: zero-shot accuracy (%).

MethodARC-EasyARC-ChallengeWinoGrandeHellaSwagOpenBookQAPIQAAvg. ↑
DuQuant++74.8750.9468.8273.3540.6076.0164.10
DuQuant++*74.9250.6869.3873.3543.0077.2664.77
DuQuant++* + micro-block75.3452.9069.6973.8141.4077.5365.11

We hope these clarifications address your concerns. Our key insight is that reducing activation magnitudes does not necessarily reduce MXFP4 quantization error. Building on this observation, we expand the transform design space by treating the transform size itself as an optimization variable. We believe this perspective provides new insights into MXFP4 quantization and can guide the design of more effective quantization methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Summary of Reviews

Dear Area Chair,

We thank the reviewers for their constructive feedback. In response to the main concerns, our rebuttal provides the following updates:

  1. Clarified the contribution and relationship to prior work. We explained how MXFP4’s lossless quantization points motivate adaptive micro-block sizes, clarified the differences from DuQuant++, PeRQ, and TORQ, and added a controlled comparison within DuQuant++, demonstrating the effectiveness of our adaptive micro-block design.

  2. Expanded the experimental evaluation. We added results across more model sizes and PTQ methods, repeated experiments with different random seeds and calibration subsets, and provided statistical analyses of downstream improvements.

  3. Strengthened the component analysis. We added ablation experiments to examine the benefits of permutation and rotation and clarified the relationship between reconstruction error and downstream performance.

  4. Added efficiency measurements. We reported calibration costs and GPU operator latency to assess the computational overhead of our method.

We hope these experiments clarify the contribution of our proposed micro-block transform designed for MXFP4 quantization. We sincerely thank the reviewers and the Area Chair again for their time and consideration.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.