- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MXAttention is a data-free, training-free framework for MXFP4 attention. It combines Universal Optimal Scaling (UOS) and Pre-Normalization Quantization (PNQ) to reduce quantization error and preserve attention normalization while retaining low-precision inference efficiency.
Critical strengths
- The paper presents training-free, efficient MXFP4 attention, which is useful for practical deployment.
- The paper is well written and easy to follow, with clear motivation and an accessible presentation of the proposed method.
Critical weaknesses
- Although the paper provides theoretical proofs, the overall novelty of UOS and PNQ appears limited. In my view, these are primarily engineering refinements.
- MXAttention does not show a clear overall accuracy advantage over NVFP4 + Sage, particularly on HunyuanVideo in Table 1.
- If MXFP4 used one shared scale per 16 elements, as in NVFP4, how would its accuracy change?
- How would the method perform on models with fewer denoising steps, such as a four-step Wan2.2 model?
- The baseline comparison is not fully fair: MXAttention uses Hadamard rotation, whereas this preprocessing is not reported for the NVFP4 baselines.
- Quantitative quality evaluation primarily relies on VBench. Could the authors report additional assessments, such as VisionReward [1] scores or human preference results, to better substantiate the generation quality?
[1] Jiazheng Xu et al. VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(13): 11269–11277, 2026.
Questions for authors
Please refer to Weaknesses.
4 comments
This comment presents the matched rotation comparison (W5) and additional quality assessment (W6).
W5. Applying the same Q/K Hadamard rotation to NVFP4 + Sage
We evaluated NVFP4 + Sage with the same Q/K Hadamard rotation on both models. Each comparison uses 48 videos with matched prompts and seeds. All methods share the same precision policy within each model: Wan2.2 uses high precision at steps 38 and 39, with no first-layer fallback; HunyuanVideo uses four-bit attention throughout denoising.
In the table, Sage denotes NVFP4 + Sage, and +H adds the same Q/K Hadamard rotation. MX denotes the complete MXAttention configuration.
| Metric | Wan Sage | Wan Sage+H | Wan MX | Hunyuan Sage | Hunyuan Sage+H | Hunyuan MX |
|---|---|---|---|---|---|---|
| Subject ↑ | 0.9459 | 0.9470 | 0.9491 | 0.9575 | 0.9566 | 0.9563 |
| Imaging ↑ | 0.6900 | 0.6934 | 0.6847 | 0.6119 | 0.6033 | 0.6050 |
| Aesthetic ↑ | 0.6348 | 0.6256 | 0.6355 | 0.6029 | 0.6079 | 0.6080 |
| Cosine ↑ | 0.8988 | 0.8835 | 0.9216 | 0.9710 | 0.9746 | 0.9655 |
| SSIM ↑ | 0.4766 | 0.4538 | 0.5190 | 0.7524 | 0.7765 | 0.7198 |
| PSNR ↑ | 13.93 | 13.40 | 15.16 | 19.74 | 20.41 | 18.59 |
| FID ↓ | 126.18 | 139.07 | 112.60 | 105.95 | 93.60 | 136.28 |
| LPIPS ↓ | 0.4378 | 0.4734 | 0.3861 | 0.2679 | 0.2433 | 0.3089 |
| MSE ↓ | 3402.3 | 3866.0 | 2462.9 | 918.7 | 793.5 | 1112.4 |
| VisionReward ↑ | 0.0832 | 0.0817 | 0.0794 | 0.0918 | 0.0972 | 0.0889 |
Rotation has model-dependent effects: it improves the five reference-based metrics on HunyuanVideo and worsens them on Wan2.2. MXAttention retains its advantage on all five Wan2.2 reference-based metrics against both NVFP4 + Sage configurations.
W6. Quality evaluation beyond VBench
Thank you for suggesting VisionReward. We have added it on both models using the same 48-video evaluation sets.
The original Table 1 also reports Cosine similarity, SSIM, PSNR, FID, and LPIPS alongside the three VBench metrics. VisionReward adds an assessment based on learned human preferences.
| Method | Wan2.2 | HunyuanVideo | Wan2.2 + first-layer fallback |
|---|---|---|---|
| Direct MXFP4 | 0.0573 | −0.0060 | 0.0716 |
| NVFP4 | 0.0842 | 0.0861 | 0.0766 |
| NVFP4 + Sage | 0.0832 | 0.0918 | 0.0825 |
| NVFP4 + Sage + Hadamard | 0.0817 | 0.0972 | — |
| MXAttention | 0.0794 | 0.0889 | 0.0859 |
The first two columns use the precision policies described above. The last column is a separate Wan2.2 comparison with the first layer kept in high precision for every method, while retaining the same step fallback. A dash indicates an unavailable result.
MXAttention improves VisionReward over direct MXFP4 on both models and achieves scores close to the original NVFP4 + Sage baseline: 0.0794 versus 0.0832 on Wan2.2, and 0.0889 versus 0.0918 on HunyuanVideo. With the same first-layer fallback, MXAttention scores 0.0859 versus 0.0825 for NVFP4 + Sage on Wan2.2.
The HunyuanVideo ablation also supports the contribution of each component. With rotation fixed, adding PNQ increases VisionReward from 0.0558 to 0.0768; adding UOS increases it further to 0.0889. Together with the reference-based metrics and the visual examples in the paper, these results support substantial quality recovery under MXFP4.
Thank you again for your helpful suggestions; we hope the additional comparisons provide a clearer picture of MXAttention's contribution and practical value, and we would be glad to discuss these results further.
We have now completed the two Wan2.2 experiments you suggested: one with 16 elements per shared scale and one with a distilled four-step model.
W3. 16 elements per shared scale. We changed the group size from 32 to 16 for Q, K, V, and P, keeping the E2M1 values and E8M0 scales. We evaluated 48 videos per configuration. All four configurations used the same Q/K Hadamard rotation, with no high-precision fallback for any denoising step or for layer 0. The baseline used the OCP scaling rule without PNQ.
| Configuration | Group size | Subject ↑ | Imaging ↑ | Aesthetic ↑ | VisionReward ↑ | PSNR ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|---|
| OCP scaling rule + rotation | 32 | 0.9250 | 0.6160 | 0.6025 | 0.0338 | 13.30 | 0.6193 |
| OCP scaling rule + rotation | 16 | 0.9309 | 0.6186 | 0.6159 | 0.0512 | 12.87 | 0.6267 |
| MXAttention | 32 | 0.9357 | 0.6730 | 0.6216 | 0.0689 | 14.98 | 0.4010 |
| MXAttention | 16 | 0.9457 | 0.6788 | 0.6248 | 0.0773 | 15.23 | 0.4024 |
For MXAttention, reducing the group size to 16 improved all three VBench scores reported in Table 1, as well as VisionReward and PSNR. LPIPS changed little. UOS and PNQ continued to help at the smaller group size: with 16 elements per scale, adding them to the rotation baseline raised Imaging from 0.6186 to 0.6788 and PSNR from 12.87 to 15.23 dB.
W4. Four-step Wan2.2. We used Wan2.2-T2V-A14B with the four-step LoRA weights from LightX2V (Seko V2.0, rank 64). We evaluated 48 videos per configuration, using videos generated by the same four-step model with high-precision attention as the reference.
| Metric | High-precision reference | Naive MXFP4 | MXAttention |
|---|---|---|---|
| Subject ↑ | 0.9515 | 0.9584 | 0.9506 |
| Imaging ↑ | 0.6743 | 0.6045 | 0.6588 |
| Aesthetic ↑ | 0.6254 | 0.6158 | 0.6226 |
| VisionReward ↑ | 0.0691 | 0.0627 | 0.0685 |
| Cosine ↑ | — | 0.9073 | 0.9191 |
| SSIM ↑ | — | 0.4037 | 0.4348 |
| PSNR ↑ | — | 13.38 | 14.30 |
| Mean per-video FID ↓ | — | 206.89 | 94.20 |
| LPIPS ↓ | — | 0.6089 | 0.4131 |
| MSE ↓ | — | 3331.3 | 2959.3 |
On this four-step model, MXAttention closed 77.9% of the gap in Imaging between naive MXFP4 and the high-precision reference. VisionReward rose from 0.0627 to 0.0685, close to the reference score of 0.0691. The videos also matched the reference more closely: LPIPS fell by 32.2% and PSNR rose by 0.92 dB relative to naive MXFP4.
Thank you for suggesting both experiments. We would be happy to answer any further questions about the settings or results.
Thank you for recognizing the practical motivation and clarity of our work. We have added matched comparisons and VisionReward evaluations on Wan2.2 and HunyuanVideo. This comment addresses W1–W3; the following comment presents the rotation and quality results.
W1. Contributions of UOS and PNQ
Our contribution starts from the analysis of two sources of MXFP4 attention error: the shared-scale trade-off and normalization mismatch.
UOS provides a theoretical basis for selecting the shared scale. Lemma 1 and Eq. (6) show that the distribution of block maxima enters the derivative of the expected error through a nonnegative factor. The sign of the derivative is determined by the E2M1 quantization grid. Theorem 1 establishes q⋆=29/4=7.25 as a global minimizer of the objective in Eqs. (3)–(4), under the stated assumptions. Corollary 1 extends this result to the specified class of weighted objectives. This gives a common scaling rule without calibration or per-layer search. In the new matched downstream experiments, changing q=7 to q=7.25 reduces mean MSE by 3.85% on Wan2.2 and 20.70% on HunyuanVideo, with rotation, PNQ, block size, and precision policy held fixed.
PNQ addresses the normalization error introduced by quantization. Section 3 and Appendix I show how inconsistent numerator and denominator weights produce a row-dependent scale error in tiled attention. Appendix R measures an average row sum of 0.9266 after offline quantization of normalized attention weights across 80 Wan2.2 tensors. This motivates preserving normalization for each row. Equation (10) uses the same quantized exponential weights in both updates; Appendices H–J analyze online rescaling and establish that the resulting weights sum to one in exact arithmetic.
INT-FlashAttention and QFlash also use quantized exponentials in both updates, and we will include them in Related Work. Our contribution is the MXFP4 error analysis, the explicit normalization analysis, and the demonstrated quality recovery.
PNQ also supports computing the row sum on the Cube unit through
Pij[Vj,1]=[PijVj,Pij1].
Our implementation uses the same quantized weights and exactly represented constant-one entries for both outputs, removing a separate Vector row reduction. Online rescaling and final normalization remain on the Vector unit. The complete UOS + PNQ + Q/K Hadamard configuration achieves 2.67× attention speedup and 1.63× total video-task speedup over BF16 on Ascend 950.
W2. Value of MXFP4 attention and comparison with NVFP4 + Sage
MXAttention recovers 92.9% and 95.8% of the Imaging Quality gap between direct MXFP4 and FP16 on Wan2.2 and HunyuanVideo, respectively. The comparison with NVFP4 + Sage is model-dependent: MXAttention leads on all five reference-based metrics on Wan2.2, while NVFP4 + Sage leads on these metrics on HunyuanVideo.
Recovering MXFP4 quality has practical value on hardware with native MXFP4 support. For example, AMD documents native MXFP4 support on MI350X/MI355X and conversion or BF16 emulation for NVFP4 checkpoints [1]. Our Ascend results provide a separate deployment example.
There is also evidence of efficient MXFP4 computation. MR-GPTQ/QuTLASS reports up to approximately 15% higher MXFP4 throughput than NVFP4 for the evaluated B200 linear-layer configuration [2]. WUSH reports up to 5.8× speedup over BF16 for evaluated RTX 5090 linear layers, including transformation and quantization [3]. These results motivate the format; our attention and video-task speedups are measured directly on Ascend.
W3. A shared scale for 16 elements
This comparison would help isolate the effect of scaling granularity. Our work targets the OCP MXFP4 format, which specifies 32 E2M1 values per E8M0 shared scale [4]. A 16-element group would double the number of scales. We have not evaluated that configuration. The reported quality recovery and acceleration use the standard 32-element format.
References
[1] AMD: NVFP4 to MXFP4.
[2] MR-GPTQ/QuTLASS, Section 5, Figure 5.
[3] WUSH, Section 5, Figure 2.
[4] OCP Microscaling Formats, Table 1.