- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RotateAttention-FP4, a unified framework for reducing quantization error in FP4 attention. It first combines grouped Hadamard rotation to redistribute activation outliers across quantization blocks with offline-calibrated Q/K channel centering to remove stable biases, alongside scale selection and probability amplification. LoRA-based QAT is then used to adapt models that remain sensitive to quantization. Experiments demonstrate that the proposed method improves fidelity to FP16 outputs and accelerates attention computation.
Critical strengths
- The experiments show practical benefits. Results on MiniMax H3 and Wan2.2 demonstrate notable improvements over SageAttention3 .
Critical weaknesses
- Insufficient novelty. The core components, rotation-based quantization and QAT, are widely used techniques, and the proposed modifications are limited.
- Incomplete experimental validation. The paper lacks quantitative ablations of its core components. Some evaluation metrics are also computed using very limited data: MiniMax H3’s VBench evaluation uses a fixed prompt, while Wan2.2’s average is based on only two videos. This limits the reliability of the experimental conclusions.
Questions for authors
- Please clarify the novelty of the proposed method.
- Please provide ablations of the core components and evaluations on a larger, more comprehensive dataset.