acceptodds
Under review as a conference paper at ICLR 2027

RotateAttention-FP4: A Unified Training-Free and Data-Efficient QAT Framework for Quantized Attention

Abstract

Attention incurs substantial computation and memory traffic in video diffusion models, motivating 4-bit quantization for efficient inference. However, existing training-free 4-bit attention methods remain limited in both generation quality and runtime efficiency, while quantization-aware training (QAT) often incurs substantial training costs. We propose RotateAttention-FP4, a unified NVFP4 attention framework supporting accurate and efficient video diffusion through both training-free inference and lightweight adaptation. Our key insight is that activation outliers and stable channel-wise biases amplify quantization error, degrading the accuracy of training-free inference and increasing the burden on model adaptation. We refer to these distribution-induced errors as structural errors and suppress them through grouped Hadamard rotation and calibration-based channel centering using offline-estimated offsets. We further optimize quantization scales for both QKV and attention weights: rotation-aware QKV scale selection reduces FP4 rounding error, while probability amplification reduces underflow of attention-weight scales. Together, these corrections improve training-free inference accuracy and reduce the residual error that sensitive models must compensate for through lightweight QAT. Experiments on Wan2.2 and MiniMax H3 demonstrate generation quality close to 16-bit attention without any fine-tuning in conventional multi-step settings and on eight-step MiniMax H3. For the quantization-sensitive four-step Wan2.2 model, QAT with low-rank adaptation (LoRA) recovers generation quality using only 30 randomly sampled training examples. Our fused implementation further delivers a – end-to-end attention speedup over FlashAttention and – preprocessing throughput over SageAttention3.

Primary area:
generative models
Keywords:
Quantized Attention, 4 bit, generative model

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_rQd9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RotateAttention-FP4, a unified framework for reducing quantization error in FP4 attention. It first combines grouped Hadamard rotation to redistribute activation outliers across quantization blocks with offline-calibrated Q/K channel centering to remove stable biases, alongside scale selection and probability amplification. LoRA-based QAT is then used to adapt models that remain sensitive to quantization. Experiments demonstrate that the proposed method improves fidelity to FP16 outputs and accelerates attention computation.

Critical strengths
  1. The experiments show practical benefits. Results on MiniMax H3 and Wan2.2 demonstrate notable improvements over SageAttention3 .
Critical weaknesses
  1. Insufficient novelty. The core components, rotation-based quantization and QAT, are widely used techniques, and the proposed modifications are limited.
  2. Incomplete experimental validation. The paper lacks quantitative ablations of its core components. Some evaluation metrics are also computed using very limited data: MiniMax H3’s VBench evaluation uses a fixed prompt, while Wan2.2’s average is based on only two videos. This limits the reliability of the experimental conclusions.
Questions for authors
  1. Please clarify the novelty of the proposed method.
  2. Please provide ablations of the core components and evaluations on a larger, more comprehensive dataset.
Reviewer_Nxu3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents RotateAttention-FP4, an NVFP4 attention framework for video diffusion built on one idea: split quantization error into structural error (outliers, stable channel biases), which offline rotation and calibration remove, and residual numerical error, which lightweight LoRA-based QAT then learns on as few as 30 samples. Training-free models use the corrected kernels as-is. Reported fidelity to FP16 beats SageAttention3 (MSE down 19–68%), with attention throughput 2.11–2.33x over FlashAttention.

Critical strengths

S1. A clean organizing idea, engineered to be free at runtime. The structural vs residual error split gives the paper a spine: deterministic distortions get removed offline, learning handles only what remains. Every component respects this. The V rotation is absorbed into the projections, centering fuses into the same matrix multiply as rotation, and probability amplification costs one exponent shift that cancels in the final division. The unified deployment path, where training-free and post-QAT models share the same inference procedure, is the part I would actually use.

S2. A thoughtfully constrained QAT recipe. Fake-quantizing QKV while keeping probabilities in FP16 during training means the backward pass is stock FlashAttention, with no custom kernel work. LoRA-only updates on 30 randomly sampled videos make adaptation cheap enough to be a routine step. The training dynamics support the paper's central claim: with structural errors suppressed, the high-noise expert's final loss drops 54.9% relative to Attn-QAT under an equal budget, so the remaining learning problem is indeed smaller.

Critical weaknesses

W1. The quality evidence is too thin for the claims. The VBench evaluation uses one fixed fox prompt for MiniMax H3 and two videos for Wan2.2 (Sec. 5.1). At that sample size, the headline QAT gap in Table 2 (0.8669 vs 0.8607) is noise. There are no qualitative samples anywhere in the paper, which is hard to accept for a video generation method. And the fidelity story has a crack the paper does not discuss: MSE drops 19 to 68% over SageAttention3, yet VBench ties or slightly trails it in all three training-free rows of Table 1. If paired fidelity to FP16 does not translate into benchmark or perceptual gains, the main quality metric needs justification.

W2. The speed framing, and the missing controls. The headline 2.11–2.33x is against FlashAttention; against the actual FP4 baseline SageAttention3 the end-to-end gain is 0.3–1.9%, and Attn-QAT is faster at shorter sequences (10.5% at 8K). The honest claim is speed parity with better fidelity, and the paper would be stronger saying so. Beyond framing, no design choice is ablated with numbers: rotation group size, the r=4 scale target, centering, and amplification are all asserted. The FP16 fallback (first/last blocks of Wan2.2, final five of MiniMax H3) is not ablated either, so I cannot tell how much of the retained quality comes from the method versus the fallback. SageAttention2 and rotation-based diffusion PTQ (vRot, DiRotQ) are cited but never run.

Questions for authors
  1. Can you run VBench at a meaningful sample size (the full suite, or at least ~100 prompts) for the training-free comparison and for Table 2? Two videos cannot carry a 0.006 difference.

  2. How much do the FP16 fallback blocks contribute to quality and to latency? What happens w/o any fallback?

  3. Why does VBench stay flat while MSE drops by up to 68%? A per-dimension breakdown, or a small human preference study, would clarify what the fidelity gains buy.

  4. Can you show ablations for the rotation group size (16-channel vs two-group vs full), the r=4 target, centering, and probability amplification, even at small scale?

  5. How sensitive is the QAT result to the 30-sample selection, and how do SageAttention2 and vRot/DiRotQ compare on your setup?

Reviewer_HozNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Summary. The paper targets the cost of attention in video diffusion models with a 4-bit (NVFP4) attention implementation. Its premise is that quantization error splits into a "structural" component — activation outliers and stable per-channel biases — removable by deterministic transforms, and a "residual" numerical component that learning must absorb. The pipeline applies grouped Hadamard rotation to Q/K/V, per-channel centering of Q/K from offline calibration, a fixed FP4 scale target of r=4, and a ×768 amplification of attention probabilities implemented as a softmax exponent shift; models that remain sensitive are then adapted with LoRA-QAT while base weights, rotations and offsets stay frozen. Experiments cover Wan2.2 and MiniMax H3 in both original and distilled variants, reporting paired fidelity against FP16, VBench, and kernel throughput on a non-causal MHA benchmark, where the method reaches 299–324 TOPS — 2.11–2.33× over FlashAttention, but only 0.3–1.9% over SageAttention3.

Critical strengths

S1. The problem is well chosen. Few-step distilled video diffusion is precisely where 4-bit attention is hardest, and the paper correctly identifies the PTQ/QAT split as the practical obstacle rather than assuming a single deployment regime.

S2. The kernel engineering work is respected. The paper implement customized preprocessing and attention kernels.

S3. Attribution is honest. §2 credits every borrowed mechanism explicitly, and Algorithms 1–2 are complete enough to reimplement. This matters: the weaknesses below concern evidence and positioning, not integrity.

Critical weaknesses

W1. No hardware peak, so no utilization figure. §5.1 names the device (RTX PRO 6000D) but never states its peak throughput, so 299–324 TOPS cannot be converted into utilization. The gap is not fillable by the reader: the 6000D does not appear in NVIDIA's public catalogue, and the advertised 4000 AI TOPS is an FP4 sparse figure for the non-D card (dense 2015.2). For a paper whose central claim is efficiency, giving the denominator is standard practice.

W2. No FP8 baseline. FP4 offers 2×(or 3x on some GPUs) FP8's dense peak and 4× BF16's, yet the measured gain over BF16 FlashAttention is only 2.11–2.33× — 53–58% of that nominal advantage. Whether 4 bits repay their accuracy cost over 8 bits needs an FP8 comparison. Also the paper cites SageAttention2 (it has INT8 + FP8 and INT4 + FP8 version) in §2 but evaluates it in neither Figure 2 nor Table 1.

W3. No model-level speedup. All timings come from a standalone H=24/D=128 non-causal MHA microbenchmark; latency, FPS and seconds never appear in the paper. The FP16/BF16 fallback blocks (the first two and last two blocks in Wan2.2, the last 5 in MiniMax H3) are excluded from the benchmark although they cap the achieved gain. The abstract's "2.11–2.33× end-to-end attention speedup" is readily read as a model-level result.

W4. Most methodological component is prior art; the only remaining claim is unmeasured. All four techniques in §4 pre-exist and are attributed as such in §2: Hadamard rotation (QuaRot, QuIP#, SpinQuant), grouped rotation (Egiazarian et al. 2026; Choi et al. 2025), Q/K smoothing with probability scaling (SageAttention3), and NVFP4 scale selection (Four Over Six, ScaleSearch). §2 consistently frames the paper's own move as substitution and simplification — "We use wider Q/K rotation groups", "We use offline-calibrated Q/K offsets ... to simplify runtime processing", "a fixed target magnitude of 4 ... captures most of the benefit of per-block search". What remains is therefore (i) combination and kernel fusion, and (ii) the structural/residual decomposition. (i) is engineering, and (ii) is never measured: no experiment quantifies how much of the error is structural versus residual, and there is no ablation at all — nothing isolates rotation, centering, r=4, or α=768, while §4 asserts effects for two of them without supporting data. The paper therefore has little validated technical contribution beyond the fusion itself.

Questions for authors

Q1. What is the peak FP4 throughput of the RTX PRO 6000D used in Figure 2, and what utilization does 299–324 TOPS correspond to? If no public figure exists, can you instead perform benchmark on SOTA GEMM from cublas/cutlass to get the peak achievable throughout?

Q2. Can you add an 8-bit attention baseline — e.g. the INT4/FP8 and INT8/FP8 path of SageAttention2, which you cite in §2 — on both Figure 2 and Table 1? Since FP4 delivers only 2.11–2.33× over FlashAttention while offering 2× the peak of FP8, the marginal value of 4 bits over 8 bits is currently unquantified.

Q3. Can you report model-level timing — seconds per generated video, or FPS for different models, workload and attention methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.