- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper presents a FP4 attention construction where the main attention forward and backward matmuls are done with NVFP4. It is not clear what the academic contribution of this paper is. It presents itself as a technical whitepaper where the authors describe an apparatus they implemented (FP4 attention) with no real technical novelty (i.e. no special rounding techniques beyond naive SR) on the ML side. Most of the empirical results focus on loss gaps to higher precision baselines, but since this paper is presenting a kernel, I would have expected analysis on the speed of the kernel, implementation bottlenecks, and other things more aligned with the engineering-focused nature of kernel construction. The writing is also hard to follow and seems to be heavily AI edited (which the authors do acknowledge). The empirical results themselves seem OK and in line with what one would expect from FP4 attention projections. I would like to see more analysis on the L2 error of the attention operator itself, comparisons with FP8 attention both in speedup and distortion, and other similar things.
Critical strengths
The authors achieved a wall clock time speedup with FP4 attention, which is generally hard given the difficulty of writing fast attention kernels and fast FP4 kernels. However, it is not clear how this speedup compares with FP8 attention, since the reported speedups of FP4 over BF16 are closer to what one would expect for a naive FP8:BF16 matmul speedup (of course this ignores the cost of exp and other scalar operations, which are known to be very expensive for low precision attention).
Critical weaknesses
See above. The paper is basically a tech report and not a paper, and it is hard to follow with hints of AI-heavy writing.
Questions for authors
- How does the L2 error of this FP4 attention construction compare with FP8 and BF16 vs a FP32 baseline?
- What does the kernel profile look like? How much time is taken by exp?
- Do you have any methods for reducing the error of FP4 attention? It seems like your method is a FP4 attention construction with no special care taken for error reduction. Certain things like rotating the Q K matrices with Hadamard matrices are cheap and can be done offline before the input, does this work here?
- Are there extreme numerical cases where your FP4 attention construction fails or gives catastrophic numerical failure? eg attention sinks, outlier values, extreme tokenwise gradients, etc?
2 comments
Sorry, there are several misreadings of the above comments here and also significant misunderstandings by the papers in the field by reviewer nnN3. Attn-QAT and SageAttention3 are not solely for DiTs which do not use a causal mask. They also report language modeling results. We cite specific places from there paper and their code bases below.
Our implementation does not have the autoregressive leakage described. As already explained in our earlier responses to reviewer 42Ny, fully past KV tiles use standard NVFP4 microscaling, while causal-boundary tiles use key-local V scaling, where each scale depends only on its corresponding token. The scaling factor is folded into the softmax numerator, preserving autoregressivity while retaining native NVFP4 .
The claim that Attn-QAT and SageAttention3 are exclusively for noncausal DiTs is incorrect. Attn-QAT evaluates autoregressive LLMs (Section 3.3), as does SageAttention3, including FP4 inference on Qwen2.5 (Section 5.3, Table 5).
SageAttention3's forward API, lines 131–141, and FP4 V-quantization kernel, lines 300–329, use shared scales across 16 consecutive tokens, introducing precisely the future-token dependency our construction avoids. A separate causal-mask bug (issue #410) has also been reported. Attn-QAT explicitly excludes SageAttention3 from LLM comparisons due to causal numerical errors (Section 3.1), without attributing them to a specific cause.
We also dispute the earlier claim of lack of novelty from reviewer nnN3. SageAttention3 provides FP4 inference but only 8-bit training (Sections 3–4), while Attn-QAT uses fake quantization and BF16 backward matrix multiplications (Sections 2.3–2.4). These are different scenarios and if reviewer nnN3 believed that SageAttention3 and Attn-QAT were only for the non causal, diT use case, our method should be considered even more novel by them if it were a first in the field.
I am another reviewer of this paper. I agree with the reviewer that the AR in forward V quantization is a fatal problem in LLM Attention. The authors mentioned Attn-QAT and SageAttn3 to justify this work, but those are for DiTs (which don't use a causal mask), so the success of those works cannot justify the design in this work for LLMs.
Update: those are mainly for DiTs