acceptodds
Under review as a conference paper at ICLR 2027

Croissattn: A Complete FP4 Attention Backward

Abstract

FP4 training speeds up the linear layers of language models, but at long context the cost shifts to attention. Four of attention’s six products sit in the backward pass but existing FP4 recipes keep all or part of it in higher precision. CroissAttn extends native FP4 execution to both forward and all four backward attention products. It combines stochastic rounding of gradients and saved-factor recasts with a consistent softmax correction, preserving the quantized forward’s straight- through gradient in conditional expectation. An error decomposition separates noise propagated through the backward pass from new product noise. Query-row- owned accumulation and packed storage reduce the memory required by tiled execution. On long-context workloads, native B200 attention kernels achieve 1.5–2.2× backward speedups over BF16. Training from scratch reaches 10B tokens with validation gaps of 0.0439 nats at 500M parameters and 0.0451 at 1.7B. Native Qwen3-8B continued training reaches a 0.056-nat gap. Under FP4 serving, its checkpoint achieves lower validation loss than directly cast BF16- trained weights. These results connect complete FP4 attention backward to training quality, deployment, and long-context execution.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Quantization, backward pass, FP4 training, Stochastic noise control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_42NyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents a FP4 attention construction where the main attention forward and backward matmuls are done with NVFP4. It is not clear what the academic contribution of this paper is. It presents itself as a technical whitepaper where the authors describe an apparatus they implemented (FP4 attention) with no real technical novelty (i.e. no special rounding techniques beyond naive SR) on the ML side. Most of the empirical results focus on loss gaps to higher precision baselines, but since this paper is presenting a kernel, I would have expected analysis on the speed of the kernel, implementation bottlenecks, and other things more aligned with the engineering-focused nature of kernel construction. The writing is also hard to follow and seems to be heavily AI edited (which the authors do acknowledge). The empirical results themselves seem OK and in line with what one would expect from FP4 attention projections. I would like to see more analysis on the L2 error of the attention operator itself, comparisons with FP8 attention both in speedup and distortion, and other similar things.

Critical strengths

The authors achieved a wall clock time speedup with FP4 attention, which is generally hard given the difficulty of writing fast attention kernels and fast FP4 kernels. However, it is not clear how this speedup compares with FP8 attention, since the reported speedups of FP4 over BF16 are closer to what one would expect for a naive FP8:BF16 matmul speedup (of course this ignores the cost of exp and other scalar operations, which are known to be very expensive for low precision attention).

Critical weaknesses

See above. The paper is basically a tech report and not a paper, and it is hard to follow with hints of AI-heavy writing.

Questions for authors
  1. How does the L2 error of this FP4 attention construction compare with FP8 and BF16 vs a FP32 baseline?
  2. What does the kernel profile look like? How much time is taken by exp?
  3. Do you have any methods for reducing the error of FP4 attention? It seems like your method is a FP4 attention construction with no special care taken for error reduction. Certain things like rotating the Q K matrices with Hadamard matrices are cheap and can be done offline before the input, does this work here?
  4. Are there extreme numerical cases where your FP4 attention construction fails or gives catastrophic numerical failure? eg attention sinks, outlier values, extreme tokenwise gradients, etc?
2 comments

I am another reviewer of this paper. I agree with the reviewer that the AR in forward V quantization is a fatal problem in LLM Attention. The authors mentioned Attn-QAT and SageAttn3 to justify this work, but those are for DiTs (which don't use a causal mask), so the success of those works cannot justify the design in this work for LLMs.

Update: those are mainly for DiTs

Sorry, there are several misreadings of the above comments here and also significant misunderstandings by the papers in the field by reviewer nnN3. Attn-QAT and SageAttention3 are not solely for DiTs which do not use a causal mask. They also report language modeling results. We cite specific places from there paper and their code bases below.

Our implementation does not have the autoregressive leakage described. As already explained in our earlier responses to reviewer 42Ny, fully past KV tiles use standard NVFP4 microscaling, while causal-boundary tiles use key-local V scaling, where each scale depends only on its corresponding token. The scaling factor is folded into the softmax numerator, preserving autoregressivity while retaining native NVFP4 PVPV.

The claim that Attn-QAT and SageAttention3 are exclusively for noncausal DiTs is incorrect. Attn-QAT evaluates autoregressive LLMs (Section 3.3), as does SageAttention3, including FP4 inference on Qwen2.5 (Section 5.3, Table 5).

SageAttention3's forward API, lines 131–141, and FP4 V-quantization kernel, lines 300–329, use shared scales across 16 consecutive tokens, introducing precisely the future-token dependency our construction avoids. A separate causal-mask bug (issue #410) has also been reported. Attn-QAT explicitly excludes SageAttention3 from LLM comparisons due to causal numerical errors (Section 3.1), without attributing them to a specific cause.

We also dispute the earlier claim of lack of novelty from reviewer nnN3. SageAttention3 provides FP4 inference but only 8-bit training (Sections 3–4), while Attn-QAT uses fake quantization and BF16 backward matrix multiplications (Sections 2.3–2.4). These are different scenarios and if reviewer nnN3 believed that SageAttention3 and Attn-QAT were only for the non causal, diT use case, our method should be considered even more novel by them if it were a first in the field.

Reviewer_nnN3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work provides CroissAttn, which extends FP4 training to the attention backward pass. Building on the quantized attention forward and the high-precision auxiliary softmax correction of Attn-QAT [1], it replaces the attention backward with NVFP4 using stochastic rounding, with attention scores replayed via a two-pass forward. Experiments report from-scratch and continued-training loss gaps to BF16, kernel speedups over BF16, and long-context memory savings.

Critical strengths

This work provides native B200 NVFP4 kernels and native speed results on B200, and validates the feasibility of the training method on 1.7B and Qwen3-8B models as well as from-scratch runs up to 10B tokens.

Critical weaknesses
  1. The contribution and novelty of this work remain insufficiently justified. The work mainly extends Attn-QAT [1] which tackles the optimization challenge of forward-quantized Attention training to fully FP4 training with standard SR, but has not clearly demonstrated the additional contribution and the novelty, especially:
    • What are the challenges from the previous work (e.g., Attn-QAT [1], Quartet-II [3]) to fully FP4-Attention training, why do previous works fail to solve them, and how does this work solve the challenges rather than trivial engineering composition?
    • What is the motivation to adopt fully FP4 training in this work? Is it for the training speedup? How do the authors balance the training performance?
  2. The writing and the evidence in this paper do not clearly support the authors' claim. The writing of this work cannot show me the answers to the two core research questions above, and the current evidence shows that this work has not solved the problems and has not achieved the motivations:
    • The paper only shows the speedup over BF16, lacking comparison to standard FP8 baselines (e.g., FlashAttention-3 [2]) to justify the speedup motivation. Moreover, the training loss gap to full-precision is large, and the authors do not provide clear comparison to previous FP4 methods (e.g., Attn-QAT [1], Quartet-II [3]) to show where the improvement is.
    • Moreover, the paper lacks clear analysis on the overhead and the algorithm details of "two-pass forward" .

[1] Attn-QAT: 4-Bit Attention With Quantization-Aware Training. Zhang et al., ICML 2026

[2] Flashattention-3: Fast and accurate attention with asynchrony and low-precision., Shah et al., NeurIPS 2024

[3] Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation, Panferov et al., ICML 2026.


(Update: Nov. 7th on the authors' response) Thanks for your response. It only partly addresses my concerns. I would like to re-summarize as follows to make my concerns more explicit.

  1. The novelty and contributions are still unclear. TetraJet-v2 (Sec 2.3) has already used backward stochastic rounding (SR) for both operands in linear layers, with layouts adapted for backward. Attn-QAT (Alg. 2 and 3) already addresses forward/backward consistency. It is unclear what the methodological contribution is beyond engineering development on those principles.
    Regarding the zero-row-sum problem, Attn-QAT's correction actually already satisfies this in exact arithmetic. Using O′=PVfO'=PV_f and dP⋆=dOVf⊤dP^\star=dOV_f^\top:
    Di⋆:=⟨dOi,Oi′⟩=∑jPij⟨dOi,Vf,j⟩=∑jPijdPij⋆.D_i^\star := \langle dO_i,O_i^{\prime}\rangle = \sum_j P_{ij}\langle dO_i,V_{f,j}\rangle = \sum_j P_{ij}dP^\star_{ij}.
    dSij⋆=Pij(dPij⋆−Di⋆)/d,∑jdSij⋆=0.dS^\star_{ij} = P_{ij}(dP^\star_{ij}-D_i^\star)/\sqrt{d}, \qquad \sum_j dS^\star_{ij}=0.
    It is unclear what the proposed fix adds beyond Attn-QAT, and a comparison on the actual dSdS row-sum value is needed, rather than only dQdQ errors.

  2. Unclear efficiency improvement over Attn-QAT. My question is: how much does FP4 attention backward provide for the overall acceleration? Moreover, two-pass forward introduces overhead, so an efficiency and accuracy comparison with online-softmax-style AttnQAT forward and further forward-backward latency breakdown (especially for the added operations) are needed to clarify that the added overhead is worthwhile.

  3. Concerns on presentation. Even after careful reading, the paper does not clearly connect the design choices to their novelty and computational costs. This concern is similar to Reviewer 42Ny.

Questions for authors
  1. Can the authors explain why the from-scratch loss gap in Table 3 grows from 0.038 at 1B tokens to 0.044 at 10B tokens? Is it worthwhile to adopt fully FP4 given such a growing gap?

  2. Can the authors provide clear results and discussion against standard FP4 attention baselines (especially Attn-QAT) on training accuracy, speedup, and memory? This comparison is crucial to justify the motivations: the fully FP4 training should provide meaningful speedup over QAT while maintaining the performance.

  3. Can the authors explain why there is only <2×<2\times speedup over BF16 with fully FP4? The authors only provide speedup with FP4 for all the components, so how much speedup is from FP4 attention backward design?

  4. I suggest the writing of this work be improved. Some terms are not clearly defined before using them, like "two-pass forward", "score replay", "HP (higher precision)" (what precision?). Can the authors clearly explain them?

2 comments
Reviewer_nnN3OpenReview ↗

Justification for my rating after rebuttal

Thanks for the response. I appreciate the engineering contribution of implementing a complete native NVFP4 attention backward and demonstrating from-scratch training. However, the rebuttal does not resolve my central concerns regarding methodological novelty and the accuracy–efficiency trade-off, and some of the key contributions newly emphasized in the rebuttal are not sufficiently articulated and supported in the submitted paper; my recommendation remains unchanged.

We're confused by your continued assessment of novelty, especially given your comment in reviewer 42Ny's thread:

"The authors mentioned Attn-QAT and SageAttn3 to justify this work, but those are for DiTs (which don't use a causal mask)."

After we pointed out that both papers evaluate autoregressive LLMs and pointed out issues with the codebases, cited sections in the papers, you updated your comment to say "mainly" for DiTs. However, you left the rest unchanged, including your agreement that autoregressive leakage was a fatal issue, despite our earlier explanation of how our implementation avoids it.

Your original review already acknowledged our native NVFP4 kernels, speedups, and from-scratch training. Yet you characterized our method as mainly extending Attn-QAT, cited it again in your November 7 response to question novelty, and then described it as a method for noncausal DiTs. We find it difficult to reconcile these assessments, particularly when Attn-QAT retains higher-precision backward matrix multiplications and our method implements the complete backward in native NVFP4.

We have also explained the mathematical contributions, including dependent stochastic products, conditional-mean guarantees, and error propagation. We are concerned that these distinctions remain unaddressed while the review continues to question methodological novelty.

Thank you for your time and participation in the review process.

Reviewer_2qUNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work builds on Attn-QAT’s matching of forward and backward quantization and high-precision auxiliary output O', extending its higher-precision backward pass to FP4. This is quite an interesting and useful work that extends Attn-QAT to make training completely fp4 native.

CroissAttn runs both forward and all four backward attention products with FP4 operands. It combines stochastic rounding, forward replay, and a higher-precision softmax correction to preserve STE gradient. A tiled execution schedule reduces memory use. experiments report small loss gap compare to BF16, while B200 kernels achieve 1.5-2.2x backward speedups. Quality is evaluated at 2K context and longer-context tests focus on performance and numerical behavior.

Critical strengths
  1. The backward ablation directly tests the contributions. Table 2 holds the FP4 forward fixed and changes the backward precision. The 10B-token runs and 8B continued-training results provide further evidence.

  2. The paper is clear on explaining implementation choices. recasting, normalization, and softmax correction all affect which gradient is computed.

Critical weaknesses
  1. Long-context training quality remains untested. All quality experiments use 2k-token contexts, even though long context is the main motivation. The longer-context backward checks use synthetic rows and exact saved factors, so they leave out the recasting noise introduced by the full method. They also omit accumulated dK and dV errors.

  2. The BF16 memory baseline expands KV to 32 heads. The paper uses compact gqa layout. Complete-step timings also include FP4 projections, and the longest-context comparison changes checkpointing. The paper discloses these choices, but the results leave the incremental benefit of complete FP4 attention unclear. A matching compact-GQA BF16 baseline would help clarify this.

Questions for authors
  1. Which BF16 attention implementation are used in Table 5? Please provide the workload shapes and clarify whether FP4 timings include the two-pass forward, stochastic recasts, auxiliary correction, and every pass needed to produce dQ, dK, dV.

  2. Comparison needed to assess the systems contribution: What speed and memory gains remain with compact GQA K/V and matched checkpointing? How much faster is the complete method than the same FP4-projection setup with BF16 attention?

  3. can you provide a longer-context quality experiment using the complete native recipe? If a training run is infeasible, actual model tensors at longer contexts would also help.

  4. could the authors clarify their choices around the usage of smoothing?

Thanks

1 comment
AuthorsOpenReview ↗

Response to reviewer 2qUN

Thank you for the thoughtful review! We agree that the original submission could better isolate the systems benefits. We also wanted to give sufficient space to the mathematical challenges of controlling error throughout the complete FP4 backward.

1. Kernel implementation and timing.

Table 5 reports native B200 causal attention kernels (head dimension 128), with 1.5–2.2× backward speedup over BF16. Our implementation uses two-pass forward normalization, score replay from the original Q/K codes, stochastic saved-factor recasts, auxiliary softmax correction, and query-row-owned dQ accumulation. Complete-step timings include projections, attention, packing, auxiliary correction, optimizer work, and exposed communication.

The BF16 kernel used is CuDNN.

2. Memory and systems contribution.

CroissAttn uses 0.49× the peak memory of cuDNN BF16 for a full attention training step (861 MB vs. 1,753 MB; B300, B=4, 32 query / 8 KV heads, T=8,192, D=128, causal; auxiliary recomputed in the backward).

Query-row ownership avoids a global partial-dQ buffer, while packed NVFP4 storage reduces retained activations. The original 512K comparison also differs in GQA expansion and checkpointing, so we agree it should be interpreted as a capacity comparison.

3. Long-context training quality.

We have now trained Qwen3-8B with the complete native FP4 recipe at 32K and 128K contexts, using three paired seeds:

Training contextGap to BF16 (nats)RULER difference (points)
2K0.0555 ± 0.0005—
32K0.0609 ± 0.0023−0.6 ± 0.3
128K0.0656 ± 0.0040−1.1 ± 0.1

These directly address the lack of long-context training evidence. We also expanded the Qwen3-8B frozen-tensor diagnostics to three checkpoints with 10,000 stochastic draws each, including saved-factor recasts and dQ/dK/dV. The measured/predicted squared-error slope is 1.008 ± 0.021, with R² = 0.984 ± 0.004.

4. Smoothing.

We use product-preserving diagonal smoothing (α = 0.3) to control activation outliers in FP4 projections. It is independent of the attention backward method, and the same projection recipe is used in the projection-only controls. Across ten seeds at 1.7B, smoothing reduces the complete-recipe gap from 0.355 to 0.128 nats, with an isolated attention increment of 0.071 ± 0.003 nats.

We appreciate these suggestions and will clarify the distinction between measured kernel performance, peak execution memory, and full-system benefits.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Responses

Final Author Comment: Revisions and Rebuttal Summary

We thank the reviewers and Area Chairs for their feedback. We have revised and uploaded the manuscript to improve clarity and explanations, particularly to highlight the mathematical contributions, causal forward construction, and relationship to prior work. The underlying method and main contributions are unchanged. We have also included some of the additional experiments from our rebuttal responses.

  • Novelty and numerical analysis (nnN3, 42Ny): We clarified why complete FP4 attention backward requires more than applying stochastic rounding to individual matrix products, including the gradient target, sampled centering, saved-factor recasting, and error propagation. These were present in the original submission but are now explained clearly in the main text.

  • Causal forward (42Ny, nnN3): We added the scaling algorithm, tilewise factorization, and also added the prefix-invariance proof to the main presentation, along with small numerical checks and boundary-scaling ablations.

  • Training accuracy (42Ny, nnN3, 2qUN): We added attention-only training through 30B tokens, reaching an approximately 0.011-nat gap to BF16, and further evaluations across 8B–32B models. These complement our existing complete-FP4 training results and help separate forward quantization error from backward error. We included the long-context training and RULER experiments discussed in our response to 2qUN.

  • Performance and ablations (all reviewers): We addressed the forward/backward timing breakdown, memory comparisons, and comparisons with prior methods, and included the requested FP8 error, Hadamard, and numerical stability experiments.

We appreciate the reviewers' suggestions on presentation and believe the revised manuscript makes the original contributions considerably easier to evaluate. We would also like to draw the Area Chairs' attention to how some assessments evolved during the discussion. In particular, several contributions initially characterized as lacking technical novelty were later acknowledged, including the complete native FP4 backward and causal forward construction (42Ny). Reviewer 42Ny also indicated that acceptance could be appropriate following a rewrite, which we have now done. We have also clarified factual concerns surrounding Attn-QAT, SageAttention3 in our exchanges with nnN3. We hope the final assessment will reflect these clarifications and the revised manuscript, rather than concerns based on the earlier presentation or interpretations that were corrected during the discussion.

Thanks for the follow-up and for acknowledging the contributions of the native FP4 backward and the causal forward construction! We have revised the manuscript to address the presentation concerns you raised. The causal scaling algorithm, tilewise factorization, prefix-invariance proof, and backward derivation are now written fully in the main text. We have also included some of the numerical ablations discussed above. We also wanted to address your remaining concern about forward accuracy. We have added attention-only training experiments, keeping all projections in BF16. At 500M parameters, native FP4 attention reaches 30B training tokens with an average loss gap of ~0.011 nats across three paired seeds. We also evaluated 8B, 14B, and direct casting on 32B models, reaching gaps of approximately 0.009–0.014 nats after 200M tokens on the reported long-text evaluation.

In terms of the 32K experiment, comparing FP4 endpoint against initial BF16 loss does not account for the substantial improvement from the initial FP4 evaluation. The FP4-served loss decreases from approximately 2.676 to 2.521, while the matched BF16 run decreases from 2.504 to 2.490. We fully agree that the remaining gap matters, but these results do show that this method can be practical.

We appreciate your willingness to reconsider acceptance following a rewrite. We hope the revised manuscript (uploaded) and additional experiments address your remaining concerns, and would appreciate your reconsideration in light of these results. Thank you for your reviews and your time during this rebuttal process, we appreciate the feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.