acceptodds
Under review as a conference paper at ICLR 2027

Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs

Abstract

Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Precision Training, Quantization, Attention, Large Language Model, Efficient AI, Numerical Stability

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_QgpzOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates FP8 training and identifies a failure mode called stale delta. The FP8 attention failure can be attributed to the saved-output correction that uses quantized operands, causing mismatches between probabilities and gradients. Therefore, this paper proposes the Delta-Matching method, which computes the correction directly from the pre-quantization probabilities and the products of FP8 backward operands. Experiments across several architectures, model sizes, and context lengths demonstrate substantial recovery of training quality relative to naive FP8 attention.

Critical strengths
  1. This paper shows a clear diagnostic contribution to FP8 attention. Decomposing the failure into probability-quantization and output-gradient-quantization mismatches provides a concrete explanation for the saved-output shortcut's failure.
  2. Delta-Matching is derived from the softmax backward invariant, avoids BF16 fallbacks or an auxiliary forward output, and is integrated into FP8 attention kernels while keeping all seven attention-core matmuls in FP8.
  3. This paper presents extensive empirical validation. The evaluation covers multiple hybrid architectures, attention variants, model sizes from 569M to 5.29B, and context extension to 64K.
Critical weaknesses
  1. The efficiency benefit is not fully demonstrated. Delta-Matching adds a separate correction pass, increasing backward time by roughly 21–28% over stale delta. At head dimension 128, it is slower than cuDNN BF16. The paper reports kernel-level throughput but no end-to-end training throughput or memory results.
  2. The proof assumes normalized probabilities, identical effective V operands, FP32 accumulation, and no rounding. The experiments stop at 5.29B parameters and 30B tokens. It is unclear whether the fix remains sufficient at larger scales and longer training.
Questions for authors
  1. What is the measured end-to-end training speedup and memory overhead of Delta-Matching versus BF16 and cuDNN/TE FP8?
  2. How does Delta-Matching behave beyond 5.29B parameters and longer token budgets? Does it remain robust on larger settings?
3 comments
AuthorsOpenReview ↗

End-to-end Results and Scaling Up

1. The efficiency benefit is not fully demonstrated. Delta-Matching adds a separate correction pass, increasing backward time by roughly 21–28% over stale delta. At head dimension 128, it is slower than cuDNN BF16. The paper reports kernel-level throughput but no end-to-end training throughput or memory results.

Thank you for the reminder. We have now added end-to-end throughput measurements for training with the Megatron framework. It is up to 1.163× over cuDNN BF16 for the 9.3B model at 256K context on eight GB200 GPUs with context parallelism of eight. Throughput increases from 7,747 to 9,011 tokens/s/GPU. On H200, the 9.3B model reaches up to 1.073× FA3 BF16 throughput. Table 27 in Appendix C.9 reports all 24 configurations, including the 1.67B cases.

Table 27. End-to-end training throughput in tokens/s/GPU. Speedup is the ratio of Delta-Matching throughput to BF16 throughput. GPU counts are totals across nodes.

ModelGPUs / baselineDP/CPContextTokens/GPUMBSBF16DM-FP8Speedup
9.3B32 H200 / FA332/18K16K210,20310,3011.010×
9.3B32 H200 / FA332/116K16K19,7719,8721.010×
9.3B32 H200 / FA34/8128K16K15,7136,0131.053×
9.3B32 H200 / FA34/8256K32K13,3543,6001.073×
9.3B32 H200 / FA32/16256K16K13,7794,0301.066×
9.3B32 H200 / FA31/32512K16K12,4032,4981.040×
9.3B4 GB200 / cuDNN4/18K16K219,58319,8561.014×
9.3B4 GB200 / cuDNN4/116K16K118,75819,2661.027×
9.3B4 GB200 / cuDNN4/132K32K115,80816,2891.030×
9.3B8 GB200 / cuDNN8/18K16K219,61119,8681.013×
9.3B8 GB200 / cuDNN8/116K16K119,04919,5471.026×
9.3B8 GB200 / cuDNN8/132K32K115,94816,6471.044×
9.3B8 GB200 / cuDNN8/164K64K112,64813,6191.077×
9.3B4 GB200 / cuDNN1/464K16K114,49215,3761.061×
9.3B4 GB200 / cuDNN1/4128K32K110,63711,5051.082×
9.3B8 GB200 / cuDNN1/8128K16K111,25312,5541.116×
9.3B8 GB200 / cuDNN1/8256K32K17,7479,0111.163×
1.67B8 H200 / FA38/18K32K444,25943,8990.992×
1.67B8 H200 / FA38/116K32K242,00641,6280.991×
1.67B8 H200 / FA38/132K32K137,70036,8890.979×
1.67B8 H200 / FA32/464K32K228,91928,8160.996×
1.67B8 H200 / FA31/8128K32K222,12121,7460.983×
1.67B16 H200 / FA31/16256K32K214,41714,0460.974×
1.67B32 H200 / FA31/32512K32K28,7828,1810.932×

DP and CP denote data and context parallelism; MBS is micro-batch-size. Measurements average five steps after three warm-up steps, excluding data loading and checkpoint saves. Training settings are matched, with normalization and RoPE additionally fused in the FP8 path.

2. The proof assumes normalized probabilities, identical effective V operands, FP32 accumulation, and no rounding. The experiments stop at 5.29B parameters and 30B tokens. It is unclear whether the fix remains sufficient at larger scales and longer training.

The core of the training dynamics experiments is showing that Delta-Matching prevents the outlier-channel growth observed under stale delta, which is why we are confident it works when scaling up. We have reorganized this experiment from the stale delta and Delta-Matching section into the experiments section to make this clearer (Section 4.3, Figure 5).

To further empirically verify this, we pretrained the 1.67B model on the 60B-token FinePDFs-Edu dataset and scaled pretraining to 9.27B parameters. We tested the 9.27B model at both the same peak learning rate as the smaller models and a lower rate to account for the larger model size. Here are the validation CE results.

ExperimentPeak learning rateBF16Delta-MatchingStale FP8
9.27B GDN/GQA, 30B Nemotron-CC tokens2.4×10−32.4\times10^{-3}1.31411.31521.8728
9.27B GDN/GQA, 30B Nemotron-CC tokens1.2×10−31.2\times10^{-3}1.29931.29851.5595
1.67B GDN/GQA, 60B FinePDFs-Edu tokens2.4×10−32.4\times10^{-3}1.75291.75392.6692

Each comparison uses one initialization per method and matched data order because of pretraining cost. At 9.27B and 2.4×10−32.4\times10^{-3}, all nine commonsense scores are within 0.86 percentage points of BF16, although RULER-8K is lower at 51.99% versus 58.83%. Tables 2 and 4(a) give the downstream results, and Appendix C.8.1 compares both learning rates.

Dear Reviewer Qgpz,

Thank you again for your thoughtful review and helpful suggestions. We have added experiments and revised the paper to address your comments. We would be grateful to hear your thoughts on our responses and updated results, and would be happy to discuss any remaining questions.

With thanks and best regards,

The Delta-Matching authors

Thank you for your efforts and timely response, which have addressed my concerns. I have raised my scores accordingly.

Reviewer_gXBdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper finds that a mismatch in the softmax backward would cause optimization problem in native 8-bit attention training. This mismatch violates the mathematical invariant: the row-sum of gradient of softmax score should be zero, but incorrect implementation gives a non-zero result. This work fixes this problem through "delta-matching" and improves training over naive FP8 baselines.

Critical strengths
  1. The paper provides concrete numerical analysis and derivation of the 8-bit attention backward, and solves the gradient-invariant mismatch problem of the naive implementation.
  2. The paper validates the method across architectures and scales, and provides solid evidence that ensuring this optimization invariant is important for training performance.
Critical weaknesses
  1. The proposed correction method has limited conceptual novelty beyond prior attention corrections (especially, Attn-QAT [1]), and lacks essential discussion in the paper. Especially, I find this correction mathematically equivalent to Attn-QAT [1] at high-precision backward, and Attn-QAT method can simply extend to the low-precision backward. The authors should delineate the difference and clarify the contribution.

    This is my derivation for your reference: consider one query row and use column vectors. Let P\mathbf P denote the probabilities before probability quantization, P^\widehat{\mathbf P} their quantized-dequantized representation. The quantized output and Attn-QAT auxiliary output are

    \qquad \mathbf O'=\widehat{\mathbf V}^{\top}\mathbf P.$$ **With high-precision backward**, Attn-QAT computes $$\mathrm{d}\mathbf P=\widehat{\mathbf V}\mathrm{d}\mathbf O, \qquad \delta^{\text{Attn-QAT}} =\mathrm{d}\mathbf O^{\top}\mathbf O'.$$ Here is a mathematical equivalence: \$\\delta^{\\text{Attn-QAT}} =\\mathrm{d}\\mathbf O^{\\top}\\mathbf O'=\\mathrm{d}\\mathbf O^{\\top}\\widehat{\\mathbf V}^{\\top}\\mathbf P=(\\widehat{\\mathbf V}\\mathrm{d}\\mathbf O)^{\\top}\\mathbf P=\\mathbf P^{\\top}\\mathrm{d}\\mathbf P.\$ This is exactly the matched correction that solves "the probability mismatch", which shows that prior work has already solves this problem and the authors should clearly discuss this. **In native FP8 backward**, adapt the same correction for $\delta$ by replacing $\mathrm{d}\mathbf O$ with its quantized-dequantized operand $\widehat{\mathrm{d}\mathbf O}$. Keeping the same value matrix, the probability gradient is $$\widehat{\mathrm{d}\mathbf P} =\widehat{\mathbf V}\widehat{\mathrm{d}\mathbf O}.$$ The adapted Attn-QAT correction is then mathematically equivalent to Delta-Matching: $$\begin{aligned} \delta^{\text{Attn-QAT-Adapted}} &=\widehat{\mathrm{d}\mathbf O}^{\top}\mathbf O'\\\\ &=\widehat{\mathrm{d}\mathbf O}^{\top} \widehat{\mathbf V}^{\top}\mathbf P\\\\ &=(\widehat{\mathbf V}\widehat{\mathrm{d}\mathbf O})^{\top}\mathbf P\\\\ &=(\widehat{\mathrm{d}\mathbf P})^{\top}\mathbf P\\\\ &={\mathbf P^{\top}\widehat{\mathrm{d}\mathbf P}} ={\delta^{\text{Delta-Matching}}}. \end{aligned}$$ Consequently, using the same probability gradient, both give the same softmax score gradient: $$\begin{aligned} \mathrm{d}\mathbf S^{\text{Attn-QAT-Adapted}} &=\mathbf P\odot\left( \widehat{\mathrm{d}\mathbf P} -\delta^{\text{Attn-QAT-Adapted}}\mathbf 1\right)\\\\ &=\mathbf P\odot\left( \widehat{\mathrm{d}\mathbf P} -\delta^{\text{Delta-Matching}}\mathbf 1\right) =\mathrm{d}\mathbf S^{\text{Delta-Matching}}. \end{aligned}$$ My concern is that: although the native-FP8 implementation and training validation are useful, the contribution in the proposed correction rule needs to be clarified.
  2. Comparisons with these prior correction approaches are needed to establish the benefit of FP8 backward. The Attn-QAT with the same FP8 forward with BF16 backward is an important baseline, answering whether low-precision backward provides a useful advantage over retaining higher-precision backward. An adapted Attn-QAT FP8 backward using dO^\widehat{\mathrm{d}\mathbf O} to compute the δ\delta would also directly compare two algebraically equivalent correction paths:

    =\mathbf P^{\top}\widehat{\mathrm{d}\mathbf P}, \qquad \delta^{\text{Attn-QAT-Adapted}} =\widehat{\mathrm{d}\mathbf O}^{\top}\mathbf O'.$$ Therefore, the paper should compare with Attn-QAT baseline on training quality and efficiency.

[1] Attn-QAT: 4-Bit Attention With Quantization-Aware Training. Zhang et al., ICML 2026

Questions for authors

The above two weaknesses are main concerns that affects my recommendations. Corresponding questions for the authors:

  1. Can the author provide more analysis and evidence to clarify the contribution over prior work (Attn-QAT [1])? Especially, clarify the equivalence in the methodology, and how the authors' implementation differs from the Attn-QAT approach (See weakness 1).
  2. Do Attn-QAT (with high-precision backward) and adapted Attn-QAT (with low-precision backward, see weakness 1) already solves the invariant problem in this work? The authors should clearly discuss in the paper.
  3. What is "Push Consistency" "Sharpened-Row Preference" "gain increase" in Figure 3(b)? Can you define in the main text or caption? Also, the trajectory of prior correction methods (like Attn-QAT) could be included in those figures to establish the contribution clearer.
  4. Can you explain the d=128 FA3 forward throughput difference in Table 11 (Forward 386.9 TFLOP/s in this work's reproduction versus 668 TFLOP/s in the official paper [2, Figure 5(d)])? Clarifying this difference may provide stronger evidence that fully FP8 training can provide meaningful speedup benefits.

Overall, I think the paper's numerical implementation and training evidence are useful to show the optimization matching is important for 8-bit attention training, but the paper lacks a clear discussion on the method's novelty upon prior work which already equivalently resolves this mismatch.

Also, the evidence for the speedup benefit could be further strengthened, so that the speedup motivation of adopting low-precision backward can be justified. Otherwise, the practical motivation for applying the proposed FP8 backward over a BF16 backward remains unclear.

[1] Attn-QAT: 4-Bit Attention With Quantization-Aware Training. Zhang et al., ICML 2026

[2] FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. Shah et al., 2024

7 comments
AuthorsOpenReview ↗

Elaboration of contribution and notation

1. Can the author provide more analysis and evidence to clarify the contribution over prior work (Attn-QAT [1])? Especially, clarify the equivalence in the methodology, and how the authors' implementation differs from the Attn-QAT approach (See weakness 1).

We referred to Attn-QAT briefly in our background and related work section, but we do agree that the method deserves a more detailed discussion. The takeaway is that the consistency principle is shared, while the computational realizations differ and we look at different setup. Delta-Matching avoids the auxiliary higher-precision forward matmul.

Attn-QAT focuses on post-training QAT for low-bit inference. It uses simulated NVFP4 quantization with BF16 attention matmuls during training, reserving native low-bit matmuls for inference. Delta-Matching targets native FP8 training, keeping all attention-core matmuls in FP8.

We have expanded this discussion in Sections 2 and 3.3 and Appendix A.3.1. Table 5 now compares the complete attention core. For MM valid query–key pairs per head and sequence and head dimension dd, published Attn-QAT's algorithms require 16Md16Md BF16 GEMM FLOPs, compared with 18Md18Md FP8 GEMM FLOPs for Delta-Matching. At H100's approximately 2×2\times FP8-to-BF16 peak throughput ratio, this gives Delta-Matching about 44% less nominal matmul time.

A hypothetical native-FP8 adaptation of Attn-QAT would use 14Md14Md FP8 plus 2Md2Md BF16 GEMM FLOPs, approximately tying our nominal matmul time on H100. Delta-Matching avoids its auxiliary higher-precision matmul and saved output, while its correction pass recomputes softmax exponentials. These peak-rate estimates exclude scalar work and memory traffic.

2. Do Attn-QAT (with high-precision backward) and adapted Attn-QAT (with low-precision backward, see weakness 1) already solves the invariant problem in this work? The authors should clearly discuss in the paper.

Yes, it also resolves the issue in exact arithmetic, with the extra higher-precision forward product discussed above. For one query, using column vectors and matched operands,

\delta_i^{\mathrm{aux}} =\widehat{dO}_i^\top O'_i =P_i^\top\widehat{dP}_i =\delta_i^{\mathrm{matched}}.$$ With normalized $P_i$, both give the same score gradient and zero row sum before the $dS$ cast. The high-precision-backward case uses $dP_i=\widehat V\,dO_i$ in the same identity. Appendix A.3.1 gives the derivation. ### 3. What is "Push Consistency" "Sharpened-Row Preference" "gain increase" in Figure 3(b)? Can you define in the main text or caption? Also, the trajectory of prior correction methods (like Attn-QAT) could be included in those figures to establish the contribution clearer. These are specifically defined in Appendix B.4. We have now added brief definitions to the caption of Figure 5, originally Figure 3. - **Sharpened-row preference** is $\bar\Delta_c/(s_c/\sqrt{192})$, where $\bar\Delta_c$ and $s_c$ are the mean and sample SD of 192 window/head differences. Each difference compares mean channel contributions in rows with $\varepsilon_i<0$ and $\varepsilon_i>0$. It is a descriptive statistic because heads within a window can be correlated. - **Push consistency** is $-m/\sqrt v$ from each run's actual Adam moments for the key RMSNorm gain. Positive values indicate upward update pressure. - **Gain** is the learned key RMSNorm weight $g_{k,c}$ before RoPE, initialized to one. Panel (c) shows its growth. For BF16 and Delta-Matching, the row split uses a hypothetical stale-FP8 residual, while push consistency uses the actual optimizer state. Our Attn-QAT comparison is analytical in Appendix A.3.1; Figure 5 contains the measured BF16, stale-FP8, and Delta-Matching runs. ### 4. Can you explain the d=128 FA3 forward throughput difference in Table 11 (Forward 386.9 TFLOP/s in this work's reproduction versus 668 TFLOP/s in the official paper [2, Figure 5(d)])? Clarifying this difference may provide stronger evidence that fully FP8 training can provide meaningful speedup benefits. Sure! The two numbers come from different benchmark configurations. The published 668 TFLOP/s point uses 16-head MHA; our 386.9 TFLOP/s result uses native GQA with 16 query and four KV heads through the training autograd wrapper of the FA3 3.0.0 wheel. Both use BF16, causal 8K attention, batch size two, head dimension 128, and H100 SXM hardware. We now state these settings in Appendix C.2 and Table 12. Our forward latency is $1.421\pm0.010$ ms, with one SE over five timing rounds. The same workload reaches $601.6\pm0.7$ TFLOP/s with cuDNN BF16. The measurements do not isolate GQA as the cause of the full gap. We also added training-step measurements, reaching 1.163× cuDNN BF16 throughput at 9.3B and 256K context on GB200 (Appendix C.9, Table 27).

Table 27. End-to-end training throughput in tokens/s/GPU. Speedup is the ratio of Delta-Matching throughput to BF16 throughput. GPU counts are totals across nodes.

ModelGPUs / baselineDP/CPContextTokens/GPUMBSBF16DM-FP8Speedup
9.3B32 H200 / FA332/18K16K210,20310,3011.010×
9.3B32 H200 / FA332/116K16K19,7719,8721.010×
9.3B32 H200 / FA34/8128K16K15,7136,0131.053×
9.3B32 H200 / FA34/8256K32K13,3543,6001.073×
9.3B32 H200 / FA32/16256K16K13,7794,0301.066×
9.3B32 H200 / FA31/32512K16K12,4032,4981.040×
9.3B4 GB200 / cuDNN4/18K16K219,58319,8561.014×
9.3B4 GB200 / cuDNN4/116K16K118,75819,2661.027×
9.3B4 GB200 / cuDNN4/132K32K115,80816,2891.030×
9.3B8 GB200 / cuDNN8/18K16K219,61119,8681.013×
9.3B8 GB200 / cuDNN8/116K16K119,04919,5471.026×
9.3B8 GB200 / cuDNN8/132K32K115,94816,6471.044×
9.3B8 GB200 / cuDNN8/164K64K112,64813,6191.077×
9.3B4 GB200 / cuDNN1/464K16K114,49215,3761.061×
9.3B4 GB200 / cuDNN1/4128K32K110,63711,5051.082×
9.3B8 GB200 / cuDNN1/8128K16K111,25312,5541.116×
9.3B8 GB200 / cuDNN1/8256K32K17,7479,0111.163×
1.67B8 H200 / FA38/18K32K444,25943,8990.992×
1.67B8 H200 / FA38/116K32K242,00641,6280.991×
1.67B8 H200 / FA38/132K32K137,70036,8890.979×
1.67B8 H200 / FA32/464K32K228,91928,8160.996×
1.67B8 H200 / FA31/8128K32K222,12121,7460.983×
1.67B16 H200 / FA31/16256K32K214,41714,0460.974×
1.67B32 H200 / FA31/32512K32K28,7828,1810.932×

DP and CP denote data and context parallelism; MBS is micro-batch-size. Measurements average five steps after three warm-up steps, excluding data loading and checkpoint saves.

Reviewer_gXBdOpenReview ↗

Reply to Authors' Rebuttal

I appreciate the authors' prompt feedback and additional discussion; it addresses part of my concerns, but not my most important ones. I am still concerned about insufficient comparison to a strong enough baseline, since setting differences cannot be a convincing justification for missing the actual efficiency and training quality comparison on Attn-QAT. Attn-QAT with native FP8 forward and its BF16 backward, or a simple FP8 backward adaptation, which I mentioned in my original review, should be considered as a baseline to solidify the contribution.

Technically, the delta δ\delta of this work is based on the reduction in token dimension NN (and based on dP^\widehat{dP}), while FA and Attn-QAT adopt δ\delta based on reduction in hidden dimension dd (and based on OO / O′O' and dOdO / dO^\widehat{dO}). The computation graph is highly different, and the difference here would be a concern for long context if N≫dN \gg d, so the efficiency claim should not only be based on peak GEMM FLOPs. But currently, I can only see the GEMM complexity analysis against this baseline without actual efficiency & accuracy results and a rigorous comparison.

I should clarify that this is not an additional requirement, but is already presented in "the paper should compare with Attn-QAT baseline on training quality and efficiency" in my original review. The technical detail I present here should be considered by the authors as well when the authors thoroughly compare this baseline.

Again, I still appreciate the authors' contribution, but according to the current paper and discussion, I do not suggest this paper be accepted this time, because the critical concern is still not resolved. I think a thorough and fair conceptual & experimental comparison to the closest prior method is necessary for a good paper, but currently I suggest that this paper be largely improved in this aspect. (Especially, Attn-QAT has already largely solved a proposed mathematical challenge which is claimed as a major contribution of this work, and the authors acknowledge this in the rebuttal)


(Update after another round of discussion: I raise my recommendation after the authors give new revision and results. See below.)

Reviewer_omsJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper mainly identifies the numerical inconsistency between the quantized results used in the forward and backward of FP8 Attention. Since FlashAttention uses the saved forward output to compute the Softmax backward correction term, leading to the Stale Delta problem, which violates the Zero-Row-Sum Invariant of the Softmax backward gradients. To address this issue, the paper proposes Delta-Matching, which recomputes the correction term using the attention probabilities and probability gradients actually involved in the backward pass, thereby restoring the Zero-Row-Sum Invariant of the gradients before quantization. Finally, experiments demonstrate that this method significantly improves the training stability of native FP8 Attention, bringing the training loss and overall downstream performance close to those of BF16/FP32 mixed-precision training while preserving the computational efficiency advantages.

Critical strengths
  1. This paper provides a rigorous mathematical derivation and proof of the origin of Stale Delta, revealing that the quantization inconsistency between attention probabilities and output gradients causes the Softmax backward gradients to deviate from the Zero-Row-Sum Invariant. It further analyzes the impact of Stale Delta on long-term training, which is also partially validated by experiments. This establishes the rationale and importance of the study.
  2. The Delta-Matching method proposed to address Stale Delta is also supported by rigorous mathematical derivations, demonstrating that it can restore the Zero-Row-Sum Invariant of the Softmax backward gradients under the specified numerical precision conditions. The paper also introduces Consistent dO as an ablation method, further demonstrating the role of Delta-Matching in restoring gradient correctness. Subsequent experiments further validate the rationale and effectiveness of Delta-Matching.
Critical weaknesses
  1. Although the experiments in this paper cover comparisons with existing FP8 Attention training methods, different attention variants, hybrid architectures, model scales, and context lengths, there are still certain limitations in validating architectural generalization. For example, the attention variants and hybrid architectures are evaluated at the same model scale, and no experiments are provided to demonstrate that Delta-Matching maintains its stability and performance advantages under combinations of two or more conditions.
  2. The experiments only include single-GPU Attention Kernel throughput measurements, without quantitative results on computational efficiency and memory savings for full training. Consequently, the practical benefits of Delta-Matching for end-to-end training remain unclear.
Questions for authors
  1. Could you provide comparisons between Delta-Matching and BF16/FP32 mixed-precision training under combinations of at least two conditions to further demonstrate the stability advantages of this method?
  2. Can you provide results on computational efficiency and memory savings for the full training process?
  3. Could you add comparative experiments across more context lengths and more types of tasks? Alternatively, could you provide analyses and comparisons involving training on different datasets or using more optimization steps?
  4. Can you provide diagrams illustrating Stale Delta and Delta-Matching to help readers understand them more clearly?
3 comments
AuthorsOpenReview ↗

Added experiment and update diagram

1.Could you provide comparisons between Delta-Matching and BF16/FP32 mixed-precision training under combinations of at least two conditions to further demonstrate the stability advantages of this method?

Yes, we run another pretrain experiment with Kimi Delta Attention + GQA AND scaling up to 5.29B, changing both architecture and model size from the 1.67B GDN/GQA base model. The BF16, stale-FP8, and Delta-Matching runs each use 30B Nemotron-CC tokens with one shared initialization and data order.

MethodValidation CERULER-4KRULER-8KCS9
BF16/FP321.326571.7160.9763.34
Delta-Matching1.326672.8859.0963.43
Stale FP81.920530.4322.6546.28

Delta-Matching matches the BF16 reference's validation loss and overall downstream performance, while stale FP8 degrades every reported benchmark. Section 4.5 and Table 3 now include these results; Appendix C.8.2 gives the setup and paired validation analysis. Benchmark scores above are percentages.

2. Can you provide results on computational efficiency and memory savings for the full training process?

Yes, we added the updated end-to-end measurements. Normal training already gives non-trivial end-to-end throughput improvement, while training with context parallel brings the largest winning because we are passing 8-bit kv when modern pretraining are largely bottlenecked by communication. We also observed that the benefits of FP8 become more pronounced as training scales up, since the increased computational savings increasingly outweigh the quantization overhead.

It reach 1.163× cuDNN BF16 throughput at 256K context on eight GB200 GPUs with context parallelism of eight at the 9.3B experiment, increasing throughput from 7,747 to 9,011 tokens/s/GPU. Against FA3 BF16 on H200, speedups range from 1.010× to 1.073×, with measurements through 512K context. Table 27 in Appendix C.9 reports all 24 configurations.

Table 27. End-to-end training throughput in tokens/s/GPU. Speedup is the ratio of Delta-Matching throughput to BF16 throughput. GPU counts are totals across nodes.

ModelGPUs / baselineDP/CPContextTokens/GPUMBSBF16DM-FP8Speedup
9.3B32 H200 / FA332/18K16K210,20310,3011.010×
9.3B32 H200 / FA332/116K16K19,7719,8721.010×
9.3B32 H200 / FA34/8128K16K15,7136,0131.053×
9.3B32 H200 / FA34/8256K32K13,3543,6001.073×
9.3B32 H200 / FA32/16256K16K13,7794,0301.066×
9.3B32 H200 / FA31/32512K16K12,4032,4981.040×
9.3B4 GB200 / cuDNN4/18K16K219,58319,8561.014×
9.3B4 GB200 / cuDNN4/116K16K118,75819,2661.027×
9.3B4 GB200 / cuDNN4/132K32K115,80816,2891.030×
9.3B8 GB200 / cuDNN8/18K16K219,61119,8681.013×
9.3B8 GB200 / cuDNN8/116K16K119,04919,5471.026×
9.3B8 GB200 / cuDNN8/132K32K115,94816,6471.044×
9.3B8 GB200 / cuDNN8/164K64K112,64813,6191.077×
9.3B4 GB200 / cuDNN1/464K16K114,49215,3761.061×
9.3B4 GB200 / cuDNN1/4128K32K110,63711,5051.082×
9.3B8 GB200 / cuDNN1/8128K16K111,25312,5541.116×
9.3B8 GB200 / cuDNN1/8256K32K17,7479,0111.163×
1.67B8 H200 / FA38/18K32K444,25943,8990.992×
1.67B8 H200 / FA38/116K32K242,00641,6280.991×
1.67B8 H200 / FA38/132K32K137,70036,8890.979×
1.67B8 H200 / FA32/464K32K228,91928,8160.996×
1.67B8 H200 / FA31/8128K32K222,12121,7460.983×
1.67B16 H200 / FA31/16256K32K214,41714,0460.974×
1.67B32 H200 / FA31/32512K32K28,7828,1810.932×

DP and CP denote data and context parallelism; MBS is micro-batch-size. Measurements average five steps after three warm-up steps, excluding data loading and checkpoint saves.

3. Could you add comparative experiments across more context lengths and more types of tasks? Alternatively, could you provide analyses and comparisons involving training on different datasets or using more optimization steps?

Yes, we added full pretraining of 1.67B GDN/GQA on 60B FinePDFs-Edu tokens, changing the corpus and doubling the original 30B-token budget. The runs use 57,220 optimizer updates with one shared initialization and data order.

Delta-Matching reaches in-domain validation CE 1.7539, compared with 1.7529 for BF16 and 2.6692 for stale FP8. It reduces the stale-delta excess CE by more than 99.8% on both FinePDFs-Edu and the separate Nemotron-CC validation stream. Delta-Matching, BF16, and stale FP8 score 64.86%, 57.80%, and 9.29% on RULER-8K. Table 4(a) also reports extraction, TriviaQA, MMLU, and CS9.

Table 4(a). FinePDFs-Edu pretraining, 60B tokens. Single-initialization results. Validation CE is measured on held-out FinePDFs-Edu data; benchmark scores are percentages. Bold marks the best FP8 result.

MethodVal CE ↓RULER-4KRULER-8KExtractionTriviaQAMMLUCS9
BF16/FP32 MP1.752966.1257.8068.6923.4626.0861.88
Naive FP8 (Stale Delta)2.669217.569.2916.310.0923.3739.42
Delta-Matching (ours)1.753973.3764.8671.1722.8627.4961.40

4. Can you provide diagrams illustrating Stale Delta and Delta-Matching to help readers understand them more clearly?

Of course, the new Figure 3 in Section 3.3 shows the seven attention-core matmuls, FP32 softmax operations, and quantization points. The red path uses the saved forward output and original output gradient. The green path computes the correction from pre-cast probabilities and the FP32 dP^\widehat{dP} computed with FP8 operands. Both feed the same softmax-backward computation.

Figure 5, formerly Figure 3, shows the corresponding attention-row updates, channel preference and optimizer pressure, and gain trajectories.

Dear Reviewer omsj,

Thank you again for your thoughtful review and helpful suggestions. We have added experiments, and the paper is revised to address your questions. We would be grateful to hear your thoughts on our responses and updated results, and would be happy to discuss any remaining questions.

With thanks and best regards,

The Delta-Matching authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.