- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates FP8 training and identifies a failure mode called stale delta. The FP8 attention failure can be attributed to the saved-output correction that uses quantized operands, causing mismatches between probabilities and gradients. Therefore, this paper proposes the Delta-Matching method, which computes the correction directly from the pre-quantization probabilities and the products of FP8 backward operands. Experiments across several architectures, model sizes, and context lengths demonstrate substantial recovery of training quality relative to naive FP8 attention.
Critical strengths
- This paper shows a clear diagnostic contribution to FP8 attention. Decomposing the failure into probability-quantization and output-gradient-quantization mismatches provides a concrete explanation for the saved-output shortcut's failure.
- Delta-Matching is derived from the softmax backward invariant, avoids BF16 fallbacks or an auxiliary forward output, and is integrated into FP8 attention kernels while keeping all seven attention-core matmuls in FP8.
- This paper presents extensive empirical validation. The evaluation covers multiple hybrid architectures, attention variants, model sizes from 569M to 5.29B, and context extension to 64K.
Critical weaknesses
- The efficiency benefit is not fully demonstrated. Delta-Matching adds a separate correction pass, increasing backward time by roughly 21–28% over stale delta. At head dimension 128, it is slower than cuDNN BF16. The paper reports kernel-level throughput but no end-to-end training throughput or memory results.
- The proof assumes normalized probabilities, identical effective V operands, FP32 accumulation, and no rounding. The experiments stop at 5.29B parameters and 30B tokens. It is unclear whether the fix remains sufficient at larger scales and longer training.
Questions for authors
- What is the measured end-to-end training speedup and memory overhead of Delta-Matching versus BF16 and cuDNN/TE FP8?
- How does Delta-Matching behave beyond 5.29B parameters and longer token budgets? Does it remain robust on larger settings?
3 comments
Dear Reviewer Qgpz,
Thank you again for your thoughtful review and helpful suggestions. We have added experiments and revised the paper to address your comments. We would be grateful to hear your thoughts on our responses and updated results, and would be happy to discuss any remaining questions.
With thanks and best regards,
The Delta-Matching authors
Thank you for your efforts and timely response, which have addressed my concerns. I have raised my scores accordingly.
End-to-end Results and Scaling Up
1. The efficiency benefit is not fully demonstrated. Delta-Matching adds a separate correction pass, increasing backward time by roughly 21–28% over stale delta. At head dimension 128, it is slower than cuDNN BF16. The paper reports kernel-level throughput but no end-to-end training throughput or memory results.
Thank you for the reminder. We have now added end-to-end throughput measurements for training with the Megatron framework. It is up to 1.163× over cuDNN BF16 for the 9.3B model at 256K context on eight GB200 GPUs with context parallelism of eight. Throughput increases from 7,747 to 9,011 tokens/s/GPU. On H200, the 9.3B model reaches up to 1.073× FA3 BF16 throughput. Table 27 in Appendix C.9 reports all 24 configurations, including the 1.67B cases.
Table 27. End-to-end training throughput in tokens/s/GPU. Speedup is the ratio of Delta-Matching throughput to BF16 throughput. GPU counts are totals across nodes.
DP and CP denote data and context parallelism; MBS is micro-batch-size. Measurements average five steps after three warm-up steps, excluding data loading and checkpoint saves. Training settings are matched, with normalization and RoPE additionally fused in the FP8 path.
2. The proof assumes normalized probabilities, identical effective V operands, FP32 accumulation, and no rounding. The experiments stop at 5.29B parameters and 30B tokens. It is unclear whether the fix remains sufficient at larger scales and longer training.
The core of the training dynamics experiments is showing that Delta-Matching prevents the outlier-channel growth observed under stale delta, which is why we are confident it works when scaling up. We have reorganized this experiment from the stale delta and Delta-Matching section into the experiments section to make this clearer (Section 4.3, Figure 5).
To further empirically verify this, we pretrained the 1.67B model on the 60B-token FinePDFs-Edu dataset and scaled pretraining to 9.27B parameters. We tested the 9.27B model at both the same peak learning rate as the smaller models and a lower rate to account for the larger model size. Here are the validation CE results.
Each comparison uses one initialization per method and matched data order because of pretraining cost. At 9.27B and 2.4×10−3, all nine commonsense scores are within 0.86 percentage points of BF16, although RULER-8K is lower at 51.99% versus 58.83%. Tables 2 and 4(a) give the downstream results, and Appendix C.8.1 compares both learning rates.