- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes GIFT, a geometry-aware method for FP8 gradient communication during LLM pretraining. It observes that LLM gradients are highly anisotropic, with geometry that varies substantially across layers. Based on this observation, this paper derives gradient transformations from a K-FAC approximation with error feedback. Experiments on 600M- and 2B-parameter Llama-style models show reduced gradient reconstruction error and competitive downstream performance.
Critical strengths
- Using gradient geometry to choose communication coordinates is reasonable. The transformation is confined to communication, preserving the optimizer’s update rule.
- This paper implements GIFT in Megatron-LM and evaluates both quantization error and actual training quality.
Critical weaknesses
- GIFT shows limited practical advantage over simpler FP8 communication. At 64 GPUs, GIFT is essentially tied with Layer-wise Euclidean FP8 for Llama-600M (10.87% vs. 10.88%) and is slower for Llama-2B (17.05% vs. 18.60%). At 16 GPUs, GIFT is slower than FP32. Given the added geometry computation and profiling run, the systems' benefit over a much simpler FP8 baseline is not compelling.
- Experiments are limited to 600M and 2B models, so generalization to larger models remains unclear.
Questions for authors
- What is the benefit of GIFT over a cheap decorrelation baseline, such as Hadamard rotation?
We thank Reviewer nA9x for the constructive assessment and for recognizing the soundness of our approach. We apologize that the original presentation did not make some of these distinctions sufficiently clear. We respond to each concern below.
Weakness 1: Practical Benefit over Euclidean FP8
The key question is whether improved gradient fidelity justifies GIFT's additional geometric computation. GIFT targets the fidelity–efficiency trade-off, rather than a higher raw FP8 speedup. In Table 1 (Section 5.2), the practical rank-32 transformation reduces relative reconstruction error by 52.7% over Euclidean FP8. Figure 4(b) and Appendix E.1 show the corresponding downstream results:
Table 1: Downstream results reported in Fig. 4(b) and Appendix E.1.
At 64 GPUs, GIFT achieves 10.87% (600M) and 17.05% (2B) end-to-end step-time improvements over FP32 communication (Fig. 4(a)).
The 16-GPU results reflect a regime where geometric overhead outweighs communication savings. However, a clear scaling trend emerges: from 16 to 64 GPUs, GIFT's step-time improvement increases from -2.33% to 10.87% for 600M and from -1.00% to 17.05% for 2B (Appendix E.2). As both model size and GPU count increase, communication savings can increasingly offset geometric overhead, suggesting greater potential benefits for larger-scale pretraining.
Regarding the profiling overhead, GIFT requires only a one-time 200-step profiling run, which typically takes 5–8 minutes in our experiments and is negligible compared with the tens of hours required for full pretraining. The selected layers remain fixed throughout the subsequent 60,000-step pretraining run, so no additional profiling is required.
Weakness 2: Generalization to Larger Models
The primary constraint on evaluating larger models is the available GPU memory capacity. Our experiments were conducted on NVIDIA GH200 Superchips with 96 GB of HBM3 memory per GPU (Section 5.1). Even for the 2B model, peak GPU memory usage reaches 92.86 GiB (Appendix E.3), approaching the hardware's memory capacity under our current training configuration.
Question 1: Hadamard-Based Smoothing Baseline
Our HS-FP8 baseline implements a block-sized Hadamard transform for gradient smoothing, together with its quantization scaling strategy, adapted to FP8 gradient communication (Section 5.1 and Appendix D.3).
Table 2: Existing Llama-600M comparison (Fig. 4(b); Appendix E.1).
GIFT achieves a higher average downstream score than HS-FP8, with the same 8/14 task-preservation count. As reported in Section 5.3, HS-FP8 is also approximately 1.8× slower than GIFT in our PyTorch-level implementation. We will highlight this existing Hadamard-based comparison more explicitly in the revised manuscript.