acceptodds
Under review as a conference paper at ICLR 2027

GIFT: Geometry-Informed Low-Precision Gradient Communication for LLM Pretraining

14 views

Abstract

Gradient communication is a major scaling bottleneck in large language model (LLM) pretraining. Low-precision formats such as FP8 can substantially reduce communication cost, but often at the expense of gradient fidelity. Existing approaches either quantize gradients directly in their original coordinates or apply data-independent transformations to suppress outliers. Our study shows that both approaches overlook a fundamental source of quantization distortion: LLM gradients are highly anisotropic, with geometry that varies substantially across layers, making low-precision error strongly direction dependent. This observation suggests that gradient fidelity depends not only on the quantizer itself, but also on the coordinates in which quantization is performed. Motivated by this insight, we design GIFT, a geometry-informed gradient communication method that transforms gradients into more isotropic coordinates before low-precision quantization. To make this transformation practical, GIFT uses a one-sided, low-rank approximation of the geometry and applies it only to layers that are most sensitive to quantization errors, while leaving the optimizer, training recipe, communication collective, and low-precision format unchanged. Across Llama-style models with approximately 600M and 2B parameters, GIFT achieves downstream task performance comparable to Muon training with FP32 gradients. On 64 NVIDIA GH200 Superchips, GIFT reduces end-to-end training step time relative to FP32 gradient communication by 10.87% for Llama-600M and 17.05% for Llama-2B.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, distributed pretraining, gradient communication, communication compression, lowprecision communication, geometry-aware communication, K-FAC

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nA9xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GIFT, a geometry-aware method for FP8 gradient communication during LLM pretraining. It observes that LLM gradients are highly anisotropic, with geometry that varies substantially across layers. Based on this observation, this paper derives gradient transformations from a K-FAC approximation with error feedback. Experiments on 600M- and 2B-parameter Llama-style models show reduced gradient reconstruction error and competitive downstream performance.

Critical strengths
  1. Using gradient geometry to choose communication coordinates is reasonable. The transformation is confined to communication, preserving the optimizer’s update rule.
  2. This paper implements GIFT in Megatron-LM and evaluates both quantization error and actual training quality.
Critical weaknesses
  1. GIFT shows limited practical advantage over simpler FP8 communication. At 64 GPUs, GIFT is essentially tied with Layer-wise Euclidean FP8 for Llama-600M (10.87% vs. 10.88%) and is slower for Llama-2B (17.05% vs. 18.60%). At 16 GPUs, GIFT is slower than FP32. Given the added geometry computation and profiling run, the systems' benefit over a much simpler FP8 baseline is not compelling.
  2. Experiments are limited to 600M and 2B models, so generalization to larger models remains unclear.
Questions for authors
  1. What is the benefit of GIFT over a cheap decorrelation baseline, such as Hadamard rotation?
1 comment

We thank Reviewer nA9x for the constructive assessment and for recognizing the soundness of our approach. We apologize that the original presentation did not make some of these distinctions sufficiently clear. We respond to each concern below.


Weakness 1: Practical Benefit over Euclidean FP8

The key question is whether improved gradient fidelity justifies GIFT's additional geometric computation. GIFT targets the fidelity–efficiency trade-off, rather than a higher raw FP8 speedup. In Table 1 (Section 5.2), the practical rank-32 transformation reduces relative reconstruction error by 52.7% over Euclidean FP8. Figure 4(b) and Appendix E.1 show the corresponding downstream results:

Table 1: Downstream results reported in Fig. 4(b) and Appendix E.1.

ModelMethodAvg. score ↑Tasks ≥ FP32 ↑
600MEuclidean FP80.51865/14
600MGIFT0.52138/14
2BEuclidean FP80.50994/14
2BGIFT0.52317/14

At 64 GPUs, GIFT achieves 10.87% (600M) and 17.05% (2B) end-to-end step-time improvements over FP32 communication (Fig. 4(a)).

The 16-GPU results reflect a regime where geometric overhead outweighs communication savings. However, a clear scaling trend emerges: from 16 to 64 GPUs, GIFT's step-time improvement increases from -2.33% to 10.87% for 600M and from -1.00% to 17.05% for 2B (Appendix E.2). As both model size and GPU count increase, communication savings can increasingly offset geometric overhead, suggesting greater potential benefits for larger-scale pretraining.

Regarding the profiling overhead, GIFT requires only a one-time 200-step profiling run, which typically takes 5–8 minutes in our experiments and is negligible compared with the tens of hours required for full pretraining. The selected layers remain fixed throughout the subsequent 60,000-step pretraining run, so no additional profiling is required.


Weakness 2: Generalization to Larger Models

The primary constraint on evaluating larger models is the available GPU memory capacity. Our experiments were conducted on NVIDIA GH200 Superchips with 96 GB of HBM3 memory per GPU (Section 5.1). Even for the 2B model, peak GPU memory usage reaches 92.86 GiB (Appendix E.3), approaching the hardware's memory capacity under our current training configuration.


Question 1: Hadamard-Based Smoothing Baseline

Our HS-FP8 baseline implements a block-sized Hadamard transform for gradient smoothing, together with its quantization scaling strategy, adapted to FP8 gradient communication (Section 5.1 and Appendix D.3).

Table 2: Existing Llama-600M comparison (Fig. 4(b); Appendix E.1).

MethodAvg. score ↑Tasks ≥ FP32 ↑
Euclidean FP80.51865/14
HS-FP80.50438/14
GIFT0.52138/14

GIFT achieves a higher average downstream score than HS-FP8, with the same 8/14 task-preservation count. As reported in Section 5.3, HS-FP8 is also approximately 1.8× slower than GIFT in our PyTorch-level implementation. We will highlight this existing Hadamard-based comparison more explicitly in the revised manuscript.

Reviewer_9XQfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

GIFT introduces geometry-informed gradient FP8 quantization for communication compression over distributed LLM pretraining. It uses a K-FAC-based coordinate transformation using a rank-32 input-side approximation. This compression strategy is selectively applied to sensitive layers based on an initail sensetivity assesment (before trainng begines, selection stays fixed while only KFAC factors are updated every 50 steps). The authors report a 52.7% reduction in relative gradient reconstruction error, measured using FP8 round-trip experiments on five MLP projections sampled at initialization. They evaluate Llama-style 600M and 2B models trained for 60,000 steps on OpenWebText, reporting improved downstream scores over Euclidean FP8 communication. On 64 GH200 GPUs, GIFT reduces training step time by 10.87% and 17.05% relative to FP32 communication, compared with 10.88% and 18.60% for standard Euclidean FP8.

Critical strengths
  1. The authors addres an important practical bottleneck using a well-defined geometric formulation without modifying the optimizer or communication format.
  2. The proposed approximation and selective application offer a plausible approach to balancing gradient fidelity and computational overhead.
  3. Provides useful reconstruction ablations, multiple FP8 baselines, and system-level scaling measurements across GPU counts and two model sizes.
Critical weaknesses
  1. Novelty and geometric justification: Coordinate transformations and smoothing are established approaches to quantization. Although GIFT derives its transformation from K-FAC, the derivation does not establish that Fisher-normalized coordinates improve or optimize Euclidean reconstruction error compared to simpler spectral/random transformations.

  2. Efficiency and scalability: The low-rank geometry approximation still uses dense Cholesky factorization and triangular solves. While end-to-end step times are reported, the timing methodology does not explicitly establish whether measurements are averaged over complete 50-step geometry-refresh cycles, including covariance synchronization, eigendecomposition, and factorization. In addition the experiments are limited to 600M and 2B models with context length fixed at 2048, leaving the computational tradeoff at larger widths and sequence lengths unclear.

  3. Statistical reliability of quality improvements: Downstream evaluations lack multi-seed variance or confidence intervals. The average improvement over Euclidean FP8 is small for Llama-600M (0.5213 versus 0.5186), while the Llama-2B advantage is concentrated in a few tasks, particularly BoolQ and CB, alongside deterioration on others. Furthermore, the primary reconstruction evidence comes from five projections at initialization rather than distributed gradients throughout training.

  4. Stability of layer sensitivity: Sensitive layers are selected once during initial profiling and remain fixed for 60,000 steps, although their geometry is refreshed periodically. The paper does not demonstrate that their relative quantization sensitivity remains stable as training progresses.

Questions for authors

Q1. can the authosr How does the method compare with simpler PCA-based or random orthogonal transformations under matched communication and error-feedback settings? Q2 Does the reported speedup include the periodic update costs or only the application of projection? Q3. Are the quality improvements statistically reproducible, and does the reconstruction advantage persist during pretraining? Q4. What is the total training wall-clock cost including the KFAC recomputation? Can the authors provide reference point where the compute budget of the gpu is fully utilized to clarify overhead costs (e.g. larger model, longer sequence length, larger batch size etc)? Q5. How does fixed selection compare with periodic reassessment, and how much of GIFT's benefit depends on this initial selection?

1 comment

We thank Reviewer 9XQf for recognizing the motivation and geometric formulation of GIFT. We address each concern below.


Weakness 1 & Question 1: Novelty and Simpler Coordinate Transformations

Our evaluation includes HS-FP8, using a block-sized Hadamard transform and quantization scaling (Section 5.1; Appendix D.3). We additionally evaluate PCA to test whether GIFT's benefits extend beyond generic decorrelation.

Table 1: Llama-600M downstream performance.

MethodAvg. ↑
HS-FP80.5043
PCA (rank 32)0.4990
GIFT0.5213

GIFT outperforms both HS-FP8 and PCA, suggesting that its geometry-informed communication design provides benefits beyond generic orthogonal transformations and spectral decorrelation.


Weakness 2 & Questions 2, 4: Efficiency, Geometry Refresh, and Scalability

End-to-end overhead.

The timing results reported in Appendix D.5 are averaged over two complete 50-step cycles, including both geometry factor updates and all method-specific transformation overhead. Thus, step times include geometry refresh.

For Llama-2B, Euclidean FP8 achieves an 18.60% speedup over the FP32 baseline, while GIFT achieves a 17.05% speedup with geometry refresh included. Excluding the two geometry update steps, GIFT's speedup increases to 18.53%, nearly matching Euclidean FP8.

Initial profiling is a one-time 200-step run taking 5–8 minutes, negligible relative to full pretraining. The selected layers remain fixed throughout the subsequent 60,000-step training run.

Scalability and hardware constraints.

The primary constraint on evaluating larger models, longer sequences, and larger micro-batches is GPU memory capacity. On NVIDIA GH200 Superchips with 96 GB HBM3, our Llama-2B model already reaches 92.86 GiB peak GPU memory usage (Appendix E.3), leaving very limited headroom. This limits further scaling under our configuration.


Weakness 3 & Question 3: Statistical Reproducibility and Reconstruction Fidelity

Multi-seed evaluation.

As Reviewer nQ82 pointed out, CB has only 56 validation examples, making its evaluation potentially unstable. The reliability of WSC evaluation has also been questioned in prior studies (Trichelair et al., 2019; Sakaguchi et al., 2020). We therefore additionally report averages over the remaining 12 downstream tasks. To assess reproducibility, we conducted two additional full Llama-2B GIFT pretraining runs with different random seeds, keeping the training configuration and evaluation protocol unchanged.

Table 2: Llama-2B performance across pretraining seeds.

MethodSeed12-task Avg. ↑
Muon FP32Original0.5474
Euclidean FP8Original0.5330
GIFTOriginal0.5381
GIFTSeed 10.5424
GIFTSeed 20.5392
GIFT Mean ± SD—0.5399 ± 0.0022

The results show relatively consistent GIFT performance across three independent pretraining seeds, with all three runs outperforming the single-seed Euclidean FP8 reference. Mean and SD quantify seed-level variability.

Obtaining reliable confidence intervals for cross-method comparisons would require substantially more independent pretraining runs. However, the computational cost is prohibitive: each Llama-2B run requires 32 GH200 GPUs for approximately 28 hours, equivalent to 896 GPU-hours and an estimated cost of $2,240 (at $2.50 per GPU-hour). We therefore report multi-seed descriptive statistics.

Reconstruction fidelity during pretraining.

The reconstruction results in Table 1 and Appendix B demonstrate GIFT's advantage at initialization. During pretraining, GIFT refreshes its geometry factors every 50 training steps to adapt to changes in gradient geometry, rather than relying on a fixed initial transformation.


Weakness 4 & Question 5: Stability of Layer Sensitivity and Selection

Layer-selection stability.

We additionally evaluated layer-selection stability at training steps 1,000 and 5,000, finding that 7 of the 13 initially selected layers remained in the sensitivity-based selection at both checkpoints. Due to time constraints, we were unable to extend this analysis to later training stages. Nevertheless, our downstream results demonstrate that the fixed-selection strategy remains effective in practice, as GIFT outperforms competing FP8 communication methods without requiring layer reselection. In future work, we plan to further investigate dynamic layer selection during pretraining.

Reviewer_nQ82OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that FP8 gradient-communication error in LLM pretraining depends on the coordinates used for quantization, because gradients are highly anisotropic and their geometry varies by layer. GIFT maps gradients into more isotropic coordinates before low-precision quantization. For practicality, it’s applied only to sensitive layers, and it adds error feedback in the transformed coordinates. In experiments, GIFT matches or beats Muon FP32 on more downstream tasks, than layer-wise Euclidean FP8. On 64 GH200s, GIFT cuts training time versus FP32 communication by 10.87% (600M) and 17.05% (2B).

Critical strengths
  • It’s novel and principled to treat the quantization coordinate system as a design choice: It goes beyond the usual scaling and Hadamard smoothing. The dual-coordinate derivation gives the transform a clear justification, which matters because it could carry over to other compression formats.
  • Practical and convincing experiment setup in a real system: GIFT runs inside Megatron-LM with system measurements up to 64 GPUs. The setup makes the experiment setup concrete.
Critical weaknesses
  • No variance reporting and results seem to be weak: The downstream averages are driven by small high-variance tasks: in Table 6, CB accuracy swings from 0.09 to 0.41 across methods, and plain Euclidean FP8 (0.5186) already beats Muon FP32 (0.5060) on average. This makes the results less convincing.
  • Missing ablations for error feedback: Euclidean FP8 with error feedback does worse than without it. Can you show GIFT without error feedback, to isolate where the gains come from?
Questions for authors
  • Quality evidence: Can you report final validation losses with multiple seeds, and confidence intervals for the downstream scores? Without these, it’s difficult to tell whether GIFT's gains over Euclidean FP8 are real, or from noise.
  • Figures: Several figures (e.g., Figures 1 and 3) are low resolution and appear to have shadow artifacts, which makes the content a bit hard to read. Could you provide higher-resolution versions?
1 comment

We thank Reviewer nQ82 for recognizing the novelty of our geometry-informed communication design and the practical evaluation in Megatron-LM. We address the concerns below.


Weakness 1 & Question 1: Statistical Reliability and Multi-Seed Evaluation

We agree that downstream evaluations can exhibit substantial variability, particularly for CB, which has a small validation set of only 56 examples, and WSC, whose evaluation reliability has been questioned in prior studies (Trichelair et al., 2019; Sakaguchi et al., 2020).

To reduce the influence of these potentially unstable evaluations, we additionally report the average over the remaining 12 downstream tasks. We also conducted two additional full Llama-2B GIFT pretraining runs with two different random seeds, in addition to the original run, to assess sensitivity to pretraining randomness while keeping the training configuration and evaluation protocol unchanged.

Table 1: Llama-2B downstream performance on 12 tasks across different pretraining seeds.

MethodSeed12-task Avg. ↑
Muon FP32Original0.5474
Euclidean FP8Original0.5330
GIFTOriginal0.5381
GIFTSeed 10.5424
GIFTSeed 20.5392
GIFT Mean ± SD—0.5399 ± 0.0022

The results show relatively consistent GIFT performance across the three pretraining seeds, with all three runs outperforming the single-seed Euclidean FP8 reference. We additionally report the mean and standard deviation across independent pretraining runs to quantify GIFT's seed-level variability.

Obtaining reliable confidence intervals for comparisons across pretraining seeds would require a substantially larger number of independent runs. However, the computational cost is prohibitive: each Llama-2B pretraining run requires 32 GH200 GPUs for approximately 28 hours, equivalent to 896 GPU-hours and an estimated cost of $2,240 (at $2.50 per GPU-hour). Therefore, we report the available multi-seed results and their descriptive statistics instead.


Weakness 2: Error Feedback Ablation

To isolate the effect of error feedback, we conducted an additional Llama-600M experiment with GIFT's error feedback disabled, while keeping all other training settings unchanged.

GIFT without EF achieves a 12-task average of 0.5281, compared with 0.5220 for Euclidean FP8. This result demonstrates that GIFT's advantage is driven by its geometry-aware compression design rather than error feedback alone.


Question 2: Figure Resolution and Readability

We have replaced the low-resolution figures with higher-resolution versions and corrected the reported visual artifacts. The updated manuscript PDF has already been uploaded, including the revised Figures 1 and 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.