acceptodds
Under review as a conference paper at ICLR 2027

Where Low Precision Bends Gradients

Abstract

Low-precision training reduces training time and memory use for transformers. Realizing these gains requires deciding which tensors, operations, and matrix-multiplication passes retain higher precision. Existing precision-allocation methods have focused on local reconstruction error or forward-loss sensitivity. These criteria measure changes in tensor values or the forward loss, but do not directly measure the resulting change in the parameter gradient that drives optimization. We reveal this mismatch by measuring how quantization errors in forward and backward computation change the full model's parameter gradient: Similar local errors at different tensor locations produce gradient shifts whose magnitudes differ by more than tenfold. These shifts mainly rotate, or bend, the gradient. To preserve gradient direction during low-precision training, we introduce bending-guided precision allocation. The method allocates high precision to selected operation groups or individual matrix-multiplication passes where quantization rotates the gradient. Using only a small amount of calibration data, the proposed method determines how much high precision is needed and where to allocate high precision to bring the gradient directions of 4- and 8-bit training close to the bfloat16 (BF16) reference. In GPT-2 continued pretraining, our calibration probe ranks precision allocations before training, closely matching the allocations' final-loss ranking with a Spearman correlation of 0.95, compared with 0.35 for local reconstruction error. Under BF16 mixed-precision evaluation, a model trained in 8-bit floating point (FP8) with our allocation matches the loss of a BF16-trained model. These allocations use 11% to 24% less high-precision activation-cache memory than a fixed policy that stores the output activations of the token embedding, output head, and every normalization layer in high precision. In matrix-multiplication simulations of NVIDIA's NVFP4 format, our probe identifies forward passes as the main source of bending. Executing forward multiplications in BF16 uses high precision for just one third of the transformer-block matrix-multiplication work and closes 99% of the loss gap to BF16 training. At a matched high-precision compute budget, allocating BF16 to selected forward matrix multiplications closes 52% of the gap, compared with at most 20% for NVIDIA-style block protection. The method also improves low-precision training across four vision backbones.

Primary area:
optimization
Keywords:
Low-precision training, Mixed-precision training, Neural network quantization, Gradient-based optimization, Numerical stability, Large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PHqzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how low-precision computations affect gradient and optimizer-update directions. It proposes using directional deviation from a higher-precision reference to guide precision allocation. On calibration data, a greedy procedure selects operations whose restoration to higher precision provides favorable reductions in directional deviation relative to their cost. The resulting precision configuration is then fixed during training.

Critical strengths
  1. The paper examines propagated gradient and optimizer-update effects rather than relying exclusively on local tensor reconstruction error. Its direction-versus-magnitude interventions and optimizer-aware analysis provide useful evidence that similarly sized numerical errors can have different consequences for optimization.

  2. The paper connects its diagnostic measurements to precision allocation and subsequent training outcomes. The reported ranking correlations and fixed-configuration training experiments provide evidence that calibration-time directional measurements contain useful information about later training quality.

Critical weaknesses
  1. The independent contribution of the directional criterion to precision allocation remains insufficiently isolated. The selection framework is a conventional greedy budgeted search, so the methodological contribution depends substantially on the scoring criterion. Correlation between directional deviation and final loss does not by itself demonstrate that optimizing this criterion produces better configurations. The existing comparisons do not fully establish a consistent advantage over alternative propagated-error or optimizer-aware criteria under matched candidate sets, budgets, and search procedures.

  2. The continued suitability of a configuration selected before training is not sufficiently characterized. Successful fixed-configuration training provides useful evidence and should not be dismissed, but does not show how sensitivities or preferred protection sets change across training stages. The paper leaves uncertainty about when the initial allocation remains effective and when recalibration would materially improve the quality–cost tradeoff.

Questions for authors
  1. Under identical candidate operations, budgets, and selection procedures, how do configurations selected using directional deviation compare with those selected using local reconstruction error and full-gradient or optimizer-update error magnitudes? Which existing experiments most directly isolate the benefit of the directional criterion?

  2. Do intermediate-checkpoint measurements show that the initially selected protection set remains effective? Please distinguish evidence that a fixed configuration can train successfully from evidence that its allocation remains suitable throughout training.

7 comments

Thank you for these questions. The controlled experiments connect preservation of the learning direction to better precision choices and better trained models. Changing only the score gives update bending the lowest mean policy and common-BF16 losses among four criteria at 355M. Direction/norm interventions identify the useful error component, and the exact error decomposition explains why full-update L2 captures the same directional information. Checkpoint experiments establish a practical role for the criterion throughout training: they validate reuse of the initial allocation in continued pretraining and identify a reallocation that improves both test losses in from-scratch training at unchanged per-step BF16 work.

W1 / Q1. Changing the allocation criterion improves the trained model at matched cost

Update bending improves the selected allocation and final loss with candidates, budget, search, and training held fixed. We first test four scores at 355M, then extend the comparison to full-gradient and full-update error criteria at 124M. The scores measure the following changes relative to BF16:

ScoreWhat it measures
Local reconstruction errorChanges in intermediate computation values.
Activation-FisherIntermediate errors weighted by the loss sensitivity of the affected activations.
Executed forward lossThe change in the current calibration loss under the candidate precision policy.
Gradient-norm distortionThe change in the gradient's length.
Full-gradient / full-update relative L2The full vector difference, including changes in both length and direction.
Gradient / update bendingThe change in the vector's direction.

Bending selects protections that bring the learning direction closer to its BF16 reference per unit of high-precision work. We first compare update bending, conditional Fisher, conditional reconstruction, and executed calibration loss at pretrained GPT-2 355M weights. All four objectives are recomputed for every candidate allocation. They share the same 144 layer–operation–pass candidates, four calibration batches, GEMM-work costs, gain-per-cost greedy search, and comparison with the best affordable single action. Each allocation uses exactly 1/18 of block-GEMM work in BF16 (approximately 5.56%).

Each selected mask is fixed before training for 1,024 updates over 2.10M tokens, using simulated NVFP4-style GEMMs and a BF16 LM head. Validation is disjoint from calibration. Policy loss evaluates each model with its training precision settings; common-BF16 loss evaluates each model's unchanged learned weights using the same BF16 arithmetic. Entries are means ± sample SD over six paired seeds:

CriterionPolicy lossCommon-BF16 loss
Update bending3.05239 ± 0.001123.02537 ± 0.00130
Conditional activation-Fisher3.05367 ± 0.001423.02743 ± 0.00167
Conditional reconstruction3.06396 ± 0.001373.03223 ± 0.00094
Executed calibration loss3.05303 ± 0.001273.02797 ± 0.00084

Update bending achieves the lowest mean loss under both evaluations among these four criteria. Relative to conditional Fisher, bending lowers policy loss by 0.00128 ± 0.00133 and common-BF16 loss by 0.00206 ± 0.00135 (mean ± SD of six paired differences). The common-BF16 reduction has a paired 95% t-confidence interval of [0.00064, 0.00347]. Both losses improve on 5/6 seeds, including all three additional seeds evaluated after the allocations were fixed. On those three confirmation seeds, the common-BF16 reduction is 0.00225 ± 0.00050. Relative to executed calibration-loss allocation, common-BF16 loss falls by 0.00260 ± 0.00112, improving on 6/6 seeds.

The score changes the protection decisions themselves: reconstruction selects ten Dgrad, one Wgrad, and one Fprop action, whereas bending selects twelve Fprop actions. The controlled experiment therefore connects the objective to a different allocation and then to lower final loss, at the same BF16 work. Improvement under both evaluations establishes better performance with the selected precision policy and better learned weights under common arithmetic.

The 124M comparison extends the test to eight criteria, including full-gradient and full-update L2. It uses GPT-2 124M with simulated NVFP4-style GEMM operands and a BF16 head. All eight objectives share:

  • the same 72 layer–operation–pass actions, covering Fprop, Dgrad, and Wgrad;
  • the same eight calibration batches and GEMM-work costs;
  • the same full gain-per-cost greedy framework and best-single-action safeguard;
  • exactly 1/6 of block-GEMM work in BF16 (approximately 16.67%);
  • frozen-mask training for 1,024 updates over 2.10M tokens, with three paired seeds.

Full-vector and forward-loss objectives score each executed candidate mask. Fisher and reconstruction use their defined modular residual risks in the same search framework. Evaluation uses all complete WikiText-103 test blocks, separate from calibration, under both the selected precision policy and common BF16 arithmetic. Entries are means ± sample SD over three paired seeds:

ScorePolicy test lossCommon-BF16 test lossPolicy-gap closed
Update bending3.34450 ± 0.000873.32177 ± 0.0013752.2 ± 0.6%
Gradient bending3.34269 ± 0.000823.32281 ± 0.0004854.6 ± 1.0%
Full-gradient relative L2 error3.34330 ± 0.000953.32284 ± 0.0007653.8 ± 0.9%
Full-update relative L2 error3.34450 ± 0.000873.32177 ± 0.0013752.2 ± 0.6%
Gradient-norm distortion3.36679 ± 0.000823.33452 ± 0.0012522.6 ± 0.5%
Executed forward-loss change3.34151 ± 0.000553.32441 ± 0.0006356.2 ± 2.2%
Activation-Fisher3.34342 ± 0.000243.32068 ± 0.0005053.7 ± 1.3%
Local reconstruction error3.38348 ± 0.001973.34361 ± 0.001040.5 ± 1.2%

At identical granularity, BF16 work, and search, bending recovers more of the training-loss gap than norm-only and local reconstruction criteria. With 16.67% of block-GEMM work in BF16, gradient bending recovers 54.6 ± 1.0% of the loss gap between all-low and BF16 training. It lowers policy test loss by 0.02411 ± 0.00064 versus norm distortion and 0.04079 ± 0.00187 versus local reconstruction, each on 3/3 paired seeds. A separate 131,072-token calibration-disjoint validation slice preserves these comparisons. At the other tested budget, 5.56% BF16 work, update bending and full-update error select the same mask and close 24.0% of the complete-test policy gap, versus −1.3% for local reconstruction.

The controls identify the information that makes these allocations effective. At 124M, update bending and full-update L2 select the same masks at both budgets, and both bending scores achieve lower common-BF16 loss than allocation by executed forward loss in the eight-criterion comparison. At 355M, update bending achieves the lowest mean loss in both evaluations among the four criteria. The decomposition below identifies the directional component shared by bending and full-update L2 and quantifies its dominance in the measured update error.

W1 / Q1, continued. Directional error explains the effective optimizer-aware scores

Bending isolates the directional component that dominates the measured optimizer-update error. Full-vector L2 contains both length and direction changes; norm distortion measures length alone. The exact decomposition connects these criteria and explains why full-update L2 and bending select the same protections in several comparisons.

The exact decomposition makes this distinction explicit. For r=∥v~∥/∥v∥r = \lVert \widetilde{v} \rVert / \lVert v \rVert, the ratio of the low-precision and BF16 vector lengths, and Dv=1−cos⁡(v,v~)D_v = 1 - \cos(v, \widetilde{v}),

= \underbrace{(r - 1)^2}_{\text{length mismatch}} + \underbrace{2rD_v}_{\text{direction mismatch}}.$$ Matching the two vector lengths makes $r = 1$, so the length term vanishes and squared relative L2 error equals exactly twice the bending. This identity shows precisely how full-vector error incorporates the direction-preservation objective. Full-update L2 and update bending select the same 355M mask and the same 124M masks at both budgets. Identical masks share the same training runs. Direction accounts for 99.89–99.97% of squared update error in measurements with actual accumulated checkpoint moments across nine GEMM checkpoint/mask cells. Each value averages 64 held-out batches at archived 124M or 355M states from seed 17. Thus, in these measurements, full-update L2 measures almost entirely the error component that bending explicitly targets. The agreement between the selectors has a quantitative explanation in the learning-direction geometry, connecting the paper's analysis to the observed allocation choices.

Scoring the AdamW update makes precision allocation sensitive to the optimizer's actual learning direction. AdamW combines the gradient with its accumulated history and rescales individual coordinates to produce the parameter update. Update bending measures the directional effect of quantization after this transformation, aligning the allocation objective with the parameter change used for learning.

A controlled 2 × 2 ablation separates two choices: which vector to measure (raw gradient or AdamW update) and which difference to measure (direction through bending, or the full vector through relative L2). Within each model, all four combinations share the complete candidate set, calibration, executed-mask re-evaluation, greedy rule, safeguard, and identical 5.56% BF16 work. Means ± sample SD over three paired seeds, with calibration-disjoint validation, are:

Scored vector / error355M policy loss355M common-BF16 loss774M policy loss774M common-BF16 loss
Gradient / bending3.05677 ± 0.000543.02802 ± 0.001942.88861 ± 0.001702.86782 ± 0.00180
Gradient / relative L23.05512 ± 0.001153.02789 ± 0.001422.88624 ± 0.001472.86740 ± 0.00128
AdamW update / bending3.05153 ± 0.000813.02456 ± 0.001432.88434 ± 0.001382.86633 ± 0.00089
AdamW update / relative L23.05153 ± 0.000813.02456 ± 0.001432.88378 ± 0.002442.86613 ± 0.00065

For both error metrics, scoring the AdamW update gives lower mean policy and common-BF16 losses than scoring the raw gradient at both scales. Update bending improves both loss endpoints over raw-gradient L2 on every paired seed at 355M and 774M. Its policy-loss reductions are 0.00360 ± 0.00143 at 355M and 0.00190 ± 0.00030 at 774M. Within update space, L2 and bending select the same mask at 355M and give close results at 774M. These controls establish the value of following quantization error through the optimizer and targeting the direction of the resulting update.

Preserving the BF16 gradient direction maintains learning under large norm changes. Section 4.2 / Table 15 isolates direction by changing gradient norm or direction separately during training, tuning the learning rate, disabling clipping, and evaluating the learned weights without quantizers:

Gradient supplied to the optimizerAdamW final lossMomentum-SGD final loss
BF16 reference3.248 ± 0.0013.292 ± 0.004
BF16 direction, quantized-gradient norm3.249 ± 0.0013.259 ± 0.002
Quantized-gradient direction, BF16 norm3.645 ± 0.0553.973 ± 0.012

Keeping the BF16 direction preserves learning even when the gradient length changes substantially. Restoring only the BF16 length leaves a large loss penalty when the direction is distorted. Together, the experiments establish the mechanism and its use in allocation: the interventions identify the importance of direction, the score swaps connect that information to better protection choices, and the training runs verify the resulting loss improvements. The update-error decomposition explains why full-update L2 also captures this useful directional information.

The submitted experiments also show training gains from bending-guided allocation at matched granularity and cost. Table 26 compares bending-guided and reconstruction-error allocation over the same 72 individual GEMM-pass candidates at 5.56% protected work, assessing gradient and update bending on held-out batches. Table 28 reports three-seed continued-pretraining outcomes under the submitted validation protocol: at the same work budget, bending-guided allocation closes 23.6 ± 2.2% of the loss gap, versus 0.6 ± 2.1% for reconstruction-error allocation. The new score swaps above additionally hold the selection procedure fixed and evaluate full-gradient and full-update error objectives.

The direction criterion also predicts a useful precision budget before training. Table 28's first mask below the BF16 raw-gradient target uses 29.6% work and learns weights within 0.004 of BF16 on average. Bending therefore connects the learning mechanism to both allocation decisions: which computations to protect and how far to extend protection. We have connected the intervention, direct allocation controls, and budget prediction in Sections 4–6 and Appendix H.1.

Reviewer_kYY2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how quantization errors propagate through neural networks and affect parameter gradients during low-precision training. The authors show that similar local quantization errors can produce substantially different gradient distortions depending on where they occur. They further distinguish changes in gradient magnitude from changes in gradient direction and argue that the latter are more difficult to compensate for through optimizer adjustments. Based on these observations, the paper proposes bending-guided precision allocation, which selectively retains high precision for operations that contribute most to gradient-direction distortion under a given resource budget. Experiments on vision models and GPT-2 demonstrate that gradient bending can guide precision allocation and predict subsequent training performance.

Critical strengths
  1. The paper addresses an important problem in low-precision training. The paper analyzes how quantization errors propagate to the parameter gradients used by the optimizer rather than relying solely on local tensor reconstruction errors. The observation that similar local errors can lead to different gradient distortions provides a meaningful motivation for reconsidering conventional precision-allocation criteria.

  2. The paper provides a detailed analysis of how quantization affects gradient magnitude and direction. In particular, the controlled experiments that independently modify gradient norms and directions help explain why directional distortion may be more harmful to training than magnitude changes. The additional analysis of forward and backward quantization, including the effects of ReLU mask changes and AdamW updates, further strengthens the understanding of the underlying mechanisms.

  3. The paper conducts extensive experiments across different precision formats, neural network architectures, and precision-allocation budgets. The authors evaluate their approach on GPT-2 continued pretraining and several vision backbones, and provide comparisons with reconstruction-error, Hessian-based, and other sensitivity-based methods.

Critical weaknesses
  1. While the paper demonstrates substantial improvements over reconstruction-error-based allocation, its advantages over stronger sensitivity-based approaches are less clear. Also, the actual improvement that comes from the proposed bending metric remains unclear.
  • For example, Table 11 reports a Spearman correlation of 0.98 for activation-Fisher and 0.95 for gradient bending under the same memory-budget setting. This suggests that activation-Fisher is a strong baseline and slightly outperforms gradient bending in predicting final training performance rankings.
  • Some comparisons involve different allocation granularities. Bending-guided selection operates on individual GEMM passes, whereas certain sensitivity-based baselines operate on larger operation groups. Thus, it is difficult to determine how much of the improvement comes from the proposed bending metric rather than the finer allocation granularity or the iterative re-evaluation of selected masks.
  1. The end-to-end performance evaluation is limited.
  • The paper is motivated by improving the efficiency of low-precision training, but the reported resource savings are primarily based on high-precision activation-cache footprints and protected GEMM work. Although Appendix E.7 provides native-kernel timing measurements, these measurements exclude operand quantization and non-GEMM operations.
  • The main FP4 training outcomes are obtained through numerical simulation rather than a complete hardware-executed FP4 training pipeline.
  1. The experimental evaluation has limited validation of scalability and generalization. It is unclear whether the selected precision allocations and their relative advantages remain consistent as model size and architecture change.
  • The main precision-allocation experiments focus on GPT-2 with 124M parameters.
  • Although Appendix E.10 extends the analysis to models with up to 1.56B parameters, the largest-scale results primarily evaluate gradient geometry at fixed model states rather than complete training trajectories.
  • The 355M from-scratch experiments transfer a precision mask derived from the 124M model rather than independently evaluating the allocation procedure at the larger scale.
  1. The proposed method has limited benefits beyond protecting the LM head in boundary-quantization experiments.
  • In the GPT-2 boundary-quantization experiments, most of the reported improvement comes from protecting a single component: the LM-head output. As shown in Figure 2, quantizing the logits alone causes a much larger gradient shift than quantizing any other site and substantially increases the forward loss. Moreover, Table 7 shows that protecting only the LM head already closes 99.79% of the FP8 loss gap and 95.83% of the NVFP4-style loss gap. While the proposed method can further reduce high-precision memory usage, its additional benefit beyond this single protection (LM head) remains unclear.
Questions for authors

Please address the weaknesses listed above. In particular, please answer the following questions:

  1. Could the authors report actual peak GPU memory usage and end-to-end training time, including calibration overhead?
  2. Could the authors evaluate the proposed allocation method on larger models using independently selected precision masks?
  3. Given that LM-head protection alone closes most of the loss gap, what additional benefits does bending-guided allocation provide beyond LM-head-only protection?
7 comments

Thank you for the detailed assessment of the propagation analysis, controlled interventions, and breadth of experiments. The results below establish the value of the allocation criterion through matched-cost training comparisons, independent selection at larger scales, and measured end-to-end efficiency. They also show how the same criterion selects both the locations and the extent of protection, with gains inside transformer blocks and across boundary formats.

W1. Controlled ablations of the allocation criterion

A controlled score ablation at 355M gives update bending the lowest mean policy and common-BF16 losses among all four criteria at identical 5.56% BF16 work. To isolate the score's contribution, we change only the allocation criterion while holding candidate operations, calibration data, search, and training fixed.

Table 11's 0.98/0.95 correlations measure how Fisher and bending rank existing boundary masks. The new comparison directly tests the allocation decision: each score selects a mask under the same budget, and the selected masks are trained under identical conditions. This isolates the score's effect on the final model.

In the GPT-2 Medium GEMM comparison, update bending, conditional activation-Fisher, conditional reconstruction, and executed forward loss share 144 layer–operation–pass candidates, four calibration batches, cost-normalized greedy search, and comparison with the best affordable single action. Fisher and reconstruction are recomputed for every candidate allocation. All four masks use exactly 1/18 of block-GEMM work in BF16 (approximately 5.56%).

Each mask is fixed before 1,024 updates over 2.10M training tokens. These runs simulate NVFP4-style GEMMs with the LM head in BF16. Final validation is disjoint from calibration; entries are means ± sample SD over six paired seeds:

CriterionPolicy lossCommon-BF16 loss
Update bending3.05239 ± 0.001123.02537 ± 0.00130
Conditional activation-Fisher3.05367 ± 0.001423.02743 ± 0.00167
Conditional reconstruction3.06396 ± 0.001373.03223 ± 0.00094
Executed calibration loss3.05303 ± 0.001273.02797 ± 0.00084

Bending lowers policy loss by 0.00128 ± 0.00133 and common-BF16 loss by 0.00206 ± 0.00135 relative to conditional Fisher (mean ± SD of paired differences). Both improve on 5/6 seeds, including all three additional seeds evaluated after the masks were fixed. On those three confirmation seeds, the reductions are 0.00114 ± 0.00091 and 0.00225 ± 0.00050, respectively. Against reconstruction, both losses improve on 6/6 seeds; against executed calibration loss, common-BF16 loss improves on 6/6. With granularity, spending, and search held fixed, these comparisons isolate the effect of the allocation score.

Preserving the BF16 direction maintains BF16-level learning under large gradient-norm changes. Table 15 independently changes gradient norm and direction: after learning-rate tuning, BF16, norm-only, and direction-only AdamW training reach 3.248 ± 0.001, 3.249 ± 0.001, and 3.645 ± 0.055, respectively, over three seeds. These runs disable clipping and evaluate without quantizers. The direction-preserving intervention maintains learning, identifying the value of the information that bending measures.

The allocation ablation connects direction preservation to precision selection: at 124M, eight scores share 72 pass candidates, the search framework, and exactly 16.67% BF16 work. Complete-test evaluation over three paired seeds gives 54.6 ± 1.0% / 52.2 ± 0.6% policy-gap closure for gradient/update bending, versus 22.6 ± 0.5% for norm distortion and 0.5 ± 1.2% for local reconstruction. Full-gradient and full-update relative errors close 53.8 ± 0.9% and 52.2 ± 0.6%. The error decomposition identifies the directional information shared by these effective scores: after norm matching, squared relative error equals exactly twice the bending. Bending isolates this component and makes it an explicit allocation objective.

A 2 × 2 gradient/update × bending/L2 ablation at 355M and 774M additionally separates the choice of vector from the error metric, with candidates, search, and 5.56% work fixed. Update bending improves both loss endpoints over raw-gradient L2 on all three paired seeds at each size. Within update space, bending and L2 select the same 355M mask. Together, these controls establish the practical value of measuring the optimizer's learning direction and explain how full-update L2 captures the same useful information. The full tables and error decomposition appear in our response to PHqz.

W2 / Q1. Actual peak GPU memory, end-to-end time, and calibration overhead

Bending-guided native FP8 training recovers 83.7 ± 6.5% of the all-FP8 policy-loss gap to BF16 while retaining the all-FP8 peak allocation of 27.28 GiB. It reduces complete training time by 4.6% and peak allocation by 12.5% relative to BF16, including calibration. Llama-3.2-1B trains for 33.55M tokens, batch 8, length 1,024, and 4,096 updates on GH200, with three paired seeds. Bending selects Fprop protection at the model's own initial weights from the four feasible pass-type policies under a one-third-work cap.

Native training policyMean step (ms)Complete training including calibration/startup (s)Peak allocated (GiB)Policy lossCommon-BF16 loss
BF16177.9 ± 0.6772.2 ± 4.931.182.32394 ± 0.000982.32394 ± 0.00098
All FP8156.6 ± 0.7691.3 ± 1.827.282.33006 ± 0.001762.32610 ± 0.00176
Bending: Fprop in BF16167.4 ± 1.0736.5 ± 10.227.282.32502 ± 0.001072.32502 ± 0.00107

The selected policy lowers all-FP8 policy loss by 0.00504 ± 0.00134, closing 83.7 ± 6.5% of the policy-loss gap. It saves 35.6 ± 9.7 seconds relative to BF16, approximately 1.05× faster, and reduces peak allocation by 3.91 GiB (12.5%). Policy loss improves over all-FP8 and elapsed time over BF16 on 3/3 paired seeds. Common-BF16 loss improves over all-FP8 by 0.00107 ± 0.00143, with lower loss on 2/3 seeds.

Calibration over the four pass-type policies takes 4.59 ± 0.04 seconds, included in the total and accounting for 0.62% of it. Timing includes pre-tokenized data/model loading, observed compilation/startup, operand quantization, all GEMM and non-GEMM work, clipping, AdamW, and CPU handling. Final disjoint evaluation is timed separately; common offline tokenization and imports are excluded. Step time averages the last 1,024 updates. The complete timer uses the existing persistent compiler cache; peak allocation covers calibration and training.

At 355M, coarse-to-fine calibration returns the same 12-action path and final mask while halving selection time. The procedure from Appendix C.1 reduces calibration wall time from 369.55 s to 182.80 s (50.5%) under the same eight-A100 setup. Both modes use identical operand caching, worker arithmetic, and all 144 singleton checks. Calibration is paid once; frozen-mask training requires no reference-gradient probes. The detailed search measurements and the single-A100 calibration-through-evaluation costs for the W1 comparison are provided below under Additional measurements for W2 / Q1.

For the separate boundary-cache claim, we implemented actual E4M3 saved tensors, retaining protected tensors in BF16. At batch 8 and sequence length 1,024:

GPT-2 124M boundary policyExtra activation memory above all-FP8 cacheReduction versus priorPeak allocation per training step
Production-style prior543.4 MiB—10.459 GiB
Three-group bending mask489.4 MiB9.9%10.406 GiB
Module-level bending mask417.4 MiB23.2%10.336 GiB

This storage benchmark uses synthetic tokens and the same memory-efficient cross-entropy for every policy. It verifies the cache savings at the quantity priced by the memory budget; the last column measures the peak over a complete forward, backward, and AdamW step.

The simulated NVFP4 allocation gain is stable under FP64 reconstruction and multiplication: selected-pass gap closure is 51.7 ± 1.0% with the original simulator and 51.9 ± 0.5% with FP64 reconstruction and multiplication, using the same FP4 codes and scales and BF16 output rounding. Preserving the gain under higher-accuracy simulator arithmetic strengthens the numerical evidence for the selected allocation. The native FP8 trajectories establish complete hardware training benefits, and Table 27 supplies native NVFP4 GEMM timings.

W3 / Q2. Larger models with independently selected masks

Independently selected bending allocations achieve lower mean policy loss than Fisher at all three GPT-2 scales—355M, 774M, and 1.56B—under the shared 5.56% BF16-work budget. Each mask is selected at its model's own weights and trained for 1,024 updates over 2.10M tokens, extending Appendix E.10's geometry analysis to continued-pretraining trajectories.

At 355M and 774M, full searches consider 144 and 216 individual layer–operation–pass actions, respectively, using update bending. At 1.56B, Appendix C.1's two-stage procedure first compares pass types, then selects individual Fprop actions using raw-gradient bending. Fisher shares the candidate granularity and cost constraint in each comparison. All GEMM comparisons keep the LM head in BF16.

Every mask below uses exactly 1/18 of block-GEMM work in BF16 (approximately 5.56%). Entries are means ± sample SD over six paired seeds at 355M and 774M, and three at 1.56B:

ModelSelectorPolicy lossCommon-BF16 loss
355MUpdate bending3.05239 ± 0.001123.02537 ± 0.00130
355MConditional Fisher3.05367 ± 0.001423.02743 ± 0.00167
774MUpdate bending2.88461 ± 0.001072.86703 ± 0.00100
774MStatic Fisher2.88572 ± 0.000712.86707 ± 0.00087
1.56BGradient bending2.77523 ± 0.000882.75887 ± 0.00102
1.56BStatic Fisher2.77669 ± 0.000892.75990 ± 0.00100

The 355M/774M rows use calibration-disjoint validation; the 1.56B rows use all 1,096 complete test blocks, or 280,576 input tokens. Results for both evaluated XL budgets, including 16.67% work, appear below under Additional budget results for W3 / Q2. The XL Fisher masks equal the exact modular knapsack optima at both budgets.

At 1.56B and 5.56% work, the paired reductions are 0.00146 ± 0.00019 in policy test loss and 0.00104 ± 0.00092 in common-BF16 test loss, both on 3/3 seeds. A separate 131,072-token calibration-disjoint validation slice confirms the ordering at both endpoints. At 774M, policy loss falls by 0.00111 ± 0.00099 on 5/6 seeds. Together with the 355M results, these independent selections consistently improve mean policy loss across the three model sizes at the shared budget.

On Llama-3.2-1B, independently selected Fprop protection achieves the lowest mean policy loss among the three pass types at identical BF16 work. Own-model calibration selects Fprop protection before the 33.55M-token native FP8 trajectories reported above. In a separate equal-work experiment, each policy protects exactly one third of GEMM work and trains for 33.55M tokens. Entries are means ± sample SD over three paired seeds:

Protected pass type, native Llama-3.2-1BPolicy lossCommon-BF16 loss
Fprop, selected by bending2.32524 ± 0.001112.32524 ± 0.00111
Dgrad2.32890 ± 0.000792.32511 ± 0.00054
Wgrad2.33051 ± 0.001542.32604 ± 0.00158

At the same one-third BF16-work budget, Fprop protection lowers policy loss by 0.00367 ± 0.00067 versus Dgrad protection and 0.00528 ± 0.00043 versus Wgrad protection, improving on 3/3 paired seeds in each comparison. Common-BF16 evaluation separately compares the learned weights; backward-only protection retains FP8 arithmetic in policy evaluation. Fresh calibration returns the original Fprop choice at all 15 seed/checkpoint states, with held-out assessment using actual AdamW moments.

For the submitted 355M from-scratch experiment, independent selection at two 355M random initializations recovers the same eight actions used by the original 134.22M-token trajectories. Target-model calibration thus independently supports that allocation. The new continued-pretraining results above additionally train masks selected at each larger model's own weights.

We have added these training and independent-selection results in Appendix H.2 and linked them from Appendix E.10, with the precise candidate space and scoring variant stated for each model.

Reviewer_rykvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how low-precision computation changes gradient directions during training. The authors show that local quantization error doesn't always predict its impact on optimization and propose gradient bending as a criterion for deciding which operations should stay in higher precision.

Critical strengths
  • The GPT-2 example is interesting: quantizing the logits and final LayerNorm output gives similar local errors (2.6% vs. 2.8%), but the resulting gradient changes are 62% vs. 3%. This makes a good case for looking beyond local reconstruction error.

  • In the NVFP4 experiments, keeping forward GEMMs in BF16 closes 99% of the training loss gap while using high precision for only one-third of the block GEMM work. That's a useful finding for precision allocation.

Critical weaknesses
  • The FP4 training results mostly use simulated quantization. The native NVFP4 experiments focus on GEMM timings, so it's still unclear how much end-to-end training speedup the method would provide.

  • In Table 11, Activation-Fisher actually has a higher correlation than bending for boundary masks (0.98 vs. 0.95). The advantage of bending over this stronger baseline isn't fully clear in that setting.

Questions for authors

Given that Activation-Fisher reaches 0.98 correlation compared with 0.95 for bending in Table 11, could you show a matched-budget training case where bending actually gives a better final loss?

2 comments

Thank you for highlighting the local-error example and the value of forward-GEMM protection.

W2 / Q1. Matched-budget training comparison with activation-Fisher

At 355M and identical 5.56% BF16 work, update bending achieves the lowest mean policy and common-BF16 losses among all four criteria, with candidate operations and the search procedure held fixed.

Activation-Fisher scores quantization errors by their sensitivity to loss. Bending scores their effect on the direction of the parameter update that drives training. It evaluates forward and backward quantization through their shared effect on that update, providing a common scale for allocating precision across the three GEMM passes.

Table 11's 0.98/0.95 correlations measure how Fisher and bending rank existing boundary masks. We test the allocation decision directly by measuring the quality of the model selected at a fixed cost. We change only the score, select masks from identical candidates with the same budget and search, and train them with the same protocol and paired seeds. Bending improves both loss endpoints in the GEMM comparison below and improves learned-weight loss in Table 11's boundary setting, also evaluated at matched cost.

At GPT-2 Medium's pretrained weights, only the allocation score changes: all four criteria use the same 144 layer–operation–pass candidates, four calibration batches, GEMM-work costs, and gain-per-cost greedy search. Each candidate protects Fprop, Dgrad, or Wgrad for a layer's attention or MLP GEMMs. Every allocation executes exactly 1/18 of block-GEMM work in BF16 (approximately 5.56%).

Fisher, reconstruction error, and forward loss are recomputed for every candidate allocation. Conditional Fisher weights the current forward quantization residuals by activation loss gradients obtained with BF16 backward computation; conditional reconstruction measures the current forward and backward GEMM residuals. Each search also compares its greedy result with the best affordable single action.

Each allocation is fixed before training and used for 1,024 updates over 2.10M WikiText-103 tokens. These runs simulate NVFP4-style GEMMs and keep the LM head in BF16. Final validation uses data disjoint from calibration; results are mean ± SD over six paired seeds:

Criterion, identical 5.56% workPolicy lossCommon-BF16 loss
AdamW update bending3.05239 ± 0.001123.02537 ± 0.00130
Conditional activation-Fisher3.05367 ± 0.001423.02743 ± 0.00167
Conditional reconstruction error3.06396 ± 0.001373.03223 ± 0.00094
Executed calibration loss3.05303 ± 0.001273.02797 ± 0.00084

Bending improves both mean losses and wins on 5/6 paired seeds for each metric, including all three additional seeds evaluated with the allocations fixed. The paired improvements over conditional activation-Fisher are 0.00128 ± 0.00133 in policy loss and 0.00206 ± 0.00135 in common-BF16 loss (mean ± SD of paired differences across six seeds). With granularity, search, and spending matched, these gains directly establish the effect of the allocation criterion on the trained model.

The separate 1.56B comparison also improves policy test loss over static activation-Fisher by 0.00146 ± 0.00019 on 3/3 seeds at 5.56% BF16 work. The larger-model results, including both tested budgets and evaluation precisions, appear in our response to kYY2, W3 / Q2.

In Table 11's NVFP4-style boundary setting, both selectors use the same 13 candidates, four calibration batches, cache costs, greedy search, and comparison with the best single action. Both allocate BF16 to 89.65% of the eligible activation-cache footprint. Fisher's greedy allocation also matches the exact knapsack solution for its scores. Each allocation is fixed before training and used for 8.39M tokens with three paired seeds. Evaluation on all complete WikiText-103 test blocks gives:

Selector, identical 89.65% BF16 cache footprintPolicy test lossCommon-BF16 test loss
Bending3.23875 ± 0.001053.23270 ± 0.00118
Activation-Fisher3.23861 ± 0.001383.23445 ± 0.00029

In Table 11's boundary setting, bending learns better weights at identical activation-cache cost: common-BF16 test loss improves by 0.00175 ± 0.00133 on all three seeds. This evaluation holds each model's learned weights fixed and uses identical BF16 arithmetic, directly comparing the training outcome as in Section 6. Policy losses remain comparable (3.23875 versus 3.23861). A separate calibration-disjoint validation slice confirms the learned-weight improvement, 0.00339 ± 0.00117, also on 3/3 seeds. The two matched-cost experiments therefore connect bending-based selection to improved training outcomes in both allocation settings.

W1. End-to-end training time and memory with native FP8

On Llama-3.2-1B, bending-guided native FP8 training recovers 83.7 ± 6.5% of the all-FP8 policy-loss gap to BF16 while retaining the all-FP8 peak allocation of 27.28 GiB. Relative to BF16 training, it reduces end-to-end time by 4.6% and peak allocated GPU memory by 12.5%, with calibration included.

Training uses 33.55M WikiText-103 tokens, batch 8, sequence length 1,024, and 4,096 AdamW updates on GH200, with three paired seeds. Calibration at the starting weights compares all-low FP8 with Fprop, Dgrad, and Wgrad protection under a one-third block-GEMM-work cap, and selects Fprop protection. Eligible low-precision GEMMs execute native E4M3 kernels.

Training policyTime including calibration/startup (s)Peak allocated GPU memory (GiB)Policy lossCommon-BF16 loss
BF16772.2 ± 4.931.182.32394 ± 0.000982.32394 ± 0.00098
All FP8691.3 ± 1.827.282.33006 ± 0.001762.32610 ± 0.00176
Bending-selected Fprop protection736.5 ± 10.227.282.32502 ± 0.001072.32502 ± 0.00107

The selected policy lowers all-FP8 policy loss by 0.00504 ± 0.00134 and saves 35.6 ± 9.7 seconds relative to BF16, with both improvements on 3/3 paired seeds. Peak allocated memory falls by 3.91 GiB (12.5%), and complete training is approximately 1.05× faster than BF16. Calibration over the four pass-type policies takes 4.59 ± 0.04 seconds, included in the total and accounting for 0.62% of it. Common-BF16 loss improves over all-FP8 by 0.00107 ± 0.00143, with lower loss on 2/3 seeds.

Timing starts before pre-tokenized data/model loading and includes calibration, compilation/startup during the measured runs, operand scaling and casting, GEMMs, non-GEMM operations, backward computation, clipping, AdamW, and CPU data handling. Common offline tokenization and framework imports are excluded; final evaluation on 131,072 calibration-disjoint tokens is timed separately. These fixed-token measurements use the existing persistent compiler cache. Peak memory covers calibration and training.

The simulated NVFP4 allocation gain is stable under FP64 reconstruction and multiplication. At 16.67% BF16 work, selected-pass gap closure is 51.7 ± 1.0% with the original simulator and 51.9 ± 0.5% with FP64 reconstruction and multiplication, using the same FP4 codes and scales and BF16 output rounding (GPT-2 124M, three paired seeds, 2.10M training tokens per run). Preserving the gain under higher-accuracy simulator arithmetic strengthens the numerical evidence for the selected allocation. The native FP8 trajectories establish complete hardware training benefits, and Table 27 supplies native NVFP4 GEMM timings.

Comments on the paper

1 comment

We thank the reviewers for recognizing the importance of tracing quantization error to the gradient used for learning, the direction-versus-norm interventions, and the forward-GEMM allocation result.

Bending-guided allocation turns preservation of the learning direction into a rule for choosing where to spend high precision and how much is needed. Calibration measures how candidate protections restore the gradient or optimizer-update direction per unit cost. The resulting allocation path identifies useful protections within the budget, and the BF16 reference target identifies the cost of reaching the desired direction fidelity. This gives a common decision rule across precision formats, model sizes, and protection granularities.

We measure both performance under the selected precision policy and the quality of the learned weights. Each trained model is evaluated twice, keeping its learned weights unchanged:

  • Policy loss evaluates the model using the precision settings used during training. It measures performance with the selected precision policy.
  • Common-BF16 loss disables quantizers and evaluates every model using the same BF16 arithmetic, as in Section 6. It compares the learned weights under identical evaluation conditions.

The additional experiments establish the contribution of this decision rule through controlled allocation, training, and hardware measurements:

  • Criterion: changing only the allocation score, with candidates, calibration data, BF16 work, search, and training held fixed, gives update bending the lowest mean policy and common-BF16 losses among four criteria at 355M. Direction/norm interventions and gradient/update controls show why preserving the learning direction benefits training; the error decomposition explains the agreement between bending and full-update L2.
  • Protection budget: calibration identifies LM-head protection as the target-level FP8 boundary allocation and selects the extra protections that remove 75.0% of the remaining learned-weight loss gap in the NVFP4-style boundary setting. With the LM head already in BF16, selecting GEMM passes closes substantially more of the loss gap than equal-work block protection.
  • Scale: masks selected independently at each model's own weights achieve lower mean policy loss than Fisher at 355M, 774M, and 1.56B under the shared 5.56% BF16-work budget. Native Llama-3.2-1B training extends the allocation principle to another architecture.
  • During training: checkpoint comparisons validate reuse of the initial allocation in continued pretraining, where it preserves the update direction as effectively as fresh selection. In from-scratch training, bending identifies a productive reallocation: switching protection locations improves both final test losses on every paired seed at the same BF16 work per step.
  • Systems: native FP8 training recovers 83.7 ± 6.5% of the all-FP8 policy-loss gap, retains the all-FP8 peak memory, and reduces complete training time by 4.6% and peak allocation by 12.5% relative to BF16, including calibration. Reproduction under FP64 arithmetic independently supports the numerical fidelity of the simulated NVFP4 allocation gain.

Loss is token cross-entropy in nats; lower is better. Unless otherwise specified, tables report means ± sample standard deviations over three paired training seeds. For improvements, we first subtract the two methods' losses within each seed, then report the mean ± sample SD of these paired differences. Gap closure is the fraction of the all-low-to-BF16 loss gap recovered by a policy, computed within each seed and then averaged.

The two allocation settings retain their definitions from Section 2: boundary protection is priced by activation-cache footprint; GEMM protection is priced by the fraction of transformer-block GEMM work executed in BF16. Fprop, Dgrad, and Wgrad denote the forward, input-gradient, and weight-gradient matrix multiplications. The NVFP4-style boundary quantizer uses FP32 block scales, whereas the simulated NVFP4-style GEMM core uses E2M1 values, 16-element blocks, E4M3 block scales, and an FP32 tensor scale. We have added the matched-budget training comparisons and native FP8 results to Section 6 (Tables R1–R2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.