- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how low-precision computations affect gradient and optimizer-update directions. It proposes using directional deviation from a higher-precision reference to guide precision allocation. On calibration data, a greedy procedure selects operations whose restoration to higher precision provides favorable reductions in directional deviation relative to their cost. The resulting precision configuration is then fixed during training.
Critical strengths
-
The paper examines propagated gradient and optimizer-update effects rather than relying exclusively on local tensor reconstruction error. Its direction-versus-magnitude interventions and optimizer-aware analysis provide useful evidence that similarly sized numerical errors can have different consequences for optimization.
-
The paper connects its diagnostic measurements to precision allocation and subsequent training outcomes. The reported ranking correlations and fixed-configuration training experiments provide evidence that calibration-time directional measurements contain useful information about later training quality.
Critical weaknesses
-
The independent contribution of the directional criterion to precision allocation remains insufficiently isolated. The selection framework is a conventional greedy budgeted search, so the methodological contribution depends substantially on the scoring criterion. Correlation between directional deviation and final loss does not by itself demonstrate that optimizing this criterion produces better configurations. The existing comparisons do not fully establish a consistent advantage over alternative propagated-error or optimizer-aware criteria under matched candidate sets, budgets, and search procedures.
-
The continued suitability of a configuration selected before training is not sufficiently characterized. Successful fixed-configuration training provides useful evidence and should not be dismissed, but does not show how sensitivities or preferred protection sets change across training stages. The paper leaves uncertainty about when the initial allocation remains effective and when recalibration would materially improve the quality–cost tradeoff.
Questions for authors
-
Under identical candidate operations, budgets, and selection procedures, how do configurations selected using directional deviation compare with those selected using local reconstruction error and full-gradient or optimizer-update error magnitudes? Which existing experiments most directly isolate the benefit of the directional criterion?
-
Do intermediate-checkpoint measurements show that the initially selected protection set remains effective? Please distinguish evidence that a fixed configuration can train successfully from evidence that its allocation remains suitable throughout training.
7 comments
The 124M comparison extends the test to eight criteria, including full-gradient and full-update L2. It uses GPT-2 124M with simulated NVFP4-style GEMM operands and a BF16 head. All eight objectives share:
- the same 72 layer–operation–pass actions, covering Fprop, Dgrad, and Wgrad;
- the same eight calibration batches and GEMM-work costs;
- the same full gain-per-cost greedy framework and best-single-action safeguard;
- exactly 1/6 of block-GEMM work in BF16 (approximately 16.67%);
- frozen-mask training for 1,024 updates over 2.10M tokens, with three paired seeds.
Full-vector and forward-loss objectives score each executed candidate mask. Fisher and reconstruction use their defined modular residual risks in the same search framework. Evaluation uses all complete WikiText-103 test blocks, separate from calibration, under both the selected precision policy and common BF16 arithmetic. Entries are means ± sample SD over three paired seeds:
| Score | Policy test loss | Common-BF16 test loss | Policy-gap closed |
|---|---|---|---|
| Update bending | 3.34450 ± 0.00087 | 3.32177 ± 0.00137 | 52.2 ± 0.6% |
| Gradient bending | 3.34269 ± 0.00082 | 3.32281 ± 0.00048 | 54.6 ± 1.0% |
| Full-gradient relative L2 error | 3.34330 ± 0.00095 | 3.32284 ± 0.00076 | 53.8 ± 0.9% |
| Full-update relative L2 error | 3.34450 ± 0.00087 | 3.32177 ± 0.00137 | 52.2 ± 0.6% |
| Gradient-norm distortion | 3.36679 ± 0.00082 | 3.33452 ± 0.00125 | 22.6 ± 0.5% |
| Executed forward-loss change | 3.34151 ± 0.00055 | 3.32441 ± 0.00063 | 56.2 ± 2.2% |
| Activation-Fisher | 3.34342 ± 0.00024 | 3.32068 ± 0.00050 | 53.7 ± 1.3% |
| Local reconstruction error | 3.38348 ± 0.00197 | 3.34361 ± 0.00104 | 0.5 ± 1.2% |
At identical granularity, BF16 work, and search, bending recovers more of the training-loss gap than norm-only and local reconstruction criteria. With 16.67% of block-GEMM work in BF16, gradient bending recovers 54.6 ± 1.0% of the loss gap between all-low and BF16 training. It lowers policy test loss by 0.02411 ± 0.00064 versus norm distortion and 0.04079 ± 0.00187 versus local reconstruction, each on 3/3 paired seeds. A separate 131,072-token calibration-disjoint validation slice preserves these comparisons. At the other tested budget, 5.56% BF16 work, update bending and full-update error select the same mask and close 24.0% of the complete-test policy gap, versus −1.3% for local reconstruction.
The controls identify the information that makes these allocations effective. At 124M, update bending and full-update L2 select the same masks at both budgets, and both bending scores achieve lower common-BF16 loss than allocation by executed forward loss in the eight-criterion comparison. At 355M, update bending achieves the lowest mean loss in both evaluations among the four criteria. The decomposition below identifies the directional component shared by bending and full-update L2 and quantifies its dominance in the measured update error.
W1 / Q1, continued. Directional error explains the effective optimizer-aware scores
Bending isolates the directional component that dominates the measured optimizer-update error. Full-vector L2 contains both length and direction changes; norm distortion measures length alone. The exact decomposition connects these criteria and explains why full-update L2 and bending select the same protections in several comparisons.
The exact decomposition makes this distinction explicit. For , the ratio of the low-precision and BF16 vector lengths, and ,
= \underbrace{(r - 1)^2}_{\text{length mismatch}} + \underbrace{2rD_v}_{\text{direction mismatch}}.$$ Matching the two vector lengths makes $r = 1$, so the length term vanishes and squared relative L2 error equals exactly twice the bending. This identity shows precisely how full-vector error incorporates the direction-preservation objective. Full-update L2 and update bending select the same 355M mask and the same 124M masks at both budgets. Identical masks share the same training runs. Direction accounts for 99.89–99.97% of squared update error in measurements with actual accumulated checkpoint moments across nine GEMM checkpoint/mask cells. Each value averages 64 held-out batches at archived 124M or 355M states from seed 17. Thus, in these measurements, full-update L2 measures almost entirely the error component that bending explicitly targets. The agreement between the selectors has a quantitative explanation in the learning-direction geometry, connecting the paper's analysis to the observed allocation choices.Scoring the AdamW update makes precision allocation sensitive to the optimizer's actual learning direction. AdamW combines the gradient with its accumulated history and rescales individual coordinates to produce the parameter update. Update bending measures the directional effect of quantization after this transformation, aligning the allocation objective with the parameter change used for learning.
A controlled 2 × 2 ablation separates two choices: which vector to measure (raw gradient or AdamW update) and which difference to measure (direction through bending, or the full vector through relative L2). Within each model, all four combinations share the complete candidate set, calibration, executed-mask re-evaluation, greedy rule, safeguard, and identical 5.56% BF16 work. Means ± sample SD over three paired seeds, with calibration-disjoint validation, are:
| Scored vector / error | 355M policy loss | 355M common-BF16 loss | 774M policy loss | 774M common-BF16 loss |
|---|---|---|---|---|
| Gradient / bending | 3.05677 ± 0.00054 | 3.02802 ± 0.00194 | 2.88861 ± 0.00170 | 2.86782 ± 0.00180 |
| Gradient / relative L2 | 3.05512 ± 0.00115 | 3.02789 ± 0.00142 | 2.88624 ± 0.00147 | 2.86740 ± 0.00128 |
| AdamW update / bending | 3.05153 ± 0.00081 | 3.02456 ± 0.00143 | 2.88434 ± 0.00138 | 2.86633 ± 0.00089 |
| AdamW update / relative L2 | 3.05153 ± 0.00081 | 3.02456 ± 0.00143 | 2.88378 ± 0.00244 | 2.86613 ± 0.00065 |
For both error metrics, scoring the AdamW update gives lower mean policy and common-BF16 losses than scoring the raw gradient at both scales. Update bending improves both loss endpoints over raw-gradient L2 on every paired seed at 355M and 774M. Its policy-loss reductions are 0.00360 ± 0.00143 at 355M and 0.00190 ± 0.00030 at 774M. Within update space, L2 and bending select the same mask at 355M and give close results at 774M. These controls establish the value of following quantization error through the optimizer and targeting the direction of the resulting update.
Preserving the BF16 gradient direction maintains learning under large norm changes. Section 4.2 / Table 15 isolates direction by changing gradient norm or direction separately during training, tuning the learning rate, disabling clipping, and evaluating the learned weights without quantizers:
| Gradient supplied to the optimizer | AdamW final loss | Momentum-SGD final loss |
|---|---|---|
| BF16 reference | 3.248 ± 0.001 | 3.292 ± 0.004 |
| BF16 direction, quantized-gradient norm | 3.249 ± 0.001 | 3.259 ± 0.002 |
| Quantized-gradient direction, BF16 norm | 3.645 ± 0.055 | 3.973 ± 0.012 |
Keeping the BF16 direction preserves learning even when the gradient length changes substantially. Restoring only the BF16 length leaves a large loss penalty when the direction is distorted. Together, the experiments establish the mechanism and its use in allocation: the interventions identify the importance of direction, the score swaps connect that information to better protection choices, and the training runs verify the resulting loss improvements. The update-error decomposition explains why full-update L2 also captures this useful directional information.
The submitted experiments also show training gains from bending-guided allocation at matched granularity and cost. Table 26 compares bending-guided and reconstruction-error allocation over the same 72 individual GEMM-pass candidates at 5.56% protected work, assessing gradient and update bending on held-out batches. Table 28 reports three-seed continued-pretraining outcomes under the submitted validation protocol: at the same work budget, bending-guided allocation closes 23.6 ± 2.2% of the loss gap, versus 0.6 ± 2.1% for reconstruction-error allocation. The new score swaps above additionally hold the selection procedure fixed and evaluate full-gradient and full-update error objectives.
The direction criterion also predicts a useful precision budget before training. Table 28's first mask below the BF16 raw-gradient target uses 29.6% work and learns weights within 0.004 of BF16 on average. Bending therefore connects the learning mechanism to both allocation decisions: which computations to protect and how far to extend protection. We have connected the intervention, direct allocation controls, and budget prediction in Sections 4–6 and Appendix H.1.
Thank you for these questions. The controlled experiments connect preservation of the learning direction to better precision choices and better trained models. Changing only the score gives update bending the lowest mean policy and common-BF16 losses among four criteria at 355M. Direction/norm interventions identify the useful error component, and the exact error decomposition explains why full-update L2 captures the same directional information. Checkpoint experiments establish a practical role for the criterion throughout training: they validate reuse of the initial allocation in continued pretraining and identify a reallocation that improves both test losses in from-scratch training at unchanged per-step BF16 work.
W1 / Q1. Changing the allocation criterion improves the trained model at matched cost
Update bending improves the selected allocation and final loss with candidates, budget, search, and training held fixed. We first test four scores at 355M, then extend the comparison to full-gradient and full-update error criteria at 124M. The scores measure the following changes relative to BF16:
Bending selects protections that bring the learning direction closer to its BF16 reference per unit of high-precision work. We first compare update bending, conditional Fisher, conditional reconstruction, and executed calibration loss at pretrained GPT-2 355M weights. All four objectives are recomputed for every candidate allocation. They share the same 144 layer–operation–pass candidates, four calibration batches, GEMM-work costs, gain-per-cost greedy search, and comparison with the best affordable single action. Each allocation uses exactly 1/18 of block-GEMM work in BF16 (approximately 5.56%).
Each selected mask is fixed before training for 1,024 updates over 2.10M tokens, using simulated NVFP4-style GEMMs and a BF16 LM head. Validation is disjoint from calibration. Policy loss evaluates each model with its training precision settings; common-BF16 loss evaluates each model's unchanged learned weights using the same BF16 arithmetic. Entries are means ± sample SD over six paired seeds:
Update bending achieves the lowest mean loss under both evaluations among these four criteria. Relative to conditional Fisher, bending lowers policy loss by 0.00128 ± 0.00133 and common-BF16 loss by 0.00206 ± 0.00135 (mean ± SD of six paired differences). The common-BF16 reduction has a paired 95% t-confidence interval of [0.00064, 0.00347]. Both losses improve on 5/6 seeds, including all three additional seeds evaluated after the allocations were fixed. On those three confirmation seeds, the common-BF16 reduction is 0.00225 ± 0.00050. Relative to executed calibration-loss allocation, common-BF16 loss falls by 0.00260 ± 0.00112, improving on 6/6 seeds.
The score changes the protection decisions themselves: reconstruction selects ten Dgrad, one Wgrad, and one Fprop action, whereas bending selects twelve Fprop actions. The controlled experiment therefore connects the objective to a different allocation and then to lower final loss, at the same BF16 work. Improvement under both evaluations establishes better performance with the selected precision policy and better learned weights under common arithmetic.