- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies where to allocate numerical precision during low-precision training, how precision should vary over training time, and how precision requirements scale with model size and compute. It uses a random-feature linear regression model with power-law data and target structure, explicitly modeling forward and backward rounding errors.
For constant-step-size SGD with last-iterate evaluation, the findings suggest that forward computations benefit more from higher precision, while rounding errors in weight-gradient computation can be masked by existing SGD noise. Precision allocated later in training is more valuable because the resulting errors have less time to dissipate. Parameter averaging changes the optimal allocation and may favor allocating backward precision earlier.
Critical strengths
- The paper provides a clear model of how low-precision training affects scaling laws. Its conclusions offer useful insights into both the location and timing of precision allocation, with potentially valuable implications for practical training.
Critical weaknesses
- Although the paper proposes precision-allocation strategies, the real-network experiments do not directly validate these strategies. This limits the evidence that the theoretical prescriptions improve practical training under a fixed compute budget.
Post-rebuttal update: The additional transformer experiments provide useful support for the predicted importance of later forward precision. I appreciate the paper’s originality and theoretical insights, which I believe outweigh the remaining empirical limitations. I have raised my rating to 4.
Questions for authors
- Could the authors evaluate at least one real neural network under the same compute budget, comparing a well-tuned fixed-precision baseline, a precision-schedule-only approach, and an approach that jointly optimizes the learning-rate and precision schedules?
- When parameter averaging is used, can the theoretically predicted change in precision allocation—particularly the shift toward earlier backward precision—also be observed empirically?
Official Comment by Authors
We thanks the the reviewer's constructive review. We now address some comments and present additional simulation results. All evaluations disable added SR, keeping FP32 for the MLP and BF16/FP32 for the transformer. Bits mean mantissa bits of ideal binary SR.
W1. We agree that the original location ablations did not establish an allocation benefit under a common network-training budget. We therefore ran a three-way selection comparison and a backward-timing test on the MLP, and additional transformer experiments for further explanations. We distinguish scheduling, learning-rate changes and additional training below.
Q1. We tested a CIFAR-10 MLP (3072–1024–10, momentum SGD, batch 128), rounding the activation operands in forward and weight-gradient products. Each candidate training run had the same modeled cost cap, equivalent to 9,750 updates at two bits for both copies. At p bits, products cost (p+9)/16 relative to no added SR; input-gradient products and a proxy for other work are full price. Fixed three bits affords 8,962 updates. All use 200-update warmup and decay over the last 20%. Tuning costs are additional; we tuned peak rates, not the decay fraction.
Policies were fixed one/two/three bits or midpoint switches 1→3 and 3→1, with the same bits on both independently rounded copies. Selection used one tuning seed and last-iterate loss on a 5,000-example validation split: (A) fixed policies × five peak LRs; (B) all policies at A's selected LR; (C) all policies × LR multipliers {0.5,1,2} around that LR. Selected policies were tested on four new seeds using the separate 10,000-example test set (±1 seed SE). Full averaging includes all post-update parameters, including warmup:
No switching schedule was selected for final tests. C−A is −0.0721 ± 0.0009 for the last iterate but +0.0472 ± 0.0005 for the average, using paired seeds. This is an LR effect: precision remains fixed. A and C chose the smallest rate available in their searches; C's smaller rate was unavailable to A. With one tuning seed and a limited policy family, this does not establish an advantage over a fully tuned fixed baseline.
On the 30M transformer (AdamW; SR excludes the output head), at 200M tokens with learning-rate decay over the final 20%, we compared fixed two-bit forward-activation SR with one bit for the first half and three for the second, at three peak learning rates (three paired seeds, ±2 SE). At the best tested rate, 0.0012, losses are 4.1861 and 4.1926; schedule minus fixed is +0.0065 ± 0.0076. The best learning rate was the largest tested. No robust scheduling advantage is established.
A separate test on the same transformer uses the default learning rate 0.0006 and rounds both activation copies to two bits. Under equal modeled operand cost, it affords 290.9M tokens versus 200M without added SR (three paired seeds, ±2 SE). Losses are 4.1360 versus 4.1846, a difference of −0.0486 ± 0.0050. This supports a finite benefit from longer training, not precision scheduling or hardware speedup. The reference was not retuned; longer runs draw more with replacement from the same finite corpus and extend cooldowns.
Q2. We tested this directly on the MLP with forward SR disabled and a constant LR after warmup. After 200 common warmup updates, both arms have two 4,775-update phases, using backward three bits then one, or the reverse, at identical modeled cost. The learning rate, 0.005, was selected by the no-SR reference's validation last-iterate loss and was the smallest of three candidates.
The differences are small relative to seed variation; neither order is clearly better. The sign change in the means does not establish the predicted reversal.
On the 30M transformer at 200M tokens, with constant learning rate after warmup, both forward schedules use two bits in the first half; the final quarters use three then one, or one then three (three paired seeds, ±2 SE). Higher precision early gives loss 0.0195 ± 0.0021 higher for the last iterate, 0.0349 ± 0.0038 for EMA and 0.0554 ± 0.0230 for parameters averaged over the second half. Thus later precision beats the reverse order, but averaging did not reduce the estimated timing difference. These forward-rounding tests use EMA or a second-half average; they do not establish the backward-timing reversal under full uniform averaging.