acceptodds
Under review as a conference paper at ICLR 2027

Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding

Abstract

Low-precision training with stochastic rounding has become popular for large language models (LLMs). However, there is currently no theory to guide how much precision each matrix product needs or how it should be scheduled over training. We study one-pass SGD on random-feature regression with power-law spectra, where the features in the forward propagation and the weight-gradient products are stochastically rounded with time-varying precision. We derive an exact recursion for the rounded loss curve and scaling laws for the extra loss due to rounding. For the final weights of a training run, performed at a constant learning rate, these laws answer three questions. Where: rounding in the forward pass limits accuracy, while rounding in the weight-gradient product is hidden by the noise of SGD. When: the bits are best spent late in training. How much: the number of bits needed grows only logarithmically with compute budget, whereas a fixed precision lowers the exponent of the scaling law. Iterate averaging of the output and learning-rate decay remove the noise of SGD and thereby expose the rounding error in the weight-gradient product. This indicates that the learning-rate and precision schedules must be designed together.

Primary area:
learning theory
Keywords:
stochastic rounding, low-precision training, scaling laws, stochastic gradient descent, random features, precision schedules, learning-rate schedules, compute-optimal scaling, mixed-precision training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pDxcOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies where to allocate numerical precision during low-precision training, how precision should vary over training time, and how precision requirements scale with model size and compute. It uses a random-feature linear regression model with power-law data and target structure, explicitly modeling forward and backward rounding errors.

For constant-step-size SGD with last-iterate evaluation, the findings suggest that forward computations benefit more from higher precision, while rounding errors in weight-gradient computation can be masked by existing SGD noise. Precision allocated later in training is more valuable because the resulting errors have less time to dissipate. Parameter averaging changes the optimal allocation and may favor allocating backward precision earlier.

Critical strengths
  1. The paper provides a clear model of how low-precision training affects scaling laws. Its conclusions offer useful insights into both the location and timing of precision allocation, with potentially valuable implications for practical training.
Critical weaknesses
  1. Although the paper proposes precision-allocation strategies, the real-network experiments do not directly validate these strategies. This limits the evidence that the theoretical prescriptions improve practical training under a fixed compute budget.

Post-rebuttal update: The additional transformer experiments provide useful support for the predicted importance of later forward precision. I appreciate the paper’s originality and theoretical insights, which I believe outweigh the remaining empirical limitations. I have raised my rating to 4.

Questions for authors
  1. Could the authors evaluate at least one real neural network under the same compute budget, comparing a well-tuned fixed-precision baseline, a precision-schedule-only approach, and an approach that jointly optimizes the learning-rate and precision schedules?
  2. When parameter averaging is used, can the theoretically predicted change in precision allocation—particularly the shift toward earlier backward precision—also be observed empirically?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thanks the the reviewer's constructive review. We now address some comments and present additional simulation results. All evaluations disable added SR, keeping FP32 for the MLP and BF16/FP32 for the transformer. Bits mean mantissa bits of ideal binary SR.

W1. We agree that the original location ablations did not establish an allocation benefit under a common network-training budget. We therefore ran a three-way selection comparison and a backward-timing test on the MLP, and additional transformer experiments for further explanations. We distinguish scheduling, learning-rate changes and additional training below.

Q1. We tested a CIFAR-10 MLP (3072–1024–10, momentum SGD, batch 128), rounding the activation operands in forward and weight-gradient products. Each candidate training run had the same modeled cost cap, equivalent to 9,750 updates at two bits for both copies. At pp bits, products cost (p+9)/16(p+9)/16 relative to no added SR; input-gradient products and a proxy for other work are full price. Fixed three bits affords 8,962 updates. All use 200-update warmup and decay over the last 20%. Tuning costs are additional; we tuned peak rates, not the decay fraction.

Policies were fixed one/two/three bits or midpoint switches 1→3 and 3→1, with the same bits on both independently rounded copies. Selection used one tuning seed and last-iterate loss on a 5,000-example validation split: (A) fixed policies × five peak LRs; (B) all policies at A's selected LR; (C) all policies × LR multipliers {0.5,1,2} around that LR. Selected policies were tested on four new seeds using the separate 10,000-example test set (±1 seed SE). Full averaging includes all post-update parameters, including warmup:

SearchSelected policy; peak LRLast-iterate lossFull-average loss
A: fixed precision/LRFixed 3 bits; 0.00251.3675 ± 0.00141.2845 ± 0.0012
B: precision policySame run as ASame as ASame as A
C: joint policy/LRFixed 3 bits; 0.001251.2954 ± 0.00181.3317 ± 0.0009

No switching schedule was selected for final tests. C−A is −0.0721 ± 0.0009 for the last iterate but +0.0472 ± 0.0005 for the average, using paired seeds. This is an LR effect: precision remains fixed. A and C chose the smallest rate available in their searches; C's smaller rate was unavailable to A. With one tuning seed and a limited policy family, this does not establish an advantage over a fully tuned fixed baseline.

On the 30M transformer (AdamW; SR excludes the output head), at 200M tokens with learning-rate decay over the final 20%, we compared fixed two-bit forward-activation SR with one bit for the first half and three for the second, at three peak learning rates (three paired seeds, ±2 SE). At the best tested rate, 0.0012, losses are 4.1861 and 4.1926; schedule minus fixed is +0.0065 ± 0.0076. The best learning rate was the largest tested. No robust scheduling advantage is established.

A separate test on the same transformer uses the default learning rate 0.0006 and rounds both activation copies to two bits. Under equal modeled operand cost, it affords 290.9M tokens versus 200M without added SR (three paired seeds, ±2 SE). Losses are 4.1360 versus 4.1846, a difference of −0.0486 ± 0.0050. This supports a finite benefit from longer training, not precision scheduling or hardware speedup. The reference was not retuned; longer runs draw more with replacement from the same finite corpus and extend cooldowns.

Q2. We tested this directly on the MLP with forward SR disabled and a constant LR after warmup. After 200 common warmup updates, both arms have two 4,775-update phases, using backward three bits then one, or the reverse, at identical modeled cost. The learning rate, 0.005, was selected by the no-SR reference's validation last-iterate loss and was the smallest of three candidates.

Paired test-loss difference: higher backward precision early minus lateMean ±1 SE, four seeds
Last iterate+0.0146 ± 0.0095
Full uniform average−0.0003 ± 0.0008

The differences are small relative to seed variation; neither order is clearly better. The sign change in the means does not establish the predicted reversal.

On the 30M transformer at 200M tokens, with constant learning rate after warmup, both forward schedules use two bits in the first half; the final quarters use three then one, or one then three (three paired seeds, ±2 SE). Higher precision early gives loss 0.0195 ± 0.0021 higher for the last iterate, 0.0349 ± 0.0038 for EMA and 0.0554 ± 0.0230 for parameters averaged over the second half. Thus later precision beats the reverse order, but averaging did not reduce the estimated timing difference. These forward-rounding tests use EMA or a second-half average; they do not establish the backward-timing reversal under full uniform averaging.

Reviewer_wESiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies stochastic rounding in one-pass SGD for random-feature linear regression with power-law spectra, independently quantizing the forward and backward feature copies. It derives precision-dependent scaling laws for prediction risk and uses them to characterize where, when, and how many bits to allocate. The predictions are validated through numerical simulations and small-scale neural network experiments.

Critical strengths

1. Theoretical characterization of stochastic rounding.

The theoretical scaling laws are established using practical quantization model of stochastic rounding, offering a theoretical foundation for analyzing precision-dependent prediction risk.

2. Predictions for precision allocation.

The paper derives concrete predictions concerning where and when higher precision is beneficial, as well as how precision requirements scale with compute. These results could provide useful guidance for designing mixed-precision training schedules.

Critical weaknesses

1. Theoretical guarantees rely on an unproven conjecture.

Theorem 4 establishes the main scaling laws for the deterministic-equivalent dynamics, but their transfer to actual SGD with joint forward and backward rounding relies on the unproven Conjecture 2. Moreover, the conjecture covers only constant training controls and a restricted time window, leaving the theoretical guarantees for dynamic precision allocation and some compute-optimal scaling predictions unclear.

2. Theoretical presentation and interpretation.

The theoretical development is notation-heavy and difficult to follow, particularly the transition from the exact recursion to the deterministic-equivalent dynamics. Moreover, although the paper qualitatively explains the sources of forward and backward rounding errors, it provides limited intuition for why they contribute the particular terms and scaling exponents in Theorem 4. This makes the main theoretical insights difficult to assess.

Questions for authors
  1. Can the authors establish the main scaling laws for the actual rounded SGD algorithm without relying on Conjecture 2?
  2. Can the authors provide a more intuitive derivation of the scaling-law terms in Theorem 4? Specifically, why does forward rounding contribute a parameter-norm-dependent term qfwkta−b+1/Mq_{\mathrm{fw}}k_t^{a-b+1}/M, while backward rounding contributes a residual-dependent term qbwrtkt/Mq_{\mathrm{bw}}r_tk_t/M? Moreover, why does iterate averaging change the forward exponent and make the forward–backward interaction term appear at leading order?
  3. Could the authors validate the theoretical findings on larger-scale language models? Specifically, given that existing studies [1, 2] suggest precision does not affect scaling exponents, how well do these theoretical results hold in practice?

References:

[1] Scaling Laws for Precision.

[2] Scaling Laws for Floating–Point Quantization Training.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank reviewer's helpful review and references. We now address questions and comments below.

W1/Q1. Our revised Theorem 1 gives matching risk bounds for actual SGD's last iterate and uniform average, with time-varying schedules without using the old conjecture. More explicitly, we use the exact second-moment recursion and positive-map comparisons bound error accumulation directly to bound risk within fixed multiplicative constants, without requiring or proving Conjecture 2's vanishing relative error.

The bounds require Gaussian data with power-law spectra, independent binary feature SR, input dimension proportional to width, positive label noise, and learning rate and precision schedules fixed before training. For large width MM and spectral exponent aa, the sum of learning rates must lie between a fixed lower threshold and MaM^a, with small enough steps and learning-rate/batch ratios. Risk averages over target, data, and SR, with high probability over the Gaussian feature map. This follows the same setting of the previous neural scaling law theory with exact arithmetic case.

Although we did not explore EMA scaling law, our general scaling law can deal with EMA by inserting its exponential weights into the same output response and check the spectral and feedback bounds. Their scaling depends on its decay and whether the effective averaging window stays fixed or grows with training length. We leave those estimates as future theoretical work.

W2. In the revision, we improve the presentation and interpretation of our results by simplifying notations, putting the technical lemmas in the appendix, and giving more explanations for our theory. Section 3 explains the main theorem before the constant-learning-rate laws. Section 4 uses these bounds to explain precision choices, learning-rate design and compute allocation.

Q2. Let zz be the feature vector, vv the parameters, r=z⊤v−yr=z^\top v-y the residual, and ξ,ζ\xi,\zeta the forward/backward rounding errors. Gradient expands as (z+ζ)(r+ξ⊤v)=zr+z(ξ⊤v)+ζr+ζ(ξ⊤v).(z+\zeta)(r+\xi^\top v)=zr+z(\xi^\top v)+\zeta r+\zeta(\xi^\top v). Here zrzr is the exact gradient. Forward error ξ⊤v\xi^\top v has variance given by a weighted sum of squared parameters. Backward error ζr\zeta r depends on squared residuals. The product term is their interaction: it propagates like backward noise but has the parameter dependence of forward noise. The SR errors are independent and mean zero conditional on the features.

Recall that g1≥…≥gMg_1\ge \ldots \ge g_M are the eigenvalues of covariance of the feature. Forward noise carries zz, where the risk bounds involve gj2g_j^2 weights, versus gjg_j for backward noise. With feature-covariance eigenvalues gj≍j−ag_j\asymp j^{-a}, learning times are of order 1/gj1/g_j, hence about τ1/a\tau^{1/a} directions are learned by cumulative learning rate τ\tau. The power-law target gives a parameter-energy factor τ(a−b+1)/a\tau^{(a-b+1)/a}; bb is the target-energy exponent.

Uniform averaging combines correlated later iterates and suppresses rapidly learned directions more than slow ones. At constant learning rate, the forward-only term gains a factor τ1/a−1\tau^{1/a-1}, giving power (2−b)/a(2-b)/a, whereas backward-only penalties keep the same order. The interaction exists for both outputs. And earlier rounding also changes later noise. It can become relatively more important after forward and SGD noise decrease.

Q3. Our transformer remains 30M parameters; larger models were not tested. At equal modeled operand cost, we compare 200M tokens without added SR with 290.9M tokens using two-mantissa-bit SR on both activation copies (output head excluded). Last-iterate validation uses SR off, native BF16/FP32 and three paired seeds. Losses are 4.1846 and 4.1360, respectively; the paired difference is −0.0486 ± 0.0050. This is a finite-budget benefit, not a hardware speedup or an LLM exponent measurement. The reference learning rate was not retuned; longer runs reuse the same corpus and extend cooldowns.

For the linear model with (a,b)=(3,2.5), noise variance 16, unrounded forward features and zero backward mantissa bits, we evaluated uniform-average risk by exact recursion. Taking the best risk across widths and durations, fitted risk-versus-compute slopes are −0.392 without SR and −0.306 with backward SR (predicted −0.375/−0.273). Risk decreases more slowly with backward rounding over these budgets.

Additionally, for references the reviewer mentioned, [1] models fixed precision through multiplicative constants and preserve their compute exponent. Our uniform-average forward-only law preserves the exact-arithmetic exponent. Moreover, Eq. (1) in [2] adds a penalty coupling model size, training data and floating-point format. Thus fitted exponents alone do not fix its optimized compute exponent. Their cross-entropy models use different quantization rules and losses from our feature-SR squared-risk model; our theorem does not refute their fits. Appendix A of the revision gives this comparison.

Thanks for the response, my main concerns are addressed. But for publication, I still feel the presentation and writing (e.g., notations, interpretations) should be improved substantially. For example, some key interpretations of why current empirical scaling laws and theoretical scaling laws do not claim an exponent transfer under precision are required. Moreover, current notations are heavy, there are a lot of notations that are not necessary, so I might keep my score.

Reviewer_SXADOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies stochastic rounding (SR) in low-precision training using one-pass SGD on random-feature linear regression with power-law spectra. At each step, the forward copy of the features (used for the prediction) and the backward copy (used in the weight gradient) are rounded independently, with precision allowed to change over training. The authors derive an exact second-moment recursion for the loss of rounded SGD and reduce it to a per-mode deterministic-equivalent system, which is conjectured to be accurate. From that system they obtain scaling laws for the rounding penalty, consisting of a forward term, a backward term, and an interaction term that only the averaged iterate sees. For backward rounding alone, they prove the law holds for the actual algorithm up to log factors. They then turn the laws into practical rules. A constant-LR last iterate is limited by forward rounding, and backward rounding only matters once averaging or a cooldown removes SGD noise. Bits are worth more late in training. The required bits grow logarithmically with compute, and a fixed precision bends the compute-optimal exponent. Finally, precision and LR schedules should be chosen together. The theory is checked against bit-exact rounded SGD and the exact recursion, and with small experiments on a CIFAR-10 MLP and a 30M-parameter transformer.

Critical strengths
  1. The exact recursion in Theorem 1 is clean and useful. It shows that SR leaves the mean on the exact-GD path and adds three distinguishable noise terms with different drivers and geometry. The monotonicity result (more bits never hurt any fixed-weight output) is proved through a positive-map argument that needs no step-size condition, which is a nice piece of analysis.

  2. For backward rounding, Theorem E.12 proves that the scaling law holds for the actual algorithm, not just the surrogate dynamics, for any schedule and both output rules. Many papers in this line stop at the deterministic equivalent, so this matters.

  3. The paper is careful about what is proved and what is not. Conjecture 2 is labeled as a conjecture, the obstructions to proving it are discussed (Sec. D.3, D.5), and the appendix states the scope of each result. Validation of the recursion against thousands of bit-exact SGD runs is thorough.

  4. The main qualitative message is useful for practitioners. The precision a run needs depends on the output rule (last iterate vs. average/EMA vs. cooldown), so a constant-LR ablation can say backward rounding is harmless when it is not. This suggests treating precision as a schedule tied to the LR schedule rather than as one global format.

Critical weaknesses
  1. The rounding sites studied are not the ones that matter most in practice. Only the activation copies are rounded. Weights in the forward product, the error signal, and the gradients are kept exact, and the grid has no overflow, underflow, or block scaling. Real FP8/FP4 recipes quantize the weights in the forward GEMM and rely on per-block scales.

  2. Several headline claims are not established for the real algorithm. The forward and interaction terms transfer only through Conjecture 2, which covers constant controls near the end of the run. The cooldown and timing results are not covered, and the small-gain assumption carries an untracked constant. The claim in the abstract that fixed precision lowers the scaling exponent comes from a regime outside the conjecture’s window. It is also not observed in the experiments: the envelope exponent is −0.172 in exact arithmetic and −0.165 at 1 bit. The abstract should be toned down to match what is proved and observed.

  3. The network experiments test little of what is new. The validation loss uses the rounded forward pass, so the forward gap includes rounding at inference time. The backward null result is what the theory predicts at these scales regardless, so it does not discriminate. Timing, the cooldown/EMA effect, and the LR–precision interaction, which are the most actionable predictions, are not tested on networks at all.

Questions for authors
  1. Can the analysis be extended to rounding the weights in the forward product, given that Figure 8(c) shows this site dominates? If not, please scope the claims so they do not read as statements about forward-pass quantization in general.

  2. Which claims in the abstract hold for the actual algorithm rather than the DE dynamics, and over which horizon?

  3. Is the backward-to-SGD-noise ratio 0.09·4^{-p} as stated in Prop. 5(i), or about 0.3·4^{-p} as in Appendix F.2? This changes the FP8 number quoted in the main text.

  4. What are the forward gaps in the MLP/transformer experiments if validation uses an exact forward pass?

  5. Do the conclusions change with per-block scaling (MX/NVFP4-style) or with round-to-nearest in the forward pass, as is common in practice?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the thoughtful review and questions. We now address questions and comments in your review.

W1. We agree that our results cover only rounding for forward prediction and the weight gradient with unbounded exponent and no block scaling. This scope makes the theory clear and clean and also captures the main effects of rounding in backward and forward propagations. Other rounding sites are compared empirically in Appendix F.4–F.5 (Table 5). We will further add more experiments on grids with overflow, underflow, or block scaling and compare the empirical results with our theorems in the revision.

W2. Our revised Theorem 1 uses the exact second-moment recursion and positive-map comparisons to prove matching risk bounds for actual SGD without Conjecture 2. We clarify in the abstract that for uniform averaging, forward-only rounding preserves the exact-arithmetic compute exponent; backward rounding changes it under the stated conditions and our definition of compute. The submitted slopes −0.172/−0.165 do not demonstrate this change. We therefore evaluated the exact second-moment recursion in another linear-model setting with stronger backward rounding. Over the tested budgets, prediction risk decreased more slowly with compute under backward rounding: the fitted slopes were −0.392 without SR and −0.306 with SR. This supports the predicted effect.

W3. The backward null result alone did not test these prescriptions. For the 30M transformer (200M tokens), both forward schedules use two bits until halfway, then exchange one/three bits in the final quarters, at constant learning rate after warmup. Higher precision early raises loss relative to higher precision late by 0.0195 ± 0.0021 for the last iterate and 0.0554 ± 0.0230 for the second-half uniform average. Averaging, including EMA, did not reduce the estimated difference. Separately, with learning-rate decay over the final 20%, we compared fixed two-bit forward SR with one bit in the first half and three in the second, at three rates. The schedule has loss 0.0065 ± 0.0076 higher at the best rate, which was the largest tested: no advantage is resolved. For CIFAR-10 MLP (constant learning rate after warmup, forward SR off), exchanging one/three backward bits between equal phases did not clearly favor either order under full uniform averaging.

Q1. We have not proved a weight-rounding law. This law does not just rely on first two moments. Matching lower bounds need new control beyond Gaussian-feature moment bounds. A shared weight copy introduces common batch noise. In linear SGD at width 64, two mantissa bits and batch 256, we compared one weight copy rounded per batch with independent feature rounding. The mean last-iterate SR-induced risk increase was about 53 times larger for weights.

Q2. The bounds cover last and uniformly averaged iterates. Risk averages over target, data and SR, with high probability over the feature map. Compute optima use constant learning rates and precisions. We assume fresh Gaussian data, power-law spectra, input dimension proportional to width, positive label noise and schedules chosen before training. For large width MM and spectral exponent aa, total learning rate lies between a lower threshold and MaM^a. Constructed thresholds bound learning rates, their ratios to batch size and amplification of later noise by earlier rounding. We use these bounds for decay instead of the old cooldown formula. EMA needs risk bounds for its different averaging weights.

Q3. We conflated coefficients for scalar variance (0.09) and propagated risk (0.3 in the reported experiment). Propagation depends on spectral weights and residual histories. We removed the universal coefficient and FP8 percentage. For backward-only SR, we bound the last-iterate increase by a constant times 4−p4^{-p} times full exact-arithmetic risk, not its SGD noise floor.

Q4. For CIFAR-10 MLP and 30M transformer (200M tokens), we train with forward-activation SR only and evaluate last iterates with added SR off. Mean loss increases over training without added SR are:

Mantissa bitsMLP30M transformer
1+0.0039+0.0562 ± 0.0029
2+0.0012+0.0172 ± 0.0033
3+0.0014+0.0068 ± 0.0032

For the same one-bit MLP parameters, loss is 1.6237 with SR off and 1.6721 with it on: most of that gap is evaluation rounding. The transformer training gap remains. Evaluation retains native FP32/BF16; these are not matched reevaluations of submitted checkpoints.

Q5. We have not established these laws for MX/NVFP4-style block scaling or round-to-nearest. A block scale depends on all block entries, so our Gaussian rounding bounds must be derived jointly with the scale. Finite exponent ranges require control of overflow, underflow and clipping. Round-to-nearest generally changes the mean update, whereas our proof uses unbiasedness. We have not proved the needed joint-moment and bias bounds.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to the reviewers and Area Chair

We thank the reviewers and Area Chair for the careful reviews and considerations. We have uploaded a revision of our paper to establish prediction-risk bounds for actual SGD without assuming Conjecture 2, clearer presentation, and more neural network experiments. The revision does not change the model setup and where/when/how-many questions.

Presentation. Our new Section 3 now states the main theorem for SGD, explains the role of its assumptions and gives a short explanation of the proof before presenting the constant-learning-rate results. Section 4 uses these results to discuss rounding location, timing, bit requirements, learning-rate design and compute allocation. This connects the risk bounds to the paper's practical questions. Detailed proofs and experimental settings remain in the appendices, and detailed simulations on real neural network and data-reuse case are in Appendix F.

In summary, forward rounding changes the prediction used in the update, whereas backward rounding changes the features multiplied by the residual. Averaging parameters changes how these errors contribute to the output. This explains why the last iterate and uniform average can require different precisions and different timing. Appendix E gives the derivations, and our response to wESi explains the same steps through the stochastic gradient.

Informativeness. The revised paper compares additional rounding locations and shared versus independent rounding, and separates rounding during training from rounding during evaluation. In our new Table 6, the same MLP parameters are evaluated with and without SR. For the MLP trained with one-bit forward rounding, most of the gap comes from evaluation rounding. This comparison helps distinguish an effect on the learned parameters from an effect on prediction after training. The model experiments compare width and training duration under a fixed compute budget. For data reuse, we explain how the number of available examples changes along with repeated access.

The additional experiments and empirical results are reported only in the rebuttal below. For example, on the 30M-parameter transformer, two-bit SR at both activation copies allows more training under the paper's compute model and gives lower mean validation loss than the tested training without SR. This illustrates how the benefit of longer training can outweigh the added rounding error. It is a comparison under the compute model, not a measurement of hardware savings, and the reference learning rate was not re-tuned.

The additional experiments clarify which practical conclusions are supported. On the MLP experiments, changing the learning rate improved the last iterate but worsened the uniform average, while the selected precision remained fixed. Our current experiments did not show an advantage from precision scheduling or resolve the predicted change in backward-precision timing under averaging. We therefore distinguish the results proved and illustrated in the linear model from the observations on neural networks. The individual responses give the comparisons and remaining limitations.

Soundness. In our initial submission, Conjecture 2 connected the deterministic-equivalent joint-rounding law to actual SGD. Our new Theorem 1 instead starts from the exact second-moment recursion (Proposition B.3) and uses positive-map comparisons to bound the risk directly. Assumption C.1 controls how earlier rounding amplifies later noise. Matching upper and lower bounds establish scaling orders without requiring or proving Conjecture 2's vanishing relative error. The proof in Appendix D.1 covers the last iterate and uniform average under the stated schedule conditions. Precision monotonicity remains Corollary B.10.The main neural scaling law now follows directly from the actual SGD updates without using Conjecture 2. The revised laws also include interaction terms between forward and backward rounding for both the last iterate and the uniform average.

We have also made the conclusions more precise. The compute scaling results use the cost model in Eq. (23) in the revision, and the schedule results apply under the assumptions in their statements. For backward rounding, we state the comparison proved in Proposition E.1 without a fixed numerical coefficient or FP8 percentage. We omit the separate analysis of rounding inputs before the fixed first layer, and the earlier predictions for other rounding locations and data reuse to keep the theoretical conclusions within the scope of the revised guarantees. The revised paper still includes experiments on other rounding locations and data reuse. For learning-rate decay, we state what follows from Theorem 1 and compare with the learning-rate design derived for exact-arithmetic SGD. These changes keep our conclusions tied to the analysis and distinguish them from observations on empirical results.

AuthorsOpenReview ↗

Summary of the revision

Below we summarize the theoretical results of the revision and compare them with our initial submission.

ResultInitial submissionRevision
Risk bounds for both outputsDeterministic-equivalent (DE) laws; forward/joint transfer relied on Conjecture 2Theorem 1: direct bounds for actual SGD under admissible schedules, without Conjecture 2
Learning-rate decay and cooldownBackward-only DE cooldown formulaCovered by Theorem 1
Where to roundDE-based location comparisonsProposition E.1: comparisons for actual SGD
When to use higher precisionDE-based allocation rulesPropositions E.2–E.4: allocation guarantees within the stated schedule classes
How many bitsDE-based precision requirementsCorollary E.5: necessary and sufficient requirements, up to additive constants
Compute-optimal exponentsDE-based results, partly outside Conjecture 2’s windowPropositions D.5–D.7, E.8 and Theorem E.9: actual-SGD results under the stated compute model
Learning-rate design under backward roundingDE-based schedule designTheorem E.7: actual risk within a constant factor of the best admissible schedule

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.