acceptodds
Under review as a conference paper at ICLR 2027

Freeze or Dither: Rounding-Aware Precision Floors for Low-Precision Training

Abstract

Convergence theory for low-precision training has produced only upper bounds, in a worst-case relative-error model blind to the rounding rule. We give the first lower bounds for training with floating-point weight storage, organized by one principle: a coordinate survives round-to-nearest (RN) when its update magnitude either exceeds the local grid spacing or carries noise at that scale, and freezes or floors when it does neither — freeze or dither. Sign-type updates can never dither. Below an explicit threshold they freeze bitwise, weight growth stops at a hard “binade wall,” and their movement-weighted noisy floor is exactly under both RN and stochastic rounding (SR), for arbitrary predictable step-size schedules, so at the stationary floor SR provably cannot help sign descent. SGD self-dithers: RN's rounding bias collapses as , which puts homogeneous-noise SGD–RN on the universal scale. But un-ditherable low-noise coordinates escape the rescue, and under heterogeneous noise with a shared constant step SGD–RN floors at , strictly worse than SGD–SR, so no rounding-blind analysis can be tight for both rules. A representation floor separately forces mantissa bits; with the sufficiency of tang2026convergence the requirement is . The floors are measured to their constants, and the wall is visible in real training: in a M-parameter GPT-2 storing weights in bf16 with no master copy, RN pins every LayerNorm gain at exactly its initialization of (a binade boundary) in of (optimizer, seed) cells, SR pins none; at the tuned budget the held-out RN/SR ratios order, in every seed, as ditherability predicts, and SR recovers the fp32 master's loss at a third of the weight memory. All code, logs and figures behind every number reported here are in the supplementary material and will be released publicly upon acceptance.

Primary area:
optimization
Keywords:
low-precision training, quantization, stochastic rounding, round-to-nearest, lower bounds, convergence analysis, mantissa bits

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_68HvOpenReview ↗
Rating
3(+2)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The headline claims are appealing, but I am not convinced that the paper establishes a clear and important practical contribution. It covers rounding theory, freezing in neural networks, a diagnostic method, and memory savings. The connection between these parts is not sufficiently demonstrated. In particular, the precise results for simple quadratic problems do not establish that the same mechanism explains the observed neural-network behavior.

My biggest concern is the reliability of the experimental comparisons. The selective compensation experiment is described as diagnosing and correcting the same training run. However, the code performs a separate diagnostic run and then starts training again from scratch. The compared runs also use different logging frequencies. Because training and validation sampling share a random-number generator, this changes the training data order, contrary to the claimed matched data order. In addition, the MLP baseline labeled “FP32 master weights” actually uses FP64 arrays. These discrepancies affect what the experiments test and weaken the conclusions drawn from them.

The experiments also do not establish the added value of the diagnostic method. It selects exactly the same 25 LayerNorm gain tensors as the simple LayerNorm-only rule already evaluated in the paper. The results therefore show that this set works, but do not show that running the diagnosis provides an additional benefit. The reported “18 residual standard deviations” result also depends on extrapolating to a parameter budget about 309 times smaller than the smallest random control. This extrapolation provides a weak basis for a strong efficiency claim.

Finally, the weight-storage calculations do not establish an overall training advantage. Peak GPU memory, training throughput, and time to reach a target loss are not measured, and the extra diagnostic run adds cost. The 124M model is trained on only about 1% of the Chinchilla reference token budget. With such short runs, the results may be sensitive to training noise, and it is unclear whether the reported gains persist with longer training.

Overall, the practical impact remains unclear, and several key claims require corrected and better-controlled experiments. I do not support acceptance in the paper’s current form.

Critical strengths

The quadratic analysis is interesting. The toy experiments agree with the predicted scaling and give a fairly clear picture of how rounding affects small updates. This is the part I found most useful.

Critical weaknesses
  1. The main contribution is hard to identify. The paper makes several claims, but I could not tell which result is the main advance and why it matters in practice.

  2. The experiments are small and short. The neural-network tests use small models and extremely limited training budgets (1%-3% of 1xChinchilla). These results alone do not show that the same benefits hold for larger models trained for much longer.

  3. The code tests a different procedure from the one described. The paper says diagnosis and compensation happen within one training run. The code runs a separate diagnosis, then trains a new model from scratch. This changes the procedure and adds training cost.

  4. The compared runs do not use the same training batches. Training and validation sampling share the same random-number stream. Running validation at different intervals changes the later training batches. The claimed matched data order is therefore not preserved.

  5. The MLP “FP32 master” baseline actually uses FP64. The master weights in the supplied MLP code are stored as 64-bit numbers, not 32-bit numbers. These results therefore do not establish how the stated FP32 baseline performs.

  6. The diagnosis has no demonstrated benefit over a simple rule. It selects all 25 LayerNorm gain tensors, exactly the same set as the LN-only baseline. The paper does not show what the extra diagnostic run adds over directly choosing those tensors.

  7. The measurement does not show that every LayerNorm gain stayed frozen. A tensor’s maximum can remain at 1.0 while other entries change. For example, both [1.0, 1.0] and [1.0, 0.9] have the same maximum. This measurement cannot support the claim that every element stayed unchanged.

  8. The “18 residual SD” number depends on an unchecked prediction. The smallest random control compensates about 5.9 million parameters, while LN25 uses only 19,200. The authors extend a fitted relationship down to this much smaller budget. There are no random controls at that budget to check whether the prediction is reliable.

  9. The overall training cost is unclear. The paper gives weight-storage counts, but no matched measurements of peak GPU memory, throughput, or total runtime including diagnosis. We therefore cannot tell whether the loss improvement is worth the extra work.

Questions for authors
  1. What is the main contribution? Is it the precision-floor theory, an explanation of freezing in neural networks, or the diagnostic method? I found the main message hard to identify. Could you state the central claim more clearly and point to the experiment that directly supports it?

  2. Why is the training budget sufficient? In Appendix G.6, the 124M model is trained on about 25M tokens. This is roughly 1% of the usual Chinchilla token-budget reference for this model size [1]. You state that the runs are past their transient. What evidence supports this? Does the held-out gap persist with longer budgets and learning-rate schedules matched to those budgets?

  3. Why are different checkpoints used in the comparison? For LN25 versus full SR in Appendix G.2, I recomputed the training-loss difference from your logs using the common steps 2600, 2800, and 3000. I get −0.0470 ± 0.0602, compared with the reported −0.0504 ± 0.0263 (mean ± sample SD over three seeds). The spread is much larger. How does this affect the conclusion that selective compensation recovers at least as much benefit as full SR?

  4. How is the data order matched? In exp12_gpt2small_bf16.py, training and validation sampling share the same RNG. Full SR logs every 100 steps, while LN25 logs every 200 steps. Validation sampling therefore changes the subsequent training batches. Why is this described as a comparison with matched data order? Does the result hold after fixing this issue?

  5. What is the actual diagnostic overhead? exp27_selective.py runs a separate 300-step diagnosis and then restarts training, although Appendix G.2 says there is no second training run. Could you explain this discrepancy and report the extra computation, wall-clock time, and peak GPU memory? After including diagnosis and SR overhead, is there still an advantage in loss at equal total compute or equal wall-clock time?

  6. What does diagnosis add over the LN-only rule? The diagnostic top25 is exactly the same set as the 25 LayerNorm gain tensors. What evidence shows that running the diagnosis adds value over simply selecting those tensors directly?

  7. How do the quadratic results carry over to real training? Which quantitative prediction of the theory is checked in the GPT experiments? Wen et al. [2] show that early loss advantages can disappear or reverse later in training. What evidence shows that your conclusions survive larger models and longer training, rather than reflecting this particular short-budget setup?

  8. What precision does the MLP master baseline actually use? The NumPy implementation appears to keep the master arrays in float64, while the paper calls this an FP32-master baseline. Could you confirm the dtype and explain whether the result holds with actual FP32 master weights?

  9. Presentation suggestion. Please explain the notation when it first appears. For example, the introduction uses Q(x)Q(x) on line 48 without first explaining what QQ and xx represent. I found the notation and transitions between theoretical settings difficult to follow. Defining the symbols and assumptions before using them would make the paper much easier to read.

[1] Hoffmann et al. (2022), Training Compute-Optimal Large Language Models.

[2] Wen et al. (ICLR 2026), Fantastic Pretraining Optimizers and Where to Find Them, Section 4.2.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 68Hv (1/3)

TL;DR. Every discrepancy you found is correct, and we reproduced each of them, including your recomputation of −0.0470 ± 0.0602. We fixed the protocol and reran the main 124M experiments under it. Where the corrected data required it we narrowed claims: LN25 does not match full SR, and the "18 SD" extrapolation is gone. The new runs answer your other questions: diagnosis cost, matched random controls, longer budgets with matched schedules, and measured memory and throughput. Theorem numbers refer to the revised PDF.

Main contribution

The main contribution is the theory: the first lower bounds for training with floating-point weight storage and no master copy. They show that the rounding rule, not only the precision, sets the error floor:

  • the relative-error model cannot be tight for both RN and SR (Cor. 6.5);
  • on the optimum's binade, sign descent floors at exactly √(π/2)/2·uσ/L under both rules (Thm 6.3);
  • SGD under RN fails exactly on the coordinates that cannot dither (Thms 5.2, 5.4);
  • without a master copy, M = Θ(log T) mantissa bits are necessary and, for sign descent with SR storage, sufficient (Cor. 4.2).

The diagnostic is a corollary, not a claimed contribution. The introduction now states this in a "Main claim" paragraph, and a new Table 1 lists each quantitative prediction, its test, and the outcome.

Which prediction is checked in GPT-2. A bitwise one. LayerNorm gains start at exactly 1.0, a bf16 binade boundary. Absorption (Lemma 4.3) plus a new bound on Adam's step (Lemma A.2) imply that under RN no Adam step size below 1.68×10⁻³ and no sign step size below 2⁻⁹ can ever move a gain coordinate. For sign descent, step sizes in (2⁻⁹, 2⁻⁸) can move gains only downward. The non-obvious ingredient is Lemma A.2: Adam's normalized step is at most 1.165η at β = (0.9, 0.95) for every gradient sequence, so the freeze is certified without looking at gradients (our runs keep fp32 moment buffers, and the logged maximum, 1.137, confirms the margin and shows the bound is nearly tight).

Outcome: it holds coordinate by coordinate. In every Adam and sign seed under RN, not one of the 19,200 gain coordinates ever leaves 1.0; under SR all of them do. A step-size sweep brackets both predicted switches to within 3%: sign descent switches at 1.9–2.0×10⁻³ and 3.8–4.0×10⁻³, against 1.95 and 3.91×10⁻³, and Adam at the same values of its realized update. The loss follows at matched budget: switching exactly these 19,200 coordinates to SR recovers 50% of the RN–SR gap, while 40 random draws of about the same size recover 5–6% on average, each less than the gains.

AuthorsOpenReview ↗

Response to Reviewer 68Hv (2/3)

Experimental reliability: all three issues confirmed and fixed

  1. Separate diagnosis run. Correct: exp27 ran a 300-step RN diagnosis, then restarted. The new protocol (exp41) runs it inside one training run. The "diag" arm trains on RN, collects the per-tensor φ<1 fraction over its first 300 steps, switches the flagged tensors to SR at step 300, and continues the same run. Its 12 diagnosis steps, timed in full (including the optimizer step they wrap), took 1.2 s of a 342 s run (0.35%), an upper bound on the diagnosis itself.
  2. Data order. Correct: train and validation draws shared one generator, and the arms logged at different intervals. Training batches now have their own generator, consumed exactly once per step, so for a given seed every arm sees the identical batch sequence however it logs or evaluates. Model initialization is also identical across arms.
  3. Checkpoints and the held-out estimator. Correct: the arms' tail windows covered different steps, and each validation reading was a single 8-sequence batch. Every arm is now evaluated at the same steps on one fixed 0.5M-token held-out set and a fixed training subset.
    • Under the corrected protocol, LN25 against all-SR gives +0.0220 ± 0.0019 held-out (3 seeds, paired), so LN25 is worse than blanket SR. Your recomputation pointed in that direction, and we withdraw the claim that the flagged set recovers at least as much as blanket SR.
    • The main 124M comparison: held-out RN/SR is 1.0227 ± 0.0029 (Adam), 1.0165 ± 0.0036 (SGD) and 0.994 ± 0.007 (sign). The first two exceed 1 in every seed; sign is within one seed SD of 1. SR/master is 1.001, 0.997 and 1.001.
  4. FP64 "FP32 master". Correct: the master arrays were float64. We reran the master control with a genuine float32 master, rounding the sum to float32 on every step (exp39), on every cell of the MLP table (Table 2 of the submission, Table 3 now) and now also for Adam. It matches full precision in every cell, with tail-loss ratios master32/fp of 0.96–1.04, and agrees with the earlier float64 master. Across all 70 master cells that train it is within seed noise of full precision (|z| ≤ 1.1, 5 seeds per arm). The conclusion is unchanged, but the label is now correct.

Two auxiliary comparisons (error feedback / 8-bit states; Adam hyperparameters) still use the earlier logging protocol. Within each, all arms logged at identical steps and so saw identical batches (App. H.7).

Diagnosis vs. the LN-only rule; the "18 SD" claim

We agree. On GPT-2 the diagnostic picks exactly the 25 gains, so on this model it adds nothing beyond the LN rule, and we now say so. What it shows is that the theory's criterion recovers that set without architecture knowledge: the gains have φ<1 fraction 1.000 against 0.44 for the next tensor. The "18 residual SD" extrapolation is removed. In its place are random controls at matched budget:

  • 20 random subsets of exactly 19,200 coordinates;
  • 20 random sets of whole non-gain tensors totalling about 19,200 parameters.

Results (seed 1337, held-out loss; all-RN 1.9978, all-SR 1.9497, LN25 1.9708):

  • Random coordinates at the same budget. All 20 draws are worse than LN25 (mean 1.9956 ± 0.0027 over draws). They recover about 5% of the RN–SR gap (random small tensors about 6%); the gains recover about half.
  • Random whole small tensors at the same budget. All 20 draws are worse.
  • 25 random whole tensors (600–3100× more parameters) beat LN25 in 4 of 10 draws.

So at matched budget, which coordinates are compensated matters, and the comparison no longer relies on extrapolation. Across three seeds, LN25 recovers 50% of the RN–SR gap (−0.0223 ± 0.0050 vs. all-RN), not all of it.

One setting where a diagnostic could add value over a static rule is late in a cosine schedule, as the step anneals and more tensors fall below the freeze threshold. An "online" arm re-diagnoses every 500 steps and switches newly exposed tensors to SR. It flags 38.6M parameters at step 1500, 4.7M at 2000 and 69.8M at 2500. It beats LN-only by −0.0093 ± 0.0016 (t = −10.2) and recovers 71% of the gap against 50%, compensating 26% of the parameter-steps that blanket SR does. At 250M tokens it recovers 90% (the gains alone 58%), but there it compensates 72% of the parameter-steps, so it shows that online compensation is feasible rather than that its selection rule is better. Blanket SR remains best, and we recommend it where its throughput cost is acceptable.

AuthorsOpenReview ↗

Response to Reviewer 68Hv (3/3)

Budget, longer training, Wen et al., and larger models

We agree that 25M tokens (about 1% of Chinchilla) cannot establish that a loss gap persists. New FineWeb runs, each with a cosine schedule matched to its budget:

  • 250M tokens (10× the original budget, about 0.1× Chinchilla): RN / SR / fp32 master / LN-only / online, 3 seeds;
  • 2.5B tokens (about 1× Chinchilla for 124M): RN / SR / master at 3 seeds, LN-only at 1.

At 250M tokens (3 seeds) the RN penalty does not wash out. Held-out RN/SR is 1.035 ± 0.012 and RN/master 1.042 ± 0.008, every seed above 1, and SR stays within seed noise of the master (1.007 ± 0.016). Within each run the ratio is flat (about 1.014) for the first half and then climbs as the cosine schedule anneals: from step 26,000 to the end, seed-mean held-out loss goes 4.422 → 4.414 for RN, against 4.293 → 4.264 for SR and 4.268 → 4.234 for the master. Once the annealed step falls below the freeze threshold of more and more coordinates, RN stops learning, which is the decaying-schedule prediction of Section 4. We tested the mechanism directly at the original budget: flooring the cosine at 0.5× of the peak instead of 0.1× (3 seeds) lowers the bitwise-frozen fraction of RN updates at the end of training from 0.90 to 0.65, and RN/SR falls from 1.023 to 1.011 ± 0.001, lower in every seed.

At 2.5B tokens (64k-token batches, three paired seeds), RN ends 1.7%, 2.1% and 1.6% above the fp32 master (1.018 ± 0.003), with the same shape as at 250M, and no gain coordinate ever moves under RN. SR ends 0.46% and 0.39% above the master at seeds 1338 and 1339. At seed 1337 it tracked the master to 2.2B tokens and then rose, on both held-out and training loss, to end 1.9% above it and 0.2% above RN. To find out why, we reran that seed's SR arm with a second SR stream, keeping the initialization and the batch sequence: it descends to the end and finishes 0.45% above the master. So the rise is not a property of the data order or the initialization; we keep the first run in the paper's table. RN/SR is 0.998, 1.016 and 1.012 (1.013 at seed 1337 with the second stream). So at 1× Chinchilla the RN penalty against the master persists in every seed, and SR stays within 0.5% of the master except in that one stream. At this batch SR's throughput cost falls to 2%.

Larger models. We added GPT-2 medium (350M parameters, 25M tokens, Adam at 3×10⁻⁴, three seeds; App. H.7). Under RN none of its 50,176 gain coordinates ever moves, in any Adam or sign run (largest normalized Adam step 1.142), while under SR all of them do. Held-out RN/SR for Adam is 1.045 ± 0.002, above 1 in every seed, and SR matches the fp32 master (1.000 ± 0.002); for sign descent it is 0.980 ± 0.008, so sign again loses nothing (at this shared step sign descent is far from converged, held-out 2.88 against 1.93 for Adam, so this is not a stationary comparison). We did not train beyond 350M; the bitwise prediction depends only on η, κ and the storage format, not on width or depth.

Cost: memory, throughput, training time

Every new run records peak GPU memory, static memory after setup, training throughput and training time (evaluation and checkpointing excluded, diagnosis included). Measured on one RTX 4090 D at batch 8×1024 (App. H.7):

  • Memory. Dropping the master saves its 4 bytes per parameter: static memory is 1.17 GiB against 1.64 GiB for Adam. At this batch activations dominate peak memory, so the saving is 5% of peak (8.82 against 9.28 GiB), and 2.7% at the 64k-token batch. We now state it that way, not as an overall training-memory advantage.
  • Throughput. Our SR is unfused (one random draw per element) and costs 14–16% (63.7k against 73.9k tokens/s for Adam); a fused rounding kernel may reduce this. Compensating only the gains runs at 72.1k tokens/s, 2.5% below RN. The online arm runs at 70.0k tokens/s, including at most 3.6% for its diagnosis steps.
  • Loss at equal training time. At 250M tokens SR is 11% slower per token but still reaches RN's final held-out loss in 0.66–0.71 of RN's training time; the fp32 master does so in 0.56–0.60.

Notation

Q, x, q, M, T, σ, u, L, 𝓕_t and dist are now defined before first use.

We hope the corrected protocol and the new runs restore confidence in the experiments, and would be grateful if you would reconsider your assessment, in particular soundness and presentation.

Thank you for the effort to address the reviews. I acknowledge the protocol corrections and the longer 124M runs. However, the long-budget comparisons cover Adam only, while the 350M experiments remain very limited, leaving some concerns about the broader empirical conclusions unresolved.

The extensive manuscript revisions also change theoretical conditions, withdraw several claims, and replace important experimental evidence, with roughly 60% of the main text appearing to have been revised in only 7 days. Thoroughly assessing these changes requires a substantial fresh review of the manuscript and supporting material. Within the remaining discussion period, alongside my other reviewing commitments, I am unable to complete that reassessment with sufficient care. I therefore cannot confirm that the revised claims and new evidence fully resolve my concerns. I respectfully defer the final decision to the AC. I thank the AC for their time and careful consideration of this case.

I am raising my overall rating to 3 and am curious to see the AC’s final assessment.

Reviewer_DJx9OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies rounding in low-precision training, where weights are stored in low precision after each update without full-precision master weights. It shows that round-to-nearest (RN) can absorb small updates, while SGD can use gradient noise to cross rounding boundaries through self-dithering.

Critical strengths
  • The analysis of noise heterogeneity and storage rounding is insightful. The two-coordinate construction in Theorem 5.6 is particularly interesting. A shared learning rate must balance the deterministic stalling error of a noiseless coordinate against the stochastic variance of a noisy coordinate, leading to different error scalings for RN and stochastic rounding (SR). This shows that two quantizers satisfying the same relative-error bound can behave quite differently in optimization.
  • The synthetic experiments examine both scaling exponents and leading constants, and show how learning rate grids and insufficient training horizons can affect the results. The Transformer experiments identify LayerNorm gains as particularly sensitive to rounding. Comparisons with fp32 master weights, error feedback, and quantized optimizer states also help isolate weight storage rounding as a separate bottleneck.
Critical weaknesses
  • The self-dithering analysis does not account for binade boundaries, and the stationary analysis needs further justification on the actual floating-point grid. Appendix B.5 treats the RN residual as a globally u-periodic sawtooth to derive the exponential bias bound in Theorem 5.4. However, floating-point spacing changes across binades. Gaussian updates can cross these boundaries even when the current iterate and optimum lie in the same binade. For example, consider M=7,u=2−7,L=σ=1,η=u,μ=1+u2,xt=1+u.M=7,u=2^{-7},L=\sigma=1,\eta=u,\mu=1+\frac{u}{2},x_t=1+u. These parameters satisfy the stated step-size condition, with ρ=ησ/u=1\rho=\eta\sigma/u=1. Direct Gaussian integration over the actual floating-point rounding cells gives E[rt∣xt]u≈−7.574×10−3\frac{\mathbb E[r_t\mid x_t]}{u}\approx -7.574\times10^{-3}, whereas Theorem 5.4 bounds its absolute value by approximately 8.516×10−108.516\times10^{-10}. The discrepancy comes from crossing into the neighboring binade. Similarly, Appendix C.4 derives the stationary constant using an infinite uniform-lattice birth–death chain. Applying this result to the actual floating-point process requires a localization argument or explicit boundary corrections. These issues matter because the paper aims to characterize floating-point storage rather than uniform-grid quantization.
  • The claimed Θ(log⁡T)\Theta(\log T) tightness combines a no-master lower bound with an upper bound for a different storage setting. Corollary 5.2 establishes an Ω(log⁡T)\Omega(\log T) lower bound for grid-valued iterates. However, the cited sufficiency result from arXiv:2510.21314 retains full-precision master weights. Quantized weights are used for gradient computation, but the master updates are not rounded back to the low-precision grid. This differs from Eq. (2), where storage rounding directly determines the next iterate. The master-weight setting avoids the update-absorption mechanism studied in this paper. Therefore, the cited upper bound does not directly establish tightness for no-master training.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer DJx9 (1/2)

TL;DR. Both points are correct, and both are fixed: Thm 5.2 (App. B.5, H.6), Thm 6.3 (new App. C.5, Lemma C.3), and Cor 4.2 (new no-master upper bound, App. E). Theorem numbers refer to the revised PDF: submission Thm 5.4 is now Thm 5.2, Thm 5.6 is now Thm 5.4, and Cor 5.2 is now Cor 4.2.

1. Binade boundaries in the self-dither theorem (now Thm 5.2)

We reproduced your number exactly. At M=7, η=u, L=σ=1, μ=1+u/2, x_t=1+u, integrating the Gaussian over the actual bf16 rounding cells at 40-digit precision gives E[r_t | F_t] = −7.574×10⁻³u, against the stated 8.5×10⁻¹⁰u. The proof applied the uniform-lattice sawtooth to a grid whose spacing changes at the binade edge. The revision repairs this in three steps (App. B.5).

(a) Per-step bias. Let B=[G,2G) be the optimum's binade. A pre-rounding point inside B has the residual of the extended lattice G+uℤ. If the pre-rounding mean c_t lies in B, a point outside B adds at most 3u/2 + (uησ/2G)|ξ|. Integrating against the Gaussian:

|E[r_t | F_t]| ≤ (u/π)e^{−2π²ρ²}/(1−e^{−6π²ρ²}) + 3u Φ̄(d_t/ησ) + (uησ/G) φ(d_t/ησ),

where d_t is the distance from c_t to the edge. The first term is the lattice (Sripad–Snyder) term. The other two form the edge term, which decays at a Gaussian rate in d_t/ησ. For c_t outside B the spacing at the pre-rounding point can be much larger than u, and the proof uses bounds that scale with it.

  • At your configuration the bound is 0.481u.
  • Over ρ ∈ {0.5, 1, 2}, d ∈ [0.3u, 64u] and both edges, the exact bias is at most 0.951 times the bound (App. H.6).
  • Far from the edge the exact bias equals the lattice prediction.

(b) Stationary bound via localization. Near the optimum the lattice bias is proportional to the error: |E ŝ_t| ≤ 2ηL|e_t|Σ_k e^{−2π²k²ρ²}, so the contraction absorbs it. The edge terms are Gaussian-small when |e_t| ≤ D/2, where D is the distance from μ to the edge. Elsewhere we use the cruder bounds and Chebyshev, whose cost in the contraction is absorbed when η ≥ c₀u/(LD) and D² ≥ c₁(uσ/L)log(e+σ/(Lu)). The theorem states the step-size window explicitly: η ∈ [u·max{1/σ, c₀/(LD)}, min{1/(2L), G/σ}], non-empty when σ ≥ 2Lu. On it,

V ≤ C(ησ²/L + u²/(ηL)) + C(σ/L)² exp(−D²/(16η²σ²)),

and tuning η gives V = O((uσ/L)·max{1, σ/(LD)}). The edge costs nothing when the optimum is farther from it than the noise scale σ/L, and at most a factor σ/(LD) otherwise.

(c) Size of the effect, measured. We moved the optimum from 16 stationary spreads to 0.05 spreads from the edge; at the closest point the chain is on the other side half of the time. The measured SGD–RN floor at ρ=1 moved by at most 3.4% (App. H.6), so for this setting the localization conditions are sufficient, not necessary.

AuthorsOpenReview ↗

Response to Reviewer DJx9 (2/2)

2. The stationary constant on the floating-point grid (App. C.4, Thm 6.3)

We agree that the infinite birth–death chain needed a localization argument. The revision adds one (new App. C.5).

(a) Stopped lower bound. Let τ be the first step at which the iterate comes within (m_t+1)u (RN) or η_t+u (SR) of the binade edge. The telescoping identity holds for the process stopped at τ, because 1[t<τ] is F_t-measurable and every per-step identity holds exactly on that event. The movement-weighted bound therefore holds with weights w_t·1[t<τ], with no assumption on the trajectory. The submitted statement is the special case P[τ ≥ T] = 1.

(b) Exit is exponentially rare (Lemma C.3). Take a constant step (m = 1 under RN, any η < u under SR), θ := Lu/σ ≤ ½, the optimum at distance D ≥ 4u from the binade edge, K := ⌊D/u⌋ − 3, and a run started within √(uσ/L) of μ. Then

P[τ ≤ T] ≤ 2e³(T+1)/(1 − e^{−3θK/2}) · exp(−¾θK²), with θK² ≥ L(D−4u)²/(uσ).

The proof uses the chain's sub-Gaussian stationary law, π(k) ≤ π(0)e^{−3θk²/4}, together with P_j ≤ P_π/π(j). The exponent is the squared distance to the edge in units of the stationary spread √(uσ/L).

(c) The attained constant transfers, at finite T. The floating-point process coincides with the lattice chain before τ. With δ_T = P[τ ≤ T], for every finite T the pre-exit average lies between C⋆(uσ/L)·(1 − R_T/(uwT(1−δ_T))) and a_T/(1−δ_T), where a_T is the lattice chain's running average and R_T an endpoint correction. It therefore sits at the common constant when exit is unlikely, the endpoint correction is small, and T exceeds the chain's mixing time.

Scope. At fixed precision the iterate eventually leaves the binade, and the theorem says nothing after τ: it characterizes the floor over horizons on which exit is unlikely. For the mid-binade optimum of our experiments at bf16 with σ=L, the exponent is about 21–22, and Lemma C.3 gives P[τ ≤ T] ≤ 0.01 up to T ≈ 4×10⁵ steps. On the grid of Figure 4 the exponent exceeds 90.

Measured. Moving μ to within 0.05 spreads of the edge changes the RN constant at η=u from 0.6269 to 0.6272, because one RN step below the edge spans two half-size cells. For SR the edge does matter: two or more spreads from it, SR and RN agree to 0.1%, but at 0.05 spreads SR's floor is 26% lower (App. H.6). Mode equivalence (Cor 6.4) is therefore stated for the localized regime.

3. Θ(log T) tightness

You are right. arXiv:2510.21314 keeps full-precision master weights, and its rate bounds ∇F at the master weights. Our representation floor constrains grid-valued iterates, which in that scheme is the quantized forward copy, not the master. Combining the two was incorrect.

The revision closes the no-master case with our own upper bound: new Corollary E.2, built on Theorem E.1, for sign descent with SR storage on a general L-smooth nonconvex f under the standard sign-probability margin.

  • Sufficiency: M ≥ ½log₂T + O(1) attains the full-precision signSGD rate O(√(LdΔ/T)) on its own metric (Σᵢ min{|∇ᵢf|, |∇ᵢf|²/σ}) when the iterates are bounded independently of T. Without that assumption, M ≥ log₂T + O(1) suffices, since SR storage keeps ‖x_t‖∞ ≤ e(‖x₀‖∞ + ηT) once 2^M ≥ T.
  • Necessity: on the same metric, the representation floor requires M ≥ ¼log₂T − O(1).

Both bounds concern the same grid-valued iterates, so at fixed dimension and without a master copy the requirement is Θ(log T). We now say explicitly that Tang et al. give the analogous sufficiency for the master-weight scheme and that it does not combine with our floor. The abstract, the introduction, Corollary 4.2 and the related-work paragraph are corrected accordingly.

We are glad the two-coordinate construction was useful. We hope these corrections resolve both concerns, and would be grateful if you would reconsider your assessment, in particular soundness.

Reviewer_sV53OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the precision limits of training with floating-point weight storage and no high-precision master copy. It analyzes how round-to-nearest and stochastic rounding affect sign descent and SGD, including update absorption, a binade wall, and noise-dependent error floors. The paper derives a representation lower bound on mantissa length, characterizes a movement-weighted noisy floor for sign descent, and studies how gradient noise can reduce rounding bias in SGD. Synthetic experiments and neural network training experiments examine the predicted behavior and the usefulness of the proposed diagnostics.

Critical strengths

The paper provides a useful rounding-aware explanation of how low-precision storage affects sign descent and SGD differently. The absorption and momentum analyses also offer practical diagnostic criteria. The experiments examine initialization, training horizon, and step-size tuning, and include stochastic-rounding and master-weight controls. Results across multiple seeds and two text corpora strengthen the empirical evidence.

Critical weaknesses

Theorems 5.4 and 6.3 rely on uniform-lattice arguments, but floating-point spacing changes at binade boundaries. The analysis does not sufficiently account for boundary crossings when transferring the rounding-bias bound and stationary constant to floating-point storage.

The LayerNorm results do not fully demonstrate complete freezing. A tensor’s maximum absolute value remaining at 1.0 does not establish that all coordinates stay at initialization, since some could decrease while the maximum stays unchanged.

The description of Table 2 is inconsistent: Appendix F specifies one reference architecture, while Appendix G.5 describes averaging four architectures. The extended-budget results also leave unclear which reported ratios represent stationary floors.

Questions for authors

What additional conditions or boundary corrections are needed for Theorems 5.4 and 6.3 to apply to floating-point storage?

Could the authors report each LayerNorm gain tensor’s maximum deviation from initialization and the fraction of coordinates that remain unchanged?

Which architectures contribute to Table 2, and what convergence evidence supports interpreting the reported ratios as stationary floors?

3 comments
AuthorsOpenReview ↗

Response to Reviewer sV53 (1/2)

TL;DR. All three questions point to real gaps. Q1: both theorems now carry explicit binade-edge conditions with proofs, and the effect is measured. Q2: per-coordinate logs show that under RN not one of the 19,200 gain coordinates ever moves, as a new lemma predicts. Q3: the table is the reference architecture alone and is a fixed-budget ranking; stationary floors are measured in a new experiment. Theorem numbers refer to the revised PDF: submission Thm 5.4 is now Thm 5.2, and Thm 6.3 keeps its number.

Q1. What conditions or boundary corrections do Theorems 5.4 and 6.3 need on floating-point storage?

Self-dither theorem (now Thm 5.2).

  • Per-step bias. Let B=[G,2G) be the optimum's binade. Whenever the pre-rounding mean c_t lies in B, |E[r_t | F_t]| ≤ (u/π)e^{−2π²ρ²}/(1−e^{−6π²ρ²}) + 3u Φ̄(d_t/ησ) + (uησ/G) φ(d_t/ησ), where d_t is the distance from c_t to the edge. Exact 40-digit integration over the real bf16 cells confirms it at every point we checked (App. H.6).
  • Stationary bound. It carries an explicit localization condition, D² ≥ c₁(uσ/L)log(e+σ/(Lu)), where D is the distance from the optimum to the edge, and an explicit step-size window η ∈ [u·max{1/σ, c₀/(LD)}, min{1/(2L), G/σ}] (non-empty when σ ≥ 2Lu), plus an additive edge term C(σ/L)²exp(−D²/16η²σ²). The tuned floor is O((uσ/L)·max{1, σ/(LD)}).

Exact sign floor (Thm 6.3).

  • Lower bound. It is stated for the process stopped at τ, the first approach to the binade edge, and holds with no assumption on the trajectory.
  • Exit (new Lemma C.3). For a constant step, an optimum at distance D ≥ 4u from the edge and a run started within √(uσ/L) of it, P[τ ≤ T] ≤ C_θK·(T+1)·exp(−¾·L(D−4u)²/(uσ)), with the prefactor explicit in the lemma. Exit is exponentially rare once the optimum is several stationary spreads √(uσ/L) from the edge, so the theorem describes the optimum's binade.
  • Attainment. The attained constant transfers to the floating-point process through the coupling up to τ, with an explicit finite-T sandwich.

How large the effect is (App. H.6). We placed the optimum between 16 and 0.05 stationary spreads from the edge. The sign–RN constant at η=u moved by under 0.1%, and the SGD–RN floor at ρ=1 by at most 3.4%. For sign–SR the edge does matter: at 0.05 spreads its floor is 26% below RN's, because SR takes half-size steps on the finer grid below the edge. So mode equivalence holds for an optimum localized in its binade, which is how the revision states it.

Q2. Per-tensor maximum deviation of the LayerNorm gains, and fraction of unchanged coordinates

Per-coordinate logs at every step, 25 gain tensors (19,200 coordinates), three seeds each (ranges over seeds; full per-tensor table in App. H.7):

optimizerstoragecoordinates ever movedtensors never movedmax |w−1|at 1.0 at the end
AdamRN0%25 / 250100%
AdamSR100%00.49–0.593.0–3.3%
signRN0%25 / 250100%
signSR100%00.37–0.443.9–4.0%
SGDRN3.3–3.5%5–110.93–0.9896.5–96.7%
SGDSR99.5–99.6%00.90–0.9613.1–13.4%

Under the fp32 master, 100% (Adam) and 99.99% (sign) of the gain coordinates move.

Why RN cannot move them. A bf16 gain at 1.0 sits on a binade boundary, with spacing 2⁻⁷ above and 2⁻⁸ below. By Lemma 4.3, any update smaller than 2⁻⁹ is absorbed, whichever optimizer produced it. New Lemma A.2 shows that Adam's normalized step satisfies |m̂/(√v̂+ε)| ≤ (1−β₁)/√((1−β₂)(1−β₁²/β₂)) = 1.165 at (0.9, 0.95), for every gradient sequence. So under RN:

  • Adam at every peak step size below 1.68×10⁻³, which covers our whole Adam grid (3e-4, 6e-4, 1.2e-3), cannot move a gain coordinate;
  • sign descent cannot below η = 2⁻⁹;
  • SGD carries the gradient's magnitude, so the theory predicts nothing bitwise there.

The largest pre-rounding update any gain received was 5.3×10⁻⁴ (Adam) and 3.0×10⁻⁴ (sign), against 2⁻⁹ = 1.95×10⁻³. The largest normalized Adam step was 1.137, against the bound 1.165. A step-size sweep brackets both predicted switches. Sign descent leaves every gain at 1.0 at η ≤ 1.9×10⁻³, moves gains only downward for η ∈ [2.0, 3.8]×10⁻³, and upward as well from η = 4.0×10⁻³; the predictions are 2⁻⁹ = 1.95×10⁻³ and 2⁻⁸ = 3.91×10⁻³. Adam switches at the same two values of its realized update.

AuthorsOpenReview ↗

Response to Reviewer sV53 (2/2)

Q3. Which architectures contribute to Table 2, and are its ratios stationary floors?

Architectures. Table 2 of the submission (Table 3 in the revision) is the reference architecture alone (64-32-10 ReLU MLP on digits), as the experimental-details appendix states (App. G). The sentence in the submission's Appendix G.5 saying that it "averages four architectures" was wrong and is corrected. The other three digits architectures enter only the 440-run grid, and a four-architecture table is added.

Stationarity. We checked directly whether the table's cells are at a floor (new EXP40), rerunning every reference cell to 8× its budget and reading the tail at 1×, 2×, 4× and 8× from the same trajectory. Most are not at a floor, and the revision says so:

  • Full precision and SR keep descending (digits is interpolable): SGD-fp 1.2×10⁻³ → 8.8×10⁻⁵, Adam-fp 4×10⁻⁴ → 3×10⁻¹⁰. The SGD ratio at M=7 goes 2.2 → 4.1 → 8.4 → 16.9.
  • RN moves much less: SGD changes by −6% to −10% between 4× and 8×.

The table is now labeled a fixed-budget loss ratio (caption and header), and for SGD and Adam its ratios are lower bounds on the long-run gap.

Floors on real data (new EXP42). An L2 penalty gives the same task an interior optimum and a near-stationary objective: every one of the 48 arms moves by under 1.2% between half and full budget, while the excess over full precision reaches 60%. A float32 master matches full precision in every cell. Sign descent has the smallest RN/SR excess ratio at every M (2–6×). For SGD, the RN excess falls with the grid spacing at log–log slope 0.56 against 1.23 for SR, so the RN/SR ratio grows from 11× at M=3 to 51× at M=6. At bf16 the SR excess is within two standard errors of zero, so the ratio there is not resolved. This is the direction of the separation in Thm 5.4; we do not claim its 2/3 exponent. At stationarity SGD rather than Adam is the most fragile for M ≥ 4, so the revision claims only that the theory puts sign descent last.

The 124M runs are fixed-budget comparisons with budget-matched schedules, not floors. At 250M tokens RN/SR is 1.035 ± 0.012, with SR within noise of the master. At 2.5B tokens RN ends 1.6–2.1% above the master in all three paired seeds, and RN/SR is 0.998, 1.016 and 1.012.

We hope the conditions, logs and new measurements answer the three questions, and would be grateful if you would reconsider your assessment, in particular soundness.

The revised manuscript and rebuttal largely resolve my concerns about binade boundaries, LayerNorm freezing, and the interpretation of the MLP results. Based on these revisions, I am raising my soundness score from 1 to 2 and my overall rating from 2 to 3.

I would also like to draw the Area Chair’s attention to the unusually extensive revisions made during rebuttal. By my estimate, more than half of the manuscript has been revised. For example, the revision narrows the scope of a central theorem and shifts the interpretation of the main MLP comparison from a claim about a stationary floor to a result at a fixed budget. My updated scores reflect the revised manuscript, and I ask the Area Chair to consider both the scale of the revisions and their effect on the paper’s central claims in reaching a final decision.

Reviewer_5ufkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work is related to precision aware lower bounds for the floating points weight storage across training iterations. Two rounding techniques are used round to nearest (RN) and stochastic rounding (SR) for illustrating the point. The work proposes lower-bounds for the weight storage based on a principle named by authors as "freeze or dither". For a smaller 124M gpt2 model, experimental results find that RN pins layernorm gain to 1.0 while SR does not.

Critical strengths
  1. The paper has proposed lower bounds for the floating point weight storage in LLM training which is interesting across the steps horizon
  2. Reviewer went over the proofs (at high level) of certain theorems and propositions, and appreciate the details in appendix
Critical weaknesses

Overall, I would like to say that the paper is not reading well in its current form. While the reviewer has experience in low-precision LLM training and SR related nuances, and even then a lot of back-and-forth, plus refs to appendix is required. For a general reader, I would say that the paper presentation is below the ICLR threshold. A lot of places the manuscript is reading like dump of scratch notes with no coherent story in sections. Specific questions are listed below:

  1. Define "T" in line49 before using it
  2. what is noise sigma2 in para spanning lines 74-85?
  3. Line 102, sripad-synder argument: insert reference,. Its there in later part of the work but not here where talked about first
  4. line 147-148, where its mentioned that tightness is obtained, as far as reviewer understood, its for signSGD like training only.
  5. line 182, drop the lemma, at best its only definitions and pretty much standard
  6. line 198: what is \matcal{F}_t ?
  7. line 203-204, the loosely put dist(.,.) needs to be defined properly that its a distance in the F_M grid
  8. Thm 5.1 is used in Thm 4.2, infact the second part is using it completely. Change order and either relegate Thm 5.1 in appendix, or more it before Thm 4.2
  9. Section-7 does not read well at all. There is no continuity, there is sudden jump to Prop7.1, and then ending para appears. Reader has no idea what to make of this section
  10. line 453-455: 10a: where are the training loss curves? 10b: what does this statement means "Freezing is a cost the generalization gap may refund."? what is generalization gap here?
  11. line 429-437: The layernorm gain sticking to 1, and then reference to Thm 4,2 is made, But the theorem is for sign-gradients like approaches, does that mean this para also did sign-gradients? its not clear from the writing
  12. Section-6 could be made much more compact with putting wordy corollaries in appendix and may be moving figures to main text, like loss cures or even Fig4 which is referenced in results.
Questions for authors

pls see the weakness section

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5ufk

We thank the reviewer for a detailed reading and for the line-by-line list. We agree that the submission read too much like notes. The revision restructures the paper around one stated claim and addresses every item below; changed passages are marked in blue in the revised PDF. The fastest way to see the new structure is p. 2 (Main claim, Contributions) and Table 1 on p. 8, which ties every experiment to a stated result. Theorem numbers refer to the revised PDF.

The story

The introduction now opens with a "Main claim" paragraph. The contribution is theoretical: the first lower bounds for floating-point weight storage without a master copy, showing that the rounding rule (RN vs. SR), not only the precision, sets the error floor. A new Table 1 lists each quantitative prediction, the experiment that tests it, and the outcome, so that every experiment is tied to a result.

Point by point

  1. T is undefined (l.49). T is now defined at first use as the number of optimizer steps, in both the abstract and the introduction.

  2. What is σ² (ll.74–85)? σ² is the per-coordinate variance of the stochastic-gradient noise, g_t = ∇f(W_t) + σξ_t. It is now defined where it first appears (abstract and Section 3), together with u (the local grid spacing) and L (the curvature of the quadratic instance).

  3. Sripad–Snyder reference (l.102). The reference is now given at the first mention.

  4. Tightness (ll.147–148). Agreed: the exact (tight) characterization is for sign descent. The contribution list now says so explicitly: for SGD we claim the matched 2/3 floor on the heterogeneous instance and, under homogeneous noise, only an upper bound (contribution (3) and Thm 5.2).

  5. Drop the lemma (l.182). Done. Lemma 3.1 (basic facts) moved to Appendix A. Only its one consequence the paper relies on stays in the setup: RN and SR satisfy the same relative-error assumption.

  6. What is 𝓕_t (l.198)? 𝓕_t is now defined in Section 3 as the history up to the start of step t: iterates, past noise, and SR draws.

  7. dist(·,·) (ll.203–204). It is now defined as dist(y, 𝔽_M) = min over z ∈ 𝔽_M of |y − z|, the distance to the floating-point grid.

  8. Theorem order (Thm 5.1 is used in Thm 4.2). Reordered. Section 4 now opens with the representation floor (Thm 4.1) and its Θ(log T) corollary, followed by absorption, the freeze theorem (Thm 4.4, which uses Thm 4.1), and the binade wall.

  9. Section 7 does not read well. Rewritten as a single argument, in order:

    • motivation: the optimizer state is itself stored, and its accumulation m⁺ = Q(m + (1−β)(g−m)) is a storage step with the increment shrunk by 1−β;
    • the proposition;
    • what it means in numbers: a bf16 momentum ignores gradient changes up to 20% at β=0.99 and up to 195% at β=0.999, and never decays above β_crit;
    • the link to reported staleness of 8-/4-bit states;
    • the experiments that test it.
  10. Training curves and the "refund" sentence (ll.453–455).

    • (a) The main text now shows the 124M held-out loss against training step (Fig. 3, right), on a fixed held-out set, as RN/SR and SR/fp32-master ratios with the mean and range over three seeds. The RN/SR gap grows as the cosine schedule anneals. The per-run curves of both held-out loss and training loss are in Appendix H.7 (Fig. 5, top and bottom rows).
    • (b) The sentence is removed. It referred to the gap between held-out and training loss: on a small, highly redundant corpus a run that freezes some coordinates can fit the training set less and generalize no worse. The revised text states this explicitly where it is used and reports training and held-out loss separately.
  11. LayerNorm paragraph and the sign-only theorem (ll.429–437). The runs there were Adam, SGD and sign descent, and the paragraph should not have cited the sign-descent theorem. The relevant results are Lemma 4.3 (absorption) and Cor. 4.5 (binade wall), which hold for any update of a given magnitude, whatever optimizer produced it, together with new Lemma A.2 (Adam's normalized step is at most 1.165η). The paragraph now says this, and the resulting bitwise prediction is verified coordinate by coordinate (see the general response).

  12. Make Section 6 compact; move figures into the main text.

    • The unweighted-average remark and the ℓ₁ corollary moved to the appendix; the latter is replaced by one sentence.
    • The step-size landscape figure (former Fig. 4) is now in the main text (Fig. 3, left).
    • The diagnostics table moved to Appendix F, making room for the predictions table and the loss curves.

We also applied the notation suggestions of the other reviewers, for example defining Q and x before first use.

We hope the restructured paper is now readable without the back-and-forth you describe, and would be grateful if you would reconsider your assessment, in particular presentation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all four reviewers, in particular DJx9 for the concrete counterexample and 68Hv for the code audit. We respond to every concern below; where the corrected data required it, we narrowed the claim. Changed passages are blue in the revised PDF.

For the AC, in brief.

  • Main results: the two-coordinate separation (Thm 5.4) and the incompleteness of the relative-error model it implies (Cor 6.5); the exact sign-descent constant (Thm 6.3), now stated on the optimum's binade.
  • Corrected: the binade edge in Thms 5.2 and 6.3, and the Θ(log T) pairing, now closed by our own no-master upper bound.
  • Narrowed or withdrawn: "LN25 recovers at least as much as SR", the 18-SD extrapolation, "a third of the weight memory" as a training-memory claim, and the MLP table as stationary floors.
  • Rerun: the main 124M comparisons, under a corrected protocol.

Theory

  1. Binade edge (DJx9, sV53).
    • Self-dither (Thm 5.2). The per-step bias bound now holds whenever the pre-rounding mean lies in the optimum's binade, with an explicit edge term. The stationary bound carries explicit localization and step-size conditions. Your counterexample (−7.574×10⁻³u) satisfies the corrected bound (0.481u).
    • Exact sign floor (Thm 6.3). It is now stated for the process stopped at the binade edge, with no assumption on the trajectory, and the attained constant at finite T. A new lemma shows that exit is exponentially rare, so the theorem describes the optimum's binade.
    • Measured. The edge moves the sign–RN floor at η=u by under 0.1% and the SGD–RN floor at ρ=1 by at most 3.4%, even with the optimum 1.5 cells from the edge. For sign–SR it does matter: 0.05 spreads from the edge, SR's floor is 26% below RN's (half-size steps on the finer grid below it), so mode equivalence is stated for an optimum localized in its binade.
  2. Θ(log T) (DJx9). The cited sufficiency keeps a master copy, so it does not pair with our floor. We now prove the no-master upper bound (Cor E.2: sign descent with SR storage reaches the full-precision signSGD rate with M = O(log T)), so Θ(log T) rests on our own results.

Experiments: the 124M protocol, corrected and rerun

All three problems 68Hv found are confirmed and fixed. Diagnosis now runs inside one training run. Training batches have their own random stream, so all arms of a seed see identical batches. Every arm is evaluated at the same steps on one fixed held-out set. The MLP master control now uses a genuine float32 master, and the LayerNorm gains are logged per coordinate.

  • 124M, 25M tokens, 3 seeds. Held-out RN/SR is 1.023 (Adam), 1.017 (SGD) and 0.994 (sign) on TinyStories, and 1.026, 1.008 and 0.998 on WikiText-103. SR matches the fp32 master. At 350M (GPT-2 medium) Adam's RN/SR is 1.045 ± 0.002 and sign's 0.980 ± 0.008.
  • Why the gap opens late. The penalty grows as the cosine schedule anneals the step below more coordinates' freeze thresholds. Flooring the schedule at 0.5× instead of 0.1× cuts the frozen fraction from 0.90 to 0.65 and halves the penalty (RN/SR 1.011, lower in every seed).
  • Longer budgets (FineWeb, budget-matched cosine). At 250M tokens (10×, about 0.1× Chinchilla) RN/SR is 1.035 ± 0.012, every seed above 1. At 2.5B tokens (about 1× Chinchilla, three paired seeds) RN ends 1.6–2.1% above the fp32 master in every seed. SR ends within 0.5% of it in two seeds; in the third it rose late, which a rerun with a second SR stream on the same batches does not reproduce (0.45% above the master). RN/SR is 0.998–1.016 (tail means).
  • Cost. Dropping the master saves 4 B/param, 2.7–5% of peak memory at our batch sizes. Our unfused SR costs 11–16% throughput at an 8k-token batch and 2% at a 64k-token batch.

A bitwise prediction in real training (68Hv, 5ufk, sV53)

A LayerNorm gain starts at 1.0, a bf16 binade boundary. Absorption (Lemma 4.3) and a new bound on Adam's normalized step (Lemma A.2, κ = 1.165, any gradient sequence) imply that under RN no Adam step size below 1.68×10⁻³ and no sign step size below 2⁻⁹ can move a gain coordinate. In every Adam and sign seed under RN, none of the 19,200 gain coordinates ever leaves 1.0; under SR all of them do. The same holds in a new 350M model (GPT-2 medium, 50,176 gain coordinates). A step-size sweep brackets both predicted switches (2⁻⁹, 2⁻⁸) to within 3%. The loss follows at matched budget: switching exactly these coordinates to SR recovers 50% of the RN–SR gap, while each of 40 random draws of about the same size recovers less (5–6% of the gap on average).

Presentation (5ufk, 68Hv)

A "Main claim" paragraph and a new Table 1 (each prediction, its test, its outcome) open the paper. Notation is defined before use, Section 4 is reordered, Section 7 is rewritten, and loss curves (held-out and training) are in the paper.

We hope these corrections and reruns resolve the concerns, and would be grateful if the reviewers would update their assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.