- Rating
- 3(+2)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The headline claims are appealing, but I am not convinced that the paper establishes a clear and important practical contribution. It covers rounding theory, freezing in neural networks, a diagnostic method, and memory savings. The connection between these parts is not sufficiently demonstrated. In particular, the precise results for simple quadratic problems do not establish that the same mechanism explains the observed neural-network behavior.
My biggest concern is the reliability of the experimental comparisons. The selective compensation experiment is described as diagnosing and correcting the same training run. However, the code performs a separate diagnostic run and then starts training again from scratch. The compared runs also use different logging frequencies. Because training and validation sampling share a random-number generator, this changes the training data order, contrary to the claimed matched data order. In addition, the MLP baseline labeled “FP32 master weights” actually uses FP64 arrays. These discrepancies affect what the experiments test and weaken the conclusions drawn from them.
The experiments also do not establish the added value of the diagnostic method. It selects exactly the same 25 LayerNorm gain tensors as the simple LayerNorm-only rule already evaluated in the paper. The results therefore show that this set works, but do not show that running the diagnosis provides an additional benefit. The reported “18 residual standard deviations” result also depends on extrapolating to a parameter budget about 309 times smaller than the smallest random control. This extrapolation provides a weak basis for a strong efficiency claim.
Finally, the weight-storage calculations do not establish an overall training advantage. Peak GPU memory, training throughput, and time to reach a target loss are not measured, and the extra diagnostic run adds cost. The 124M model is trained on only about 1% of the Chinchilla reference token budget. With such short runs, the results may be sensitive to training noise, and it is unclear whether the reported gains persist with longer training.
Overall, the practical impact remains unclear, and several key claims require corrected and better-controlled experiments. I do not support acceptance in the paper’s current form.
Critical strengths
The quadratic analysis is interesting. The toy experiments agree with the predicted scaling and give a fairly clear picture of how rounding affects small updates. This is the part I found most useful.
Critical weaknesses
-
The main contribution is hard to identify. The paper makes several claims, but I could not tell which result is the main advance and why it matters in practice.
-
The experiments are small and short. The neural-network tests use small models and extremely limited training budgets (1%-3% of 1xChinchilla). These results alone do not show that the same benefits hold for larger models trained for much longer.
-
The code tests a different procedure from the one described. The paper says diagnosis and compensation happen within one training run. The code runs a separate diagnosis, then trains a new model from scratch. This changes the procedure and adds training cost.
-
The compared runs do not use the same training batches. Training and validation sampling share the same random-number stream. Running validation at different intervals changes the later training batches. The claimed matched data order is therefore not preserved.
-
The MLP “FP32 master” baseline actually uses FP64. The master weights in the supplied MLP code are stored as 64-bit numbers, not 32-bit numbers. These results therefore do not establish how the stated FP32 baseline performs.
-
The diagnosis has no demonstrated benefit over a simple rule. It selects all 25 LayerNorm gain tensors, exactly the same set as the LN-only baseline. The paper does not show what the extra diagnostic run adds over directly choosing those tensors.
-
The measurement does not show that every LayerNorm gain stayed frozen. A tensor’s maximum can remain at 1.0 while other entries change. For example, both [1.0, 1.0] and [1.0, 0.9] have the same maximum. This measurement cannot support the claim that every element stayed unchanged.
-
The “18 residual SD” number depends on an unchecked prediction. The smallest random control compensates about 5.9 million parameters, while LN25 uses only 19,200. The authors extend a fitted relationship down to this much smaller budget. There are no random controls at that budget to check whether the prediction is reliable.
-
The overall training cost is unclear. The paper gives weight-storage counts, but no matched measurements of peak GPU memory, throughput, or total runtime including diagnosis. We therefore cannot tell whether the loss improvement is worth the extra work.
Questions for authors
-
What is the main contribution? Is it the precision-floor theory, an explanation of freezing in neural networks, or the diagnostic method? I found the main message hard to identify. Could you state the central claim more clearly and point to the experiment that directly supports it?
-
Why is the training budget sufficient? In Appendix G.6, the 124M model is trained on about 25M tokens. This is roughly 1% of the usual Chinchilla token-budget reference for this model size [1]. You state that the runs are past their transient. What evidence supports this? Does the held-out gap persist with longer budgets and learning-rate schedules matched to those budgets?
-
Why are different checkpoints used in the comparison? For LN25 versus full SR in Appendix G.2, I recomputed the training-loss difference from your logs using the common steps 2600, 2800, and 3000. I get −0.0470 ± 0.0602, compared with the reported −0.0504 ± 0.0263 (mean ± sample SD over three seeds). The spread is much larger. How does this affect the conclusion that selective compensation recovers at least as much benefit as full SR?
-
How is the data order matched? In
exp12_gpt2small_bf16.py, training and validation sampling share the same RNG. Full SR logs every 100 steps, while LN25 logs every 200 steps. Validation sampling therefore changes the subsequent training batches. Why is this described as a comparison with matched data order? Does the result hold after fixing this issue? -
What is the actual diagnostic overhead?
exp27_selective.pyruns a separate 300-step diagnosis and then restarts training, although Appendix G.2 says there is no second training run. Could you explain this discrepancy and report the extra computation, wall-clock time, and peak GPU memory? After including diagnosis and SR overhead, is there still an advantage in loss at equal total compute or equal wall-clock time? -
What does diagnosis add over the LN-only rule? The diagnostic top25 is exactly the same set as the 25 LayerNorm gain tensors. What evidence shows that running the diagnosis adds value over simply selecting those tensors directly?
-
How do the quadratic results carry over to real training? Which quantitative prediction of the theory is checked in the GPT experiments? Wen et al. [2] show that early loss advantages can disappear or reverse later in training. What evidence shows that your conclusions survive larger models and longer training, rather than reflecting this particular short-budget setup?
-
What precision does the MLP master baseline actually use? The NumPy implementation appears to keep the master arrays in float64, while the paper calls this an FP32-master baseline. Could you confirm the dtype and explain whether the result holds with actual FP32 master weights?
-
Presentation suggestion. Please explain the notation when it first appears. For example, the introduction uses on line 48 without first explaining what and represent. I found the notation and transitions between theoretical settings difficult to follow. Defining the symbols and assumptions before using them would make the paper much easier to read.
[1] Hoffmann et al. (2022), Training Compute-Optimal Large Language Models.
[2] Wen et al. (ICLR 2026), Fantastic Pretraining Optimizers and Where to Find Them, Section 4.2.
4 comments
Response to Reviewer 68Hv (2/3)
Experimental reliability: all three issues confirmed and fixed
- Separate diagnosis run. Correct:
exp27ran a 300-step RN diagnosis, then restarted. The new protocol (exp41) runs it inside one training run. The "diag" arm trains on RN, collects the per-tensor φ<1 fraction over its first 300 steps, switches the flagged tensors to SR at step 300, and continues the same run. Its 12 diagnosis steps, timed in full (including the optimizer step they wrap), took 1.2 s of a 342 s run (0.35%), an upper bound on the diagnosis itself. - Data order. Correct: train and validation draws shared one generator, and the arms logged at different intervals. Training batches now have their own generator, consumed exactly once per step, so for a given seed every arm sees the identical batch sequence however it logs or evaluates. Model initialization is also identical across arms.
- Checkpoints and the held-out estimator. Correct: the arms' tail windows covered different steps, and each validation reading was a single 8-sequence batch. Every arm is now evaluated at the same steps on one fixed 0.5M-token held-out set and a fixed training subset.
- Under the corrected protocol, LN25 against all-SR gives +0.0220 ± 0.0019 held-out (3 seeds, paired), so LN25 is worse than blanket SR. Your recomputation pointed in that direction, and we withdraw the claim that the flagged set recovers at least as much as blanket SR.
- The main 124M comparison: held-out RN/SR is 1.0227 ± 0.0029 (Adam), 1.0165 ± 0.0036 (SGD) and 0.994 ± 0.007 (sign). The first two exceed 1 in every seed; sign is within one seed SD of 1. SR/master is 1.001, 0.997 and 1.001.
- FP64 "FP32 master". Correct: the master arrays were float64. We reran the master control with a genuine float32 master, rounding the sum to float32 on every step (
exp39), on every cell of the MLP table (Table 2 of the submission, Table 3 now) and now also for Adam. It matches full precision in every cell, with tail-loss ratios master32/fp of 0.96–1.04, and agrees with the earlier float64 master. Across all 70 master cells that train it is within seed noise of full precision (|z| ≤ 1.1, 5 seeds per arm). The conclusion is unchanged, but the label is now correct.
Two auxiliary comparisons (error feedback / 8-bit states; Adam hyperparameters) still use the earlier logging protocol. Within each, all arms logged at identical steps and so saw identical batches (App. H.7).
Diagnosis vs. the LN-only rule; the "18 SD" claim
We agree. On GPT-2 the diagnostic picks exactly the 25 gains, so on this model it adds nothing beyond the LN rule, and we now say so. What it shows is that the theory's criterion recovers that set without architecture knowledge: the gains have φ<1 fraction 1.000 against 0.44 for the next tensor. The "18 residual SD" extrapolation is removed. In its place are random controls at matched budget:
- 20 random subsets of exactly 19,200 coordinates;
- 20 random sets of whole non-gain tensors totalling about 19,200 parameters.
Results (seed 1337, held-out loss; all-RN 1.9978, all-SR 1.9497, LN25 1.9708):
- Random coordinates at the same budget. All 20 draws are worse than LN25 (mean 1.9956 ± 0.0027 over draws). They recover about 5% of the RN–SR gap (random small tensors about 6%); the gains recover about half.
- Random whole small tensors at the same budget. All 20 draws are worse.
- 25 random whole tensors (600–3100× more parameters) beat LN25 in 4 of 10 draws.
So at matched budget, which coordinates are compensated matters, and the comparison no longer relies on extrapolation. Across three seeds, LN25 recovers 50% of the RN–SR gap (−0.0223 ± 0.0050 vs. all-RN), not all of it.
One setting where a diagnostic could add value over a static rule is late in a cosine schedule, as the step anneals and more tensors fall below the freeze threshold. An "online" arm re-diagnoses every 500 steps and switches newly exposed tensors to SR. It flags 38.6M parameters at step 1500, 4.7M at 2000 and 69.8M at 2500. It beats LN-only by −0.0093 ± 0.0016 (t = −10.2) and recovers 71% of the gap against 50%, compensating 26% of the parameter-steps that blanket SR does. At 250M tokens it recovers 90% (the gains alone 58%), but there it compensates 72% of the parameter-steps, so it shows that online compensation is feasible rather than that its selection rule is better. Blanket SR remains best, and we recommend it where its throughput cost is acceptable.
Response to Reviewer 68Hv (3/3)
Budget, longer training, Wen et al., and larger models
We agree that 25M tokens (about 1% of Chinchilla) cannot establish that a loss gap persists. New FineWeb runs, each with a cosine schedule matched to its budget:
- 250M tokens (10× the original budget, about 0.1× Chinchilla): RN / SR / fp32 master / LN-only / online, 3 seeds;
- 2.5B tokens (about 1× Chinchilla for 124M): RN / SR / master at 3 seeds, LN-only at 1.
At 250M tokens (3 seeds) the RN penalty does not wash out. Held-out RN/SR is 1.035 ± 0.012 and RN/master 1.042 ± 0.008, every seed above 1, and SR stays within seed noise of the master (1.007 ± 0.016). Within each run the ratio is flat (about 1.014) for the first half and then climbs as the cosine schedule anneals: from step 26,000 to the end, seed-mean held-out loss goes 4.422 → 4.414 for RN, against 4.293 → 4.264 for SR and 4.268 → 4.234 for the master. Once the annealed step falls below the freeze threshold of more and more coordinates, RN stops learning, which is the decaying-schedule prediction of Section 4. We tested the mechanism directly at the original budget: flooring the cosine at 0.5× of the peak instead of 0.1× (3 seeds) lowers the bitwise-frozen fraction of RN updates at the end of training from 0.90 to 0.65, and RN/SR falls from 1.023 to 1.011 ± 0.001, lower in every seed.
At 2.5B tokens (64k-token batches, three paired seeds), RN ends 1.7%, 2.1% and 1.6% above the fp32 master (1.018 ± 0.003), with the same shape as at 250M, and no gain coordinate ever moves under RN. SR ends 0.46% and 0.39% above the master at seeds 1338 and 1339. At seed 1337 it tracked the master to 2.2B tokens and then rose, on both held-out and training loss, to end 1.9% above it and 0.2% above RN. To find out why, we reran that seed's SR arm with a second SR stream, keeping the initialization and the batch sequence: it descends to the end and finishes 0.45% above the master. So the rise is not a property of the data order or the initialization; we keep the first run in the paper's table. RN/SR is 0.998, 1.016 and 1.012 (1.013 at seed 1337 with the second stream). So at 1× Chinchilla the RN penalty against the master persists in every seed, and SR stays within 0.5% of the master except in that one stream. At this batch SR's throughput cost falls to 2%.
Larger models. We added GPT-2 medium (350M parameters, 25M tokens, Adam at 3×10⁻⁴, three seeds; App. H.7). Under RN none of its 50,176 gain coordinates ever moves, in any Adam or sign run (largest normalized Adam step 1.142), while under SR all of them do. Held-out RN/SR for Adam is 1.045 ± 0.002, above 1 in every seed, and SR matches the fp32 master (1.000 ± 0.002); for sign descent it is 0.980 ± 0.008, so sign again loses nothing (at this shared step sign descent is far from converged, held-out 2.88 against 1.93 for Adam, so this is not a stationary comparison). We did not train beyond 350M; the bitwise prediction depends only on η, κ and the storage format, not on width or depth.
Cost: memory, throughput, training time
Every new run records peak GPU memory, static memory after setup, training throughput and training time (evaluation and checkpointing excluded, diagnosis included). Measured on one RTX 4090 D at batch 8×1024 (App. H.7):
- Memory. Dropping the master saves its 4 bytes per parameter: static memory is 1.17 GiB against 1.64 GiB for Adam. At this batch activations dominate peak memory, so the saving is 5% of peak (8.82 against 9.28 GiB), and 2.7% at the 64k-token batch. We now state it that way, not as an overall training-memory advantage.
- Throughput. Our SR is unfused (one random draw per element) and costs 14–16% (63.7k against 73.9k tokens/s for Adam); a fused rounding kernel may reduce this. Compensating only the gains runs at 72.1k tokens/s, 2.5% below RN. The online arm runs at 70.0k tokens/s, including at most 3.6% for its diagnosis steps.
- Loss at equal training time. At 250M tokens SR is 11% slower per token but still reaches RN's final held-out loss in 0.66–0.71 of RN's training time; the fp32 master does so in 0.56–0.60.
Notation
Q, x, q, M, T, σ, u, L, 𝓕_t and dist are now defined before first use.
We hope the corrected protocol and the new runs restore confidence in the experiments, and would be grateful if you would reconsider your assessment, in particular soundness and presentation.
Thank you for the effort to address the reviews. I acknowledge the protocol corrections and the longer 124M runs. However, the long-budget comparisons cover Adam only, while the 350M experiments remain very limited, leaving some concerns about the broader empirical conclusions unresolved.
The extensive manuscript revisions also change theoretical conditions, withdraw several claims, and replace important experimental evidence, with roughly 60% of the main text appearing to have been revised in only 7 days. Thoroughly assessing these changes requires a substantial fresh review of the manuscript and supporting material. Within the remaining discussion period, alongside my other reviewing commitments, I am unable to complete that reassessment with sufficient care. I therefore cannot confirm that the revised claims and new evidence fully resolve my concerns. I respectfully defer the final decision to the AC. I thank the AC for their time and careful consideration of this case.
I am raising my overall rating to 3 and am curious to see the AC’s final assessment.
Response to Reviewer 68Hv (1/3)
TL;DR. Every discrepancy you found is correct, and we reproduced each of them, including your recomputation of −0.0470 ± 0.0602. We fixed the protocol and reran the main 124M experiments under it. Where the corrected data required it we narrowed claims: LN25 does not match full SR, and the "18 SD" extrapolation is gone. The new runs answer your other questions: diagnosis cost, matched random controls, longer budgets with matched schedules, and measured memory and throughput. Theorem numbers refer to the revised PDF.
Main contribution
The main contribution is the theory: the first lower bounds for training with floating-point weight storage and no master copy. They show that the rounding rule, not only the precision, sets the error floor:
The diagnostic is a corollary, not a claimed contribution. The introduction now states this in a "Main claim" paragraph, and a new Table 1 lists each quantitative prediction, its test, and the outcome.
Which prediction is checked in GPT-2. A bitwise one. LayerNorm gains start at exactly 1.0, a bf16 binade boundary. Absorption (Lemma 4.3) plus a new bound on Adam's step (Lemma A.2) imply that under RN no Adam step size below 1.68×10⁻³ and no sign step size below 2⁻⁹ can ever move a gain coordinate. For sign descent, step sizes in (2⁻⁹, 2⁻⁸) can move gains only downward. The non-obvious ingredient is Lemma A.2: Adam's normalized step is at most 1.165η at β = (0.9, 0.95) for every gradient sequence, so the freeze is certified without looking at gradients (our runs keep fp32 moment buffers, and the logged maximum, 1.137, confirms the margin and shows the bound is nearly tight).
Outcome: it holds coordinate by coordinate. In every Adam and sign seed under RN, not one of the 19,200 gain coordinates ever leaves 1.0; under SR all of them do. A step-size sweep brackets both predicted switches to within 3%: sign descent switches at 1.9–2.0×10⁻³ and 3.8–4.0×10⁻³, against 1.95 and 3.91×10⁻³, and Adam at the same values of its realized update. The loss follows at matched budget: switching exactly these 19,200 coordinates to SR recovers 50% of the RN–SR gap, while 40 random draws of about the same size recover 5–6% on average, each less than the gains.