- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a loss scaling law for quantized pretraining in which the forward and backward quantization choices enter separately. It builds on the Quartet law, which rescales effective model size by the forward scheme and effective data by the backward scheme inside a coupled power law, and adds two scheme-dependent constants meant to capture persistent optimization error from each pass. The motivation is a capacity-gap versus optimization-gap decomposition. As I read it the paper makes three contributions: a decomposition of the NVFP4 and Quartet II recipes into forward and backward components showing forward choices dominate the loss gap; the extended law itself; and a layer-wise extension with a compute-optimal allocation over model size, data, and the fraction of layers kept in higher precision in each pass. For validation the authors reuse the published Quartet grid and add a large new grid of 375 Qwen3 runs covering combination of forward and backward schemes at each model size, with shared initialization and data order. Under one-step size and data extrapolation the proposed law gives lower held-out error than Quartet on both grids, and the ablation favors keeping both persistent terms. The layer-wise extension, fit on smaller models, predicts held-out mixed-precision runs well. The paper concludes that compute-optimal precision rises with budget and that scarce high-precision budget should go to the backward pass first.
Critical strengths
- A controlled grid and a genuinely informative decomposition. The Qwen3 grid covers every combination of the forward and backward schemes at each size, with shared initialization and data order so that loss differences isolate the quantization choice. The decomposition itself is useful on its own: quantizing only the backward pass costs well under a percent, quantizing only the forward pass costs more, and most of the Quartet II gain over the NVFP4 recipe comes from the forward pass. Since both recipes were originally proposed as complete configurations, this breakdown is practical information, and it is the most solid evidence in the paper.
- A simple change that improves held-out prediction on independent data, with the gain traceable to the forward term. The added terms help in every extrapolation setting tested: on the published Quartet grid, size extrapolation improves from 1.6e-4 to 1.4e-5 MSE, and on the authors' own grid from 8.4e-6 to 5.3e-6. The largest win is on the published grid, which guards against overfitting to one setup. The ablation points to where the gain comes from: dropping the forward term raises the geometric-mean error roughly threefold back toward the Quartet baseline, while dropping the backward term barely changes it. That pattern is consistent with the finding that forward quantization dominates, and the fitted values tell the same story: the forward constants differ by about 0.013 between NVFP4 and MXFP8, accounting for a large share of the observed forward-only gap. This matters because accurate loss prediction is the paper's stated goal, and on that criterion the improvement is real within the tested range.
Critical weaknesses
- The headline claim rests on very short extrapolations, and the key transition sits hundreds of times beyond the data. On the authors' grid the size extrapolation is roughly 1.4x and the data extrapolation roughly 1.2x. Over such a range a constant cannot be distinguished from a slowly decaying or slowly growing term, and there are prior work deriving regimes where low-precision loss can worsen with more data. The compute-optimal figure spans many orders of magnitude beyond the largest run, and the key transition sits hundreds of times beyond the data. The budget dependence follows directly from assuming the terms are constant, while the Quartet analysis under its own cost model points the other way at larger scales. The decisive question is which law holds at scale, and the paper does not test it. The gains on the authors' own grid correspond to errors of a few thousandths of a nat with no reported seed noise, so it is unclear what exceeds run-to-run variation.
- The theory does not pin down the functional form, and the mixing analysis is thinly identified. The bound is for convex losses under SGD with a decaying step size, rather than for Adam-trained transformers. An upper bound being constant does not make the gap itself constant: the rounding-error term in the bound stays positive even at the quantized optimum, since rounding error is generically nonzero at any latent point, so a non-vanishing bound does not show a non-vanishing gap. Only the one-dimensional example gives a lower bound, and it uses an initialization exactly on a rounding boundary with exact gradients. The N-dependence is also open, since the rounding-error norm sums per-coordinate errors and the gradient bound need not be N-independent. Empirically the effective-data rescaling saturates at its upper bound for every NVFP4 backward scheme, so on this grid the backward pass is modeled almost entirely by the constant term, and dropping the backward term barely changes the error. The backward term is also attributed to gradient bias even though the backward quantizers studied are designed to be unbiased, which leaves the source of a nonzero backward constant unexplained. The mixing exponent is fit from a single interior mixing fraction, so the power-law shape that gets optimized over the full range is untested; the two passes are assumed to mix independently without jointly mixed runs; and the backward exponent is fit with one backward technique but applied to another.
Questions for authors
- What is the seed-to-seed standard deviation of final validation loss for at least one size and scheme, and are the gains on your grid outside that band?
- The effective-data rescaling sits at its upper bound for every NVFP4 backward scheme, so the backward pass is modeled almost entirely by its constant, yet the backward term is attributed to gradient bias while the quantizers studied are designed to be unbiased. Please report the corresponding Quartet-law values, and explain where the bias comes from or what otherwise accounts for a nonzero backward constant.
- If the forward constant is replaced by a slowly decaying term in model size or data, is held-out accuracy similar, and how much do the compute-optimal conclusions change? If accuracy is similar but the conclusions move a lot, they are not identified by the data.
- Does the persistent term shrink if the learning rate is retuned for quantized runs or if the decay schedule is changed?
3 comments
Thanks for the response. Most of my questions were answered and I will maintain my rating.
Dear Reviewer Jc3P,
Thank you for your response and for maintaining your positive rating. We are glad that our responses addressed most of your questions.
Rebuttal by Authors
Thank you for your positive review. We address your questions below and have uploaded a revised paper with changes highlighted in blue.
Q1. What is the seed-to-seed standard deviation of validation loss, and do the gains exceed it? (W1, Q1)
Please refer to General Response Q1, where we repeat training runs on the held-out 910M model at four token budgets. The run-to-run standard deviation of the validation loss is 5e-4 and 1e-3 for the two repeated recipes, and the improvement of our law is significant against this variability for both recipes (p≤ 2.5e-4).
Q2. Can a constant error term be distinguished from a slowly decaying one, and how much do the compute-optimal conclusions change under such alternatives? (W1, W2, Q3)
Please refer to General Response Q3, where we compare our law with variants whose persistent error terms decay or grow with model size and token count. The decaying variants recover our law, while the growing variants degrade held-out prediction on the 910M model, so we use the simpler form with constant persistent error terms. Since the decaying variants recover our law, the predicted compute-optimal allocation remains unchanged.
Q3. Validation of the predicted compute-optimal allocation through actual training runs. (W1)
Please refer to General Response Q4, where we train the configurations predicted by our law and the Quartet law, along with uniform NVFP4 and MXFP8 baselines, on the held-out 910M model under a matched compute budget. The configuration predicted by our law achieves the lowest loss among them, and our law more accurately predicts the gains over uniform NVFP4.
Q4. Remark 3.1 is a bound for convex losses under SGD rather than for Adam-trained Transformers, and the 1D example relies on initialization on a rounding boundary with exact gradients. (W2)
Remark 3.1 and Example 3.1 are intended as a minimal illustration of how quantized training can leave a persistent error. Such persistent error from weight oscillation has been widely observed beyond this simplified setting, from quantized training of larger-scale vision and language models [1, 2] to, more recently, quantized LLM pretraining [3], and our persistent error terms are further supported by their accurate held-out predictions. We have clarified the scope of Example 3.1 in the revised paper (Section 3.1).
Q5. What are the corresponding fitted values of the Quartet law for the backward penalty? (W2, Q2)
On the Quartet grid, the Quartet law captures the backward penalty with ρD of 0.875 for MXFP4 SR, whereas our law assigns it to both terms (ρD = 0.951 and ϵb = 0.0076, Table 9 in the paper). On our grid, the Quartet law captures it with ρD of 0.951 for NVFP4 (MS-EDEN) and 0.995 for MXFP8, whereas our law assigns it to ϵb (0.0076 and 0.0004, respectively). We speculate that how the backward penalty splits between the two terms depends on the data and the pretraining grid. We have clarified this in the revised paper (Appendix D).
Q6. The mixing exponent is fitted from a single mixing fraction, and the two passes are assumed to mix independently without jointly mixed runs. (W2)
Please refer to General Response Q2, where we (1) add runs keeping 25% and 75% of the blocks in MXFP8, and (2) evaluate Eq. 3 on runs mixing both passes simultaneously, which are held out from fitting. Eq. 2 remains accurate across all mixing ratios (Table A2), and Eq. 3 accurately predicts the jointly mixed runs (Table A3).
Q7. Does the persistent error shrink if the learning rate is rescaled for quantized runs or the decay schedule is changed? (Q4)
Please refer to General Response Q3, where we scale the peak learning rate with the token budget for all runs. The fitted persistent error terms are similar to those under the shared learning rate (ϵf from 0.014 to 0.017 and ϵb from 0.007 to 0.006), indicating that the persistent error stays consistent across learning rates.
References
[1] Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, Tijmen Blankevoort. Overcoming Oscillations in Quantization-Aware Training. ICML 2022.
[2] Alexandre Défossez, Yossi Adi, Gabriel Synnaeve. Differentiable Model Compression via Pseudo Quantization Noise. TMLR 2022.
[3] Yuxiang Chen, Yifan Liu, Xiaoming Xu, Pengle Zhang, Michael Beyer, Martin Rapp, Jun Zhu, Jianfei Chen. TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control. ICML 2026.