acceptodds
Under review as a conference paper at ICLR 2027

Mixed Precision Scaling Laws for Quantized Pretraining

Abstract

Since scaling laws were introduced for modern LLMs kaplan2020scaling, several works have extended them to account for quantization, tracking the emerging low-precision data types supported on AI accelerators. We develop a more accurate scaling law for quantized pretraining, grounded in optimization theory and validated empirically. Our formulation separates the effects of quantization on model capacity and on optimization efficiency. Through convergence analysis, we further observe that both forward and backward quantization can introduce an optimization error that persists with continued training, which we model explicitly in our law. To design our experimental regimes used to fit and evaluate our scaling law, we first revisit existing 4-bit training recipes and decompose them into their constituent forward and backward pass quantization choices, isolating the components that most strongly affect loss. Our Qwen3 experiments under selected 4-bit regimes, together with published Llama-style experiments, show that our law predicts loss with up to 11× lower MSE than existing laws as model size and token count scale. Finally, we extend our law to layer-wise mixed precision, and identify the optimal quantized-pretraining recipe, i.e., model size, training tokens, and per-layer forward/backward precision, under a hardware-aware compute budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, pretraining, quantized pretraining, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Jc3POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a loss scaling law for quantized pretraining in which the forward and backward quantization choices enter separately. It builds on the Quartet law, which rescales effective model size by the forward scheme and effective data by the backward scheme inside a coupled power law, and adds two scheme-dependent constants meant to capture persistent optimization error from each pass. The motivation is a capacity-gap versus optimization-gap decomposition. As I read it the paper makes three contributions: a decomposition of the NVFP4 and Quartet II recipes into forward and backward components showing forward choices dominate the loss gap; the extended law itself; and a layer-wise extension with a compute-optimal allocation over model size, data, and the fraction of layers kept in higher precision in each pass. For validation the authors reuse the published Quartet grid and add a large new grid of 375 Qwen3 runs covering combination of forward and backward schemes at each model size, with shared initialization and data order. Under one-step size and data extrapolation the proposed law gives lower held-out error than Quartet on both grids, and the ablation favors keeping both persistent terms. The layer-wise extension, fit on smaller models, predicts held-out mixed-precision runs well. The paper concludes that compute-optimal precision rises with budget and that scarce high-precision budget should go to the backward pass first.

Critical strengths
  1. A controlled grid and a genuinely informative decomposition. The Qwen3 grid covers every combination of the forward and backward schemes at each size, with shared initialization and data order so that loss differences isolate the quantization choice. The decomposition itself is useful on its own: quantizing only the backward pass costs well under a percent, quantizing only the forward pass costs more, and most of the Quartet II gain over the NVFP4 recipe comes from the forward pass. Since both recipes were originally proposed as complete configurations, this breakdown is practical information, and it is the most solid evidence in the paper.
  2. A simple change that improves held-out prediction on independent data, with the gain traceable to the forward term. The added terms help in every extrapolation setting tested: on the published Quartet grid, size extrapolation improves from 1.6e-4 to 1.4e-5 MSE, and on the authors' own grid from 8.4e-6 to 5.3e-6. The largest win is on the published grid, which guards against overfitting to one setup. The ablation points to where the gain comes from: dropping the forward term raises the geometric-mean error roughly threefold back toward the Quartet baseline, while dropping the backward term barely changes it. That pattern is consistent with the finding that forward quantization dominates, and the fitted values tell the same story: the forward constants differ by about 0.013 between NVFP4 and MXFP8, accounting for a large share of the observed forward-only gap. This matters because accurate loss prediction is the paper's stated goal, and on that criterion the improvement is real within the tested range.
Critical weaknesses
  1. The headline claim rests on very short extrapolations, and the key transition sits hundreds of times beyond the data. On the authors' grid the size extrapolation is roughly 1.4x and the data extrapolation roughly 1.2x. Over such a range a constant cannot be distinguished from a slowly decaying or slowly growing term, and there are prior work deriving regimes where low-precision loss can worsen with more data. The compute-optimal figure spans many orders of magnitude beyond the largest run, and the key transition sits hundreds of times beyond the data. The budget dependence follows directly from assuming the terms are constant, while the Quartet analysis under its own cost model points the other way at larger scales. The decisive question is which law holds at scale, and the paper does not test it. The gains on the authors' own grid correspond to errors of a few thousandths of a nat with no reported seed noise, so it is unclear what exceeds run-to-run variation.
  2. The theory does not pin down the functional form, and the mixing analysis is thinly identified. The bound is for convex losses under SGD with a decaying step size, rather than for Adam-trained transformers. An upper bound being constant does not make the gap itself constant: the rounding-error term in the bound stays positive even at the quantized optimum, since rounding error is generically nonzero at any latent point, so a non-vanishing bound does not show a non-vanishing gap. Only the one-dimensional example gives a lower bound, and it uses an initialization exactly on a rounding boundary with exact gradients. The N-dependence is also open, since the rounding-error norm sums per-coordinate errors and the gradient bound need not be N-independent. Empirically the effective-data rescaling saturates at its upper bound for every NVFP4 backward scheme, so on this grid the backward pass is modeled almost entirely by the constant term, and dropping the backward term barely changes the error. The backward term is also attributed to gradient bias even though the backward quantizers studied are designed to be unbiased, which leaves the source of a nonzero backward constant unexplained. The mixing exponent is fit from a single interior mixing fraction, so the power-law shape that gets optimized over the full range is untested; the two passes are assumed to mix independently without jointly mixed runs; and the backward exponent is fit with one backward technique but applied to another.
Questions for authors
  1. What is the seed-to-seed standard deviation of final validation loss for at least one size and scheme, and are the gains on your grid outside that band?
  2. The effective-data rescaling sits at its upper bound for every NVFP4 backward scheme, so the backward pass is modeled almost entirely by its constant, yet the backward term is attributed to gradient bias while the quantizers studied are designed to be unbiased. Please report the corresponding Quartet-law values, and explain where the bias comes from or what otherwise accounts for a nonzero backward constant.
  3. If the forward constant is replaced by a slowly decaying term in model size or data, is held-out accuracy similar, and how much do the compute-optimal conclusions change? If accuracy is similar but the conclusions move a lot, they are not identified by the data.
  4. Does the persistent term shrink if the learning rate is retuned for quantized runs or if the decay schedule is changed?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your positive review. We address your questions below and have uploaded a revised paper with changes highlighted in blue.


Q1. What is the seed-to-seed standard deviation of validation loss, and do the gains exceed it? (W1, Q1)

Please refer to General Response Q1, where we repeat training runs on the held-out 910M model at four token budgets. The run-to-run standard deviation of the validation loss is 5e-4 and 1e-3 for the two repeated recipes, and the improvement of our law is significant against this variability for both recipes (p≤p \le 2.5e-4).


Q2. Can a constant error term be distinguished from a slowly decaying one, and how much do the compute-optimal conclusions change under such alternatives? (W1, W2, Q3)

Please refer to General Response Q3, where we compare our law with variants whose persistent error terms decay or grow with model size and token count. The decaying variants recover our law, while the growing variants degrade held-out prediction on the 910M model, so we use the simpler form with constant persistent error terms. Since the decaying variants recover our law, the predicted compute-optimal allocation remains unchanged.


Q3. Validation of the predicted compute-optimal allocation through actual training runs. (W1)

Please refer to General Response Q4, where we train the configurations predicted by our law and the Quartet law, along with uniform NVFP4 and MXFP8 baselines, on the held-out 910M model under a matched compute budget. The configuration predicted by our law achieves the lowest loss among them, and our law more accurately predicts the gains over uniform NVFP4.


Q4. Remark 3.1 is a bound for convex losses under SGD rather than for Adam-trained Transformers, and the 1D example relies on initialization on a rounding boundary with exact gradients. (W2)

Remark 3.1 and Example 3.1 are intended as a minimal illustration of how quantized training can leave a persistent error. Such persistent error from weight oscillation has been widely observed beyond this simplified setting, from quantized training of larger-scale vision and language models [1, 2] to, more recently, quantized LLM pretraining [3], and our persistent error terms are further supported by their accurate held-out predictions. We have clarified the scope of Example 3.1 in the revised paper (Section 3.1).


Q5. What are the corresponding fitted values of the Quartet law for the backward penalty? (W2, Q2)

On the Quartet grid, the Quartet law captures the backward penalty with ρD\rho_D of 0.875 for MXFP4 SR, whereas our law assigns it to both terms (ρD\rho_D = 0.951 and ϵb\epsilon_b = 0.0076, Table 9 in the paper). On our grid, the Quartet law captures it with ρD\rho_D of 0.951 for NVFP4 (MS-EDEN) and 0.995 for MXFP8, whereas our law assigns it to ϵb\epsilon_b (0.0076 and 0.0004, respectively). We speculate that how the backward penalty splits between the two terms depends on the data and the pretraining grid. We have clarified this in the revised paper (Appendix D).


Q6. The mixing exponent is fitted from a single mixing fraction, and the two passes are assumed to mix independently without jointly mixed runs. (W2)

Please refer to General Response Q2, where we (1) add runs keeping 25% and 75% of the blocks in MXFP8, and (2) evaluate Eq. 3 on runs mixing both passes simultaneously, which are held out from fitting. Eq. 2 remains accurate across all mixing ratios (Table A2), and Eq. 3 accurately predicts the jointly mixed runs (Table A3).


Q7. Does the persistent error shrink if the learning rate is rescaled for quantized runs or the decay schedule is changed? (Q4)

Please refer to General Response Q3, where we scale the peak learning rate with the token budget for all runs. The fitted persistent error terms are similar to those under the shared learning rate (ϵf\epsilon_f from 0.014 to 0.017 and ϵb\epsilon_b from 0.007 to 0.006), indicating that the persistent error stays consistent across learning rates.


References

[1] Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, Tijmen Blankevoort. Overcoming Oscillations in Quantization-Aware Training. ICML 2022.

[2] Alexandre Défossez, Yossi Adi, Gabriel Synnaeve. Differentiable Model Compression via Pseudo Quantization Noise. TMLR 2022.

[3] Yuxiang Chen, Yifan Liu, Xiaoming Xu, Pengle Zhang, Michael Beyer, Martin Rapp, Jun Zhu, Jianfei Chen. TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control. ICML 2026.

Thanks for the response. Most of my questions were answered and I will maintain my rating.

Dear Reviewer Jc3P,

Thank you for your response and for maintaining your positive rating. We are glad that our responses addressed most of your questions.

Reviewer_6V27OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a scaling law for quantized LLMs pretraining that incorporates mixed precision across forward and backward passes plus across transformer blocks. The work builds on the Quartet framework by adding two additive terms (forward weights oscillation and gradient bias) to capture persistent optimization errors while previous approaches try to express the quantization error as a reduced effective model size or training token count. The authors first decompose two FP4 pretraining recipes on a Qwen3 model to select quantization schemes then fit and evaluate their scaling law. Finally, they extend the law to layer-wise MXFP8/NVFP4 mixing via a power-law interpolation exponent and solve for compute-optimal precision allocations, concluding that optimal precision increases with the compute budget.

Critical strengths
  • Section 2 cleanly shows that, perhaps surprising, "forward pass quantization is generally more sensitive than backward pass quantization" in both the Quartet II and NVFP4 settings, expanding the previous works.
  • Adding an irreducible quantization error floor interpretably corrects previous works that mostly use multiplicative adjustments.
  • The Qwen3 375-run grid uses CompleteP parameterization, matched initial checkpoints, and identical data ordering. All those measures make the experiments comparison clean.
Critical weaknesses
  • While the proposed law form and parametrization seem reasonable and well-explained, the Table 9 reveals the fit collapse: ρD\rho_D is 1.0 in most cases or close to it. While the explanations may be sound, such collapse suggests over-parametrization and makes the final MSE improvements suspicious. Furthermore, BF16 parameters on the Quartet grid are severely ill-conditioned: dropping a single token budget (D/N=800) between the model-size and data splits causes A to jump (1.1e5 to 3.5e5) and B (2.5e5 to 6.7e5). Perhaps this can be cleanly interpreted, but the explanation is missing.
  • While the scaling law explanation can be reasonable, the work in its current state lacks significance analysis. MSE reports are unreasonable without a significance test (aka excluding pure noise fit improvements).
  • The learning rate scaling is suspicious. Appendix B reports the learning rate to be scaled according to the lowest tokens budget (Bjorck et al., 2025). By Bjorck et al., increasing D by 2.8x (12.8B to 35.8B) requires significant LR reduction. Using an unscaled, overly high peak learning rate at longer token horizons inflates steady-state quantization oscillation.
  • Eq. 4 assumes identical forward and backward kernel speedups, ignoring published benchmarks (e.g., Quartet) showing backward FP4 speedups are lower than forward speedups. This biases the conclusion that the backward pass should be quantized first.
  • Remark 3.1 uses Euclidean norms for the parameters distances, which depend on the dimensionality N (parameters count). Therefore, the bound cannot be used to motivate the N-independence of the ϵ\epsilon term. If anything, it proves that it in fact depends on the model size.
Questions for authors
  • As the first and second weakness suggests, it is hard to estimate whether the proposed law form is indeed a reasonable change or just a better noise fit. Please provide statistical tests on the parameters’ fit significance.
  • Concerning the weakness 5, In Table 2 you comprehensively survey QAT scaling laws alongside PTQ and QPT laws to contrast how prior literature models quantization penalties, yet you omit Compute-Optimal Quantization-Aware Training (Dremov et al., 2026, ICLR 2026). Their formulation shares direct parallels with your core work: like your paper, they identify that existing laws fail in the large-data limit and introduce an explicit irreducible error floor alongside decay terms. Given that your parameter ρD\rho_D collapses to 1.000, how do you justify modeling ϵ\epsilon as static constants independent of model size N, and could you discuss how your conclusion that optimal pretraining precision increases with compute relates to Dremov et al.’s finding that optimal QAT fractions increase with compute scale?
  • Consider running small-scale experiments with appropriate LR scaling for the training tokens budget. This will actually prove the algorithm’s robustness in real-world scenarios.
  • Perhaps, the most important practical outcome is the prediction of compute-optimal precision. You seem not to validate how accurate your predictions are on the real tests. Have you tried to predict optimal QPT setup for a larger model, sample configurations around the predicted optima (sub-optimal neighborhood), and check after the training that the predicted optima is indeed valid?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the valuable feedback. We address your questions and concerns below and have uploaded a revised paper with changes highlighted in blue.


Q1. Is the improvement of the proposed law statistically significant? (W1, W2, Q1)

Please refer to General Response Q1, where we repeat the training of two NVFP4 quantization recipes three times each on the held-out 910M model at four token budgets. Using a t-test against run-to-run noise, the improvement of our law is significant for both recipes (p≤p \le 2.5e-4, Table A1).


Q2. Running small-scale experiments with LR scaling for the training tokens budget. (W3, Q3)

Please refer to General Response Q3, where we conduct a smaller-scale data extrapolation experiment with the peak learning rate scaled with the token budget. Our law achieves a 37% lower held-out MSE than the Quartet law (Table A5), and the fitted persistent error terms remain similar to those under the shared learning rate.


Q3. Eq. 4 assumes identical forward and backward speedups, which may bias the conclusion that the backward pass should be quantized first. (W4)

Our cost model (Eq. 4) supports separate forward and backward speedups, as each pass has its own cost term. In Figure A2, we further visualize the optimal allocation with the forward NVFP4 speedup fixed to (a) 3.0$\times$ or (b) 2.6$\times$, while varying the backward NVFP4 speedup along the y-axis. Even when the forward and backward speedups differ, the backward pass keeps NVFP4 up to larger budgets than the forward pass, unless its speedup drops close to that of MXFP8. This is because the backward pass also has a larger γb\gamma_b, which keeps the loss increase small when partially quantized, and a smaller persistent error, both independent of the speedups (Section 4.3). We have included this analysis in the revised paper (Appendix J).


Q4. The bound cannot motivate error terms that are independent of N. (W5, Q2)

Please refer to General Response Q3, where we compare our law with variants whose persistent error terms decay or grow with model size NN or token count DD (Table A4). The decaying variants recover our law, while the growing variants increase the held-out MSE on the 910M model by 72% for N and 29% for D, with largely consistent results in the other three extrapolation settings. We therefore use the simpler form with constant persistent error terms. We have included these results in the revised paper (Appendix F).


Q5. Missing discussion of Dremov et al. (2026), and how our finding that optimal precision increases with compute relates to their finding that optimal QAT fractions increase with compute. (Q2)

Thank you for pointing out this work. We have added it to Table 2 and discussed its relation to our work in the revised paper (Appendix O). As you noted, Dremov et al. (2026) also model an irreducible QAT error, which depends only on the precision and is independent of both model size and token count, like our persistent error terms. Since their final model is quantized to a fixed bit width regardless of how training is split between the full-precision and QAT phases, this error does not affect their optimal QAT fraction. In quantized pretraining, by contrast, our persistent error can be reduced by training in higher precision, which becomes optimal since this error outweighs the diminishing loss reduction from the speedup as compute grows.


Q6. Validation of the predicted compute-optimal allocation through actual training runs. (Q4)

Please refer to General Response Q4, where we train the configurations predicted by our law and the Quartet law, along with uniform NVFP4 and MXFP8 baselines, on the held-out 910M model under a matched compute budget. The configuration predicted by our law achieves the lowest loss among them, and our law more accurately predicts the gains over uniform NVFP4.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 6V27,

We thank you for your valuable feedback. As the discussion period ends in a day, we would like to kindly check whether our responses and the revised paper have addressed your concerns. If you have any further questions, please feel free to let us know.

Thank you once again for your time and effort.

Reviewer_vWywOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a scaling law for quantized LLM pretraining by incorporating persistent optimization error terms for forward and backward quantization. The authors validate improved loss prediction on Qwen3 and published Llama-style experiments, extend the formulation to layer-wise mixed precision, and derive compute-optimal precision allocations under different training budgets and hardware speedups.

Critical strengths
  1. Improved scaling-law formulation. The paper introduces a reasonable extension to existing quantization scaling laws by separately modeling forward and backward optimization errors. The proposed formulation consistently improves held-out loss prediction over the Quartet baseline across the evaluated settings.
  2. Extensive empirical investigation. The authors conduct a substantial number of pretraining experiments across model sizes, token budgets, and quantization schemes. The decomposition of FP4 training recipes and the analysis of forward/backward precision sensitivity provide useful insights into quantized pretraining.
Critical weaknesses
  1. Unclear practical significance of the prediction improvements. Although the proposed law reduces prediction MSE, the absolute improvements are relatively small in several settings (e.g., MAPE decreases from 0.076% to 0.059% for Qwen3 model-size extrapolation). The paper does not establish whether these improvements are meaningful relative to training variability or whether they lead to better precision selection decisions. Consequently, the practical value of the improved scaling law remains unclear.
  2. Insufficient depth of experimental validation. The paper presents several interesting findings, but the experiments do not sufficiently support some of its broader conclusions. In particular, the layer-wise interpolation is evaluated only at a fixed 50% NVFP4 ratio, while the final optimization considers arbitrary mixing ratios and simultaneous forward/backward mixing. Moreover, the compute-optimal allocations are derived using a simplified hardware cost model without validation through actual training. These gaps weaken the paper's most practically relevant claims.
Questions for authors
  1. Practical significance of prediction accuracy. Given the small absolute prediction improvements in several settings, how do these differences compare with run-to-run training variability? More importantly, can the authors demonstrate that the improved predictions lead to more accurate configuration rankings or better precision allocation decisions than the Quartet baseline?
  2. Generalization of mixed-precision modeling. Can the authors evaluate the proposed layer-wise interpolation at additional NVFP4 ratios (e.g., 25% and 75%) and with simultaneous forward/backward mixing? These configurations are directly relevant to the final compute-optimal analysis but are not covered by the current validation.
  3. Practical validation of compute-optimal allocations. Can the authors compare the predicted optimal configurations against uniform FP4/FP8 baselines under matched training budgets? Additionally, how sensitive are the predicted allocations to realistic end-to-end training speedups rather than assumed GEMM speedups?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the valuable feedback. We address your questions and concerns below and have uploaded a revised paper with changes highlighted in blue.


Q1. Are the prediction improvements larger than run-to-run training variability? (W1, Q1)

Please refer to General Response Q1, where we repeat training runs on the held-out 910M model at four token budgets. The run-to-run standard deviation of the validation loss is 5e-4 and 1e-3 for the two repeated recipes, and the improvement of our law is significant against this variability for both recipes (p≤p \le 2.5e-4).


Q2. Does the interpolation in Eq. 2 remain accurate at other NVFP4 ratios (e.g., 25% and 75%)? (W2, Q2)

Please refer to General Response Q2, where we add layer-wise mixed-precision runs keeping 25% and 75% of the blocks in MXFP8 for both passes, both selection strategies, and all model sizes and token budgets. Eq. 2 remains accurate across all mixing ratios, with a held-out R2R^2 of 0.991 on the 910M model, and endpoints from our law yield more accurate predictions than those from the Quartet law for every pass and strategy (Table A2).


Q3. Is the independence assumption in Eq. 3 supported when forward and backward passes are mixed simultaneously? (W2, Q2)

Please refer to General Response Q2, where we evaluate Eq. 3 on held-out runs mixing both passes simultaneously on the 910M model. Eq. 3 accurately predicts these runs without being calibrated on them, with an MSE comparable to that on the single-pass runs (Table A3), supporting the independence assumption.


Q4. Validation of the predicted compute-optimal allocations against uniform FP4/FP8 baselines under matched training budgets. (W2, Q1, Q3)

Please refer to General Response Q4, where we train the configurations predicted by our law and the Quartet law, along with uniform NVFP4 and MXFP8 baselines, on the held-out 910M model under a matched compute budget. The configuration predicted by our law achieves the lowest loss with a clear advantage over both uniform baselines (Table A6), and our law more accurately predicts the gains over uniform NVFP4 (Table A7).


Q5. Predicted allocations to realistic end-to-end speedups rather than assumed GEMM speedups. (W2, Q3)

While Figure 1 in the paper assumes the GEMM speedup for MXFP8 (1.8$\times$), our cost model takes the speedup of each format over BF16 as an input, so the optimal configuration can be computed for realistic speedups measured on a given hardware and software stack. Since end-to-end MXFP8 speedups are often smaller than their GEMM speedups (e.g., 1.28$\times$ over BF16 [1]), we further visualize the optimal allocation with the MXFP8 speedup lowered to 1.6$\times$ and 1.4$\times$ in Figure A1. Lower MXFP8 speedups shift the switch to larger budgets, but the optimal precision still increases with the budget, and the backward pass keeps NVFP4 longer than the forward pass. We have included these results in the revised paper (Appendix I).


References

[1] PyTorch Team. Accelerating 2K scale pre-training up to 1.28x with TorchAO, MXFP8 and TorchTitan on Crusoe B200 Cluster. PyTorch Blog, 2025.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer vWyw,

We thank you for your valuable feedback. As the discussion period ends in a day, we would like to kindly check whether our responses and the revised paper have addressed your concerns. If you have any further questions, please feel free to let us know.

Thank you once again for your time and effort.

The authors have adequately addressed my main concerns through additional experiments, so I increase my rating from 2 to 3.

Reviewer_3VeyOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Scaling Laws in quantized pretraining. Existing approaches primarily model quantization loss through effective parameter count and the number of training tokens, which makes it difficult to capture persistent optimization error. To address this limitation, the authors introduce persistent optimization error terms corresponding to forward and backward quantization, and validate their predictive power through both theoretical analysis and experiments. Furthermore, the model is extended to Layer-wise Mixed Precision in order to predict loss under different precision configurations and to jointly optimize model scale, training data volume, and precision allocation under a given computational budget.

Critical strengths
  1. The paper focuses on the practically important problem of computational resource allocation in quantized pretraining and incorporates model scale, training data volume, and numerical precision into a unified modeling framework.
  2. By separately modeling the effects of quantization on model capacity and optimization efficiency, and by introducing persistent optimization error terms, the work provides a reasonably interpretable extension of existing Scaling Laws.
  3. Experiments conducted on both Quartet and Qwen3 datasets demonstrate superior performance over prior methods on model-scale and training-token extrapolation tasks, achieving up to approximately an 11-fold reduction in mean squared error (MSE).
Critical weaknesses
  1. The theoretical analysis is insufficient to fully support the persistent optimization error hypothesis. The optimization error upper bound derived in Remark 3.1 contains a term that does not decay with the number of training steps; however, this does not imply that the actual error necessarily possesses a non-zero lower bound. Moreover, the one-dimensional constructive example is inadequate to establish the prevalence of this phenomenon in practical large language model (LLM) pretraining.
  2. Experimental validation of Layer-wise Mixed Precision is limited. Section 4.2 only evaluates the setting in which 50% of the blocks use low precision, which is insufficient to confirm that the power-function interpolation in Equation (2) remains accurate at other ratios—particularly given that this formula is subsequently employed for continuous precision allocation optimization.
  3. There is a lack of joint validation for simultaneous forward and backward mixed-precision configurations. Experiments only vary layer-wise precision independently in either the forward or the backward pass, yet Equation (3) assumes that the two effects can be combined independently. Because forward quantization can influence the computation of backward gradients, this independence assumption lacks direct experimental support.
  4. The conclusions regarding computationally optimal precision allocation lack empirical verification. The computational budgets considered in Figure 1 substantially exceed the range covered by the experiments, and the cost model relies primarily on theoretical FLOPs together with assumed hardware acceleration ratios, without adequately accounting for quantization overhead or actual training throughput. Consequently, it remains unclear whether the predicted optimal configurations are applicable to real large-scale pretraining.
Questions for authors
  1. Can the authors further validate the stability of the persistent optimization error terms across different model scales, numbers of training tokens, and learning-rate schedules, in order to support the assumption that these terms are independent of model scale and training data volume?
  2. Why is Layer-wise Mixed Precision evaluated only at a 50% mixing ratio? Does the proposed interpolation formula remain accurate at other ratios such as 25% and 75%?
  3. Have the authors validated configurations in which Layer-wise Mixed Precision is applied simultaneously to both the forward and backward passes? Does the independence assumption underlying Equation (3) receive experimental support?
  4. Have the optimal precision configurations predicted in Figure 1 been verified through actual training runs? Would the final conclusions remain unchanged if end-to-end training time, rather than theoretical FLOPs, were adopted as the computational budget?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your positive review. We address your questions below and have uploaded a revised paper with changes highlighted in blue.


Q1. The non-vanishing upper bound does not imply a persistent optimization error, and the example does not establish that this error occurs in larger-scale pretraining. (W1)

We note that persistent error in quantized training caused by weight oscillation, which prevents the quantized weights from converging, is not specific to our example and has been widely observed and studied empirically, from quantized training of larger-scale vision and language models [1, 2] to, more recently, quantized LLM pretraining [3]. Remark 3.1 and Example 3.1 are intended to illustrate this phenomenon in a minimal setting. We have clarified this in the revised paper (Section 3.1).


Q2. Can you support the assumption that the persistent error terms are independent of model scale and training data volume? (Q1)

Please refer to General Response Q3, where we compare our law with variants whose persistent error terms decay or grow with model size and token count. The decaying variants recover our law, while the growing variants degrade held-out prediction on the 910M model, so we use the simpler form with constant persistent error terms.


Q3. Does the interpolation in Eq. 2 remain accurate at mixing ratios other than 50% (e.g., 25% and 75%)? (W2, Q2)

Please refer to General Response Q2, where we add layer-wise mixed-precision runs keeping 25% and 75% of the blocks in MXFP8. The results show that Eq. 2 remains accurate across all mixing ratios, with a held-out R2R^2 of 0.991 on the 910M model (Table A2).


Q4. Is the independence assumption in Eq. 3 supported when forward and backward passes are mixed simultaneously? (W3, Q3)

Please refer to General Response Q2, where we evaluate Eq. 3 on additional held-out runs mixing both passes simultaneously on the 910M model. Eq. 3 accurately predicts these runs without being calibrated on them (Table A3), supporting the independence assumption.


Q5. The predicted compute-optimal configurations are not verified by actual training runs. (W4, Q4)

Please refer to General Response Q4, where we train the configurations predicted by our law and the Quartet law, along with uniform NVFP4 and MXFP8 baselines, on the held-out 910M model under a matched compute budget. The configuration predicted by our law achieves the lowest loss among them, and our law more accurately predicts the gains over uniform NVFP4.


Q6. Do the conclusions hold when end-to-end training time, rather than theoretical FLOPs, is used as the compute budget? (W4, Q4)

Our scaling law predicts loss independently of the cost model, so the FLOPs-based cost can be replaced by any measure of training cost, such as the relative end-to-end training time of each precision over BF16. As long as lower precision provides a bounded speedup, higher precision eventually becomes optimal as the budget grows, since the persistent error terms do not diminish with scale while the gains from the speedup do. The cost model only shifts the budgets at which higher precision becomes optimal, as we illustrate in Figure A1 (Appendix I in the revised paper) with the MXFP8 speedup lowered to 1.6$\times$ and 1.4$\times$.


References

[1] Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, Tijmen Blankevoort. Overcoming Oscillations in Quantization-Aware Training. ICML 2022.

[2] Alexandre Défossez, Yossi Adi, Gabriel Synnaeve. Differentiable Model Compression via Pseudo Quantization Noise. TMLR 2022.

[3] Yuxiang Chen, Yifan Liu, Xiaoming Xu, Pengle Zhang, Michael Beyer, Martin Rapp, Jun Zhu, Jianfei Chen. TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control. ICML 2026.

Dear Reviewer 3Vey,

We thank you for your encouraging comments and valuable feedback. We hope our responses and the revised paper have addressed your questions. If you have any further questions, please feel free to let us know.

Thank you once again for your time and effort.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank the reviewers for spending time and effort carefully reading our paper and providing helpful feedback. We have uploaded a revised paper, with changes since submission highlighted in blue. We address points raised by multiple reviewers below, and all other comments and questions in individual replies, indicating where each result appears in the revised paper.


Q1. Is the improvement of our law significant against run-to-run variability? (Reviewers vWyw, 6V27, and Jc3P)

To assess whether the improvement of our law exceeds run-to-run variability, we repeat the training of two NVFP4 quantization recipes three times each on the held-out 910M model at four token budgets (18.1B, 21.7B, 25.4B, and 29.9B). In Table A1, we compare the MSE of each law against the mean of the repeated runs at each budget, using a t-test under the null hypothesis that the difference in squared errors between the two laws arises only from run-to-run noise. The improvement is significant for both recipes (p≤p \le 2.5e-4). We have included these results in the revised paper (Appendix E).

ForwardBackwardRun-to-run stdMSE (Quartet law)MSE (Ours)tp
NVFP4 (4/6 + 1D)NVFP4 (MS-EDEN)5.3e-41.3e-53.2e-66.22.5e-4
NVFP4 (4/6 + 1D)NVFP4 (RHT + SR)9.6e-42.4e-51.0e-67.94.7e-5

Table A1: Held-out MSE on the 910M model over four token budgets, relative to the mean of three repeated runs, with a t-test against run-to-run noise.


Q2. Does the mixed-precision interpolation remain accurate at other mixing ratios and when both passes are mixed simultaneously? (Reviewers vWyw, 3Vey, and Jc3P)

Other mixing ratios. We add layer-wise mixed-precision runs that keep 25% or 75% of the blocks in MXFP8, on top of the 50% runs in the paper, covering both passes, both selection strategies, all four model sizes, and all four token budgets. Following the setup in Section 4.2, we fit γS\gamma_S on the mixed-precision runs at all three ratios on the three smaller models, using endpoints predicted by either the Quartet law or our law, and evaluate on the held-out 910M model. As shown in Table A2, Eq. 2 remains accurate across all mixing ratios, and endpoints from our law yield more accurate predictions than those from the Quartet law for every pass and strategy.

PassStrategyEndpointsγS\gamma_SR2R^2 ↑MSE ↓
ForwardFirst/lastQuartet law1.300.9761.8e-5
Ours1.100.9934.9e-6
ForwardMiddleQuartet law1.170.9841.2e-5
Ours1.000.9972.0e-6
BackwardFirst/lastQuartet law1.900.9672.3e-5
Ours1.600.9897.6e-6
BackwardMiddleQuartet law0.600.9622.6e-5
Ours0.500.9791.4e-5
All–Quartet law–0.9742.0e-5
Ours–0.9917.1e-6

Table A2: Fitted γS\gamma_S and held-out R2R^2 and MSE on the 910M model, with mixed-precision runs keeping 25%, 50%, and 75% of the blocks in MXFP8.

Mixing both passes simultaneously. We further train runs mixing both passes simultaneously at (pf,pb)=(0.5,0.5)(p_f, p_b) = (0.5, 0.5), keeping the first and last blocks in MXFP8, on the held-out 910M model at all four token budgets. We predict these runs with Eq. 3, using γS\gamma_S fitted only on the single-pass mixed-precision runs of the three smaller models. As shown in Table A3, Eq. 3 accurately predicts these runs without being calibrated on them, with an MSE comparable to that on the single-pass runs (Table A2), supporting the independence assumption. Endpoints from our law again yield more accurate predictions than those from the Quartet law. We have included these results in the revised paper (Section 4.2 and Appendix H).

MixingEndpointsR2R^2 ↑MSE ↓
Forward + backwardQuartet law0.9898.1e-6
Ours0.9906.9e-6

Table A3: Held-out R2R^2 and MSE of Eq. 3 on the 910M model, with runs mixing both passes simultaneously at (pf,pb)=(0.5,0.5)(p_f, p_b) = (0.5, 0.5).

AuthorsOpenReview ↗

General Response (2/2)

Q3. Do the persistent error terms depend on model size, token count, or the learning rate? (Reviewers 3Vey, 6V27, and Jc3P)

Model size and token count. We agree that the persistent error terms could in principle depend on model size NN and token count DD. In Table A4, we test this with variants whose forward and backward persistent error terms are scaled by (N/N0)±c(N/N_0)^{\pm c} or (D/D0)±c(D/D_0)^{\pm c}, allowing them to decay or grow with NN or DD, all fitted on the three smaller models and evaluated on the held-out 910M model. The decaying variants recover our law, as the fitted exponent converges to zero, while the growing variants increase the held-out MSE by 72% for NN and 29% for DD. In the revised paper (Appendix F), we further run this ablation in the other three extrapolation settings, with largely consistent results. We therefore use the simpler form with constant persistent error terms, which also extrapolates more accurately to larger models.

Persistent error termFitted exponentHeld-out MSE (×10−6\times 10^{-6})Held-out R2R^2
Constant (ours)–5.290.994
Decaying, ϵ(N/N0)−c\epsilon (N/N_0)^{-c}c→0c \to 05.290.994
Growing, ϵ(N/N0)c\epsilon (N/N_0)^{c}0.129.080.989
Decaying, ϵ(D/D0)−c\epsilon (D/D_0)^{-c}c→0c \to 05.290.994
Growing, ϵ(D/D0)c\epsilon (D/D_0)^{c}0.206.840.992

Table A4: Held-out MSE and R2R^2 on the 910M model with the persistent error terms of our law scaled with model size NN or token count DD (N0N_0 = 427M, D0D_0 = 21.8B).

Learning rate. We further conduct a smaller-scale data extrapolation experiment in which the peak learning rate is scaled with the token budget. We train the 280M, 430M, and 640M models with BF16 and with NVFP4 (4/6 + 1D forward, MS-EDEN backward) in the forward pass, backward pass, or both, fit both laws on 12.8B, 15.3B, 17.9B, and 21.1B tokens, and evaluate on the runs with NVFP4 in both passes at 25.5B tokens. As shown in Table A5, our law achieves a 37% lower held-out MSE than the Quartet law, and the fitted persistent error terms remain similar to those under the shared learning rate (ϵf\epsilon_f from 0.014 to 0.017 and ϵb\epsilon_b from 0.007 to 0.006). We have included these results in the revised paper (Appendices F and G).

LawHeld-out R2R^2 ↑Held-out MSE ↓
Quartet law0.9844.8e-5
Ours0.9903.0e-5

Table A5: Held-out R2R^2 and MSE at 25.5B tokens with the peak learning rate scaled with the token budget.


Q4. Are the predicted compute-optimal allocations validated by actual training runs? (All reviewers)

To validate the predicted allocations in practice, we train the compute-optimal configuration predicted by each law on the held-out 910M model, along with uniform NVFP4 and MXFP8 baselines. All runs use a matched compute budget of C=7.9×1019C = 7.9 \times 10^{19} FLOPs, with speedups of 2.8$\times$ for NVFP4 and 1.8$\times$ for MXFP8 over BF16. In Table A6, the configuration predicted by our law achieves the lowest loss with a clear advantage over both uniform baselines, while that of the Quartet law remains comparable. Moreover, in Table A7, our law more accurately predicts the gains of both configurations over uniform NVFP4, which the Quartet law underestimates or predicts in the wrong direction. We have included these results in the revised paper (Section 4.3 and Appendix K).

MethodConfigurationObserved loss ↓
Uniform NVFP4Fwd: NVFP4, Bwd: NVFP4, DD = 40.9B3.0142
Uniform MXFP8Fwd: MXFP8, Bwd: MXFP8, DD = 26.3B3.0388
Quartet lawFwd: 61% NVFP4, Bwd: NVFP4, DD = 38.2B3.0129
OursFwd: 11% NVFP4, Bwd: NVFP4, DD = 35.1B3.0124

Table A6: Observed validation loss on the 910M model under a matched compute budget (C=7.9×1019C = 7.9 \times 10^{19} FLOPs). Percentages denote the fraction of NVFP4 blocks in the forward pass, with the first and last blocks kept in MXFP8.

Config vs. uniform NVFP4ObservedPredicted (Quartet law)Predicted (Ours)
Quartet law's1.3e-3 better0.5e-3 better1.4e-3 better
Ours1.8e-3 better1.0e-3 worse2.4e-3 better

Table A7: Observed and predicted loss difference of each law's configuration relative to uniform NVFP4 in Table A6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.