acceptodds
Under review as a conference paper at ICLR 2027

Full-Stack NVFP4: Stable, Role-Aware 4-Bit LLM Pretraining Beyond Linear Projections

Abstract

NVFP4 pretraining typically targets Transformer projections while leaving optimizer states, optimizer computation, and attention in higher precision. Extending NVFP4 beyond projections is not a uniform quantization problem: each module has a distinct numerical role and failure mode. We present , a role-aware framework instantiated by four independently composable recipes. retains a compact BF16 projection subspace within full-shape NVFP4 computation; gradient decoupling and periodic SVD realignment stabilize its low-rank factors, reducing the linear-only loss gap from to . transforms persistent momentum states before storage, stabilizes direct NVFP4 Newton–Schulz iterations, and protects softmax-sensitive products in BF16. On 3B pretraining with 64B tokens, BF16 and Full-Stack NVFP4 reach losses of and , a gap, with similar aggregate zero-shot results. Native four-block measurements on one RTX 5090 show 2.50–2.83 Root speedups over optimized BF16 and 37.9–42.5% lower AdamW peak memory.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language model pretraining, low-precision training, 4-bit quantization, NVFP4, optimizer-state quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xh4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Full-Stack NVFP4, four role-aware recipes extending 4-bit pretraining beyond linear projections: LoRA-SVD for projection subspaces, NVFP4-state AdamW, NVFP4-native Newton-Schulz iterations, and selective FP4 attention. A 3B model trained for 64B tokens reaches within 0.838% of BF16 loss with matched zero-shot accuracy; microbenchmarks show 2.50-2.83x speedups for Q-Root and ~40% lower peak optimizer memory. Per the paper's coverage table, this is the first FP4 recipe addressing all four modules jointly.

Critical strengths
  1. The first NVFP4 recipe covering the full training stack, with end-to-end quality validation at 3B / 64B tokens -- a substantive step beyond projection-only or optimizer-only prior work.
  2. Well-chosen controls: matched budgets, per-module ablations, a BF16+LoRA capacity control, and native-hardware comparisons make the component-level attribution credible.
Critical weaknesses
  1. The efficiency evidence does not support the practical claims: quality runs use fake quantization, speed/memory numbers come from a single-GPU microbenchmark, and the 56.9-61.9% savings are extrapolations; per-module gains are highly uneven (2.83x vs. 1.27x), so the headline numbers are not representative of the full stack.
  2. Key ablation differences (0.5-0.6% single-run loss gaps) lack uncertainty estimates and may be within run-to-run noise.
Questions for authors
  1. How many seeds were run? Do the 0.61% vs. 1.12% vs. 0.56% differences exceed run-to-run variance?
  2. How is numerical equivalence between fake-quant and native NVFP4 kernels established (SR is disabled on SM120)? Is there any native-precision training supporting the quality conclusions?
  3. What is the loss gap between Q-Root and GRASP/low-bit Muon in a matched pretraining setting?
  4. Is the 0.838% gap converging or diverging within the 64B tokens, and how would it evolve at longer budgets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xh4Q (part 1 of 2)

Thank you for the positive comments on full-stack coverage (Table 1, p.2), the 3B / 64B validation (Figure 5, Table 4), and the BF16+LoRA capacity control (Table 3). We answer the questions first, then add two points from the weaknesses that the questions do not ask.

Q1. How many seeds were run? Do 0.61% vs 1.12% vs 0.56% exceed run-to-run variance?

Table 3 in the submission is seed 42; we added seed 44 (same 1B / 20B setup). Because of training cost, there is no third seed and no standard deviation.

Settingseed 42seed 44
NVFP4 4/6+1.40%+1.57%
4/6 + LoRA-SVD+0.61%+0.66%
4/6 + Q-AdamW+0.98%+1.25%
4/6 + Q-Attn+0.47%+0.83%
Full-Stack w/o Root+0.56%+0.79%
  • 0.61% is 4/6 + LoRA-SVD (seed 42); seed 44 is +0.66%, stable across seeds.
  • 1.12% is 4/6 + plain LoRA in the same table (seed 42 only), a capacity control. The quality result we keep is LoRA-SVD recovering the 4/6 gap (1.40→0.61 and 1.57→0.66). This shows the gain is not mainly from adding a LoRA branch, but from the initialization and the constraint.
  • 0.56% is Full-Stack w/o Root (seed 42); seed 44 is +0.79%. That row is under a Root (Muon-style) optimizer and is not comparable to the AdamW block above.

We do not rank modules on sub-1% gaps, especially 0.5–0.6%, and we do not call them quality improvements. Quantizing the optimizer and attention changes grad-norm fluctuation and couples modules, so the trajectory changes. The intent of Table 3 is the last sentence of §4.2: enabled independently and composed. We treat the remaining <1% residuals as close to BF16, with components that can be used together stably.

Q2a. How is numerical agreement between fake-quant and native NVFP4 established (SR off on SM120)?

First paragraph of §4.4 (p.7): this SM120 path has no efficient SR primitive. TE and we both disable SR in the timing runs so the efficiency comparison is not dominated by the rounding implementation. Quality runs still use SR in the backward pass (last paragraph of §2, p.2); that is not the same rounding as the 5090 timing runs.

Operator development uses the fake-quant path and BF16 as references. Per-component error vs BF16 is at the same level as the Rel. MSE in Figures 2–4. On 4-block forward–backward with the Table 5 shapes, relative error between fake-quant and native operators is also at the same level. We think operator quality should match the fake-quant experiments. Still, this is not native end-to-end training.

Q2b. Is there any native-precision training that supports the quality claims?

No. Quality comes from fake-quant pretraining on Ampere (§4.1). Efficiency comes from four-block runs on a single 5090 (§4.4 / Table 5). Scale-up native training on 5090 is not realistic now. The quality claims rest on fake-quant pretraining, plus the operator-level / 4-block numerical check in Q2a.

Q3. In a matched pretraining setting, what is the loss gap of Q-Root vs GRASP / low-bit Muon?

The open-source GRASP fake-quant implementation is too expensive; we have not been able to run this experiment. Isolated BF16+Q-Root vs BF16+Root is +0.34%. That does not replace a matched pretraining comparison to GRASP.

Q4a. Is the 0.838% converging or diverging within 64B tokens?

Figure 5 (§4.3, p.7) and the sentence after it: the curves stay close throughout, ending at 2.286 vs 2.267, a relative 0.838%. After zooming 56–64B, Full-Stack stays about 0.02 absolute loss above BF16, and both curves are still decreasing. This is a persistent offset, not last-step jitter, and there is no clear sign that it is blowing up.

Q4b. What happens with a longer budget?

The Limitations section already says we only have 3B / 64B. We do not extrapolate here, and we cannot support larger or longer runs. On the curves we have, the recipe buys efficiency at a small quality gap; at least for the early stage of LLM training, that holds.

AuthorsOpenReview ↗

Reponse to Reviewer xh4Q (part 2 of 2)

Additional points not covered directly by the questions

Efficiency evidence and “practical” claims. If the claim is the 56.9–61.9% in the last paragraph of §4.4, or the unqualified 2.50–2.83× from Table 5, we agree the evidence is not enough for that reading. The main result should be the 3B quality in Figure 5 / Table 4; Table 5 / Table 6 are accompanying native probes. Hardware limits us to reporting quality from Ampere fake-quant and efficiency from 4-block runs on one 5090; we cannot report a large-scale native training measurement. The camera-ready will remove or demote the matched-loss extrapolation from the main results.

Uneven per-module speedups (2.83× vs 1.27×). This is expected once modules are split by role. One of these factors should not stand in for the full stack.

  • 2.83×: Table 5 Root, d=6144d=6144 (p.8), the full-step Full-Stack recipe under the Root optimizer; the isolated optimizer in Table 6, Root row, reaches 3.23×.
  • 1.27×: Appendix F, “Attention.” (p.16), attention kernels only, vs PyTorch SDPA (torch.nn.functional.scaled_dot_product_attention; in our measurements this uses the closed-source cuDNN attention backend).
  • LoRA-SVD MLP is 11.2–17.2% slower than TE (same page, “LoRA-SVD MLP.”). Forward is faster than TE; LoRA backward fusion on SM120 is not done yet, which hurts efficiency.
  • Q-AdamW is essentially IO-bound; the gain tracks bits/param: Table 5 AdamW is the full-recipe four-block step, Table 6 AdamW is the isolated measurement.
Reviewer_bRGvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines four low-precision training recipes: a BF16 low-rank branch alongside NVFP4 projections, transformed AdamW states, NVFP4 Newton–Schulz products for Root, and a fixed mixed-precision attention schedule. Quality experiments reach 3B parameters and 64B tokens using fake quantization on A800 GPUs. The final training loss is 2.286 versus 2.267 for BF16, with close zero-shot scores. Native speed and memory are measured separately on four decoder blocks of an RTX 5090, where the largest speed improvement comes from the quantized optimizer path.

Critical strengths
  1. Extending low precision beyond projections is a useful direction. The second-moment transformation and the direct NVFP4 Root path address real costs that projection-only recipes leave behind. The 1B ablations and 3B training run provide evidence that the proposed components can be combined over the tested token budgets without a large quality loss.

  2. The paper reports useful resource breakdowns and does not hide unfavorable comparisons. Its MLP is slower than TE, and its complete AdamW step has essentially the same runtime as TE despite lower memory use. The native measurements include LoRA gradients and QR processing, and the periodic SVD cost is reported separately. These details help distinguish the contributions to the overall result.

Critical weaknesses
  1. The quality and resource results come from different implementations and settings. The 3B/64B-token run uses fake quantization with stochastic rounding, while native RTX 5090 timing disables stochastic rounding and uses four blocks with different widths and sequence length. The measured native recipe is not shown to converge comparably. FP32-master-weight references and baseline-optimizer descriptions also differ between sections and appendices. Consequently, the estimated matched-loss savings join results that have not been validated together.

  2. The marginal benefit beyond optimized projection-only quantization is unclear. The BF16 low-rank branch uses different activation views from the dense branch, and training backward is not fully fused; the measured MLP is 11.2–17.2% slower than TE. Attention speedup is against an unspecified SDPA backend rather than a matched optimized implementation. There is no native complete-step progression isolating each added component. This matters because AdamW runtime is approximately tied with TE, while the large speed gain is concentrated in Root. Four-block SVD measurements also leave distributed training costs unresolved.

Questions for authors
  1. Can quality, step time, and peak memory be validated under the same native rounding, model shape, and optimizer settings? Please reconcile the master-weight and baseline descriptions and provide component ablations against optimized projection-only training.

  2. Which low-rank and attention paths are fused today, and what activation traffic and conversions remain? Please identify the attention backend and report full-step distributed SVD costs, including communication, workspace, and factor-state realignment.

4 comments
AuthorsOpenReview ↗

Reponse to Reviewer bRGv (part 1 of 3)

Thank you for the close reading, and for noting that we did not hide that the MLP is currently slower than TE, which in practice makes the AdamW full step on par with TE (Appendix F Table 6 and Table 5). We answer the questions first, then add one design point the questions do not ask.

Q1a. Can quality, step time, and peak memory be validated together under the same native rounding, model shape, and optimizer setting?

Not at present. Quality is measured with fake quantization on Ampere (§4.1; 3B shapes in Table 7), with stochastic rounding (SR). Time / memory are measured on a single 5090 with four blocks (§4.4 / Table 5), SR off, d=4096/6144d=4096/6144, L=8192L=8192. The three quantities do not appear together in one native training run. Scale-up training on 5090 is not realistic for us yet; Ampere also does not support NVFP4.

SR is a commonly used technique, we turn SR off because this SM120 path has no efficient hardware stochastic rounding; a software implementation is much slower. TE and Ours both disable it, so the efficiency comparison is not a difference in rounding implementations. The 5090 runs use a wider dd to stress efficiency, not to match the 3B quality setup. The measurement config does not change the relative comparison to TE and torch.compile, but joining the two sides into matched-loss savings makes it easy to think they were validated together. We agree the wording was misleading. The camera-ready will demote or remove that paragraph from the main results.

Operator error vs BF16 is at the same level as the Rel. MSE in Figures 2–4. On 4-block forward–backward with the Table 5 shapes, relative error between fake-quant and native operators is about 5%. That is an operator-level check, not a proof of native convergence.

Q1b. Please reconcile the master-weight and baseline descriptions

The FP32 master-weight wording is a mistake; the main text and the appendix were inconsistent. The actual process is: BF16 base weights and LoRA are assembled into an equivalent dense FP32 matrix, randomized SVD is run in FP32, then the result is written back to the base weights and LoRA, casting to the original dtype, i.e. BF16. We will fix this in the camera-ready.

The optimizer baselines look inconsistent because they belong to different experiments, and each uses the setting that was actually run:

  • Paragraph after Table 5 (p.8): full-step BF16 / TE use Muon. Root costs more than Muon through specialized coefficients and outlier clipping; using Muon as the efficiency baseline does not bias the comparison in our favor. The recipe also relies on those two Root changes for quantization, so the full-step efficiency comparison uses the lighter Muon.
  • Appendix F, “Root.” (p.16): this isolates NVFP4 Root itself and compares to GRASP.
  • Fake-quant quality: the bottom block of Table 3 and Table 8 both use Root.

Q1c. How do the components break down efficiency gain relative to optimized projection-only?

Appendix F Table 6 (p.16) and the four paragraphs after it are isolated module measurements. Full-recipe four-block steps are in Table 5 (p.8).

  • LoRA-SVD MLP (the cleanest projection-only comparison): 17.2% / 11.2% slower than TE, 2.87× / 3.30× faster than torch.compile BF16. Forward is actually slightly faster than TE; we keep fewer resident activations than TE. Backward has not yet fused the LoRA branch well on SM120, which slows the whole module. This is currently the main efficiency gap. We acknowledge it, and it is limited by our kernel-engineering experience.
  • Attention: 1.27× faster than BF16 SDPA, 51.1% / 52.9% lower incremental memory. SDPA is a strong, practical attention implementation in PyTorch; details in Q2c.
  • Root: 3.14× / 3.23× vs BF16, 4.32× / 4.07× vs GRASP.
  • AdamW isolated: 1.40× / 1.41× faster, −56.0% / −56.1% memory.
  • The AdamW row in Table 5 is the full recipe: time essentially matched to TE, memory −37.9% / −42.5%. The matched time is mainly because LoRA-SVD MLP and attention have not yet realized the isolated gains in the full step, not because AdamW itself has no gain.
  • The Root row in Table 5 is the full recipe: 2.50× / 2.83× vs BF16, 1.83× / 2.17× vs TE, memory −21.5% / −23.1%. Isolated Root is faster (3.14× / 3.23× vs BF16). The complete-step number is smaller because LoRA-SVD MLP and attention still occupy part of the step and have not yet matched TE; unlike AdamW, Root NS dominates the step, so the full recipe still clearly beats TE.
  • We do not give a stacked complete-step breakdown of the form “projection-only (TE), then add the other modules one by one.” How much each module moves the full step depends on: which optimizer is used, how large attention is relative to MLP, hidden size, and input shape. Isolated measurements already show each module’s own speed and memory gain, independent of that mix; Table 5 then shows them used together. We think that is enough for a validation since we can only afford 4 block simulation on a single 5090.
AuthorsOpenReview ↗

Response to Reviewer bRGV (part 2 of 3)

Q2a. Which low-rank and attention paths are fused today?

There are kernels for both forward and backward. The LoRA-SVD training backward has not yet fused the LoRA kernel well on SM120, mainly because on-chip memory is tight. We do not claim this is a fully optimized implementation; we report what we can currently do on 5090 SM120. It should not be read as ideal efficiency, or as efficiency on Hopper SM100.

Q2b. SVD cost? Distributed training?

SVD is not done every step. The period, per-call cost, amortization, and extra memory are in Appendix F, “Periodic fast-SVD.” (p.16). The full step does not include this cost; it is reported separately.

  1. Period. §3.2, last paragraph, Periodic realignment (p.4): after a two-period warmup, randomized SVD runs every 2,048 steps; the first call is at step 4,096. Appendix G Table 8 (p.17): SVD interval / warmup factor = 2,048 / 2. The first paragraph of §4.4 (p.7) also says periodic fast-SVD is reported separately.
  2. Four-block cost, same measurement as Table 5. One realignment at d=4096/6144d=4096/6144 is 160.00 / 226.96 ms; amortized 0.078 / 0.111 ms per step, which is 0.026% / 0.018% of the Ours full step in Table 5 (303.84 / 618.87 ms); extra peak allocation 0.031 / 0.039 GiB (about 0.3%). So in a full step, time and workspace are negligible; that is why Table 5 does not fold them into the per-step ms.
  3. Distributed. With distributed Adam, BF16 parameters must be all-gathered before the alignment. This is not extra communication from periodic SVD: the optimizer already all-gathers to sync parameters under DDP, so we do not count it as an incremental cost of our method. Distributed training will slow the full step through communication and shrink the speedup vs the baseline; the isolated-module gains are still there, and other methods pay communication as well.

Q2c. What is SDPA? Which attention backend?

SDPA is torch.nn.functional.scaled_dot_product_attention, the standard PyTorch 2.0+ interface. It picks the best backend automatically. In our measurements it goes to closed-source cuDNN attention, which is faster than the FlashAttention-2 backend, so we use it as a strong baseline. Our implementation mainly follows the FlashAttention-2 open source and adapts it to the NVFP4 recipe. We had used an informal internal abbreviation; the full name is given here.

AuthorsOpenReview ↗

Response to Reviewer bRGV (part 3 of 3)

Additional point not covered directly by the questions

Why the low-rank branch and the dense branch do not share the same activation view. This is by design, not an omission. §3.2 equations (p.3–4): the NVFP4 GEMM uses Q(X)Q(Wres)Q(X)Q(W_{\mathrm{res}}); the subspace-constrained low-rank branch is BF16 (XL1⊤)L2⊤(XL_1^\top)L_2^\top. The low-rank ops read high-precision XX or dY\mathrm{d}Y, and recover accuracy with relatively low cost and this mechanism. The extra 16-bit traffic adds some overhead, but it is not the main slowdown now; forward can be faster than TE in our measurements. The difficulty is limited on-chip memory on SM120, so LoRA backward is not fully fused yet.

Thank you for the clarifications and additional experiments. The master-weight and attention-backend descriptions are clearer, but my main concerns remain unresolved: quality and efficiency are still evaluated with different rounding settings and implementations, without native training validation connecting them, and the marginal full-step benefits beyond projection-only quantization remain unclear. Narrowing the claims is appropriate, but does not fill these evidence gaps. I therefore maintain my rating of 2 (Weak rejection).

Reviewer_47csOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends NVFP4 pretraining beyond linear projections to optimizer states and computations, as well as attention, with four recipes: LoRA-SVD combines an NVFP4 residual weight with a BF16 low-rank subspace; Q-AdamW stores both Adam moments in NVFP4; Q-Root performs the Root optimizer's Newton–Schulz products in NVFP4; and Q-Attn uses NVFP4 for QK⊤QK^\top and dSdS products while retaining BF16 for PVPV, P⊤dOP^\top dO, and dOV⊤dO V^\top. In a 1B-parameter, 20B-token run, LoRA-SVD reduces the loss gap from 1.40% to 0.61%. With all four recipes, a 3B-parameter, 64B-token run using fake quantization on A800 GPUs reaches a 0.838% gap to BF16 with similar zero-shot scores. On four decoder blocks on one RTX 5090, the full step is 2.50–2.83× faster than BF16 with Root, and AdamW peak memory falls by 37.9–42.5%.

Critical strengths

S1. Broad coverage of the training stack. Most FP4 training work stops at linear layers. This paper studies optimizer states, Newton–Schulz iterations, and attention in one system, and shows that the recipes can be enabled one by one and combined. The 3B/64B run stays close to BF16 in both loss and zero-shot scores. To my knowledge, stable NVFP4 storage of both Adam moments in pretraining has not been shown before.

S2. Useful component-level analysis. The second-moment pipeline is well ablated (reconstruction error 0.0116 → 0.0053; removing any stage causes divergence at 1B). The attention study (Fig. 4) gives a clear reason for keeping PVPV and dOV⊤dOV^\top in BF16. The systems comparison uses strong baselines (max-autotune torch.compile BF16 and Transformer Engine 2.17).

Critical weaknesses

W1. Quality and efficiency are measured on different systems. Quality comes from fake quantization on A800 GPUs, with stochastic rounding (SR) in backward. Speed and memory come from a four-block microbenchmark on a consumer RTX 5090 with SR turned off. No native end-to-end NVFP4 training run exists, and the "matched-loss savings" of 56.9–61.9% are extrapolated from these two separate setups. The headline 2.5–2.8× speedup also depends on the optimizer taking a large share of a single-GPU step (BF16 Root step 760 ms vs. AdamW 406 ms). In distributed training with a sharded optimizer, this share is much smaller, so the end-to-end gain is likely far lower. Finally, the Root comparison runs Q-Root against BF16/TE Muon, which is a different optimizer.

W2. The ablations are hard to interpret and lack noise estimates. All runs use one seed, and the differences in Table 3 are 0.1–0.5%. Some rows are surprising. Adding Q-Attn to NVFP4 4/6 lowers the gap from 1.40% to 0.47%, which is smaller than the gap of LoRA-SVD (0.61%). Full-Stack w/o Root (0.56%) is better than 4/6 + LoRA-SVD alone. Adding more quantization should not reduce loss, so these numbers are likely within run-to-run noise. If so, the per-recipe claims in Table 3 are not supported. There is also no isolated quality result for Q-Root.

Questions for authors
  1. What is the seed-to-seed loss variance at 1B/20B? How should we read the rows in Table 3 where adding quantization lowers loss (e.g., 4/6 + Q-Attn)?
  2. Is Q-Root's quality comparable to BF16 Root alone, without the other recipes? Why does the systems baseline use Muon rather than Root?
  3. How does Q-AdamW compare with existing low-bit optimizers (8-bit Adam, 4-bit Adam (Li et al., 2023), FP8 states in COAT) in loss and memory?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 47cs (part 1 of 2)

Thank you for pointing out coverage, the second-moment ablation, and the attention precision split. We answer your questions first, then add one point that the questions do not ask directly.

Q1. How many seeds were run? How should we read the ablation gaps?

Table 3 in the submission uses seed 42. We added seed 44 under the same 1B / 20B setup. There is no third seed and no standard deviation; training cost limited this.

Settingseed 42 (submission Table 3, p.6)seed 44
NVFP4 4/6+1.40%+1.57%
4/6 + LoRA-SVD+0.61%+0.66%
4/6 + Q-AdamW+0.98%+1.25%
4/6 + Q-Attn+0.47%+0.83%
Full-Stack w/o Root+0.56%+0.79%

We do not rank modules by quality on sub-1% row-to-row gaps, especially the counter-intuitive ones, and we do not describe them as “quantization made it better.” In training we see that quantizing the optimizer and attention slightly increases grad-norm fluctuation. Below 1%, that can change how far a step travels, or change the trajectory directly. The intent of Table 3 is the last sentence of §4.2 (p.6): recipes can be enabled independently and composed. The point is composability and independent on/off switches, not ranking modules with these small deltas.

LoRA-SVD reduces the linear-only gap from 1.40% / 1.57% to 0.61% / 0.66%, in the same direction on both seeds. That is the quality result we keep. On seed 44, 4/6+Q-Attn is +0.83% and Full-Stack w/o Root is +0.79%; neither is necessarily better than adding LoRA-SVD alone.

Q2a. What is the isolated Q-Root quality?

The bottom block of Table 3 in the submission only has BF16+Root vs Full-Stack (+0.62%). We added isolated BF16+Q-Root, which is +0.34%. Switching in Q-Root alone has a small effect; after stacking the full stack the gap is still +0.62%. The added increment is small, and the pieces can be used together.

Q2b. Why does the systems baseline use Muon rather than Root?

All quality runs use Root+AdamW: the bottom block of Table 3, and Appendix G Table 8 (p.17). We chose Root because of two changes relative to Muon—shape-dependent (a,b,c)(a,b,c) and p99 percentile clipping. Both help Newton–Schulz iteration error and state compression, and make this path easier to quantize to NVFP4. Figure 3b (p.5) shows that Root-style coefficients reduce NVFP4 Newton–Schulz error relative to standard Muon coefficients. With those two changes we do not need mixed precision or stepwise SVD to stay stable. The original Root paper also reports slightly better training quality than Muon, so the quality baseline uses Root.

The efficiency comparison uses the cheaper Muon. Relative to Muon, Root only adds cost (specialized coefficients and clipping; the Newton–Schulz iteration count and the core matmuls stay the same), and our recipe uses those two Root changes for quantization. The paragraph after Table 5 (p.8) uses Muon for the full-step BF16 / TE baselines: 2.50× / 2.83× is reported against the lighter baseline, and does not inflate the speedup.

Q3a. Loss of Q-AdamW vs 8-bit Adam / 4-bit Adam (Li et al., 2023) / COAT FP8 states?

On the same 4/6 setting as Table 3, we replace Adam states with these low-bit implementations. Loss deltas relative to BF16:

  • Ours NVFP4 Q-AdamW: +0.98%
  • 8-bit Adam: +0.59%
  • The COAT FP8 optimizer kernel does not run on Ampere or on 5090. We reproduced its power scaling (Dynamic Range Expansion) with fake quantization following the paper; training diverged, and we have not yet found the cause. FP8 states without power scaling give +0.42%.
  • 4-bit Adam: +2.1%

Q3b. Memory of the three above?

The second-to-last paragraph of §3.3 (p.5): the second moment is 5.0625 bits/element; the master stays BF16, with no FP32 master. Appendix F, “AdamW.” (p.16): the BF16 baseline is a BF16 weight, a BF16 gradient, and two FP32 moments, 96 bits/param; Ours is 42.0625 bits/param. From the implementations: 8-bit Adam is about 48.25, COAT FP8 states about 48.25, 4-bit Adam 40.5.

Isolated AdamW, same accounting: Ours is 1.40× faster than BF16 with 56.0% lower peak allocation. 8-bit Adam is 1.40× faster with 49.3% lower peak allocation. 4-bit Adam is 1.37× faster with 57.8% lower peak allocation. COAT’s open-source Adam uses a CUDA architecture that 5090 does not support, and we have not reimplemented it yet; given the state bitwidth, we expect it to be close to 8-bit Adam.

AuthorsOpenReview ↗

Response to Reviewer 47cs (part 2 of 2)

Additional point not covered directly by the questions

Native end-to-end NVFP4 training, and native efficiency under distributed training. Scale-up training on 5090 is not feasible for us now. Quality runs are fake-quant DDP on Ampere only, so we cannot report distributed native efficiency.

Operator development has always used the fake-quant path and BF16 as references. Per-component error vs BF16 is at the same level as the Rel. MSE in Figure 2, Figure 3, and Figure 4. We also checked operators on a 4-block setup with the same shapes as Table 5: relative error between the fake-quant path and the native operators is about 5%. That only shows that the operators roughly match the fake-quant experiments; it is not end-to-end native training.

The 56.9–61.9% matched-loss savings are an extrapolation. The last paragraph of §4.4 (p.8) states this: a fixed-batch estimate that maps the quality gap in Figure 5 onto the single-GPU times in Table 5. §6 Limitations (p.9) says again that this is a matched-loss time estimate from combining speedup with the 3B loss curves, not an end-to-end system metric. The efficiency numbers are there to show that, while holding quality, the design still buys substantial efficiency against strong TE and torch.compile baselines. In the camera-ready we will remove or demote the matched-loss paragraph from the main results, so it is not read as a measured end-to-end number.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

The three reviews mostly ask the same things: quality and speed were not measured in one run; two reviewers asked about counter-intuitive gaps in Table 3; the systems table mixes Root and Muon; and there is no native end-to-end training.

That is fair. We use fake-quant on A800 to check whether the recipes train (Figure 5, Table 4), and four-block runs on one RTX 5090 to check native speed and memory (Table 5, Table 6). These are separate checks of the method across modules. We do not claim they add up to end-to-end native training, and we cannot provide that result. The appendix line about an FP32 master is a writing mistake of FP32 SVD operating and BF16 storage: masters always stay BF16.

What we added: a second 1B/20B seed (44); isolated Q-Root (+0.34%); the same 4/6 setup against 8-bit Adam, 4-bit Adam, and COAT-style FP8 states; and a 4-block check that fake-quant and native operators differ by about 5%. LoRA-SVD still cuts the linear-only gap from about 1.4–1.6% to about 0.6% on both seeds. The counter-intuitive Table 3 rows (loss looking better after extra quantization) are small, we believe they are likely noise, or mild trajectory drift / grad-norm fluctuation from quantizing the optimizer and attention. Seed 44 also shows that extra quantization does not reliably “improve quality.” We are not claiming that some modules raise the score. The claim is that they can be used together and still train stably.

Our claim is limited to what we actually ran. Within Table 1 this is the first NVFP4 recipe that covers projections, Adam states, Newton–Schulz, and trainable attention, and it still trains close to BF16 (3B / 64B: 0.838%, similar zero-shot). Efficiency is vs optimized BF16 and TE 2.17, including the numbers that look bad for us: MLP is slower than TE, and the AdamW full step matches TE. The Root path is 2.50–2.83× vs optimized BF16; AdamW peak memory is 37.9–42.5% lower. Larger native / distributed training is still missing. For the early pretraining we have (about 1B–3B), these recipes can be used together, stay close to BF16, and cut compute and memory. We think that already shows the value of the work; we hope the missing native end-to-end run is not treated as a reason to deny the contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.