- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents Full-Stack NVFP4, four role-aware recipes extending 4-bit pretraining beyond linear projections: LoRA-SVD for projection subspaces, NVFP4-state AdamW, NVFP4-native Newton-Schulz iterations, and selective FP4 attention. A 3B model trained for 64B tokens reaches within 0.838% of BF16 loss with matched zero-shot accuracy; microbenchmarks show 2.50-2.83x speedups for Q-Root and ~40% lower peak optimizer memory. Per the paper's coverage table, this is the first FP4 recipe addressing all four modules jointly.
Critical strengths
- The first NVFP4 recipe covering the full training stack, with end-to-end quality validation at 3B / 64B tokens -- a substantive step beyond projection-only or optimizer-only prior work.
- Well-chosen controls: matched budgets, per-module ablations, a BF16+LoRA capacity control, and native-hardware comparisons make the component-level attribution credible.
Critical weaknesses
- The efficiency evidence does not support the practical claims: quality runs use fake quantization, speed/memory numbers come from a single-GPU microbenchmark, and the 56.9-61.9% savings are extrapolations; per-module gains are highly uneven (2.83x vs. 1.27x), so the headline numbers are not representative of the full stack.
- Key ablation differences (0.5-0.6% single-run loss gaps) lack uncertainty estimates and may be within run-to-run noise.
Questions for authors
- How many seeds were run? Do the 0.61% vs. 1.12% vs. 0.56% differences exceed run-to-run variance?
- How is numerical equivalence between fake-quant and native NVFP4 kernels established (SR is disabled on SM120)? Is there any native-precision training supporting the quality conclusions?
- What is the loss gap between Q-Root and GRASP/low-bit Muon in a matched pretraining setting?
- Is the 0.838% gap converging or diverging within the 64B tokens, and how would it evolve at longer budgets?
2 comments
Reponse to Reviewer xh4Q (part 2 of 2)
Additional points not covered directly by the questions
Efficiency evidence and “practical” claims. If the claim is the 56.9–61.9% in the last paragraph of §4.4, or the unqualified 2.50–2.83× from Table 5, we agree the evidence is not enough for that reading. The main result should be the 3B quality in Figure 5 / Table 4; Table 5 / Table 6 are accompanying native probes. Hardware limits us to reporting quality from Ampere fake-quant and efficiency from 4-block runs on one 5090; we cannot report a large-scale native training measurement. The camera-ready will remove or demote the matched-loss extrapolation from the main results.
Uneven per-module speedups (2.83× vs 1.27×). This is expected once modules are split by role. One of these factors should not stand in for the full stack.
- 2.83×: Table 5 Root, (p.8), the full-step Full-Stack recipe under the Root optimizer; the isolated optimizer in Table 6, Root row, reaches 3.23×.
- 1.27×: Appendix F, “Attention.” (p.16), attention kernels only, vs PyTorch SDPA (
torch.nn.functional.scaled_dot_product_attention; in our measurements this uses the closed-source cuDNN attention backend). - LoRA-SVD MLP is 11.2–17.2% slower than TE (same page, “LoRA-SVD MLP.”). Forward is faster than TE; LoRA backward fusion on SM120 is not done yet, which hurts efficiency.
- Q-AdamW is essentially IO-bound; the gain tracks bits/param: Table 5 AdamW is the full-recipe four-block step, Table 6 AdamW is the isolated measurement.
Response to Reviewer xh4Q (part 1 of 2)
Thank you for the positive comments on full-stack coverage (Table 1, p.2), the 3B / 64B validation (Figure 5, Table 4), and the BF16+LoRA capacity control (Table 3). We answer the questions first, then add two points from the weaknesses that the questions do not ask.
Q1. How many seeds were run? Do 0.61% vs 1.12% vs 0.56% exceed run-to-run variance?
Table 3 in the submission is seed 42; we added seed 44 (same 1B / 20B setup). Because of training cost, there is no third seed and no standard deviation.
We do not rank modules on sub-1% gaps, especially 0.5–0.6%, and we do not call them quality improvements. Quantizing the optimizer and attention changes grad-norm fluctuation and couples modules, so the trajectory changes. The intent of Table 3 is the last sentence of §4.2: enabled independently and composed. We treat the remaining <1% residuals as close to BF16, with components that can be used together stably.
Q2a. How is numerical agreement between fake-quant and native NVFP4 established (SR off on SM120)?
First paragraph of §4.4 (p.7): this SM120 path has no efficient SR primitive. TE and we both disable SR in the timing runs so the efficiency comparison is not dominated by the rounding implementation. Quality runs still use SR in the backward pass (last paragraph of §2, p.2); that is not the same rounding as the 5090 timing runs.
Operator development uses the fake-quant path and BF16 as references. Per-component error vs BF16 is at the same level as the Rel. MSE in Figures 2–4. On 4-block forward–backward with the Table 5 shapes, relative error between fake-quant and native operators is also at the same level. We think operator quality should match the fake-quant experiments. Still, this is not native end-to-end training.
Q2b. Is there any native-precision training that supports the quality claims?
No. Quality comes from fake-quant pretraining on Ampere (§4.1). Efficiency comes from four-block runs on a single 5090 (§4.4 / Table 5). Scale-up native training on 5090 is not realistic now. The quality claims rest on fake-quant pretraining, plus the operator-level / 4-block numerical check in Q2a.
Q3. In a matched pretraining setting, what is the loss gap of Q-Root vs GRASP / low-bit Muon?
The open-source GRASP fake-quant implementation is too expensive; we have not been able to run this experiment. Isolated BF16+Q-Root vs BF16+Root is +0.34%. That does not replace a matched pretraining comparison to GRASP.
Q4a. Is the 0.838% converging or diverging within 64B tokens?
Figure 5 (§4.3, p.7) and the sentence after it: the curves stay close throughout, ending at 2.286 vs 2.267, a relative 0.838%. After zooming 56–64B, Full-Stack stays about 0.02 absolute loss above BF16, and both curves are still decreasing. This is a persistent offset, not last-step jitter, and there is no clear sign that it is blowing up.
Q4b. What happens with a longer budget?
The Limitations section already says we only have 3B / 64B. We do not extrapolate here, and we cannot support larger or longer runs. On the curves we have, the recipe buys efficiency at a small quality gap; at least for the early stage of LLM training, that holds.