acceptodds
Under review as a conference paper at ICLR 2027

How Far Can MXFP4 Go in Pretraining?

Abstract

While pretraining large language models in 4-bit floating point (FP4) can substantially reduce training cost, FP4 formats struggle to cover the dynamic range of the weights, activations, and gradients encountered during training. Popular FP4 formats therefore pair each small block of values with a shared scale. MXFP4 uses power-of-two block scales that are simpler to support in hardware, while NVFP4 spends mantissa bits on its local scale to reduce quantization error. The two formats thus trade hardware simplicity against scale precision, and recent work reports larger pretraining degradation for MXFP4 than for NVFP4. In this work, we propose a MXFP4 pretraining recipe, MXFP4-MARS, based on recent advanced techniques, including Hadamard rotation, stochastic rounding, and Overflow-Aware Scaling (OAS). Our recipe also leverages Macro Block Scaling (MBS), which maintains higher-precision scale at a coarser granularity while keeping hardware-friendly power-of-two per-block scales. At matched perplexity, our MXFP4 recipe closes the token-overhead gap to NVFP4 from the previously reported 36% down to 0.21% on average, a 171 reduction in additional tokens required. Also, at the same training budget, a 7B model trained using MXFP4-MARS yields a 1.4% higher average downstream score than the model trained with NVFP4. These results identify scale hierarchy as a key design dimension for low-precision training and show that MXFP4 can be quality-competitive with NVFP4 while keeping the hardware-friendly power-of-two scales, making it an attractive, portable target for FP4 pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model pretraining, low-precision training, FP4 quantization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_K2xmOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work provides a systematic evaluation of key factors in MXFP4 training, including scaling factor computation, stochastic rounding (SR), and Hadamard transformation (HT). It also adopts Macro Block Scaling (MBS), which is proposed in previous work. MBS maintains higher-precision scale at a coarser granularity beyond the MXFP4 micro-block. The best among the evaluated MXFP4 training recipes reaches performance comparable to NVFP4 training.

Critical strengths
  1. The paper provides systematic training results at a large scale. The experiment is done on training LLMs of large size (up to 7B) with at most 640B tokens.

  2. The presentation is clear, and the paper is easy to follow.

Critical weaknesses
  1. My main concern is that the MXFP4-OCP baseline is insufficiently justified, and the backward quantization implementation in the evaluated methods is unclear, which undermines the claimed contributions. In particular, an important issue about the backward pass is unclear in this paper: for the quantization of activation XX and weight WW in the backward pass, is the quantizer input aligned with the quantized-dequantized tensor from the forward pass, X^=Qfwd(X),W^=Qfwd(W)\hat X = Q_{fwd}(X), \hat W= Q_{fwd}(W)? Or just quantize the high-precision tensors X,WX, W in the backward pass?
    This alignment of forward and backward tensors (X^,W^\hat X, \hat W) has been established as an important issue in MXFP4 training, and Quartet [1] and TetraJet [2] are MXFP4 training recipes that ensure this alignment. The comparison against a forward-backward-aligned baseline is valuable for identifying the benefits of MBS, without requiring the full set of their additional proposed techniques.

  2. The methodological novelty is limited. OAS and MBS were already introduced in [3], while SR and HT are established FP4 training techniques. The authors should clarify what is new beyond composing these components, and clarify the additional insights beyond the prior work.

[1] Quartet: Native FP4 Training Can Be Optimal for Large Language Models, NeurIPS 2025
[2] Oscillation-Reduced MXFP4 Training for Vision Transformers, ICML 2025
[3] Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction, ICML 2026

Questions for authors

Please clarify Weaknesses 1 and 2. These are my main concerns affecting my rating. I have no additional questions currently.

5 comments

Thank you for your comments and for recognizing our large-scale training results. Below, we present additional experiments on forward-backward alignment and clarify the contributions and practical value of our work.

W1: Forward-backward alignment

A1: Thank you for raising this point. In the paper, backward quantization in all evaluated recipes takes the high-precision X and W as inputs, and we have clarified this in Section 4.2 of the revised paper. Following your suggestion, we implemented forward-backward alignment as in Quartet and TetraJet, where backward quantization takes the forward-quantized X̂ and Ŵ as inputs. The table reports the final loss of 300M models trained for 30B tokens, with and without alignment.

Configurationw/o alignmentw/ alignmentChange
MXFP4-OCP2.24872.2325−0.72%
MXFP4-MARS2.20252.1881−0.65%
NVFP42.18212.1728−0.43%

We find that alignment benefits FP4 training, reducing the loss of all three configurations by 0.43%-0.72%. This benefit does not change the conclusions of the paper: with alignment applied to both, MXFP4-MARS still reduces the loss by 1.99% relative to MXFP4-OCP, compared with 2.05% without alignment. Alignment also benefits MXFP4 (0.65%-0.72%) more than NVFP4 (0.43%), narrowing the loss gap between MXFP4-MARS and NVFP4 from 0.93% to 0.70%. Overall, our recipe remains effective with alignment: MXFP4-MARS closes about 75% of the loss gap between MXFP4-OCP and NVFP4 (from 2.75% to 0.70%), and the two are complementary.

W2: Contributions beyond existing techniques

A2: We agree that OAS, MBS, stochastic rounding, and Hadamard rotation build on prior work. Beyond composing them, our work provides the following insights for pretraining:

  1. Scale precision as the missing piece. To our knowledge, this is the first work to apply OAS/MBS to both the forward and backward passes of LLM pretraining. We show that smaller blocks and better power-of-two scale selection narrow the gap to NVFP4 but cannot close it, while adding a higher-precision macro-block scale with MBS closes nearly all of the remaining gap.

  2. Effectiveness of the combination in training. Our ablations show that the components are effective only under specific conditions: MBS helps only with gradient stochastic rounding, Hadamard rotation helps only for weight gradients, and 2D scaling degrades both formats. Building an effective recipe therefore requires understanding these interactions, and our ablations also rule out ineffective configurations that the community would otherwise need costly pretraining runs to test.

  3. MXFP4-NVFP4 parity at scale. To our knowledge, this is also the first work to show MXFP4 training quality comparable to NVFP4 at scale, with 7B models trained on 640B tokens.

Thanks for the additional discussion and results. My concern is only partly resolved, so I can only raise my rating from 1 to 2.

I appreciate the technical contributions to the community, but my concerns about the novelty remain. Because the core methods are from previous work, this work is a good technical report with sufficient ablation studies, but not a conference paper that brings sufficient new insights and ideas.

Thank you for raising your score and for acknowledging the technical contributions to the community. We respect your assessment and would like to clarify two insights that we believe go beyond prior work.

Prior work (Chhugani et al., 2026) introduced OAS and MBS to reduce quantization error in inference, where lower error directly translates into better quality. In pretraining, we find that this no longer holds. As shown in Figure 3(a), MBS128 lowers quantization error but increases the final loss gap relative to BF16 under round-to-nearest gradient quantization (from 6.88% to 8.54%), because it produces a larger bias in the quantized gradients that accumulates over training. Only when combined with gradient stochastic rounding does MBS128 reduce the gap (from 6.47% to 5.12%), and Appendix E analyzes how stochastic rounding removes this bias. This interaction between scale precision and gradient bias does not arise in inference, so it was not observed in prior work.

Our continued-training experiment also decomposes MXFP4's gap to NVFP4 under the same setup. Smaller blocks and OAS recover about half of the 42.42% additional tokens needed to match NVFP4 (down to 19.62%), and the remaining gap closes only when a higher-precision macro-block scale is added (down to 1.33%). This identifies scale precision as the missing piece for MXFP4 in pretraining, and shows that it can be recovered at a coarse granularity while keeping hardware-friendly power-of-two block scales.

Beyond the technical contributions you have mentioned, our work offers insights that, to the best of our knowledge, have not been reported in prior work: the counterintuitive effect of MBS under round-to-nearest gradient quantization, and the decomposition of MXFP4's pretraining gap to NVFP4.

Reviewer_hZCyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates MXFP4 pretraining for large language models. It proposes MXFP4-MARS, a pretraining recipe that combines MXFP4 blocks, Overflow-Aware Scaling, Macro Block Scaling, stochastic rounding for gradients, and Hadamard rotation. The recipe quantizes all linear layers in a 7B dense Transformer and trains for 640B tokens. The authors report that MXFP4-MARS closely tracks NVFP4, requires only 0.11%-0.32% more tokens than NVFP4 to reach matched validation perplexity, recovers 80% of the downstream gap between MXFP4-OCP and BF16, and achieves 1.4% higher average accuracy than NVFP4 across six benchmarks.

Critical strengths
  1. This paper proposes a practical FP4 pretraining recipe that combines several techniques. It also provides extensive ablations on block size, scale hierarchy, rounding mode, Hadamard placement, and 1D vs. 2D scaling. This is a useful ablation study for the FP4 pretraining community.
Critical weaknesses
  1. The core components of MXFP4-MARS are not novel. OAS comes from prior work (Chhugani et al., 2026; Yang et al., 2025), and MBS is also introduced by Chhugani et al. Stochastic rounding, Hadamard rotation, and 2D block scaling are existing techniques. The main contribution is thus an engineering recipe rather than a new quantization method. This is below the novelty bar expected at ICLR.
  2. The paper relies on fake quantization, not measuring end-to-end training latency or throughput, so the practical efficiency benefit of MXFP4-MARS remains unverified.
  3. The evaluation scope is limited. The experiments focus on dense Llama-3-style models up to 7B parameters. It is unclear whether the recipe generalizes to larger scales, such as 13B and 30+B.
Questions for authors
  1. Can the authors provide measured end-to-end latency and memory for MXFP4-MARS versus BF16, MXFP4-OCP, and NVFP4 on identical hardware?
4 comments

Thank you for your feedback and for recognizing the usefulness of our ablation study for the FP4 pretraining community. Below, we clarify our contribution, report measured end-to-end step time and peak memory, and discuss the limits of our evaluated scale.

W1: Novelty beyond existing components

A1: We agree that the individual components build on prior work. Our contribution is not a new quantization method, but an understanding of what MXFP4 needs to match NVFP4 in pretraining. Our analysis identifies scale precision as the missing piece: smaller blocks and better power-of-two scale selection narrow the gap but cannot close it, reducing the additional tokens needed to reach NVFP4 at 7B from 42.42% to 19.62%, while adding a higher-precision macro-block scale with MBS further reduces it to 1.33% (see our response to Reviewer FUFq, W3).

Our ablations also offer practical guidance on combining these components: MBS is effective only with gradient stochastic rounding, Hadamard rotation helps only for weight gradients, and 2D scaling degrades both formats. With these findings, future work can save many pretraining runs on ineffective configurations. To our knowledge, this is the first work to apply OAS/MBS to both the forward and backward passes of LLM pretraining, and the first to show MXFP4 training quality comparable to NVFP4 at scale, with 7B models trained on 640B tokens.

W2 / Q1: End-to-end latency and memory on identical hardware

A2: We have now implemented native FP4 training paths and measured end-to-end step time and peak memory on GB300 GPUs at the global batch sizes used in training. The table below reports the end-to-end step time:

Configuration300M (GBS 20)7B (GBS 384)
MXFP4-OCP334.98 ms (0.00%)2377.25 ms (0.00%)
MXFP4-16-OAS-MBS128338.43 ms (+1.03%)2749.05 ms (+15.64%)
MXFP4-MARS342.82 ms (+2.34%)2844.14 ms (+19.64%)
NVFP4354.44 ms (+5.81%)2538.18 ms (+6.77%)
NVFP4 (with Had)373.74 ms (+11.57%)2632.76 ms (+10.75%)
BF16334.20 ms (−0.23%)2702.34 ms (+13.68%)

On the 300M model, quantization rather than GEMM dominates step time, and MXFP4-MARS adds only 2.34% to MXFP4-OCP, less than NVFP4 (5.81% without and 11.57% with Hadamard rotation), which needs an extra pass over the entire tensor. On the 7B model, GEMMs dominate, and MXFP4-MARS adds 19.64% to MXFP4-OCP, compared with 10.75% for NVFP4 with Hadamard rotation. This overhead mainly comes from MBS: current Tensor Cores apply only MXFP4 block scales, not the additional macro-block scale of MBS. Applying this scale requires an extra multiplication on each partial sum, which runs on CUDA cores with about 180x lower throughput than FP4 Tensor Cores on GB300. Native Tensor Core support for this multiplication could remove most of this overhead. With MBS bringing MXFP4 close to NVFP4 in training quality, we believe this is a promising direction for future hardware. We have added the end-to-end step time measurements to Appendix D.

The table below reports the peak training memory:

Configuration300M peak memory7B peak memory
BF16118.78 GiB (1.000x)160.08 GiB (1.000x)
MXFP4-OCP118.31 GiB (0.996x)163.97 GiB (1.024x)
MXFP4-MARS118.86 GiB (1.001x)163.97 GiB (1.024x)
NVFP4118.82 GiB (1.000x)163.97 GiB (1.024x)

Compared with BF16, all FP4 configurations use similar peak memory on 300M and 2.4% more on 7B. Our implementation keeps the high-precision weights and activations needed for the backward pass and quantizes GEMM operands on the fly, so the FP4 copies add temporary buffers. Among the FP4 configurations, MXFP4-MARS has peak memory similar to MXFP4-OCP and NVFP4: all are within 0.5% of BF16 on 300M and identical on 7B.

W3: Generalization to larger models

A3: We agree that evaluating 13B and 30B+ models would further strengthen the paper. Our current evaluation covers dense Llama-3-style models up to 7B trained on 640B tokens. Training the 7B BF16 baseline took ~169 hours on 64 GB300 GPUs (~10,800 GPU-hours), or ~\175KperrunatAWSB300pricing(175K per run at AWS B300 pricing (\\16.146/GPU-hour). Assuming the same token budget and hardware efficiency, the estimated cost rises to about \325Kfor13Band325K for 13B and \\750K for 30B per run. These estimates are a lower bound. Our emulated FP4 runs with Hadamard rotation and stochastic rounding take longer than BF16, and exploring multiple recipes would multiply the cost further.

Also, we would like to mention that our training/model scale is already much larger than most prior FP4 pretraining studies. TetraJet-v2 [1] trains models up to 370M parameters on up to 212B tokens, Quartet II [2] up to 1.9B parameters on 38B tokens, Tseng et al. [3] 6.7B models on 21B tokens, and Chmiel et al. [4] a 7B model on about 200B tokens. Only NVIDIA's comparison of MXFP4 and NVFP4 [5] uses a slightly larger scale, with an 8B model trained on 1T tokens.


[1] TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control

[2] Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

[3] Training LLMs with MXFP4

[4] FP4 All the Way: Fully Quantized Training of LLMs

[5] Pretraining Large Language Models with NVFP4

Thank you for the additional discussion and results, which partially addressed my concerns, and I have therefore raised my rating from 1 to 2. However, my concerns about novelty and scalability remain.

Thank you for raising your score and for the continued engagement. We understand the remaining concern about scale, and would like to add one new result. We trained an additional 1.1B model with a similar tokens-per-parameter ratio to our 300M model, which allows us to compare MXFP4-MARS and NVFP4 across three model sizes:

ModelTokensNVFP4 PPLMXFP4-MARS PPLRelative PPL gap
300M3B15.351815.6011+1.62%
1.1B11B9.78069.8253+0.46%
7B640B5.79045.8055+0.26%

The perplexity gap between MXFP4-MARS and NVFP4 narrows, rather than grows, with scale: at a similar tokens-per-parameter ratio, it drops from 1.62% at 300M to 0.46% at 1.1B. Although longer training typically widens the gap as quantization errors accumulate, our 7B model, trained with about 9x more tokens per parameter, shows the smallest gap of 0.26%. These results suggest that MXFP4-MARS remains competitive with NVFP4 as model size increases. We will state this trend explicitly in the revision. We believe validating every training recipe variant at substantially larger scale (10B+) is beyond what most academic and industrial groups can afford, as each additional run costs hundreds of thousands of GPU-dollars. Our evaluation is already larger in scale than most prior FP4 pretraining studies. By identifying which design choices matter and which do not at the largest scale we could test, our study narrows the search space so that groups training larger models need only validate a single, well-characterized recipe.

Reviewer_FUFqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether MXFP4 can match NVFP4 in LLM pretraining when linear layers in the transformer blocks run in FP4 for forward and backward GEMMs. The proposed MXFP4-MARS combines existing techniques: 1×16 blocks, Overflow-Aware Scaling, Macro Block Scaling, stochastic rounding, and Hadamard rotation. MXFP4-MARS performs close to NVFP4 in validation perplexity, needs on average 0.21% more tokens to reach matched perplexities, and scores higher on six downstream benchmarks.

Critical strengths
  • Practical question. MXFP4 is supported by more vendors than NVFP4, so a working MXFP4 training recipe has real-world value.
  • The scale-hierarchy result is a useful insight. A coarse high-resolution scale on top of fine power-of-two scales recovers most of NVFP4's advantage.
  • The ablations give interesting findings. MBS only helps with SR, Hadamard rotation only helps WGRAD, and 2D scaling hurts both formats. The macro-block size is chosen with measured kernel throughput in mind.
Critical weaknesses
  1. OAS and MBS both come from [1], and SR and Hadamard rotation have been standard in FP4 training. The most contributions are the combination and the study of how the variants interact.
  2. NVFP4 is relatively faster. In Table 5, NVFP4 reaches 1327.6 TFLOPS on 7B shapes, while MBS128 reaches 1049.6. On Blackwell, there is no practical reason to pick MXFP4-MARS over NVFP4. The 2.6–4.5% end-to-end overhead estimate is also optimistic. It only rescales GEMM time and holds others fixed, which leaves out the Hadamard transforms, SR random number generation, the extra max pass for MBS, and re-quantization of transposed tensors. So a measured time-to-target comparison would be much more convincing.
  3. The claims may overstate the benefits. The "36% → 0.21%, 171× reduction" uses the results from NVIDIA et al, which were obtained under different models, data, token budgets, and recipes. It remains unclear the performance gap under the same settings.
  4. Baselines are incomplete. MXFP4 gets a block-size change, OAS, and a tuned MBS size, but NVFP4 only gets SR and WGRAD Hadamard. It does not get [1] or [2]. Additionally, the training data, batch size, weight decay, and even the code are not specified, which makes the results difficult to reproduce.
[1] Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction
[2] Four over six: More accurate NVFP4 quantization with adaptive block scaling
[3] Quartet II: Accurate LLM pre-training in NVFP4 by improved unbiased gradient estimation
Questions for authors

See weaknesses.

3 comments

Thank you for recognizing the practical value of our work and the insights from our analysis. We address your questions about novelty, end-to-end efficiency, the fairness of our comparison with prior results, and the strength of the NVFP4 baseline with new measurements and the missing training details below.

W1: Novelty beyond existing techniques

A1: We agree that OAS, MBS, stochastic rounding, and Hadamard rotation build on prior work. Our contribution is to adapt them to LLM pretraining and identify which design choices work and which do not. To the best of our knowledge, this is the first work to apply OAS/MBS to both the forward and backward passes of LLM pretraining, and the first to show MXFP4 training quality comparable to NVFP4 at scale, with 7B models trained on 640B tokens.

As you noted, our analysis identifies scale precision as the key factor. Smaller blocks and better power-of-two scale selection narrow the gap to NVFP4 but cannot close it: in the continued-training experiment (answer to W3), MXFP4-16-OAS reduces the additional tokens needed to reach NVFP4 from 42.42% to 19.62%, and adding MBS further reduces it to 1.33%. Equally important, our ablations show which combinations of these techniques work and which do not: MBS is effective only with gradient stochastic rounding, Hadamard rotation helps only for weight gradients, and 2D scaling degrades both formats. Since each design choice must be validated with a full pretraining run, these findings can save practitioners substantial cost in exploring the MXFP4 design space. We have revised the introduction to better highlight these contributions.

W2: End-to-end training efficiency

A2: Thank you for pointing out the limitations of the projected training overhead. We have implemented native FP4 training paths and measured end-to-end step time on GB300 GPUs at the global batch sizes used in training, as reported in the table below.

Configuration300M (GBS 20)7B (GBS 384)
MXFP4-OCP334.98 ms (0.00%)2377.25 ms (0.00%)
MXFP4-16-OAS-MBS128338.43 ms (+1.03%)2749.05 ms (+15.64%)
MXFP4-MARS342.82 ms (+2.34%)2844.14 ms (+19.64%)
NVFP4354.44 ms (+5.81%)2538.18 ms (+6.77%)
NVFP4 (with Had)373.74 ms (+11.57%)2632.76 ms (+10.75%)
BF16334.20 ms (−0.23%)2702.34 ms (+13.68%)

On the 300M model, quantization rather than GEMM dominates step time, and MXFP4-MARS adds only 2.34% to MXFP4-OCP, less than NVFP4 (5.81% without and 11.57% with Hadamard rotation), which needs an extra pass over the entire tensor. On the 7B model, GEMMs dominate, and MXFP4-MARS adds 19.64% to MXFP4-OCP, compared with 10.75% for NVFP4 with Hadamard rotation.

We agree that NVFP4 is currently faster on Blackwell GPUs. MXFP4-MARS is slower mainly because current Tensor Cores apply only MXFP4 block scales, not the additional macro-block scale of MBS. Applying this scale requires an extra multiplication on each partial sum, which runs on CUDA cores with about 180x lower throughput than FP4 Tensor Cores on GB300. Native Tensor Core support for this multiplication could remove most of this overhead. With MBS bringing MXFP4 close to NVFP4 in training quality, we believe this is a promising direction for future hardware. We have added these end-to-end measurements to Appendix D.

W3: Fairness of the 36% versus 0.21% comparison

A3: Thank you for raising this concern. We agree that the 36% result reported in [1] was obtained under a different setup. For a fair comparison using the same model and data, we continued training four MXFP4 configurations of our 7B model at the final learning rate and measured the additional tokens each needed to match NVFP4's final perplexity. Among them, we added a new baseline that uses the same scaling rule and block size of 32 as the MXFP4 recipe behind the original 36% result, which we call MXFP4-NVIDIA.

RecipeAdditional tokens
MXFP4-MARS+1.33%
MXFP4-16-OAS+19.62%
MXFP4-NVIDIA+42.42%
MXFP4-OCPN/A

Under the same setting, MXFP4-NVIDIA comes within 0.07% of NVFP4's final perplexity after 37% additional tokens, close to the 36% reported in [1], and closing the full gap takes 42.42% in total. Our enhanced MXFP4-16-OAS baseline without MBS needs 19.62%, and adding MBS reduces this to 1.33% for MXFP4-MARS. For MXFP4-OCP, additional training barely helps: after nearly 30% additional tokens, its perplexity remains 10.14% higher than NVFP4's. We have added this comparison, together with the perplexity curves during continued training, to Section 5.3 (Figure 5) of the revised paper.


[1] Pretraining Large Language Models with NVFP4

W4: NVFP4 baseline completeness and reproducibility

A4: The techniques in [1] address limitations of power-of-two scales and do not directly apply to NVFP4. OAS corrects overflow caused by rounding E8M0 scales, which is not applicable with NVFP4's E4M3 scales. MBS adds a higher-precision scale over every 128 elements, whereas NVFP4 already uses higher-precision E4M3 scales for every 16 elements.

We agree that Four Over Six [2] can improve NVFP4. As shown in the table below, applying it to activations reduces the loss of NVFP4 by 0.22%, whereas applying it to all GEMM operands slightly increases the loss (+0.02%). This gain is much smaller than those of stochastic rounding (1.16%) and Hadamard (1.22%) when each is applied alone. Four Over Six also adds overhead, as it quantizes each block twice and selects the scale with the lower error. The same scale selection can also be applied to MXFP4, but it increases the loss by 0.49% compared with OAS.

ConfigurationFinal loss (30B tokens)
BF162.1215 (0.00%)
MXFP4-OCP2.2487 (+6.00%)
MXFP4-MARS2.2025 (+3.82%)
MXFP4-MARS (OAS -> Four Over Six)2.2134 (+4.33%)
NVFP42.1821 (+2.86%)
NVFP4 + Four Over Six (activation)2.1772 (+2.63%)

Although techniques such as Four Over Six can further improve NVFP4, our contribution of substantially narrowing the gap between MXFP4 and NVFP4 still holds. Even against NVFP4 with Four Over Six, MXFP4-MARS reduces the loss gap from 3.28% for MXFP4-OCP to 1.16% at 30B tokens.

Thank you for pointing out the missing reproducibility details. We have added the training data coverage, batch size, weight decay, and optimizer settings to Appendix A.

Thanks for the detailed response and additional experiments. Most of my concerns (W2, W3, W4) have been resolved. I update my rating accordingly.

Reviewer_yoY5OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the limits of the MXFP4 low-precision format in the pre-training of large-scale models. By introducing a combinatorial optimization strategy called MXFP4-MARS, the authors successfully narrowed the training efficiency gap between MXFP4 and NVIDIA’s proprietary NVFP4 format from 36% down to 0.21%.

Critical strengths
  • This paper presents an end-to-end pre-training process for a 7B-parameter model using 640B tokens, with its effectiveness validated through comprehensive downstream evaluations and ablation studies.

  • The approach provides a systematic analysis of the design space for low-precision pre-training, identifying key mechanisms, such as the necessity of pairing block-wise scaling with stochastic rounding to eliminate gradient bias.

  • This manuscript is well-written, with a clear motivation and a rigorous Discussion section.

Critical weaknesses
  • Validation was conducted solely on a 7B-parameter dense model. It has not been demonstrated that this recipe is effective at mitigating the non-linear accumulation of quantization errors at larger scales.
Questions for authors

To control overhead, the paper opts against using Dynamic MBS. Does the current configuration reach the theoretical ceiling for MXFP4, or is there still room for further optimization?

2 comments

Thank you for your review and for recognizing our systematic analysis of the MXFP4 pretraining design space. Your comments focus on generalization beyond 7B and the remaining optimization headroom of MXFP4. Below, we explain the compute constraints on model scale and discuss where MXFP4 can still improve beyond the current recipe.

W1: Generalization beyond 7B

A1: We agree that validation beyond 7B would strengthen the paper. Within our evaluation, the 640B-token horizon allows quantization error to accumulate over a long training run, yet MXFP4-MARS closely tracks NVFP4 throughout training and reaches NVFP4's final validation perplexity with only 1.33% additional tokens. Extending this evaluation to larger models, however, requires substantial compute: training our 7B BF16 baseline on 640B tokens took ~169 hours on 64 GB300 GPUs (~10,800 GPU-hours), or ~\175KperrunatAWSB300pricing(175K per run at AWS B300 pricing (\\16.146/GPU-hour). Assuming the same token budget and hardware efficiency, training a 13B model would cost an estimated \$325K per run, and comparing multiple recipes would multiply this cost.

Also, we would like to mention that our training/model scale is already much larger than prior FP4 pretraining studies. TetraJet-v2 [1] trains models up to 370M parameters on up to 212B tokens, Quartet II [2] up to 1.9B parameters on 38B tokens, Tseng et al. [3] 6.7B models on 21B tokens, and Chmiel et al. [4] a 7B model on about 200B tokens.


[1] TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control

[2] Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

[3] Training LLMs with MXFP4

[4] FP4 All the Way: Fully Quantized Training of LLMs

Q: Further optimization headroom for MXFP4

A: Our empirical study demonstrates MXFP4's potential as a competitive alternative to NVFP4 for LLM pretraining. There is still room to improve the current recipe, for example through Dynamic MBS, selective use of higher precision, or BF16 healing. These directions are largely orthogonal to our focus on the gap between MXFP4 and NVFP4 and may improve both formats to further narrow their gaps to BF16 training.

We perform additional experiments on 300M models to explore some of these directions. Dynamic MBS reduces the loss of MXFP4-MARS by 0.62% at 3.28B tokens, but this gain does not persist at 30B tokens, suggesting that static MBS already captures most of the benefit of macro-block scaling. Keeping the first and last layers in BF16 reduces the loss by 1.27% at 30B tokens and closes about 35% of the loss gap to BF16.

However, these directions come at a computational cost: selective use of higher precision and BF16 healing reduce the fraction of FLOPs performed in FP4, and Dynamic MBS requires more expensive scale computation at every training step. Given the training costs discussed in W1, a systematic evaluation of these directions at larger scales is expensive.

Thank you again for your positive assessment of our work. As the discussion period is coming to a close, we would like to share a new result related to your concern about larger scales. We trained an additional 1.1B model with a similar tokens-per-parameter ratio to our 300M model, which allows us to compare MXFP4-MARS and NVFP4 across three model sizes:

ModelTokensNVFP4 PPLMXFP4-MARS PPLRelative PPL gap
300M3B15.351815.6011+1.62%
1.1B11B9.78069.8253+0.46%
7B640B5.79045.8055+0.26%

From 300M to 1.1B, the perplexity gap shrinks from 1.62% to 0.46%. Although longer training typically widens the gap, our 7B model, trained with about 9x more tokens per parameter, shows the smallest gap of 0.26%. These results suggest that MXFP4-MARS remains competitive with NVFP4 as models scale up, and we will highlight this trend in the revision. Extending such validation to 10B+ models would cost hundreds of thousands of GPU-dollars per run, which few academic or industrial groups can afford, while our evaluation already exceeds the scale of most prior FP4 pretraining studies. By establishing which design choices matter and which do not at the largest scale within our compute budget, our study narrows the search space, so that groups training larger models need only validate a single, well-characterized recipe.

We would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers for their constructive feedback. We have revised the paper accordingly, with all changes marked in blue. The main changes are:

  1. Same-setting token efficiency (Section 5.3, Figure 5; FUFq). We continue training four MXFP4 configurations of our 7B model beyond 640B tokens and measure the additional tokens each needs to reach the final perplexity of NVFP4. MXFP4-MARS needs only 1.33%, compared with 42.42% for the prior MXFP4 recipe under the same setup. This result replaces the original comparison in the abstract, introduction, and conclusion, which contrasted our 0.21% average overhead at matched perplexities with the 36% reported in prior work under a different training setup.

  2. Additional baselines (Section 5.1, Table 2; FUFq). We add MXFP4-NVIDIA, which follows the MXFP4 baseline of prior work, as a third FP4 baseline and MXFP8 as an 8-bit reference.

  3. Contributions (Section 1; FUFq, hZCy, K2xm). We revise the contribution list to highlight our main insights: scale precision is the missing piece for MXFP4, the recipe components are effective only under specific conditions, and this is the first work to apply MBS to both the forward and backward passes of LLM pretraining.

  4. Backward-pass quantization (Section 4.2; K2xm). We clarify that all evaluated recipes quantize the high-precision weights and activations in the backward pass.

  5. Measured end-to-end efficiency (Appendix D; FUFq, hZCy). We replace the projected overhead with measured end-to-end step time on GB300 GPUs and explain where the MBS overhead comes from.

  6. Training details (Appendix A; FUFq). We add the training data coverage, batch sizes, learning rate schedules, optimizer settings, and weight decay.

  7. Discussion (Section 6; yoY5). We clarify that our goal is to close the gap between MXFP4 and NVFP4, and that other techniques for further improving FP4 training are complementary to our recipe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.