acceptodds
Under review as a conference paper at ICLR 2027

Analysis of MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible error

Abstract

MXFP4 arithmetic can dramatically accelerate reinforcement learning (RL) post-training of large language models (LLMs), yet the quantization error introduces severe accuracy degradation. Existing work treats the quantization error as a monolithic noise term, missing the distinct mechanisms upon interpreting how quantization error damages training. We prove that this error is not monolithic but admits an exact, privileged decomposition into three additive components: scale bias from power-of-two rounding, deadzone truncation from zeroing small values, and grid noise from rounding to the 4-bit grid. The deadzone component is exactly orthogonal to the other two, and, crucially, the grid component is invariant to scale precision: refining the block scale reduces scale bias without touching grid noise, driving the total error toward an irreducible floor set by the grid and deadzone alone. Each component dominates a distinct RL failure mode: scale bias accumulates multiplicatively through the backward pass, affecting gradient accuracy; deadzone truncation degrades rollout quality; and grid noise raises the policy's entropy. Our contribution is this diagnostic framework rather than a new correction: to validate it, we use three existing, mechanism-targeted techniques as controlled interventions (Macro-block scaling for scale bias, Outlier Fallback for deadzone, and Adaptive Quantization Noise for grid noise/entropy), each removing one component and recovering its predicted RL pathway. On Qwen2.5-3B (dense) this recovers BF16 accuracy to within 0.7 pp, on Qwen3-30B-A3B-Base (MoE) to parity, and on Llama-3.1-8B (a different family) to within 3.2 pp. The same component-to-pathway mapping holds across all three, and the results generalize from GSM8K to MATH500.

Primary area:
reinforcement learning
Keywords:
quantization error, low-precision training, large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sk6eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper splits MXFP4 quantization error into three parts: scale error, deadzone error and grid error. It argues that these mainly hurt gradients, rollout quality and exploration, respectively. To test this, it applies three existing techniques, one per component. Experiments use simulated W4A4 quantization during RL training on Qwen and Llama models. The combined recipe recovers most of the GSM8K drop, while transfer and long-response results still show gaps.

Critical strengths

1.Breaking the error into parts is a useful way to diagnose low-bit RL training. The additive identity is clear, and the deadzone orthogonality holds under the stated ceiling-scale rule. Measurements on three architectures give a good empirical picture of how large each part is.

2.he recovery in training quality is substantial. On Qwen2.5-3B, the combined recipe brings GSM8K from 60.1% to 81.3%, close to BF16 at 82.0% (Table 3). The Llama study uses three seeds. I also appreciate that the paper reports the harder settings where recovery fails, and says clearly that no real-hardware speedup was measured.

3.The backward-only experiment is a useful attempt to test the explanation directly. Keeping the forward and rollout quantization settings fixed, changing only the backward scale raises GSM8K accuracy from 37.7% to 72.8% on Llama (Section 4.1 and Appendix H). It is only one seed and does not prove the proposed gradient formula, but it gives a concrete reason to investigate backward quantization separately. That is more informative than only reporting the final combined recipe.

Critical weaknesses

1.The mapping from error parts to training failures is not yet justified.

  • Section 4.1 treats a change of block scale as multiplying the layer output by s/s*. But changing the scale also changes which grid point each element rounds to, so the effect is not a simple multiplication.
  • The gradient argument multiplies scale factors across layers, but a Transformer has normalization and residual connections in the backward pass. These can break the product form, and the paper doesn't address this.
  • Appendix D.3, Eq. (23), gives a zero-mean Gaussian for a sum of Uniform[0,1) variables without subtracting their means (each has mean 1/2). The log base in Eq. (4) also needs to be stated consistently. The experiments show the recipe works. They don't show it works for the reasons given. The limitations section admits part of this, but the abstract and conclusion still make strong causal claims.

2.Some of the experimental conclusions are more certain than the evidence allows. The Qwen results use one seed, so the small remaining gaps to BF16 do not establish reliable parity. Table 4 also says seed noise cannot change the ranking against earlier methods because the full recipe has a small standard deviation. But those competing methods were run only once; the full recipe's variance tells us nothing about theirs. The large recovery over naive MXFP4 is convincing, but claims about near-BF16 performance and the exact ranking need repeated runs of the relevant baselines.

Questions for authors

1.Is the "irreducible floor" meant only as the limit when the scale goes to max|x|/6? If it is meant as a lower bound, how do you reconcile it with the 32-element example above? 2.Can you redo the gradient derivation with rounding, normalization and residual paths included, and fix the centering in Eq. (23) and the log convention in Eq. (4)? 3.Do you have interventions that change only the deadzone error, or only the grid error? If not, which claims could be narrowed to "the combined recipe helps" rather than a per-component causal story?

3 comments

We thank the reviewer for the valuable comments and suggestions. We intend to keep our response short but exact. We report new measurements (App. J of the revised PDF), including where they refute our claims.

Q1. The floor. Yes. Per Eq. 3, the floor is only the limit as the scale goes to the ideal s∗=max⁡∣x∣/6s^*=\max|x|/6 (E8M∞\mathrm{E8M}\infty). "Irreducible" means scale refinement converges to it and cannot remove it, since the E2M1 grid is fixed. OF can reduce reconstruction error below this limit by quantizing and adding back a residual in a second pass. Eq. 3 describes scale refinement for a single-pass E2M1 quantizer.

Q2. Gradient derivation. You are right, and the issue is substantive. Setup: Qwen2.5-3B, 64 held-out prompts, real backward through all 36 layers incl. norm and residual paths (App. J, Figs. 12–14).

  • Rounding: Qs/Q∗=γ⋅rQ_s/Q^*=\gamma\cdot r, where rr is the rounding-index ratio. Index shifts cancel the scale gain per element: E[γ]=1.46\mathbb{E}[\gamma]=1.46, but E[γr]=1.01\mathbb{E}[\gamma r]=1.01.
  • Propagation: the E8M0/ideal ratio of first-layer input-gradient norms has median 0.95 (P5–P95: 0.50–1.36). The log-ratio is uncorrelated with the predicted cumulative scale sum (Pearson r=0.06r=0.06).
  • What still holds: scale error degrades gradient accuracy. Removing it reduces weight-gradient NMSE by 55.6% (CI [38, 72]%), consistent with the backward-only Q∗Q^* RL result (37.7% → 72.8%).

Revision: We replace Eq. 22 with the exact blockwise expansion: Qs(Z)=Γ⊙Q∗(Z)+s⊙(ks−k∗)Q_s(Z)=\Gamma\odot Q^*(Z) + s \odot (k_s-k^{*}), i.e., a scale term plus an index-correction term, composed through the actual layer Jacobians (I+JFI+J_F for residual blocks). The scalar product is only a conditional approximation, and our data violate its condition. We drop the 0.18×0.18\times–5.6×5.6\times claim.

Eq. 23: we fix it to ∑l(δl−12)≈N(0,L/12)\sum_l(\delta_l-\tfrac{1}{2})\approx\mathcal{N}(0,L/12), matching the main text at L259. Eq. 4 uses log⁡2\log_2 throughout.

Q3. Single-component interventions. We switched each component of Def. 3.1 independently (all 8 combinations, App. J, Table 11) on a frozen checkpoint.

Removed from MXFP4Local dX NMSEKL to BF16Greedy /64 (base 19)Entropy
Scale−31%−0.06444−0.08
Deadzone−4%≈0\approx 0 (n.s.)25 (n.s.)≈0\approx 0
Grid+72%+0.14510+0.28
  • Scale → gradient: supported. It also drives most rollout damage, so it is not pathway-specific.
  • Deadzone → rollout: The ablation did not establish deadzone error as the dominant source of rollout degradation on this checkpoint.
  • Grid → entropy: sign reverses. Grid alone raises entropy by 0.09 nats, but in real MXFP4 it lowers entropy.

We narrow the claims accordingly:

  • Kept: the exact decomposition, deadzone orthogonality, the scale-refinement limit, and scale error → gradient inaccuracy.
  • Restated at the method level: "the combined recipe recovers most of the loss."
  • Downgraded to hypotheses: deadzone → rollout and grid → exploration.

Abstract, contribution 2, Sec. 4 and the conclusion will be revised.

Statistics. We agree, and will remove the Table 4 ranking sentence. New Qwen results (3 seeds, final accuracy; App. J, Table 12):

ModelBF16Naive MXFP4Full recipe
Dense83.4±1.283.4\pm1.260.3±1.060.3\pm1.081.2±0.381.2\pm0.3
MoE91.2±0.591.2\pm0.587.4±1.187.4\pm1.191.3±0.391.3\pm0.3

We hope that our response above has addressed your concern, and we would appreciate that you could consider to raise your rating. Thanks again for your time and effort.

Thanks for your reply. Most of my concerns are resolved, and i hope the author could add the new experimant results to the final version. I will keep my positive score.

Thank you for your positive feedback and for recognizing our revisions. We appreciate your constructive comments and are glad that our responses and new experiments have addressed most of your concerns. These new results will be added to the final version.

Reviewer_mhgvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes MXFP4 quantization error in LLM reinforcement learning by decomposing it into scale bias, deadzone truncation, and grid noise. It establishes structural properties of this decomposition and proposes that the components predominantly affect gradient accuracy, rollout quality, and policy entropy, respectively. The authors use three existing techniques, Macro-block Scaling (MBS), Outlier Fallback (OF), and Adaptive Quantization Noise (AQN), as diagnostic interventions. Experiments using quantize/dequantize emulation across three models show substantial accuracy recovery, approaching or exceeding BF16 performance on the Qwen models while retaining a gap on Llama.

Critical strengths

The decomposition provides a useful framework for diagnosing quantization error. The exact additive decomposition, deadzone orthogonality, and empirical analysis across three architectures offer a more specific account of MXFP4 error than aggregate reconstruction metrics alone.

The interventions recover much of the observed accuracy loss across different architectures. The combined recipe closes the GSM8K gap to 0.7 percentage points on Qwen2.5-3B and reaches BF16-level performance on Qwen3-30B-A3B.

Critical weaknesses

The evidence does not sufficiently isolate the proposed component-to-pathway relationships, particularly for OF and AQN. OF performs residual quantization and changes several error components; Table 5 reports reductions in scale, deadzone, and grid error for MBS+OF. Its accuracy benefit therefore does not isolate deadzone recovery as the cause. Similarly, AQN injects scheduled noise into LayerNorm weights rather than directly controlling grid error, so its benefits could reflect a general exploration effect. The authors acknowledge these limitations, but they concern the paper’s central diagnostic claim.

The validation leaves efficiency and sustained training stability unresolved. All experiments use quantize/dequantize emulation, with no measured throughput or memory savings and no runtime accounting for the interventions. OF’s second residual quantization pass makes this overhead particularly relevant to the practical motivation.

The successful recovery results also rely mainly on short GSM8K runs of approximately 120 steps with a maximum response length of 1,024 tokens. The appendix adds DAPO experiments and longer-response analysis, but recovery remains partial and generation quality deteriorates with response length.

Questions for authors

How should OF’s proposed role as a deadzone-targeted probe be reconciled with Table 5, where MBS+OF reduces all three error components? What evidence attributes its rollout improvements specifically to deadzone recovery?

Could you simply extend your training to longer runs with extended context length?

4 comments

We thank the reviewer for the valuable comments and suggestions. We intend to keep our response short but exact. We ran the component-isolating test the reviewer asked for, and we narrow our claims accordingly. New results are added to the PDF in Appendix J, referred below.

1. OF and the deadzone. OF corrects the first-pass residual, which includes scale, deadzone, and grid errors. The 99.0% reduction in Table 5 measures the residual error of the two-pass OF reconstruction using an ideal-scale reference at α=0.5\alpha=0.5.

We evaluated all eight combinations of the error components on a frozen Qwen2.5-3B checkpoint with 64 held-out prompts (App. J, Table 11). The relevant comparisons are:

Forward errorGreedy correct /64
BF1644
Full MXFP419
MXFP4 without deadzone error25
MXFP4 without scale error44

Removing deadzone error improved the observed greedy accuracy, and removing scale error produced the largest improvement on this checkpoint. Adding OF improved Dense GSM8K accuracy by 17.5 pp in the RL ablation. We report this as the gain from the full OF correction.

2. AQN and grid noise. Grid error alone raises entropy by 0.09 nats. But added on top of scale and deadzone error (real MXFP4), it lowers entropy by 0.28 nats. An AQN-specificity control is inconclusive. We will present AQN as an empirically useful noise schedule (Fig. 4, Fig. 7), not as evidence of a grid→entropy pathway.

3. Efficiency. We measured OF’s inference overhead on Ascend 950DT (App. J, Table 14). Relative to MXFP8, inference took 34% longer with activation residuals and 96% longer with both activation and weight residuals. Bilateral OF uses 8 data bits per element on each operand, and its explicit expansion requires four FP4 GEMM equivalents.

4. Longer training and response length. We increased the response cap from 1,024 to 4,096 tokens and continued the MoE full-recipe run to 232 updates. GSM8K accuracy reached 91.4% at 116 updates and 92.6% at 232 updates (single run, seed 123, App. J, Table 13). Responses averaged 216 tokens over the last ten updates. On DAPO-MATH, where responses were about 795 tokens, the recipe partially recovered accuracy (App. G).

We hope that our response above has addressed your concern, and we would appreciate that you could consider to raise your rating. Thanks again for your time and effort.

Dear Reviewer mhgv,

As the discussion period is approaching its end, could you please check our responses and let us know if you have further questions? If our response has resolved your concerns, we would be grateful if you could kindly consider to raise your rating.

Thank you very much for your insightful review and your effort.

Thanks for the rebuttal and additional efforts. I may raise my score.

We are encouraged by your positive reassessment and will carefully incorporate your suggestions in the final revision. Thank you again for your thoughtful evaluation!

Reviewer_StMPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper decomposes MXFP4 quantization error into scale bias, deadzone truncation, and grid noise, and connects each to a different RL failure mode. It uses three existing techniques (MBS, OF, AQN) to study these effects during GRPO training on GSM8K, across dense and MoE models.

Critical strengths
  1. The decomposition provides a useful way to reason about different sources of quantization error. I appreciate that the paper explicitly positions the interventions as existing techniques rather than new contributions.

  2. The backward-only Q* experiment (L266-269) is a useful controlled experiment. The evaluation covers multiple architectures and model families, and the negative long-CoT results in Appendix G are informative.

Critical weaknesses
  1. The theoretical claims and proposed mechanisms are not sufficiently supported. Eq. 1 is an algebraic identity, and Lemma 3.2 largely follows from the definitions and ceiling rounding. The nearly identical statistics across models (Table 1) appear to be properties of the quantizer rather than the models. More concerningly, Sec. 4.1 assumes that the scale ratio directly multiplies layer outputs (L247, Eq. 22), which is not generally true for quantization. Fig. 3 measures scale-rounding errors rather than gradient errors. The claimed gradient amplification therefore needs stronger justification.

  2. The experimental evidence does not fully establish the proposed component-to-pathway mapping. Table 5 reports a 99% reduction in grid error with MBS+OF, despite the claimed irreducible floor (L211-215). This needs clarification, especially with alpha=0.5. The cost of dense two-pass OF is also not discussed, making the efficiency claims difficult to assess. Qwen results are single-seed, and Table 2 reports peak GSM8K test accuracy rather than final accuracy (Table 6). Finally, HiFloat4 (2607.26515) and QUADS (2607.15810) are missing from related work. Both study FP4 RL quantization failures, particularly activation errors, and HiFloat4 uses a closely related residual correction. These are important omissions given the claimed novelty and the activation bottleneck acknowledged in Appendix G.

Questions for authors
  1. Can the authors directly measure gradient errors or gradient-norm ratios to validate Eq. 4 and the proposed scale-bias mechanism?
  2. How is the 99% grid-error reduction in Table 5 obtained, and which alpha is used? How does this reconcile with the irreducible error floor?
  3. What is the effective bitwidth and GEMM overhead of OF when applied to both weights and activations? How does it compare with W4A8 or NVFP4?
  4. Can the authors clarify peak versus final results in Tables 2-3 and provide multi-seed results for the Qwen experiments?
  5. How does the proposed OF approach differ from HiFloat4's residual quantization?
3 comments

We thank the reviewer for the valuable comments and suggestions. We intend to keep our response short but exact. New measurements support several concerns, and they are uploaded to the PDF in Appendix J.

Q1. Gradients. We measured real gradients on Qwen2.5-3B (64 held-out prompts, all 36 layers, shared forward/loss, paired bootstrap; figures in App. J of the revised PDF, Figs. 12–14, Table 10).

  • Supported: scale error degrades gradient accuracy. Using the ideal scale cuts weight-gradient NMSE from 0.426 to 0.189 (−55.6%, CI [38, 72]%), consistent with the backward-only Q∗Q^* result (37.7% → 72.8%).
  • Not supported: the magnitude law of Eq. 4/22. Rounding indices shift and cancel the scale gain (E[γ]=1.46\mathbb{E}[\gamma]=1.46, E[γr]=1.01\mathbb{E}[\gamma r]=1.01). The E8M0-to-ideal-scale ratio of first-layer input-gradient norms had a median of 0.95 (5th–95th percentiles: 0.50–1.36).

We will replace Eq. 22 with an exact blockwise operator expansion (scale term + index-correction term) and restate Sec. 4.1 as gradient-direction error. Table 1's consistency across models is intended: it is a format-level property (L464–466).

Q2. Table 5 and the floor. The 99.0% reduction in Table 5 was computed by comparing sums of squared residuals using a two-pass OF reference QOF∗Q_{\mathrm{OF}}^* with ideal scales and α=0.5\alpha=0.5. It measures the residual error of the composite OF reconstruction, which differs from the single-pass grid error egride_{\mathrm{grid}} in Definition 3.1.

  • Floor: Eq. 3 is the limit of scale refinement for a single-pass E2M1 quantizer. MBS behaves as predicted (Table 5: eDZe_{\mathrm{DZ}}, egride_{\mathrm{grid}} unchanged). OF is not scale refinement: it adds a second 4-bit code per element, so going below this floor is no contradiction. We will call it the "scale-refinement limit" and drop the lower-bound wording.

Q3. OF cost. With α=0.5\alpha=0.5, (A0+αAr)(W0+αWr)=A0W0+0.5ArW0+0.5A0Wr+0.25ArWr(A_0+\alpha A_r)(W_0+\alpha W_r)=A_0W_0+0.5A_rW_0+0.5A_0W_r+0.25A_rW_r.

  • Bitwidth: 8 data bits per element per operand (8.5 including scales).
  • Arithmetic: an explicit expansion needs 4 FP4 GEMMs, i.e. 4×4\times W4A4.
  • Measured inference time (Ascend 950DT, Dense-32B), relative to MXFP8: +34% with activation residual only, +96% with activation and weight residuals (App. J, Table 14).

We claim no efficiency advantage over W4A8 or NVFP4, and we have no matched timings for them. OF is an analysis probe in this paper.

Q4. Peak vs final, multiple seeds. The Table 2–3 full-recipe numbers are peaks (MoE 92.49 peak / 91.58 final). We will report final accuracy as primary. Seeds 42, 123, 2026, final accuracy, mean ±\pm SD (App. J, Table 12):

BF16Naive MXFP4Full recipe
Dense83.4±1.283.4\pm1.260.3±1.060.3\pm1.081.2±0.381.2\pm0.3
MoE91.2±0.591.2\pm0.587.4±1.187.4\pm1.191.3±0.391.3\pm0.3

Recovery over naive MXFP4 is robust. Some configurations differ across runs: in particular, MoE arms differ in attention-quantization coverage, and with attention unquantized, naive MoE reaches 90–92%. MoE "parity" is therefore descriptive only, not a claim of equivalence.

Q5. HiFloat4 and QUADS. We will add both. We do not claim residual quantization as ours: Sec. 5 credits OF and related methods. With the same quantizer, activation-only OF at α=1\alpha=1 equals HiF4's dense residual; ResQ adds 2:4 sparsity, and α\alpha blends the residual.

Single-seed Llama-3.1-8B results (final / peak): HiF4+ResQ 83.2 / 84.8; QUADS-style Dense adaptation 79.1 / 81.4 (App. J). The HiF4 + Rollout-ResQ run uses HiFloat4 quantize–dequantize emulation, while our recipe uses MXFP4. The runs therefore differ in both quantization format and residual correction.

We hope that our response above has addressed your concern, and we would appreciate that you could consider to raise your rating. Thanks again for your time and effort.

Dear Reviewer StMP,

As the discussion period is approaching its end, could you please check our responses and let us know if you have further questions? If our response has resolved your concerns, we would be grateful if you could kindly consider to raise your rating.

Thank you very much for your insightful review and your effort.

Dear Reviewer StMP,

As the discussion period is closing within hours, we would like to send another kind reminder. We would greatly appreciate it if you could take a moment to check whether our responses have addressed your concerns. If yes, we would be deeply grateful that you could consider to raise your rating.

Thank you again for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.