acceptodds
Under review as a conference paper at ICLR 2027

TetraBack: Identifying and Correcting Optimization Bias in Microscaled 4-bit Training

Abstract

Microscaled 4-bit weight and activation quantization (W4A4) accelerates LLM inference, but maintaining accuracy at such low precision remains challenging. Activation quantization is a major source of this degradation, yet its effects on training dynamics remain underexplored. In this work, we show that this accuracy gap stems partly from optimization bias in the backward approximation, rather than solely from the limitations of low-bit representation. We consistently observe sustained growth in activation magnitudes and pre-FFN RMSNorm weights during standard MXFP4 W4A4 quantization-aware training (QAT), a pattern that impairs optimization and is absent when activations remain unquantized. We trace this behavior to the conventional straight-through estimator (STE) used for activation quantization, which ignores the gradient contribution from each group's scale and biases gradients reaching the preceding RMSNorm. We propose **TetraBack** to restore this missing gradient contribution without modifying forward computation. Our method curbs the abnormal growth of activation magnitudes and RMSNorm weights, consistently improves performance across 3B–94B MoE models, and reduces the loss gap induced by activation quantization by up to **41%**.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization-Aware Training, W4A4 Quantization, MXFP4, Microscaling, Activation Quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_w57QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors consider the microscale quantization methods for 4bit that are now commonly used for training LLMs. Through empirical evaluation of activation and pre-ffn rmsnorm growth in training, they locate the issue to bias and gives evidence to it being due to approximated STE used to differentiate through the quantization function. Because quantization scales and measurment are block specific, the authors suggest an additional contribution to gradient that takes these into consideration. The authors show how this fix can close both the observed symptoms of norm growth and the actuall error gap in QAT training. Main contribution: (1) the link between activation growth, quantization bias and error outcome. (2) suggested solution by adapting gradient to improve and close quantization error gap.

Critical strengths
  • A new diagonstical lens on quantization impact for training microscaled models. This is explained and substantiated so future works can use to further advance quantization efforts.
  • A simple, computationaly cheap fix to quantization gradients that partially closes the error gap in QAT. This will probably become the baseline for any future technique in 4bit training.
Critical weaknesses
  • Experimental setting is narrow - authors should add additional models, regimes (finetuning) to further substantiate their claims.
Questions for authors
  • can you suggest other methods/models/techniques where your analysis and proposed solution may be applicable (e.g, STE is used for other methods like dictionary learning, where similar/other bias may exist)?
  • please take a look at recent finding about quantization in normalized architecture (nGPT) by Fishman et al. (https://arxiv.org/abs/2605.06067) - can you explain some of their findings with your view? I don't ask to actually train/evaluate (as it's a different architecure) but I think can be valuable to compare conceptually (normalized architecture has bounded weight+activation norm, fixed scale)
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer w57Q

We thank you for your time, positive endorsement, and insightful suggestions, as well as for recognizing our diagnostic lens and backward correction as a useful baseline for 4-bit training. Below, we extend our evaluation and discussion following your suggestions.

On Weakness: Additional Architectures and Fine-Tuning (SFT)

1. Dense/MoE architectures: We evaluated models following the Llama 3.2, Qwen3, and OLMo2 architectures trained for 20B tokens under MXFP4 W4A4 (Gap closed = reduction of W4A4-W4A16 loss gap; percentages are computed from unrounded losses). TetraBack closes 31.2–60.031.2\text{--}60.0% of the eval loss gap (and 34.334.3% with quantized Attention Q/K/V/O on MoE-3B; see our response to Reviewer vdku (W1)):

ModelMetric ↓ (gap closed ↑)W4A16 (Ref.)W4A4 (Naive)W4A4 w/ Diag. Corr.W4A4 w/ TetraBack
Llama 3.2 (1.24B dense)Train loss2.07242.11102.1020 (23.4%)2.0895 (55.6%)
Eval loss2.18352.22252.2123 (26.2%)2.1991 (60.0%)
Qwen3 (3.02B-A0.47B MoE)Train loss2.06862.09302.0895 (14.2%)2.0835 (38.8%)
Eval loss2.19052.21622.2118 (17.0%)2.2054 (42.2%)
OLMo2 (1.60B dense)Train loss2.19452.20932.2090 (2.1%)2.2061 (21.7%)
Eval loss2.31042.32212.3224 (-2.6%, worse)2.3185 (31.2%)

Naive W4A4 increases layer-mean FFN output RMS by 261–1736261\text{--}1736% and FFN Norm weight RMS by 27–10427\text{--}104% relative to W4A16, whereas TetraBack reduces their relative differences from W4A16 to within 1616% and 5.25.2%.

2. Multi-seed 94B SFT: We fine-tuned the 94B checkpoints on UltraChat-200k (1 epoch) and MetaMathQA (3 epochs) (LR=10−5\mathrm{LR}=10^{-5}, cosine decay, batch size 32; mean ± SD over 3 SFT seeds per checkpoint; SFT retains each checkpoint’s pretraining precision and TetraBack on/off setting, supporting the benefit of using TetraBack throughout pretraining and SFT without isolating the two stages). TetraBack improves over W4A4 by +2.40–3.14+2.40\text{--}3.14 points on all 0-shot metrics:

SFT Dataset0-shot Metric ↑W8A8 Ref.W4A4 (Naive)W4A4 w/ TetraBack
UltraChat-200kIFEval strict (prompt)32.47±1.1931.73±0.5934.87±1.19
UltraChat-200kIFEval loose (prompt)35.24±1.2334.69±0.2137.34±1.44
MetaMathQAGSM8K accuracy70.41±0.1965.93±0.5368.71±1.13
MetaMathQAMATH-500 accuracy26.27±1.1423.00±1.2225.40±1.74

On Q1: Broader Applicability

Our analysis is potentially applicable to other quantizers where multiple elements share an input-dependent scale, but the STE ignores this dependency. Our experiments already support its applicability on microscaling quantized training across different architectures, attention/FFN modules, and MXFP4/NVFP4 formats.

Thank you for suggesting dictionary learning: a similar issue could arise if discrete encoding uses an input-dependent shared scale. At the same time, whether such an STE bias exists there and systematically affects optimization is an interesting open question for future investigation beyond this work.

On Q2: Comparison with Normalized Architectures (nGPT [1])

Fishman et al. [1] remove learnable RMSNorms and apply hard ℓ2\ell_2 normalizations to bound weight norms and normalize hidden states and GEMM inputs. This geometry limits radial scale growth and promotes coherent signal accumulation (this promotion in signal accumulation might be a complementary mechanism to the correction of the bias), though fixed activation norms do not necessarily make group-wise scales input-independent (so shared-scale backward bias may persist, and the effect of such bias in nGPT remains open). However, to achieve improvement for such an architecture, it may require different optimization strategies (e.g., removing weight decay and warmup in their work).

In contrast, the learnable-Norm and activation drift we identify commonly appears across the tested mainstream LLM architectures and is caused by an improper backward STE for microscaled activations. TetraBack resolves the W4A4 trajectory drift across mainstream architectures without architectural or optimizer changes, which may offer broader algorithmic inspiration and possible practical value.

Thank you again for your valuable feedback, which helped us strengthen our evaluation; these additional results further validate our core findings without altering our original analysis or claims, and will be included in the final revision. We hope our responses have addressed your questions, and we are happy to discuss further should you have any additional comments.

[1] Normalized Architectures are Natively 4-Bit, Fishman et al., arXiv:2605.06067.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Response

Dear Reviewer w57Q,

Thank you again for your encouraging review and insightful suggestions. We have conducted the additional experiments and expanded our discussions to address your comments. Before the discussion period ends, we would be glad to hear your thoughts on our responses and new results, and would be happy to discuss any further questions.

With thanks and best regards,
The Authors

I thank reviewers for the response and additional experiments. The added evaluations substantially address my concern about the narrow experimental setting. The discussion of broader applicability and the comparison to nGPT are also helpful. I agree with the framing that normalized architectures may mitigate norm-growth symptoms while not necessarily removing the underlying shared-scale STE bias. This distinction strengthens the conceptual contribution. Overall, the response addresses my main questions. I will keep my positive recommendation.

AuthorsOpenReview ↗

Thank You for Your Feedback and Support

Dear Reviewer w57Q,

Thank you very much for taking the time to read our response and for your encouraging feedback! We are glad that the additional architecture and SFT evaluations and the conceptual discussion have addressed your main questions, and we will incorporate them into the final revision.

Best regards,
The Authors

Reviewer_vdkuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TetraBack studies from-scratch MXFP4 W4A4 quantization-aware training of MoE LLMs. The authors find that quantizing activations makes MoE output magnitudes and pre-FFN RMSNorm weights grow steadily during training, and they show this growth hurts optimization. They attribute it to the standard straight-through estimator, which ignores that each group's quantization scale depends on the group's largest activation, and add the missing gradient term in the backward pass only. The correction recovers part of the loss gap between W4A4 and W4A8/W8A8 on 3B to 94B MoE models and also works for NVFP4.

Critical strengths

The paper identifies a new phenomenon. The norm and activation drift under activation quantization is not reported in prior work, and the norm-constraint intervention and layer-wise gradient tracing make a convincing causal case.

Critical weaknesses
  1. The experiments cover only one model family. All models are MoEs adapted from DeepSeek-V4 and V4.1-Flash, with DeepSeek-specific components such as mHC and SwiGLU clamping. Quantization is applied only to the routed-expert FFNs. It is unclear whether the norm drift and the gain from TetraBack appear in dense models, in standard Llama-style architectures, or when attention layers are also quantized. The experiments should be expanded to at least one dense model and one non-DeepSeek architecture.

  2. The comparison with Hu et al. (2025) is too limited to judge effectiveness. The only comparison is a training-loss table on the 3B model in Appendix F.4, and it uses the authors' own reimplementation rather than the original method. No accuracy is reported for this comparison, and the closest prior method does not appear in the main benchmark table. Without downstream results against Hu et al. and other W4A4 methods, it is difficult to assess how much TetraBack actually adds.

  3. The performance gain is marginal. On the 94B model the benchmark averages improve by less than one point, and TetraBack is worse than plain W4A4 on 5 of the 21 benchmarks. All results are single runs with no variance reported, and W8A8 has the best validation loss but a lower benchmark average than TetraBack, which shows that differences of this size are within noise. It is therefore not possible to say for sure that the proposed method is effective.

Questions for authors

See weaknesses

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer vdku (Part 1/2)

We thank you for your time and constructive feedback, and for recognizing our new optimization finding and causal analysis. Below, we address your concerns.

On W1: More Architectures and Quantized Attention Projections

We trained models following the Llama 3.2 (1.24B dense, Pre Norm), Qwen3 (3.02B MoE, Pre Norm), and OLMo2 (1.60B dense, branch-output Norm) architectures (without mHC or SwiGLU clamping) for 20B tokens under MXFP4 W4A4. TetraBack closes 31.2–60.031.2\text{--}60.0% of the eval loss gap (while Diag. Corr. widens the eval loss gap on OLMo2):

Table: MXFP4 W4A4 (FFN quantization) across dense and MoE models (Gap closed = reduction of W4A4-W4A16 loss gap; percentages are computed from unrounded losses).

ModelMetric ↓ (gap closed ↑)W4A16 (Ref.)W4A4 (Naive)W4A4 w/ Diag. Corr.W4A4 w/ TetraBack
Llama 3.2 (1.24B dense)Train loss2.07242.11102.1020 (23.4%)2.0895 (55.6%)
Eval loss2.18352.22252.2123 (26.2%)2.1991 (60.0%)
Qwen3 (3.02B-A0.47B MoE)Train loss2.06862.09302.0895 (14.2%)2.0835 (38.8%)
Eval loss2.19052.21622.2118 (17.0%)2.2054 (42.2%)
OLMo2 (1.60B dense)Train loss2.19452.20932.2090 (2.1%)2.2061 (21.7%)
Eval loss2.31042.32212.3224 (-2.6%, worse)2.3185 (31.2%)

Moreover, naive W4A4 increases layer-mean FFN output RMS by 261–1736261\text{--}1736% and FFN Norm weight RMS by 27–10427\text{--}104% relative to W4A16, whereas TetraBack reduces their relative differences from W4A16 to within 1616% and 5.25.2%.

Quantizing both routed FFNs and Attention Q/K/V/O projections on MoE-3B, TetraBack closes 32.532.5% train and 34.334.3% Pile Val. loss gaps (∼3×\sim 3\times Diag. Corr.):

Model (Quantized Attn Q/K/V/O + FFN)Metric ↓ (gap closed ↑)W4A16W4A4W4A4 w/ Diag. Corr.W4A4 w/ TetraBack
MoE-3BTrain loss1.92541.94751.9451 (10.8%)1.9403 (32.5%)
Pile Val. loss2.48182.50732.5045 (10.9%)2.4985 (34.3%)

Under joint Attention and FFN quantization, naive W4A4 increases the layer-mean Pre-Attention Norm RMS, Pre-FFN Norm RMS, and MoE output RMS by 2121%, 111111%, and 490490% relative to W4A16, whereas TetraBack reduces all three relative differences from W4A16 to within 99%.

On W2: Comparison with Hu et al. (2025) [1] and Other W4A4 Baselines

We would first like to clarify the context of Hu et al. (2025) [1]: it is a preprint without released code (requiring our own implementation in App. F.4) surveying the general FP4 design space, which focused on a different research question rather than W4A4 optimization drift and did not adopt scale-gradient adjustment in their final best-loss LLM recipe. As analyzed in detail in App. F.4 of our submitted paper, for X^g=Q(Xg):R32→R32\widehat{\mathbf{X}}_g = Q(\mathbf{X}_g): \mathbb{R}^{32} \to \mathbb{R}^{32}, their element-wise loss-gradient expression is mathematically incomplete as a full group-wise surrogate rule: it approximates the non-diagonal surrogate Jacobian as a diagonal matrix, omitting cross-element shared-scale terms and failing to resolve the drift in our experiments (App. F.4, Fig. 16), which highlights the non-triviality of our work.

Following your suggestion to provide a more explicit comparison alongside other W4A4 methods (HCP and OutControl from Sec. 4.3), we evaluate them across 3 seeds on MoE-3B below. TetraBack closes ∼3–4×\sim 3\text{--}4\times as much of the train and Pile Val. loss gaps as Diag. Corr., and closes ∼3.7–6.5×\sim 3.7\text{--}6.5\times as much loss gap as HCP and OutControl (which incur forward/inference overhead) while achieving higher downstream accuracy:

Table: MoE-3B, 3 seeds (mean ± SD): loss and 0-shot accuracy (token-normalized). Diag. Corr. ([1]) follows App. F.4 (an aggregation ablation, not Hu et al.’s full recipe).

MethodTrain loss ↓ (gap closed ↑)Pile Val. loss ↓ (gap closed ↑)HellaSwag ↑ARC-E ↑ARC-C ↑Benchmark Avg. ↑
W4A161.9181±0.00212.4732±0.002853.55±0.1767.10±0.9238.51±1.3753.05±0.79
W4A41.9377±0.00152.4975±0.002252.26±0.4466.26±0.5536.80±0.4351.77±0.11
W4A4 w/ Diag. Corr. ([1])1.9356±0.0024 (10.76±4.28%)2.4935±0.0042 (16.06±8.49%)52.38±0.2965.81±0.7437.06±0.6951.75±0.15
W4A4 w/ HCP (activation-only)1.9356±0.0023 (10.42±3.53%)2.4942±0.0035 (13.31±6.81%)52.37±0.2065.30±0.9237.03±0.8451.56±0.52
W4A4 w/ OutControl1.9358±0.0026 (9.55±5.61%)2.4956±0.0030 (7.57±3.26%)52.51±0.3565.98±0.2936.52±1.6851.67±0.59
W4A4 w/ TetraBack1.9289±0.0027 (44.50±4.72%)2.4855±0.0030 (48.85±6.52%)52.93±0.1866.68±0.1437.43±0.2652.35±0.08
AuthorsOpenReview ↗

Rebuttal to Reviewer vdku (Part 2/2)

On W3: Practical Evidence of Gains and Multi-Seed SFT Evaluation

Because pretraining a 94B multimodal MoE model on 566B tokens is extremely compute-intensive, running multiple from-scratch 94B pretraining runs is computationally infeasible (which is why we report 3-seed pretraining variance on MoE-3B above). However, both the evaluations in our paper and new supplementary experiments provide complementary evidence that supports the 94B gains:

1. Consistent Pile validation loss improvement across all 22 subsets: On MoE-94B, TetraBack lowers overall Pile validation loss from 1.92161.9216 to 1.91781.9178 (recovering 26.626.6% of the W4A4-to-W8A8 gap), consistently improving every single one of the 22 Pile subsets:

Table: MoE-94B, Pile validation cross-entropy loss across all 22 subsets (1/2).

SettingArXivBookCorpus2Books3DM MathEnronEuroParlFreeLawGitHubGutenbergHackerNewsNIH ExPorter
W4A4 (Naive) ↓1.44242.39872.38841.19332.10401.70851.62740.66972.33322.44122.2173
W4A4 w/ TetraBack ↓1.43842.39312.38191.19172.10151.70611.62370.66732.32772.43662.2137

Table: MoE-94B, Pile validation cross-entropy loss across all 22 subsets (2/2).

SettingOpenSubtitlesOpenWebText2PhilPapersPile-CCPubMed Abs.PubMed CentralStackExchangeUSPTOUbuntu IRCWikipedia (en)YouTubeSub
W4A4 (Naive) ↓2.33932.24222.09122.31131.97881.56611.60001.85902.01171.98441.7672
W4A4 w/ TetraBack ↓2.33682.23712.08712.30761.97601.56211.59621.85442.00781.98051.7653

2. Paired-bootstrap SE on the 21 base-model benchmarks: In multi-benchmark base-model evaluations, minor variations on individual tasks are natural [2], so simultaneous gains on all 21 tasks are not expected. TetraBack wins on 15 of 21 benchmarks, ties on 1, and trails on only 5 in Tab. 1 of our paper. Resampling test questions with replacement within each benchmark to compute the paired-bootstrap SE of (W4A4 w/ TetraBack−W4A4\text{W4A4 w/ TetraBack} - \text{W4A4}) (quantifying test-set uncertainty for fixed checkpoints, not pretraining-seed variability) supports an overall gain beyond finite-test-set uncertainty:

MoE-94B AverageText-only (13 tasks)Multimodal (8 tasks)Overall (21 tasks)
Δ vs. Naive W4A4 ± paired-bootstrap SE (pp) ↑+0.59±0.30+0.85±0.40+0.69±0.24

3. Multi-seed Supervised Fine-Tuning (SFT) on MoE-94B:
As shown in recent evaluation studies [2], direct base-model evaluation can exhibit minor ranking discrepancies relative to validation loss, whereas fine-tuning before evaluation (train-before-test [2]) aligns downstream task rankings with pretraining loss. To further validate our findings (following Reviewer w57Q's suggestion and [2]), we fine-tuned the 94B checkpoints on UltraChat-200k (1 epoch) and MetaMathQA (3 epochs) (LR=10−5\mathrm{LR}=10^{-5}, cosine decay, batch size 32). Across 3 SFT seeds, TetraBack improves over naive W4A4 by +2.40+2.40 to +3.14+3.14 percentage points on all four metrics, confirming consistent downstream gains after SFT:

Table: MoE-94B, 0-shot evaluation after matched-precision SFT (mean ± SD over 3 SFT seeds per checkpoint; SFT retains each checkpoint’s pretraining precision and TetraBack on/off setting, supporting the benefit of using TetraBack throughout pretraining and SFT without isolating the two stages).

SFT Dataset0-shot Metric ↑W8A8 ReferenceW4A4 (Naive)W4A4 w/ TetraBack
UltraChat-200kIFEval strict (prompt)32.47±1.1931.73±0.5934.87±1.19
UltraChat-200kIFEval loose (prompt)35.24±1.2334.69±0.2137.34±1.44
MetaMathQAGSM8K accuracy70.41±0.1965.93±0.5368.71±1.13
MetaMathQAMATH-500 accuracy26.27±1.1423.00±1.2225.40±1.74

In summary, these results confirm that the optimization drift we identified is a cross-architecture problem in microscaled W4A4 training, and that TetraBack suppresses this drift with consistent performance improvements. Thank you again for your valuable feedback; these additional results further validate our core findings without altering our original analysis or claims, and will be included in the final revision. We hope our responses have addressed your questions, and we are happy to discuss further should you have any additional comments.

[1] Elucidating the Design Space of FP4 training, Hu et al., arXiv:2509.17791.
[2] Train-before-Test Harmonizes Language Model Rankings, Zhang et al., arXiv:2507.05195.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Response

Dear Reviewer vdku,

Thank you again for your valuable feedback and helpful suggestions. We have added new experiments and clarifications in response to your comments. With the discussion phase concluding soon, we would very much appreciate hearing your thoughts on our responses and updated results, and remain happy to address any further questions.

With thanks and best regards,
The Authors

Reviewer_KGzUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates optimization drift in microscaled W4A4 training, manifested as elevated pre-FFN RMSNorm weights and activation magnitudes. The authors attribute this behavior to the standard STE omitting gradient contributions through the shared, input-dependent group scale. They propose TetraBack aggregates these contributions across each quantization group and preserves forward computation. Experiments across 3B–94B MoE models show reduced scale drift and lower training losses, together with improved validation loss on the 94B model.

Critical strengths
  1. The controlled diagnostics are informative. Weight-only quantization controls, BF16 backward computation, Norm-scale interventions, and analysis of radial gradients and actual parameter updates provide useful evidence about the observed drift.

  2. Improvements extend across multiple model scales, including a 94B multimodal model. The correction is simple, preserves the forward quantizer, and introduces no additional inference path.

Critical weaknesses
  1. Limited methodological novelty. Input-dependent scale differentiation is already studied in (Hu et al. 2025). The specific increment in the contribution of this paper is cross-element aggregation under chosen surrogate derivatives. The diagonal-versus-full ablation is useful, but provides only a partial reconciliation with Hu et al.’s negative findings on gradient adjustments.

  2. The Norm-drift diagnosis needs clearer positioning relative to prior work and general scale control. FoG [R1] reports growth of QK RMSNorm gains during low-precision training and studies freezing or regularizing them. OSP [R2] reduces channel-wise amplification through single-scale normalization. Neither of these works is cited, although the paper describes the drift as “previously overlooked.” The paper neither establishes the same pre-FFN W4A4 mechanism, and should clarify the additional understanding provided by the shared-scale STE attribution. Fig. 4 reports that W4A4 training loss is reduced by 0.0041 and 0.0044 using practical Norm constraints, by 0.0074 using oracle scale matching, and by 0.0078 for TetraBack on MoE-3B. However, the superiority of TetraBack over the tested practical controls are confined to 3B without a reported sensitivity study. A modest sweep of Norm-specific regularization, ideally confirmed at another feasible scale, would better quantify its added value beyond general scale control.

  3. Robustness and competitiveness of the gains remain unclear. The absolute loss reductions are modest (0.0078 for 3B training and 0.0038 for 94B validation), although they recover a meaningful fraction of the quantization-induced gap. Independent-run variability is not reported, making it difficult to assess whether these gains consistently exceed run-to-run training variability. Section 4.3 includes forward-compensation baselines but does not evaluate MXFP4-compatible block-wise rotation [R3]. Such rotations alter the group absmax and quantization residuals through which TetraBack’s correction acts, so its benefit on top of a rotation baseline is not obvious. A matched 3B comparison of a rotation baseline with and without TetraBack would establish its incremental value; compatibility alone does not.

[R1] Alejandro Hernández Cano, Dhia Garbaya, Imanol Schlag, and Martin Jaggi. Towards Fully FP8 GEMM LLM Training at Scale. NeurIPS, 2025.

[R2] Jungwoo Park, Taewhoo Lee, Chanwoong Yoon, Hyeon Hwang, and Jaewoo Kang. Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models. ACL, 2025, pp. 12582–12600.

[R3] Yuantian Shao, Peisong Wang, Yuanteng Chen, Chang Xu, Zhihui Wei, and Jian Cheng. Block Rotation is All You Need for MXFP4 Quantization. In Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306:110275–110295, 2026.

Questions for authors
  1. Can repeated runs at 3B confirm that the full group-wise correction consistently outperforms both the diagonal variant and tuned Norm-scale controls?

  2. Under a matched 3B training budget, does TetraBack retain an additional benefit when combined with OutControl/HCP or MXFP4-compatible block-wise rotation, or does the benefit vanish?

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer KGzU (Part 1/2)

We thank you for your time and constructive suggestions. Before presenting our new experiments, we briefly clarify two points on Weaknesses 1 and 2 (W1 and W2):

(i) Regarding W1, Hu et al. [1] neither identified nor addressed the W4A4 optimization-drift problem. Their element-wise loss-gradient expression is incomplete as a group-wise surrogate backward rule, omitting cross-element contributions (analyzed in detail in App. F.4 of our submitted paper). Moreover, our contribution goes far beyond correcting this omission: we identify an unreported harmful W4A4 optimization drift, establish its causal connection to the omitted scale path, and correct it without changing the forward computation.

(ii) Regarding W2, the norm interventions are causal diagnostics rather than proposed methods. The abnormal Norm growth is a consequence of the STE-induced bias. Directly constraining Norms may help mitigate the symptom, but does not restore the omitted shared-scale gradient path, and is therefore distinct from correcting the underlying backward approximation.

On W1 & Q1 (Part 1): Core Contribution and Relation to Prior Work

1. Core contribution and distinction from [1]:

We first clarify that our contribution goes well beyond a backward formula: (a) we identify model-level W4A4 optimization drift previously unreported in the FP4 training literature;
(b) we establish its causal link to omitted scale-path terms in activation-microscaling backpropagation;
(c) we resolve it with a simple backward correction that substantially improves training without introducing inference overhead.

To our knowledge, this phenomenon and its causal attribution have not been reported in prior FP4 training literature; moreover, this persistent drift appears across diverse mainstream architectures (Llama 3.2, Qwen3 MoE, OLMo2, and DeepSeek), and TetraBack effectively suppresses the drift and improves W4A4 training across them in our experiments (see our response to Reviewer vdku (W1)).

Specifically, Hu et al. (2025) [1] is a preprint surveying the general FP4 design space, which focused on a different research question rather than W4A4 optimization drift and did not adopt scale-gradient adjustment in their final best-loss LLM recipe. Their incomplete diagonal-only approximation does not resolve the optimization drift in our experiments (App. F.4, Fig. 16 of our paper), and our method substantially outperforms this diagonal design below. This controlled comparison shows that cross-element aggregation is important for mitigating the observed drift in our setting, and highlights the non-triviality of identifying an effective correction.

2. Multi-seed quantitative comparison (Q1, Part 1):
By comparing them quantitatively on MoE-3B, TetraBack closes approximately 3–4×3\text{--}4\times as much of the quantization-induced loss gap as the diagonal-only correction does:

Table: MoE-3B, three training seeds (mean ± SD). Gap closed = the reduction of the W4A4-W4A16 loss gap (computed per seed from unrounded losses). Diag. Corr. ([1]) follows App. F.4 of our paper, isolating cross-element aggregation rather than reproducing their complete training recipe.

MethodTrain loss ↓ (gap closed ↑)Pile Val. loss ↓ (gap closed ↑)
W4A16 (Reference)1.9181±0.00212.4732±0.0028
W4A4 (Standard STE)1.9377±0.00152.4975±0.0022
W4A4 w/ Diag. Corr. ([1])1.9356±0.0024 (10.76±4.28%)2.4935±0.0042 (16.06±8.49%)
W4A4 w/ TetraBack1.9289±0.0027 (44.50±4.72%)2.4855±0.0030 (48.85±6.52%)

(On OLMo2-1.6B [see our response to vdku (W1)], Diag. Corr. widens the eval loss gap by 2.62.6%, while TetraBack closes 31.231.2%.)

[1] Elucidating the Design Space of FP4 training, Hu et al., arXiv:2509.17791.

On W2 (Part 1): Clarifying our Claim and Analysis

Our paper explicitly scopes our claim to a persistent optimization drift specific to microscaled W4A4 training (Sec. 1 & Sec. 2) rather than general scale drift. We thank you for pointing out FoG [R1] and OSP [R2] (using [R1–R3] from your review), which examine complementary problems in different settings:

  • FoG [R1] studies training instability and loss divergence in FP8 training from outlier amplification, distinct from our stable-yet-biased W4A4 trajectory.
  • OSP [R2] does not study low-precision training (QAT), instead modifying BF16 pretraining optimizers and norms for a friendlier model for post-training quantization (PTQ).

We will gladly cite and discuss [R1, R2] in the revision.

Regarding the Pre-FFN W4A4 mechanism and the additional understanding from our STE attribution, we highlight Sec. 3 (Fig. 6), Sec. 4.5 (Fig. 11), and App. F.1–F.3 (Figs. 13–15), which trace in detail how the omitted shared-scale STE gradient biases upstream Pre-FFN Norm gradients and drives scale growth across layers.

AuthorsOpenReview ↗

Rebuttal to Reviewer KGzU (Part 2/2)

On W2 (Part 2) & Q1 (Part 2): Diagnostic Norm Controls

Following your suggestion to further evaluate general Norm-scale controls, we report an expanded comparison on MoE-3B and MoE-21B below, where TetraBack outperforms the tested non-oracle controls:

Table: MoE-3B, 3 seeds (mean ± SD). Gain refers to the RMSNorm weight. Norm controls (Oracle RMS matching uses final W4A16 norms and is impractical).

W4A4 SettingTrain loss ↓Pile Val. loss ↓
Standard STE1.9377±0.00152.4975±0.0022
Gain frozen at init ([R1] inspired)1.9335±0.00232.4919±0.0045
Gain frozen after warmup1.9338±0.00192.4914±0.0024
Single-scale Norm ([R2] inspired)1.9390±0.00262.4966±0.0029
Fixed Gain RMS=0.81.9331±0.00272.4910±0.0014
Fixed Gain RMS=11.9328±0.00202.4901±0.0022
Fixed Gain RMS=1.21.9374±0.00192.4956±0.0024
Oracle RMS matching1.9309±0.00142.4862±0.0025
TetraBack1.9289±0.00272.4855±0.0030

Table: MoE-21B: transferring the best-tested practical control from 3B and applying the same oracle-matching procedure with targets from the final MoE-21B W4A8 checkpoint.

Training MethodTrain loss ↓Pile Val. loss ↓
Naive W4A41.53062.1707
W4A4 w/ Fixed Gain RMS=11.52992.1709
W4A4 w/ Oracle RMS matching1.54152.1929
W4A4 w/ TetraBack1.52722.1674

On MoE-21B, the best practical 3B control (Fixed Gain RMS=1) barely changes train loss and does not improve Pile validation loss, while the oracle RMS-matching strategy worsens both losses (indicating that such interventions are not uniformly effective across scales and may require further tuning), whereas TetraBack achieves the lowest train and validation losses among the tested settings at both 3B and 21B.

At the same time, we respectfully clarify that constraining Pre-FFN Norms in Sec. 2 was introduced as a controlled causal diagnostic to verify that the enlarged Norm scale is a harmful outcome of W4A4 optimization: abnormal Norm growth is a downstream consequence of the STE bias, so directly constraining Norms may help mitigate this symptom, but does not restore the omitted shared-scale gradient path. Thus, while exploring tuned Norm regularization is a complementary direction, our core contribution lies in identifying the W4A4 optimization drift, tracing it to the omitted shared-scale path, and resolving it at its backward root without Norm-specific tuning.

On W3 & Q2: Multi-Seed Results on the Gains of TetraBack

Both HCP and OutControl incur forward/inference overhead, whereas TetraBack achieves lower train and validation losses without forward or inference overhead (MoE-3B, 3 seeds).

Moreover, Hadamard Transform (HT) [R3] transforms the quantizer's operand to reduce MXFP4 forward quantization error, while TetraBack fixes the backward STE of the quantization operation itself; modifying the operand and correcting the operator's backward pass are complementary in our experiments.

W4A4 SettingTrain loss ↓Pile Val. loss ↓
Standard STE1.9377±0.00152.4975±0.0022
HCP (activation-only)1.9356±0.00232.4942±0.0035
OutControl1.9358±0.00262.4956±0.0030
block-wise HT [R3] only (block size=32)1.9329±0.00242.4905±0.0021
TetraBack1.9289±0.00272.4855±0.0030
block-wise HT [R3] + TetraBack1.9267±0.00312.4821±0.0039

To further evaluate the practical significance of closing 45–4945\text{--}49% of the W4A4-W4A16 loss gap on MoE-3B, we also examine its equivalent token efficiency in extended MoE-3B runs: matching constant-LR window-smoothed training-loss curves show that TetraBack uses 15.4%–20.4% fewer training tokens to reach the same training-loss level (within the conducted training range, without extrapolation):

Tokens w/ TetraBack40.0 B45.0 B50.0 B55.0 B60.0 B
Equiv. tokens w/o TetraBack (token savings ↑)47.3 B (15.4%)54.1 B (16.8%)61.0 B (18.0%)68.1 B (19.2%)75.4 B (20.4%)

In summary, our core contribution goes far beyond a backward formula or loss gains: it lies in identifying this fundamental W4A4 optimization drift across architectures, establishing the non-obvious causal link between model-level Norm/activation drift and the omitted off-diagonal Jacobian coupling in activation-microscaling backpropagation, and proposing a simple yet effective solution, TetraBack.

Thank you again for your valuable feedback; these additional results further validate our core findings without altering our original analysis or claims, and will be included in the final revision. We hope our responses have addressed your questions, and we are happy to discuss further should you have any additional comments.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Response

Dear Reviewer KGzU,

Thank you again for your thoughtful review and constructive suggestions. We have provided additional experiments and clarifications to address your comments. As the discussion period is coming to a close, we would be grateful to hear your thoughts on our responses and new results, and would be happy to discuss any remaining questions.

With thanks and best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Summary and Key Clarifications for Area Chair and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for your time and constructive feedback. Below, we briefly summarize our key findings, respectfully clarify possible misunderstandings, and highlight how our rebuttal addresses the main concerns raised in the reviews.

1. Key Findings: Addressing an Overlooked Issue in 4-bit Training

The optimization drift we identify is a previously overlooked yet common and long-existing issue in microscaled activation-quantized (W4A4) training across diverse mainstream LLM architectures (DeepSeek, Llama 3.2, Qwen3, OLMo2; see Rebuttal to Reviewer w57Q) and formats (MXFP4, NVFP4). Specifically, we show that the conventional STE widely used in existing FP4 training recipes introduces a persistent bias into the optimization trajectory: omitting the shared-scale gradient path drives upstream Norm drift and degrades training.

Beyond proposing TetraBack to suppress this drift and substantially improve W4A4 training without modifying forward computation, we transparently share our causal-diagnostic methodology, offering both reusable analytical tools and a practical correction for future low-precision training (with Reviewer w57Q noting that TetraBack "will probably become the baseline for any future technique in 4bit training").

2. Clarifying Possible Misunderstandings of Our Diagnostics and Prior Work

While all three reviewers recognized the value of our causal diagnostics and mechanistic findings, several negative comments appear to stem from two misinterpretations:

  • (1) Possibly viewing our causal-diagnostic Norm interventions as a competing training method (Reviewer KGzU): In Sec. 2, we constrained Pre-FFN Norms strictly as causal diagnostics rather than competing methods, to share our step-by-step investigation and verify that Norm drift is harmful. Constraining Norms only treats the symptom rather than the root cause and may require scale-specific tuning, whereas TetraBack directly corrects the underlying STE bias without any Norm control.

  • (2) Possibly viewing our Appendix F.4 analysis of Hu et al. (2025) as a prior solution to this problem (Reviewer KGzU, Reviewer vdku): Hu et al. studied a different research question; they neither identified nor solved W4A4 optimization drift. We specifically analyzed their formulation in App. F.4 of our submitted paper to provide a more comprehensive context, demonstrating that their formula is mathematically incomplete and fails to resolve the drift, thus highlighting the necessity of our correction.
    More importantly, our contribution goes far beyond a backward correction alone: this W4A4 optimization drift has never been identified in prior FP4 literature. Uncovering it and tracing model-level Norm drift to a subtle gradient bias in the widely adopted default STE is highly non-trivial.

Nevertheless, beyond clarifying these two points, our rebuttal also provides direct experimental evidence that TetraBack outperforms the tested baselines and practical Norm controls.

3. Summary of Additional Evidence and Responses

  • Cross-architecture and module generalization (Reviewer vdku, Reviewer w57Q): We expanded our evaluation to additional dense and MoE architectures and quantized Attention projections, confirming that this W4A4 optimization drift exists across architectures (Llama 3.2, Qwen3, OLMo2) and modules, and TetraBack effectively suppresses the drift, improves training, and outperforms the baselines.
  • Positioning relative to prior works (Reviewer KGzU): We clarify the different research problems and fundamentally distinct mechanisms relative to the prior FP8/PTQ works mentioned by Reviewer KGzU.
  • Multi-seed comparisons and statistical validation (Reviewer KGzU, Reviewer vdku, Reviewer w57Q): Multi-seed pretraining evaluations on MoE-3B, together with more detailed evaluation analyses and 3-seed SFT results on MoE-94B, further confirm the superiority of TetraBack over the baselines.

In summary, following the reviewers' constructive suggestions, we have completed the requested evaluations and extensions, all of which consistently support our core conclusions and original manuscript claims, and we will incorporate them into the final revision.

We hope these clarifications and additional results are helpful in assessing the scientific and practical contributions of our work, and we thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.