- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors consider the microscale quantization methods for 4bit that are now commonly used for training LLMs. Through empirical evaluation of activation and pre-ffn rmsnorm growth in training, they locate the issue to bias and gives evidence to it being due to approximated STE used to differentiate through the quantization function. Because quantization scales and measurment are block specific, the authors suggest an additional contribution to gradient that takes these into consideration. The authors show how this fix can close both the observed symptoms of norm growth and the actuall error gap in QAT training. Main contribution: (1) the link between activation growth, quantization bias and error outcome. (2) suggested solution by adapting gradient to improve and close quantization error gap.
Critical strengths
- A new diagonstical lens on quantization impact for training microscaled models. This is explained and substantiated so future works can use to further advance quantization efforts.
- A simple, computationaly cheap fix to quantization gradients that partially closes the error gap in QAT. This will probably become the baseline for any future technique in 4bit training.
Critical weaknesses
- Experimental setting is narrow - authors should add additional models, regimes (finetuning) to further substantiate their claims.
Questions for authors
- can you suggest other methods/models/techniques where your analysis and proposed solution may be applicable (e.g, STE is used for other methods like dictionary learning, where similar/other bias may exist)?
- please take a look at recent finding about quantization in normalized architecture (nGPT) by Fishman et al. (https://arxiv.org/abs/2605.06067) - can you explain some of their findings with your view? I don't ask to actually train/evaluate (as it's a different architecure) but I think can be valuable to compare conceptually (normalized architecture has bounded weight+activation norm, fixed scale)
4 comments
Gentle Follow-Up on Our Response
Dear Reviewer w57Q,
Thank you again for your encouraging review and insightful suggestions. We have conducted the additional experiments and expanded our discussions to address your comments. Before the discussion period ends, we would be glad to hear your thoughts on our responses and new results, and would be happy to discuss any further questions.
With thanks and best regards,
The Authors
I thank reviewers for the response and additional experiments. The added evaluations substantially address my concern about the narrow experimental setting. The discussion of broader applicability and the comparison to nGPT are also helpful. I agree with the framing that normalized architectures may mitigate norm-growth symptoms while not necessarily removing the underlying shared-scale STE bias. This distinction strengthens the conceptual contribution. Overall, the response addresses my main questions. I will keep my positive recommendation.
Thank You for Your Feedback and Support
Dear Reviewer w57Q,
Thank you very much for taking the time to read our response and for your encouraging feedback! We are glad that the additional architecture and SFT evaluations and the conceptual discussion have addressed your main questions, and we will incorporate them into the final revision.
Best regards,
The Authors
Rebuttal to Reviewer w57Q
We thank you for your time, positive endorsement, and insightful suggestions, as well as for recognizing our diagnostic lens and backward correction as a useful baseline for 4-bit training. Below, we extend our evaluation and discussion following your suggestions.
On Weakness: Additional Architectures and Fine-Tuning (SFT)
1. Dense/MoE architectures: We evaluated models following the Llama 3.2, Qwen3, and OLMo2 architectures trained for 20B tokens under MXFP4 W4A4 (Gap closed = reduction of W4A4-W4A16 loss gap; percentages are computed from unrounded losses). TetraBack closes 31.2–60.0% of the eval loss gap (and 34.3% with quantized Attention Q/K/V/O on MoE-3B; see our response to Reviewer vdku (W1)):
Naive W4A4 increases layer-mean FFN output RMS by 261–1736% and FFN Norm weight RMS by 27–104% relative to W4A16, whereas TetraBack reduces their relative differences from W4A16 to within 16% and 5.2%.
2. Multi-seed 94B SFT: We fine-tuned the 94B checkpoints on UltraChat-200k (1 epoch) and MetaMathQA (3 epochs) (LR=10−5, cosine decay, batch size 32; mean ± SD over 3 SFT seeds per checkpoint; SFT retains each checkpoint’s pretraining precision and TetraBack on/off setting, supporting the benefit of using TetraBack throughout pretraining and SFT without isolating the two stages). TetraBack improves over W4A4 by +2.40–3.14 points on all 0-shot metrics:
On Q1: Broader Applicability
Our analysis is potentially applicable to other quantizers where multiple elements share an input-dependent scale, but the STE ignores this dependency. Our experiments already support its applicability on microscaling quantized training across different architectures, attention/FFN modules, and MXFP4/NVFP4 formats.
Thank you for suggesting dictionary learning: a similar issue could arise if discrete encoding uses an input-dependent shared scale. At the same time, whether such an STE bias exists there and systematically affects optimization is an interesting open question for future investigation beyond this work.
On Q2: Comparison with Normalized Architectures (nGPT [1])
Fishman et al. [1] remove learnable RMSNorms and apply hard ℓ2 normalizations to bound weight norms and normalize hidden states and GEMM inputs. This geometry limits radial scale growth and promotes coherent signal accumulation (this promotion in signal accumulation might be a complementary mechanism to the correction of the bias), though fixed activation norms do not necessarily make group-wise scales input-independent (so shared-scale backward bias may persist, and the effect of such bias in nGPT remains open). However, to achieve improvement for such an architecture, it may require different optimization strategies (e.g., removing weight decay and warmup in their work).
In contrast, the learnable-Norm and activation drift we identify commonly appears across the tested mainstream LLM architectures and is caused by an improper backward STE for microscaled activations. TetraBack resolves the W4A4 trajectory drift across mainstream architectures without architectural or optimizer changes, which may offer broader algorithmic inspiration and possible practical value.
Thank you again for your valuable feedback, which helped us strengthen our evaluation; these additional results further validate our core findings without altering our original analysis or claims, and will be included in the final revision. We hope our responses have addressed your questions, and we are happy to discuss further should you have any additional comments.
[1] Normalized Architectures are Natively 4-Bit, Fishman et al., arXiv:2605.06067.