- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes QuantRet, a quantization-friendly LLM pretraining framework that combines spherical Transformer representations with three alternative weight retraction strategies (Frobenius, spectral, and oblique). The authors analyze quantization error propagation and evaluate the approach on models ranging from 0.3B to 1.7B parameters. Experiments demonstrate improved language modeling quality and downstream performance under the primary INT4 PTQ settings, although the benefits vary across quantization schemes.
Critical strengths
- Promising empirical results for low-bit quantization. QuantRet achieves substantial improvements over the evaluated baselines under W4A4 and W4A4KV4 quantization across multiple model sizes. These results suggest that pretraining-time architectural and geometric constraints can improve PTQ robustness.
- Insightful analysis of quantization robustness. The paper systematically investigates how different weight constraints affect perturbation amplification and error propagation. The theoretical analysis, together with layerwise measurements and controlled perturbation experiments, offers useful insights into quantization-friendly model design.
Critical weaknesses
- Limited scalability and generalization evidence. Unlike plug-and-play PTQ methods, QuantRet requires modifying the architecture and optimization process from the beginning of pretraining, making scalability particularly important. However, the largest evaluated model is only 1.7B, with a maximum Transformer depth of 28 layers. Together with the substantially weaker improvements under NVFP4, this raises concerns about whether the findings can reliably guide the pretraining of modern larger-scale LLMs.
- Insufficient isolation of the proposed contributions. Several architectural components closely resemble existing normalized Transformers, particularly nGPT. Moreover, the observed improvements may result from a combination of architectural modifications, Muon optimization, and matrix retractions. Although some ablations are provided, the current experiments do not sufficiently isolate the incremental contribution of the proposed geometric constraints, weakening the methodological novelty and causal interpretation.
Questions for authors
- Scalability: Can the authors provide stronger evidence of scalability, ideally through larger-model experiments or systematic depth-scaling evaluations? Given the substantial cost of adopting QuantRet during pretraining, such evidence would significantly affect my assessment.
- Contribution attribution: Can the authors include a directly matched nGPT-like baseline and a complete optimizer × retraction ablation under the same architecture and training budget? This would help clarify which components are responsible for the reported gains.
- Quantization generalization: Why are the improvements much smaller under NVFP4 and inconsistent under SmoothQuant? A more detailed analysis of when and why the proposed constraints improve quantization would strengthen the generalizability of the conclusions.
3 comments
Response to Reviewer Kij4 [Part 2]
Q3: Why the gains are smaller under NVFP4 and inconsistent under SmoothQuant
NVFP4 quantizes in blocks of 16 values, and QuantRet's advantage comes from per-token activation quantization. The SmoothQuant column of Table 5 also changed the weight quantizer, and that change, not smoothing, moved QuantRet. Values are FineWeb-Edu CE at 0.6B; positive differences favor QuantRet.
NVFP4. Table R1 gives W4A4 CE for the same models with an INT4 or FP4 codebook and FP32 scales at block size 16 and at per-channel / per-token granularity, with unclipped symmetric ranges and the KV cache in BF16; its per-token row is therefore harsher than Table 1's W4A4.
Table R1
| granularity | INT4 AdamW | INT4 Obl | INT4 AdamW Obl | FP4 AdamW | FP4 Obl | FP4 AdamW Obl |
|---|---|---|---|---|---|---|
| FP, unquantized | 2.692 | 2.682 | 0.010 | 2.692 | 2.682 | 0.010 |
| block 16 | 2.745 | 2.729 | 0.016 | 2.738 | 2.738 | 0.000 |
| per-channel / per-token | 6.133 | 3.389 | 2.744 | 4.061 | 2.915 | 1.146 |
| Table 14, NVFP4 | 2.741 | 2.740 | 0.001 |
QuantRet's lead is a function of activation granularity: large per token under either codebook, gone at block 16, where FP4 matches Table 14. Table 11 shows the same for activations alone, and Table 9 almost none for weights alone at channel granularity. Table 14 extends this to every scale: AdamW and QuantRet lose alike under NVFP4, so FP quality sets the order and AdamW leads at 1.7B, while Frob and Obl stay ahead of OSP; the per-token failure, by contrast, grows with scale, as Table 1 shows.
SmoothQuant. Table 5 turned smoothing on and replaced clipped W4 ranges with unclipped ones. Table R2 completes the at 0.6B W4A4KV4 with the same calibration set and .
Table R2
| W4 range | smoothing | AdamW | Obl | AdamW Obl |
|---|---|---|---|---|
| clipped | off | 3.751 | 3.056 | 0.695 |
| clipped | on | 3.201 | 3.058 | 0.143 |
| unclipped | off | 4.152 | 3.116 | 1.036 |
| unclipped | on | 3.059 | 3.110 |
Smoothing leaves QuantRet where it was and repairs AdamW; the unclipped range costs QuantRet and, once smoothing has rescaled the weight columns, no longer costs AdamW. QuantRet leads in three of the four cells, AdamW only in the one Table 5 reports. Under GPTQ, which leaves AdamW's activation error in place, QuantRet keeps its full lead; under QuaRot, which repairs it as smoothing does, Frob and Obl keep a smaller lead and Spec falls behind.
When and why. Per-token 4-bit activations leave a large error in every token. QuantRet turns it into a smaller local output error, most of all at the readers, Appendix D.3, and its state error stays smaller through the network, Section 4.3. The constraints therefore pay off where a quantizer leaves that error in place and not where the format or a repair removes it for AdamW too.
Thank you for your detailed response. My concerns have been addressed, and I am therefore raising my score to 3.
Response to Reviewer Kij4 [Part 1]
Thank you for your careful review and your constructive questions. We respond to each point below.
W1 and Q1: Limited scalability and generalization evidence; larger-model experiments or systematic depth-scaling evaluations
Thank you for suggesting larger-model experiments and a systematic depth-scaling evaluation. We ran 12, 20, 28 and 40 layers with the width, data, token budget and seed of the 0.3B runs; 40 layers exceeds the submission's maximum of 28. The table gives FineWeb-Edu CE under the PTQ pipeline of Table 1.
QuantRet-Obl has the lowest W4A4 CE at every depth, and its loss from FP does not grow from 12 to 40 layers. The baselines lose most at 12 layers and are not monotone in depth.
Please refer to our reply to your Q3 for the weaker improvements under NVFP4 and to our reply to Reviewer VaxM for the measured per-step training cost. The revised version reports the depth scan in Appendix D.7.
W2 and Q2: Isolation of the proposed contributions; a directly matched nGPT-like baseline and a complete optimizer × retraction ablation
Thank you for suggesting these two experiments. Please refer to our reply to Reviewer j9ex for what QuantRet shares with nGPT, what differs, and the directly matched nGPT baseline, which is now in Table 4. We also completed the optimizer × retraction ablation; the table gives FineWeb-Edu CE at 0.3B for one seed, with the data, token budget and W4A4 pipeline of Table 1.
With neither Muon nor the post-step retraction, QuantRet still loses less from FP to W4A4 than nGPT and both baselines. The retraction lowers that loss under either optimizer, more under Muon. Muon lowers FP CE but not that loss, and the two together give the lowest W4A4 CE.