acceptodds
Under review as a conference paper at ICLR 2027

QuantRet: Quantization-Robust Language Model Pretraining via Geometric Retractions

Abstract

Low-bit post-training quantization (PTQ) can fail when a small number of coordinates set the quantization range or when high-gain directions amplify local errors. We introduce QuantRet, a pretraining method for residual Transformer decoders. QuantRet retracts selected activations to fixed-radius spheres and constrains the norms of hidden weight matrices throughout pretraining. These operations fix the radial scale at selected activation sites and constrain how hidden linear maps rescale their inputs without changing the quantizer. Our analysis separates weight and activation perturbations, bounds their local amplification under the matrix constraints, and gives a conditional bound on their propagation through spherical residual updates. On Qwen3-like models, QuantRet lowers language-modeling loss under low-bit weight, activation, and key–value-cache quantization, and its strongest variant achieves higher zero-shot commonsense accuracy under the same quantized graphs. Mechanism measurements show that similar weight signal-to-noise ratios can produce different final dot-product signal-to-noise ratios and end-to-end degradation. These results support pretraining-time geometric constraints as a practical way to improve low-bit PTQ.

Primary area:
optimization
Keywords:
quantization-robust pretraining, post-training quantization, manifold constraints

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Kij4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QuantRet, a quantization-friendly LLM pretraining framework that combines spherical Transformer representations with three alternative weight retraction strategies (Frobenius, spectral, and oblique). The authors analyze quantization error propagation and evaluate the approach on models ranging from 0.3B to 1.7B parameters. Experiments demonstrate improved language modeling quality and downstream performance under the primary INT4 PTQ settings, although the benefits vary across quantization schemes.

Critical strengths
  1. Promising empirical results for low-bit quantization. QuantRet achieves substantial improvements over the evaluated baselines under W4A4 and W4A4KV4 quantization across multiple model sizes. These results suggest that pretraining-time architectural and geometric constraints can improve PTQ robustness.
  2. Insightful analysis of quantization robustness. The paper systematically investigates how different weight constraints affect perturbation amplification and error propagation. The theoretical analysis, together with layerwise measurements and controlled perturbation experiments, offers useful insights into quantization-friendly model design.
Critical weaknesses
  1. Limited scalability and generalization evidence. Unlike plug-and-play PTQ methods, QuantRet requires modifying the architecture and optimization process from the beginning of pretraining, making scalability particularly important. However, the largest evaluated model is only 1.7B, with a maximum Transformer depth of 28 layers. Together with the substantially weaker improvements under NVFP4, this raises concerns about whether the findings can reliably guide the pretraining of modern larger-scale LLMs.
  2. Insufficient isolation of the proposed contributions. Several architectural components closely resemble existing normalized Transformers, particularly nGPT. Moreover, the observed improvements may result from a combination of architectural modifications, Muon optimization, and matrix retractions. Although some ablations are provided, the current experiments do not sufficiently isolate the incremental contribution of the proposed geometric constraints, weakening the methodological novelty and causal interpretation.
Questions for authors
  1. Scalability: Can the authors provide stronger evidence of scalability, ideally through larger-model experiments or systematic depth-scaling evaluations? Given the substantial cost of adopting QuantRet during pretraining, such evidence would significantly affect my assessment.
  2. Contribution attribution: Can the authors include a directly matched nGPT-like baseline and a complete optimizer × retraction ablation under the same architecture and training budget? This would help clarify which components are responsible for the reported gains.
  3. Quantization generalization: Why are the improvements much smaller under NVFP4 and inconsistent under SmoothQuant? A more detailed analysis of when and why the proposed constraints improve quantization would strengthen the generalizability of the conclusions.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Kij4 [Part 1]

Thank you for your careful review and your constructive questions. We respond to each point below.

W1 and Q1: Limited scalability and generalization evidence; larger-model experiments or systematic depth-scaling evaluations

Thank you for suggesting larger-model experiments and a systematic depth-scaling evaluation. We ran 12, 20, 28 and 40 layers with the width, data, token budget and seed of the 0.3B runs; 40 layers exceeds the submission's maximum of 28. The table gives FineWeb-Edu CE under the PTQ pipeline of Table 1.

layersAdamW FPAdamW W4A4AdamW W4A4 −- FPOSP FPOSP W4A4OSP W4A4 −- FPObl FPObl W4A4Obl W4A4 −- FP
123.0413.9060.8652.9463.7390.7932.9283.2360.308
20, Table 12.9493.4540.5052.8573.4470.5902.8593.1230.264
282.9013.2350.3342.8133.4440.6312.8273.0750.248
402.8463.2540.4082.7643.2300.4662.7883.0400.252

QuantRet-Obl has the lowest W4A4 CE at every depth, and its loss from FP does not grow from 12 to 40 layers. The baselines lose most at 12 layers and are not monotone in depth.

Please refer to our reply to your Q3 for the weaker improvements under NVFP4 and to our reply to Reviewer VaxM for the measured per-step training cost. The revised version reports the depth scan in Appendix D.7.

W2 and Q2: Isolation of the proposed contributions; a directly matched nGPT-like baseline and a complete optimizer ×\times retraction ablation

Thank you for suggesting these two experiments. Please refer to our reply to Reviewer j9ex for what QuantRet shares with nGPT, what differs, and the directly matched nGPT baseline, which is now in Table 4. We also completed the optimizer ×\times retraction ablation; the table gives FineWeb-Edu CE at 0.3B for one seed, with the data, token budget and W4A4 pipeline of Table 1.

modeloptimizerpost-step matrix retractionFPW4A4W4A4 −- FP
AdamW, Table 1AdamWnone2.9493.4540.505
OSP, Table 1Muonnone2.8573.4470.590
matched nGPTAdamWunit norm2.8444.7071.863
QuantRet, newAdamWnone3.0493.2970.248
QuantRet, Table 4AdamWObl3.0303.2650.235
QuantRet, Table 4Muonnone2.8883.2390.351
QuantRet-Obl, Table 1MuonObl2.8593.1230.264

With neither Muon nor the post-step retraction, QuantRet still loses less from FP to W4A4 than nGPT and both baselines. The retraction lowers that loss under either optimizer, more under Muon. Muon lowers FP CE but not that loss, and the two together give the lowest W4A4 CE.

AuthorsOpenReview ↗

Response to Reviewer Kij4 [Part 2]

Q3: Why the gains are smaller under NVFP4 and inconsistent under SmoothQuant

NVFP4 quantizes in blocks of 16 values, and QuantRet's advantage comes from per-token activation quantization. The SmoothQuant column of Table 5 also changed the weight quantizer, and that change, not smoothing, moved QuantRet. Values are FineWeb-Edu CE at 0.6B; positive differences favor QuantRet.

NVFP4. Table R1 gives W4A4 CE for the same models with an INT4 or FP4 codebook and FP32 scales at block size 16 and at per-channel / per-token granularity, with unclipped symmetric ranges and the KV cache in BF16; its per-token row is therefore harsher than Table 1's W4A4.

Table R1

granularityINT4 AdamWINT4 OblINT4 AdamW −- OblFP4 AdamWFP4 OblFP4 AdamW −- Obl
FP, unquantized2.6922.6820.0102.6922.6820.010
block 162.7452.7290.0162.7382.7380.000
per-channel / per-token6.1333.3892.7444.0612.9151.146
Table 14, NVFP42.7412.7400.001

QuantRet's lead is a function of activation granularity: large per token under either codebook, gone at block 16, where FP4 matches Table 14. Table 11 shows the same for activations alone, and Table 9 almost none for weights alone at channel granularity. Table 14 extends this to every scale: AdamW and QuantRet lose alike under NVFP4, so FP quality sets the order and AdamW leads at 1.7B, while Frob and Obl stay ahead of OSP; the per-token failure, by contrast, grows with scale, as Table 1 shows.

SmoothQuant. Table 5 turned smoothing on and replaced clipped W4 ranges with unclipped ones. Table R2 completes the 2×22\times 2 at 0.6B W4A4KV4 with the same calibration set and α\alpha.

Table R2

W4 rangesmoothingAdamWOblAdamW −- Obl
clippedoff3.7513.0560.695
clippedon3.2013.0580.143
unclippedoff4.1523.1161.036
unclippedon3.0593.110−0.051-0.051

Smoothing leaves QuantRet where it was and repairs AdamW; the unclipped range costs QuantRet and, once smoothing has rescaled the weight columns, no longer costs AdamW. QuantRet leads in three of the four cells, AdamW only in the one Table 5 reports. Under GPTQ, which leaves AdamW's activation error in place, QuantRet keeps its full lead; under QuaRot, which repairs it as smoothing does, Frob and Obl keep a smaller lead and Spec falls behind.

When and why. Per-token 4-bit activations leave a large error in every token. QuantRet turns it into a smaller local output error, most of all at the readers, Appendix D.3, and its state error stays smaller through the network, Section 4.3. The constraints therefore pay off where a quantizer leaves that error in place and not where the format or a repair removes it for AdamW too.

Thank you for your detailed response. My concerns have been addressed, and I am therefore raising my score to 3.

Reviewer_j9exOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes QuantNet, which uses a hyperspherical architecture that normalizes the weight and activations to bounded hyperspheres. The authors observe that through the hyperspherical architecture, error propagation through quantization is bounded, and thus the quantized model is more robust towards 4-bit WA quantization. Authors pretrain several 0.3-1.7B scale LLMs and show that hyperspherical variants are indeed more robust to 4-bit quantization.

Critical strengths
  • The three retractions introduced in Section 2.3 seem new to the quantization domain.

  • The experiments using the Muon optimizer have not been studied in previous papers.

  • [post-rebuttal] The novelty of this work lies in combining channel-shared coefficients and Muon optimizers, and, from such a combination, the model is robust to 4bit INT WA quantization.

Critical weaknesses

I have strong objections to accepting this paper due to its novelty issues:

  • While the authors cited nGPT, as far as I understand, the architecture is the same as nGPT, including row weight/activation normalizations and learnable residual normalization (Eq. 3). I need to see a clear discussion of how it is different than nGPT instead of claiming everything as their own.
  • The study of how hyperspherical architecture influences the robustness of 4-bit WA quantization has been done in a previous year ICLR submission. The argument through error amplification/propagation is similar. I do not see the authors citing or acknowledging this. The paper is at: https://openreview.net/forum?id=tiqfxkYf1o

I will also refer to the previous submission's weaknesses raised by its reviewers here:

  • Current quantization is applied to 7B+ models. We need to see the experiment on 7B+ models to see if the conclusion still holds.

Post Rebuttal

  • There is substantial work to do revising the manuscript. We discussed several rounds, and I think the new insights on what actually contributed to the robustness of the model have changed. I believe that the three retraction variants are no longer important.
  • The comparisons are on small-scale (mainly 0.3B - 1.7B) models. The training is also on 20x token-over-size (Chinchilla optimal). Things can be different for the modern, larger-scale models that are excessively trained.
  • Overall, there are some insights in the paper that call for further large-scale validation, but we can't say anything definite now based on what is presented in the current manuscript.
Questions for authors
  • Please clarify the novelty weaknesses. [Mostly resolved]
6 comments
AuthorsOpenReview ↗

Response to Reviewer j9ex [Part 1]

Thank you for your careful review and your constructive questions. We respond to each point below.

W1 / Q1: Relation to nGPT

We agree that the core architecture is built on nGPT: the hyperspherical block of Section 2.1 is nGPT's, and the revised version says so from the abstract on. QuantRet is a pretraining recipe for quantization-robust checkpoints built on that block. The table lists what is the same and what is different, and a matched nGPT baseline follows; the reasons for each difference are in a second comment.

ComponentnGPTQuantRet
Activation normalization (hidden states, Q, K)unit norm (radius 1)same operation at radius d\sqrt{d} (RMS 1)
Residual update (Eq. 3)interpolation toward the normalized branch output, per-channel coefficientssame interpolation in nGPT's stored-coordinate parameterization, channel-shared scalars clipped to [0,1][0, 1]
Learned scalesper-channel vectors on attention scores and feed-forward pathhead-wise attention temperature, scalar gate scale; per-vocabulary logit scale kept
Weight constraint after every stepunit norm along the embedding dimension (reader rows, writer columns)Obl: same axes, writer-column radius dout/din\sqrt{d_{\mathrm{out}}/d_{\mathrm{in}}}; Frob: ∥W∥F\lVert W\rVert_F; Spec: ∥W∥2\lVert W\rVert_2; all scale-matched
OptimizerAdamW, then renormalizationambient Muon, then hard retraction to the chosen set
Vocabulary boundarynormalized tables, learned logit scalesame, plus learned orthogonal maps and stop-gradient logit scaling, fused at deployment
Quantization–error analysis (Section 3); PTQ evaluation against AdamW and OSP (Section 4)

In other words: at the level of the inference graph, QuantRet is nGPT's block up to parameter granularity and the radius convention; the differences are in what is constrained during training and how, and everything about quantization is new.

Direct comparison (matched nGPT baseline, 0.3B). We trained the baseline you and Reviewer Kij4 asked for: nGPT's block with nGPT's own training recipe, matched to our 0.3B runs in GQA shape, tied vocabulary, data and token budget, and quantized with the pipeline of Table 1. FineWeb-Edu CE / WikiText-2 PPL:

FPW4A16W4A4W4A4KV4
matched nGPT2.844 / 23.02.906 / 24.64.707 / 148.94.725 / 152.8
QuantRet-Obl (Table 4)2.859 / 23.92.910 / 25.63.123 / 32.13.132 / 32.4
Δ\Delta CE from own FP, nGPT / Obl–0.062 / 0.0511.863 / 0.2641.881 / 0.273

nGPT is slightly better in FP and ordinary at W4A16; the collapse comes from 4-bit activations, and not from the vocabulary rotation: QuantRet-Obl trained without it has Δ\Delta 0.307 at W4A4.

Where the gap comes from (per-token INT4 activation quantizer of Table 1, same validation text):

nGPTQuantRet-Obl
Normalized MSE of INT4 on the Q/K/V and gate/up inputs, mean over 20 layers0.140.015 (OSP 0.015; AdamW 0.04–0.06)
Same, o_proj / down_proj inputs0.015 / 0.140.010 / 0.15
Energy share of the largest hidden-state channel, layer 10 / 1552% / 54%0.5% / 0.4%
Per-token max-abs/RMS; excess kurtosis, layers 3–1525–27; >100>1005–6; <0.1<0.1

The nGPT hidden state keeps its norm but concentrates its direction on one coordinate; a per-token 4-bit grid whose range is set by that coordinate leaves one or two levels for the other channels, the regime that the directional analysis of Section 3 addresses. The o_proj and down_proj inputs are not the cause: they quantize about equally well for every method. Which of the differences in the table above prevents the massive channel is not isolated.

Changes in the revised version. The abstract and the introduction present QuantRet as a pretraining method built on nGPT, Section 2.1 opens with "The block follows nGPT", the first contribution is training an nGPT-like block under three alternative hidden-matrix constraints, and the matched nGPT baseline is in Table 4.

AuthorsOpenReview ↗

Response to Reviewer j9ex [Part 2]

W1, continued: why each difference exists (0.3B, FineWeb-Edu CE, single seed; Δ\Delta = quantized −- FP CE).

  1. Which matrix quantity is constrained (Section 2.3). nGPT's single rule fixes the norm of every vector along the embedding dimension. For us, which quantity to fix is the open question: Section 3 relates the constraint to the energy of a quantization perturbation and to the change it causes in a writer's normalized output direction, so Frob, Spec and Obl are three testable answers with scale-matched radii, not three spellings of nGPT. They are not interchangeable: Spec has the smallest W4A4 gap at all three scales at the highest FP CE (Table 1), and constraining every matrix row-wise instead, the all-row control of Table 4, gives W4A4 CE 3.158 against 3.123 for Obl at equal FP CE, so the axis on which the writer is constrained matters.

  2. Keeping the constraint at the quantized checkpoint (Section 2.4). nGPT also renormalizes after every step; QuantRet keeps that element but changes its target and its partner. After each ambient Muon step the hidden matrices are retracted exactly onto the chosen set, so the analysis of Section 3 applies to the checkpoint that is quantized. The ablation shows that the shared element is load-bearing under Muon and that the optimizer is not the source of robustness: removing the retraction raises W4A4 CE from 3.123 to 3.239, while replacing Muon by projected AdamW gives FP 3.030 and W4A4 3.265, a gap of 0.235 against 0.264 with Muon.

  3. Scale conventions and coefficient granularity. Radius d\sqrt{d} instead of 1 is a rescaling, not a claimed contribution; it puts hidden states, Q and K at RMS 1, so the per-token quantizers used for the AdamW and OSP baselines apply unchanged, and matrix radii follow the same convention so that the equal-energy comparison of Section 3 is meaningful. Channel-shared coefficients clipped to [0,1][0, 1] are the form in which Theorem 1 bounds the propagation of normalized quantization error as a scalar recurrence. Scalar rather than per-channel learned scales are a measured trade (Table 12): nGPT-style per-channel gate/up scales worsen W4A4 CE in every matched pair, vector residual coefficients lower FP CE but widen the FP$\to$W4A4 gap in every pair, and only the coordinate attention scales lower W4A4 CE, slightly.

  4. Quantization analysis and protocol. The directional error analysis and conditional propagation bound (Section 3), the perturbation and single-site measurements (Section 4.3), and the PTQ evaluation at 0.3B/0.6B/1.7B against AdamW and OSP (Section 4) have no counterpart in nGPT; related low-precision work on nGPT-style models is addressed in our reply to W2.

W2: The work at OpenReview tiqfxkYf1o

It is related work we should have cited. The revised version cites it in the introduction, where normalized architectures are first mentioned, and discusses it in the related work in Appendix A. That paper establishes, before us, that pretraining nGPT's hyperspherical architecture in place of a standard Transformer makes 4-bit weight-and-activation PTQ far less damaging, and argues it through error amplification; we position QuantRet relative to that result. The difference is in what is studied: that paper evaluates nGPT as released under block-scaled MXFP4/NVFP4 at 0.5–1B, whereas QuantRet asks which constraint and retraction to train with and evaluates them under per-token/per-channel INT4 at 0.3–1.7B. In that setting the matched nGPT baseline in our reply to W1 does not hold up, which is why the choice matters.

The forwarded point on 7B+ models

We ran this experiment at 8B: AdamW and QuantRet-Frob trained from scratch in the shape of Qwen3-8B, each on a complete schedule of 1.07B training tokens with one seed. The table gives FineWeb-Edu CE under the PTQ pipeline of Table 1.

8BFPW4A4W4A4 −- FP
AdamW3.0813.4080.327
QuantRet-Frob2.9103.0910.181
Reviewer_j9exOpenReview ↗

Thanks for the additional experiments but now I'm more confused.

Thanks for running the additional experiments and providing the rebuttal. I didn't ask for the 0.3B table because I know nGPT does not work with per-channel-ish INT4 WA quantization, and it is exactly because of the energy concentration problem you observed. The advantage of nGPT lies in NVFP4 (as the two papers I referred to) because NVFP4 constrains that big peak of energy in a local area so that it does not wipe out the entire channel.

Please do not report the 4B and 8B results. They can be very misleading. Training on 1B tokens does not even converge, and quantization on a non-converged model is meaningless.

But I remain very confused. My biggest confusion is why this architecture even works for INT4 WA quantization, whereas we know that no other architecture can (without PTQ tricks like Hadamard transform). Correct me if I'm wrong. You suggest that energy is not concentrated on any dimension in the architecture, but how is that possible? A residual connection, no matter how it is constructed, will carry the outlier dimension (i.e., the outlier channel observed in many previous works) to the next layers, which will then cause this energy concentration behavior. The proposed architecture magically avoids this, which leads to its success on the INT4 quantization. I think none of the differences provided in your contrast table are fundamental to this energy concentration problem, besides possibly the Muon optimizer. I guess you also cannot answer this, as the rebuttal says, "Which of the differences in the table above prevents the massive channel is not isolated."

Here is my final takeaway. If the final reason is because of the Muon optimizer, then this work is applying Muon to an nGPT-style architecture and shows that it can be better than the original OSP recipe. This is good to know but nothing fascinating, as it is just another proof of Muon's effectiveness in quantization and another proof of nGPT's effectiveness over standard transformer architecture. If it is something else, then at least for now, I cannot infer what it is from the manuscript.

I have raised my score to a 2 and have adjusted the informativeness and soundness scores accordingly. This year's scoring system is not very friendly, and I feel sorry for the authors, but I do not think this work is an accept yet without my confusion settled.

Reviewer_VaxMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • QuantRet constrains weight matrices and keep selected activations on fixed-radius spheres during pre-training. This in turn improves qunatization robustness (PTQ). They compare Frobenius, spectral, and reader/writer normalization.
  • Along wtith the activation magnitudes, the analysis studies the direction of errors after normalization and how residual mixing propagates these into changes in final output.
  • Results cover Qwen3-like models upto 1.7B, weight/activation/KV quantization, several PTQ methods are shown, and a smaller transfer experiment is demonstrated showing post-training application too.
Critical strengths
  • Empirical gains are substantial. at W4A4KV4 there is a huge ppl gain of 23.84 from 41.45 of OSP despite having worse full precision ppl. This is an useful evidence that pretraining can improve quantization robustness without explicitly quantizing the model during training.
  • Activation magnitude and local reconstruction error alone can miss what ultimately damages predictions. The perturbation studies and reader/writer ablation help make this distinction concrete.
Critical weaknesses
  • The paper states the error bounds (conditions on complete-branch sensitivity, local error, etc) conditions clearly, but does not show whether the trained models satisfy them. The evidence for the proposed explanation is therefore less direct than the evidence for improved quantized quality.
  • The practical training trade-off is only partly evaluated. Wall-clock and memory overhead from retractions and boundary rotations are not measured. With inference evaluated through fake quantization, the results currently establish quality for the tested graphs, leaving the benefit per unit of compute unclear.
Questions for authors
  • What are the per-step time and peak memory costs of each variant relative to OSP and AdamW? How should the quality gains be interpreted at a matched training-compute budget?
  • QuantRet improves sensitivity to the leading-direction perturbations but has higher KL under isotropic noise at down-projection inputs (Section 4.3). Are actual errors concentrated in the protected directions, or does subsequent propagation explain the improvement?
1 comment
AuthorsOpenReview ↗

Response to Reviewer VaxM

Thank you for your review and your constructive questions. We respond to each point below.

W1: Whether the trained models satisfy the conditions of the error bounds We would like to clarify that the conditions in your parenthesis are not conditions of the error bound: Theorem 1 holds for any model with its realized branch sensitivity and local error, and the conditions on complete-branch sensitivity, local error, output gain and the mixture norm belong to Proposition C.5, which states when QuantRet's upper bound is no larger than an additive-residual model's. To avoid this ambiguity the revised version removes this proposition and the Section 3 sentence that cites it: its conditions can only be tested against an additive-residual model that differs from QuantRet in nothing else, which AdamW and OSP are not, and it compares upper bounds that sit far above the realized errors, so it says little about the models; the realized comparison is the one in Section 4 and Table 3.

W2 and Q1: Per-step time and peak memory costs; quality gains at a matched training-compute budget Thank you for raising this point. We profiled the training step of the five 0.6B models with the paper's training configurations on four A100-40GB; the table gives the time and peak memory of one full step, retractions and boundary rotations included.

per step at 0.6BAdamWOSPFrobSpecObl
time, ms34512997332033803340
of which forward and backward34062837314031353162
of which optimizer step38152156156156
of which retraction––168115
peak memory per GPU, GiB26.322.923.923.923.9

A QuantRet step costs less than AdamW's and more than OSP's in both time and memory, and Table 1 matches tokens. The W4A4 gains do not depend on this difference in cost: both baselines at 1.7B, the largest budget in the paper, are behind every QuantRet variant at 0.3B, the smallest, in CE and in PPL.

Q2: Are actual errors concentrated in the protected directions, or does subsequent propagation explain the improvement? Your reading of Section 4.3 is right, and we measured the realized errors to answer it. They are not concentrated in the leading directions. The table below gives how much W amplifies the realized per-token A4 error relative to isotropic noise of equal energy, so 1 means isotropic. Values are means over the 28 layers, with the A4 quantizer of Table 1 applied to the FP activations.

inputAdamWOSPFrobSpecObl
down projection1.201.061.3111.651.21
attention output1.011.210.940.990.99

Every model is close to 1 except Spec at the down projection, whose inputs concentrate in those directions. Propagation alone does not explain the improvement either. When one writer input is quantized at a time, QuantRet's output KL is far lower than AdamW's in the early layers and higher in most later layers, Appendix D.4. Local errors are smaller at the readers: the local W4A4 output SNR there is higher for all three variants, Appendix D.3. Their MaxAbs/RMS at the attention inputs is also far below AdamW's. Table 1 shows where each method loses. OSP loses most from FP to W4A16 and AdamW from W4A16 to W4A4. QuantRet loses about as little as AdamW in the first step and less than both baselines in the second.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Summary of the new experiments and of the revised paper

We thank all reviewers for their comments. In response, we ran new experiments and revised the paper. This comment summarizes both; the details are in our replies to each reviewer.

Relation to nGPT. Our reply to Reviewer j9ex on W1 gives a detailed, component-by-component comparison of QuantRet with nGPT. We also trained an nGPT baseline matched to our 0.3B setting. In full precision the two models are close, with CE 2.844 for nGPT and 2.859 for QuantRet-Obl. Under W4A4 the CE of nGPT rises to 4.707 and that of QuantRet-Obl to 3.123.

Where the gain comes from. The matched nGPT baseline concentrates the energy of its residual stream in one outlier channel, which sets the range of a per-token quantizer. Two of QuantRet's differences from nGPT reduce this channel, and they act jointly: channel-shared residual coefficients and Muon on the hidden matrices. We retrained the nGPT baseline with each change alone. Each lowers its loss from FP to W4A4, and neither reaches QuantRet, which uses both. We also completed the optimizer ×\times retraction ablation inside QuantRet, and the retraction reduces the loss from FP to W4A4 under both AdamW and Muon. Please see our follow-up reply to Reviewer j9ex and our reply to Reviewer Kij4, W2 and Q2.

Deeper models. We varied the depth from 12 to 40 layers, and QuantRet-Obl has the lowest W4A4 CE at every depth. Please see our reply to Reviewer Kij4, W1 and Q1.

Training cost. We measured the time and peak memory of one training step. A QuantRet step costs less than an AdamW step and more than an OSP step. Please see our reply to Reviewer VaxM, W2 and Q1.

Changes in the revised paper. We present QuantRet as a pretraining method built on nGPT, from the abstract on. We add a matched nGPT baseline and two retrained versions of it to Table 4, one with channel-shared residual coefficients and one with Muon. In the new Section 4.5, we show that the nGPT baseline concentrates the energy of its residual stream in one outlier channel and that each of the two changes reduces this channel, and we suggest matching the granularity of the residual coefficients to that of the quantizer. We report the depth scan in Appendix D.7. We add a paragraph on limitations and future work to the conclusion. We highlight the new and rewritten text in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.