acceptodds
Under review as a conference paper at ICLR 2027

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

10 views

Abstract

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide a simple way to improve the pretraining-to-quantization transition. But the mechanism behind weight averaging remains insufficiently explained. This leads to inconsistent and fragile performance gains, thereby preventing practitioners from applying such a technique confidently. As a response, we formulate weight averaging as a trade-off between retaining training progress and improving robustness under perturbation. We further derive a continuous family of averaging kernels that unifies conventional strategies and achieves the Pareto frontier between the two competing goals. Critically, a theoretical framework for performing weight averaging under PTQ is developed. It can be shown that coarser quantization is more susceptible to perturbations, whereas finer quantization could be less affected. Thus, our results could provide unified theoretical guidance for performing weight averaging under different PTQ conditions. Experiments validate both the predicted behavior and the proposed averaging strategy.

Primary area:
optimization
Keywords:
LLM, Pretraining, PTQ, Weight Averaging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NPGQOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how weight averaging affects post-training quantization and explains the effect through a trade-off between temporal lag and spatial noise. Based on this view, the authors propose LNWA and analyze how the preferred averaging strategy changes with quantization precision. The experiments show consistent trends across different averaging methods and bit-widths.

Critical strengths
  1. The paper gives a clear explanation for the effect of weight averaging on PTQ by separating it into temporal lag and spatial noise.

  2. The paper derives a family of averaging weights from the lag–noise trade-off and connects existing methods such as LAWA, WMA, and EMA within the same framework.

  3. The observed preference shift between 2-bit and 4-bit quantization, as well as the comparison with learning-rate decay, provides consistent empirical evidence for the proposed lag–noise trade-off.

Critical weaknesses
  1. The lag–noise decomposition is insightful, but the treatment of temporal correlation between nearby checkpoints could be explained more clearly. Since these checkpoints are collected from the same training trajectory, it would be useful to clarify under what assumptions the spatial-noise term can be characterized by R(w)=∑dwd2R(w)=\sum_d w_d^2, and how sensitive the analysis is to temporal correlations.

  2. The paper uses the deviation from the latest latent state μT\mu_T to characterize temporal lag, but the connection between this distance and actual training progress is not directly validated. The authors should show whether a larger temporal lag indeed corresponds to worse full-precision loss or model quality on the checkpoints used in the experiments.

  3. Theorem 4.3 predicts a precision-dependent shift in the optimal lag–noise trade-off, but the experiments mainly demonstrate this trend qualitatively through the preference reversal between 2-bit and 4-bit quantization. It would strengthen the paper to compare the theoretically predicted optimum or critical precision with the empirical optimum observed in Figure 4.

  4. The empirical evaluation is mainly limited to OPT models up to 1.3B parameters. The authors should consider evaluating larger or more recent LLMs to further verify the generality of the conclusions.

Questions for authors
  1. How does the analysis account for the correlation between nearby checkpoints from the same training trajectory? In particular, under what conditions is the spatial-noise term R(w)=∑dwd2R(w)=\sum_d w_d^2 a good approximation?

  2. How is the LNWA hyperparameter λ\lambda selected in the experiments? Can it be determined from the theoretical quantities, or does it require a post-quantization validation sweep?

  3. The paper argues that different bit-widths prefer different averaging strategies. Can the theory predict where this change happens, and does that prediction match the experimental results?

5 comments
AuthorsOpenReview ↗

Response to Official Review of Reviewer NPGQ, Part (1/3)

The authors sincerely thank you for your valuable comments and suggestions. We have addressed all main points raised in your review. We have carefully revised and supplemented the paper accordingly.

  • We clarify the spatial noise assumption and provide empirical support for its use (W1, Q1).
  • We connect temporal lag to model quality using full-precision OPT-125M results (W2).
  • We explain how Theorem 4.3 guides the choice of λ\lambda for real-world deployments (W3, Q2, Q3).
  • We broaden the empirical evaluation with experiments on another model family and on zero-shot downstream tasks (W4).

Please see the detailed responses below.

Spatial-Noise Approximation (W1, Q1)

Thank you for asking about our spatial noise assumption. We clarify the conditions of R(w)R(\boldsymbol{w}) and present empirical evidence that the approximation is robust to the temporal correlations in practice.

To derive the spatial noise factor R(w)R(\boldsymbol{w}), we start with the retained spatial noise ∑dwdϵd\sum_d w_d\boldsymbol{\epsilon}_d, where ϵd=θT−d−μT−d\boldsymbol{\epsilon}_d=\boldsymbol{\theta}_{T-d}-\boldsymbol{\mu}_{T-d} is the spatial noise in the saved weights at age dd. With Γde=Cov⁡(ϵd,ϵe∣F0)\boldsymbol{\Gamma}_{de}=\operatorname{Cov}(\boldsymbol{\epsilon}_d,\boldsymbol{\epsilon}_e\mid\mathcal F_0), its total variance is

=\operatorname{Tr}\!\left(\sum_{d,e} w_d w_e\boldsymbol{\Gamma}_{de}\right) =\sum_d w_d^2\operatorname{Tr}(\boldsymbol{\Gamma}_{dd}) +2\sum_{d<e}w_dw_e\operatorname{Tr}(\boldsymbol{\Gamma}_{de}).$$ To simplify this variance, we use Assumption A.1, which specifies a common noise covariance $\boldsymbol{\Gamma}_{dd}=\boldsymbol{\Sigma}$ across retained weights and sets $\boldsymbol{\Gamma}_{de}=\boldsymbol{0}$ for $d\ne e$. The cross terms vanish, leaving $$V(\boldsymbol{w})=\operatorname{Tr}(\boldsymbol{\Sigma})\sum_d w_d^2 =\operatorname{Tr}(\boldsymbol{\Sigma})R(\boldsymbol{w}).$$ We consider this assumption reasonable for two reasons. First, LLMs are often pretrained for a single epoch, with each batch used once. Avoiding repeated data can substantially reduce noise correlations. Second, we save weights at large intervals, which can further reduce these correlations. We also test this assumption in the MNIST MLP and OPT-125M experiments, where the measured temporal noise correlations approach near zero as the saving interval increases (Figures 7 and 11(a) of the **revised manuscript**, respectively). To test whether the approximation is robust to temporal correlations in practice, we compare the predicted increase $R(\boldsymbol{w})-1/M$ with the measured increase over LAWA, as shown in Table 1. *Table 1. Theoretically predicted and experimentally measured increases in retained spatial noise relative to LAWA for OPT-125M.* | Kernel | $\lambda$ | Predicted noise increase | Measured noise increase | |:---|---:|---:|---:| | LAWA | $0$ | 0.000 | 0.000 | | LNWA | $0.02$ | 0.011 | 0.040 | | LNWA | $0.1$ | 0.100 | 0.176 | | LNWA | $0.4$ | 0.232 | 0.287 | For the LNWA kernels in Table 1, the measured noise increases exceed the predictions by up to $0.076$. The measured increases follow the predicted ordering in all six windows, indicating that this qualitative conclusion is insensitive to the temporal correlations observed in the experiment.
AuthorsOpenReview ↗

Response to Official Review of Reviewer NPGQ, Part (2/3)

How Temporal Lag Relates to Full-Precision Model Quality (W2)

Thank you for asking how temporal lag relates to full-precision model quality. According to our theory, reducing temporal lag can improve full-precision model quality within the tolerance interval, as shown in Table 2a. Beyond the interval, the trend reverses because the cost of additional spatial noise outweighs the benefit of reducing temporal lag (Table 2b).

We evaluate OPT-125M in full precision on FineWeb-Edu. Temporal lag is m(w)=∑ddwdm(\boldsymbol{w})=\sum_d d w_d. On the evaluated grid, FP PPL stays below the LAWA baseline through λ=0.04\lambda=0.04 and first exceeds it at λ=0.06\lambda=0.06.

Table 2a. Temporal lag and full-precision perplexity within the tolerance interval, including the LAWA baseline.

Kernelλ\lambdaTemporal lagFP PPL
LAWA004.500032.03
LNWA0.0020.0023.862631.90
LNWA0.0040.0043.383431.84
LNWA0.010.012.500031.79

Table 2b. Temporal lag and full-precision perplexity beyond the tolerance interval.

Kernelλ\lambdaTemporal lagFP PPL
LNWA0.060.061.250032.08
LNWA0.10.11.000032.25
LNWA0.40.40.555632.82
Stable LR∞\infty0.000035.58

The Rule of Choosing λ\lambda (W3, Q2, Q3)

Thank you for asking how to choose λ\lambda and whether its exact value can be computed directly from Theorem 4.3. Theorem 4.3 provides a principled guidance for how to choose λ\lambda at a target precision rather than directly derives the specific experimental setting, reducing the need for a blind search.

In our experiments, the preference generally reverses between 2-bit and 4-bit precision. Practitioners often avoid 3-bit quantization because inference kernels offer limited support for it [1]. We therefore provide practical guidance on choosing λ\lambda for 2-bit and 4-bit deployments. We recommend starting with a small λ\lambda or LAWA (λ=0\lambda=0) at 2-bit precision to favor denoising, and exploring larger λ\lambda values at 4-bit precision and above to reduce temporal lag. This guidance agrees with the PTQ choices in Figure 6 (λ=0.02\lambda=0.02 at 4-bit precision and λ=0.004\lambda=0.004 at 2-bit precision).

AuthorsOpenReview ↗

Response to Official Review of Reviewer NPGQ, Part (3/3)

More Empirical Evaluation (W4)

Thank you for suggesting that we broaden our empirical evaluation. We have expanded the empirical evaluation in two directions: validation on another model family and zero-shot downstream evaluation.

Cross-model Validation

We additionally pretrain GPT-2 models at three scales (124M, 355M, and 774M). These results have been added to Appendix C.3, “Cross-Model Validation,” and Figures 18–19 of the revised manuscript. Across all three scales, performance at 4-bit precision initially improves and then degrades as λ\lambda increases, while the λ\lambda–PPL correlations are negative at 4-bit precision and positive at 2-bit precision, consistent with the OPT results. For your convenient reference, we provide brief numerical summaries below.

Table 3. ΔNLL\Delta\mathrm{NLL} relative to LAWA for GPT-2 models under GPTQ on held-out RefinedWeb. Values are in units of 10−310^{-3}. Negative values indicate improvement; positive values indicate degradation.

Precisionλ\lambdaGPT-2 124MGPT-2 355MGPT-2 774M
4-bit0.006−2.50−4.01−4.92
4-bit1.0+34.57+37.08+22.28
2-bit0.006+48.42+59.26+55.48
2-bit1.0+184.42+356.62+128.68

Table 4. λ\lambda–PPL correlations for GPT-2 models under GPTQ. Each cell reports Pearson rr over the six averaging kernels used in Figure 19.

Evaluation dataPrecisionGPT-2 124MGPT-2 355MGPT-2 774M
RefinedWeb4-bit−0.712−0.843−0.948
RefinedWeb2-bit+0.873+0.754+0.370
FineWeb-Edu4-bit−0.957−0.909−0.937
FineWeb-Edu2-bit+0.870+0.746+0.371

Zero-shot Accuracy Validation

We have also added zero-shot evaluation on six downstream tasks for OPT-1.3B after 2-bit quantization, reported in Table 2 of the revised manuscript. LNWA outperforms Stable LR and both LR-decay baselines on all six tasks under both GPTQ and AWQ. These results show that LNWA’s robustness to low-precision quantization extends beyond perplexity to downstream task performance.

Table 5. Zero-shot accuracy (%; higher is better) of OPT-1.3B after 2-bit PTQ. Avg. is the mean over the six reported tasks. Best results within each PTQ method are in bold.

PTQMethodPIQAARC-EasyHellaSwagCOPAOBQASciQAvg.
GPTQStable LR60.1748.4829.3460.0015.0074.7047.95
GPTQLR decay (1%)60.1748.1528.7964.0016.8072.8048.45
GPTQLR decay (0%)59.3047.3528.1958.0015.4072.2046.74
GPTQLNWA63.2252.7430.4166.0019.0078.7051.68
AWQStable LR67.0360.1434.1367.0020.8079.6054.78
AWQLR decay (1%)66.6560.7734.7167.0021.2080.8055.19
AWQLR decay (0%)66.9260.8634.4662.0021.0080.0054.21
AWQLNWA68.2363.5135.4073.0023.0082.8057.66

Regarding model size, our experiments use scales comparable to those in related theoretical studies of LLM pretraining and weight averaging [2,3,4]. We chose this range to align our experimental setup with prior work and make it easier to compare findings across studies in the community. The consistent results across OPT models from 125M to 1.3B already support LNWA across model scales, while the additional GPT-2 experiments extend this evidence to another model family. We sincerely appreciate your suggestion and will further extend the experiments and analysis to larger models in future work.

[1] Liu et al. ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. NeurIPS, 2025.

[2] Wen et al. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. ICLR, 2025.

[3] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. NeurIPS, 2024.

[4] Meterez et al. Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging. arXiv, 2026.

Reviewer_t9vtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies why averaging the last M checkpoints of a constant-learning-rate LLM pretraining run helps post-training quantization (PTQ), and how the averaging kernel should be chosen.

Critical strengths

The question is well chosen and practically relevant. Prior work (Catalan-Tatjer et al., ICLR 2026) showed empirically that LAWA can beat LR decay after PTQ. The reversal of the WMA/LAWA ranking between 4-bit and 2-bit (Fig. 1a, consistent with Fig. 4) is a concrete observation that motivates a kernel-level theory. Figure 1 and 2 are well drawn and helps to guide the reader through the paper.

  • The lag-noise formulation and the LNWA family are clean and correct.
  • The closed form (Eq. 9 and 38), the support knots of Lemma A.4, and the strict monotonicity in Eq. (10) all hold.
  • The m2-scalarization recovers every budget-constrained solution;
  • The Table 1 coordinates of LAWA, WMA and EMA are exact.
  • The family gives a single interpretable knob that places LAWA, WMA, EMA and the last iterate on one curve, which is useful vocabulary for the community.
Critical weaknesses

1- Gap to the experiments. In the experiments is the main weakness in my opinion, T ≈ 15.3k steps / 500 ≈ 30, and the observed W4 tolerance region in Fig. 3 extends to λ ≈ 0.03–0.07, i.e. x ≈ 0.64–0.74. That lies beyond the first support segment (x ≤ 11/27 ≈ 0.41, λ ≤ 1/120). The proofs place τ_b and x_b inside that segment for large T (l. 1866–1868; Lemma A.14). So the experiments are not in the regime the theorems describe, and the claimed agreement is only qualitative. Neither T0 (Lemma A.11), b_crit nor τ_b is ever estimated for the trained models.

I recommend the authors to resolve this perhaps by

  • a scaling in which lag and noise are comparable (e.g. σ² = Θ(1/T²)), so that τ_b and b_crit have nondegenerate limits; or
  • a finite-T statement with an explicit T0, together with estimates of b_crit and τ_b from the OPT runs, compared with where the reversal is actually observed (between W3 and W2 for GPTQ-125M).

2- The empirical validation is qualitative, and its statistical support is weak. Single runs. No seeds or variances are reported for any OPT experiment. This matters all the more because the paper's subject is run-to-run noise. Small effects. At W4 the whole effect being explained is under 1% PPL: 49.40–49.83 for GPTQ-125M in Fig. 4, and 26.10–26.41 for AWQ-1.3B in Fig. 13.

Weak correlations. The correlation evidence uses n = 6 kernels per panel. Several of the correlations cited as support are not significant at conventional levels (two-sided Pearson tests):

GPTQ-125M W4: r = −0.716, p ≈ 0.11; GPTQ-125M W2: r = +0.756, p ≈ 0.08; GPTQ-1.3B W2: r = +0.364, p ≈ 0.48 (Fig. 12); AWQ-350M W2: r = +0.494, p ≈ 0.32 (Fig. 13).

The claim at l. 2422–2427 ("At every model scale, PPL is positively correlated … at W2 and negatively … at W4 under both methods") is true for the signs but not for their reliability.

Questions for authors

-Can you estimate the quantities in Eq. (23) (µ̄^⊤u, u^⊤G(µ_T − µ̄), c_G) from the OPT-125M run, and check whether the predicted b_crit lies between 2 and 4 bits for GPTQ, where the observed reversal occurs? Even a rough agreement would greatly strengthen the paper.

-Does the w-dependence of the quantization term survive deterministic rounding with per-group min-max scales? A direct measurement of the actual GPTQ/AWQ quantization error as a function of R(w) and ‖µ_T^w‖ on the trained models would answer this.

-How many seeds were run for the OPT experiments? Can you add error bars to Figs. 3–4, and test the λ–PPL correlations across seeds?

-For the OPT runs, is A.9 satisfied? That is, does the weight norm ‖θ_t‖ increase over the averaging window under AdamW with weight decay 0.1?

3 comments
AuthorsOpenReview ↗

Response to Official Review of Reviewer t9vt, Part (1/3)

The authors sincerely thank you for your valuable comments and suggestions. We have addressed all main points raised in your review. We have carefully revised and supplemented the paper accordingly.

  • We analyze a joint scaling of lag and noise following your advice and show how the tolerance interval can extend across support segments (W1).
  • We repeat the OPT quantization experiments across seeds and report variation in the tolerance and λ\lambda–PPL results (W2, Q3).
  • We explain the difficulty of estimating the precision threshold for practical GPTQ and give guidance on choosing λ\lambda for real-world deployments (Q1).
  • We measure weight norms, quantization scales, reconstruction error, and perplexity across OPT averaging kernels to assess the mechanism under groupwise GPTQ (Q2, Q4).
  • We assess the key assumptions on OPT-125M with different data orders (Q4).

Please see the detailed responses below.

Gap to the Experiments (W1)

Thank you for pointing out a potential limitation of taking T→∞T\to\infty in our theoretical analysis. The OPT experiments necessarily use finite training budgets due to computational constraints, so some differences from the asymptotic results are expected. Asymptotic analyses of T→∞T\to\infty are nevertheless well established in the community[1, 2]. Moreover, practical LLM pretraining often uses substantially larger token budgets, including in overtraining regimes [3], which further motivates studying the behavior at large TT.

We appreciate your suggestion to keep temporal lag and spatial noise comparable, and we explore this possibility under the following scaling.

\qquad \max_{0\le d < M} \left| \lVert \boldsymbol{\mu}_{T-d}\rVert_2^2-\lVert \boldsymbol{\mu}_T\rVert_2^2 \right| \le\frac{C_\mu}{T^2},$$ where $\sigma_0^2$ and $C_\mu$ are independent of $T$. Together with the existing local regularity assumption, this scaling keeps the initial error slope and curvature comparable: $$\mathcal E_b'(0^+)=O(T^{-2}), \qquad \mathcal E_b''(0^+)=\Theta(T^{-2}).$$ The quadratic expression on the first support segment gives $$\mathcal E_b(x_{\mathrm F})-\mathcal E_b(0)=x_{\mathrm F} \left[\mathcal E_b'(0^+)+\frac{x_{\mathrm F}}2\mathcal E_b''(0^+)\right],$$ where $x_{\mathrm F}=(M+1)/[3(M-1)]$ denotes the endpoint of the first support segment. The slope term is at most of order $T^{-2}$, and the curvature term is of order $T^{-2}$, so the sign of their sum depends on their relative magnitudes. When their sum is negative, the error remains below its initial level at $x_{\mathrm F}$, and continuity ensures that the tolerance interval extends beyond this boundary. Under this scaling, the tolerance interval can extend beyond the first support segment even for large $T$. Your suggestion offers an elegant way to connect the asymptotic analysis with the results at finite training budgets. In future work, we plan to study this scaling in more detail and derive conditions under which the tolerance interval extends across support segments.
AuthorsOpenReview ↗

Response to Official Review of Reviewer t9vt, Part (2/3)

Statistical Support of Empirical Validation (W2, Q3)

Thank you for suggesting that we check the OPT results across seeds and quantify their variability. The original OPT PTQ experiments used one seed. We have extended the experiments to four seeds, keeping the training checkpoints, quantization settings, and evaluation data fixed. In the revised manuscript, we have added these results to the subsection “Multi-seed Validation,” including multi-seed versions of Figs. 3–4 with error bars and shaded bands. The main conclusions of the paper continue to hold across these seeds. And we provide a subset of the GPTQ results below for your convenient reference.

Table 1. Paired ΔNLL\Delta\mathrm{NLL} relative to LAWA under GPTQ on held-out RefinedWeb. Values are in units of 10−310^{-3}. Negative values indicate improvement; positive values indicate degradation. The last column reports the mean ±\pm sample standard deviation.

Precisionλ\lambdaSeed 1Seed 2Seed 3Seed 4Mean ±\pm SD
4-bit0.008−3.15−4.22−2.81−1.45−2.91 ± 1.14
4-bit1.0+36.72+37.11+37.97+40.72+38.13 ± 1.81
2-bit0.008+79.45+59.69+29.28+52.27+55.17 ± 20.73
2-bit1.0+195.73+208.36+206.87+266.43+219.34 ± 31.89

Across the four seeds, moderate adjustment improves performance at 4-bit precision, whereas a larger adjustment leads to degradation. The same moderate adjustment already degrades performance at 2-bit precision.

Table 2. λ\lambda–PPL correlations under GPTQ. Each seed cell reports Pearson rr. The last column gives the mean and sample standard deviation across seeds.

Evaluation dataPrecisionSeed 1Seed 2Seed 3Seed 4Mean rr ±\pm SD
RefinedWeb4-bit−0.716−0.750−0.717−0.781−0.741 ± 0.031
RefinedWeb2-bit+0.756+0.880+0.659+0.620+0.729 ± 0.116
FineWeb-Edu4-bit−0.930−0.933−0.930−0.932−0.931 ± 0.002
FineWeb-Edu2-bit+0.717+0.861+0.681+0.745+0.751 ± 0.078

The correlation direction is consistent across the four seeds on both datasets: negative at 4-bit precision and positive at 2-bit precision. This supports the observed precision-dependent preference for reducing temporal lag at higher precision and stronger denoising at lower precision.

Where the Observed Reversal Occurs (Q1)

Thank you for asking how our framework can guide the choice of λ\lambda at a target precision. The main difficulty in directly estimating bcritb_{\mathrm{crit}} is that cGc_G is defined under our simplified global-scale quantization model and is difficult to compute precisely in practical groupwise GPTQ. Reliable estimation therefore requires additional modeling and calibration. This gap is common in the training theory literature, where idealized models often provide insights and guide experimental choices at a qualitative level [2, 4], with specific numerical settings depending on the model and experimental setup. Thus, Theorem 4.3 primarily provides qualitative insight and guidance for kernel selection, and the predicted shift in preference is consistent with our experimental results.

In our experiments, the preference generally reverses between 2-bit and 4-bit precision. Practitioners often avoid 3-bit quantization because inference kernels offer limited support for it [5]. We therefore provide practical guidance on choosing λ\lambda for 2-bit and 4-bit deployments. We recommend starting with a small λ\lambda or LAWA (λ=0\lambda=0) at 2-bit precision to favor denoising, and exploring larger λ\lambda values at 4-bit precision and above to reduce temporal lag. This guidance agrees with the practical PTQ choices in Figure 6 (λ=0.02\lambda=0.02 at 4-bit precision and λ=0.004\lambda=0.004 at 2-bit precision).

AuthorsOpenReview ↗

Response to Official Review of Reviewer t9vt, Part (3/3)

Weight Norm and Quantization Scales (Q2, Q4)

Thank you for asking how OPT weight norms vary and whether the quantization error still depends on the averaging kernel in practice. Regarding Q4, yes, Table 3 shows increasing weight norms across averaging kernels from the original OPT-125M run. Our decomposition gives Q(w)=E∥θTw∥22=∥μTw∥22+Pσ2R(w)\mathcal Q(w)=\mathbb E\lVert \theta_T^w\rVert_2^2=\lVert \mu_T^w\rVert_2^2+P\sigma^2R(w). Under Assumptions A.8 and A.9, the first term is nondecreasing, while the second increases with λ\lambda through R(w)R(w), so the weight norm increases. The measured checkpoint norms show the corresponding trend. This shows that the weight-norm growth predicted by the simplified GPTQ model is also observed in our OPT experiments.

Regarding Q2, yes, the kernel dependence persists under deterministic rounding with per-group min–max scales. In our simplified GPTQ model, higher parameter energy leads to larger scales and a higher quantization error at fixed precision through s(w)2=Q(w)/Ps(w)^2=\mathcal Q(w)/P and Eb(w)−P0(w)=cGQ(w)/(2b−1)2\mathcal E_b(w)-\mathcal P_0(w)=c_G\mathcal Q(w)/(2^b-1)^2. Our actual measurements with 2-bit GPTQ show corresponding overall increases in groupwise scales, quantization error, and PTQ perplexity, as reported in Table 3. Thus, the energy–scale–reconstruction relationship predicted by the simplified model is also observed under practical groupwise GPTQ. Here, S=N−1∑gngsg2S=N^{-1}\sum_g n_gs_g^2 summarizes the actual groupwise scales.

Table 3. Weight norms, groupwise scales, reconstruction error, and perplexity for OPT-125M under 2-bit GPTQ. Each column corresponds to an averaging coefficient. Output MSE is reported as a percentage increase relative to LAWA.

Measurementλ=0\lambda=0 (LAWA)0.0010.0010.010.010.10.111
Weight norm ∥θTwλ∥2\lVert \theta_T^{w_\lambda}\rVert_2835.379838.769859.116879.192888.719
Groupwise scales SS (×10−2\times10^{-2})2.0362.0962.1192.1672.212
quantization error increase (%)0.002.184.1110.9613.46
2-bit PPL73.9476.2675.8982.5789.92

The full mechanism measurements and OPT assumption-validation results have been added to Figures 5 and 11 of the revised manuscript, respectively.

Empirical Validation of Key Assumptions (Q4)

Thank you for asking whether Assumption A.9 is supported by the OPT experiments. We have also extended the assumption validation to OPT-125M in the revised manuscript (Figure 11). Using the original training configuration, we analyze the Q, K, V, FC1, and FC2 weights and biases in the last decoder layer. Figure 11(a) shows that temporal noise correlations decrease with the saving interval and become weak at larger intervals. Figure 11(b) shows that the first ten fitted directions capture only a small fraction of the held-out noise variance, consistent with spatial-noise dispersion. Figure 11(c) shows positive inner-product estimates for all 36 increment pairs in each module. Figure 11(d) shows positive alignment with the window-start latent mean in most cases. These results provide empirical support for our key assumptions to LLM training.

[1] Oleg Filatov, Jan Ebert, Jiangtao Wang, and Stefan Kesselheim. Time Transfer: On Optimal Learning Rate and Batch Size in the Infinite Data Limit. arXiv, 2024.

[2] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A Dynamical Model of Neural Scaling Laws. ICML, 2024.

[3] Samir Yitzhak Gadre et al. Language Models Scale Reliably with Over-Training and on Downstream Tasks. ICLR, 2025.

[4] Wen et al. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. ICLR, 2025.

[5] Liu et al. ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. NeurIPS, 2025.

Reviewer_XgKbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies weight averaging at the end of stable learning rate pretraining as a tradeoff between temporal lag, measured by the mean age of the averaged weights, and spatial noise, measured by the squared norm of the averaging kernel. Minimizing the spatial noise plus λ times the squared temporal lag gives LNWA, a closed form kernel family that assigns linearly decaying coefficients to the most recent saved weights and runs from LAWA through WMA to the Stable endpoint, the last saved weight. Using a simplified GPTQ model with randomized dither rounding and a single global RMS scale, the paper shows a tolerance to degradation transition as the kernel shifts toward more recent weights, and a precision threshold at or below which LAWA is optimal. Experiments on OPT models from 125M to 1.3B pretrained from scratch and quantized with GPTQ and AWQ at 4, 3, and 2 bits report perplexity trends consistent with these predictions.

Critical strengths
  • LNWA places LAWA, WMA, and the Stable endpoint on one frontier controlled by a single coefficient λ, with EMA as an approximation, and describes each kernel by two lag noise coordinates, the mean age and the squared norm of the averaging kernel.

  • The paper shows that the preferred averaging kernel depends on the target precision. Across three model sizes and both GPTQ and AWQ, larger averaging coefficients lower perplexity at 4 bit and raise it at 2 bit.

Critical weaknesses
  • In the simplified GPTQ model, the paper uses randomized dither rounding and a single global scale computed from the RMS magnitude instead of groupwise scales, so the reconstruction error introduced by quantization depends only on the parameter energy. GPTQ and AWQ in the experiments use groupwise scales, and the paper does not measure the parameter energy or the group scales across averaging kernels to show that this mechanism explains the measured perplexity. The final Hessian trace in Figure 15 also shows lower local curvature for the averaged kernels, which the theory does not capture because the calibration matrix is fixed.

  • The paper explains the precision dependent reversal in preference as lower precision favoring less spatial noise, but in the theory this reversal is not decided by the spatial noise. At LAWA, spatial noise has no first order contribution, so whether to shift toward more recent saved weights depends only on whether the decrease in the unquantized error offsets the increase in the quantization term from the latent mean norm. Accordingly, the precision threshold in Theorem 4.3 has no spatial noise term and becomes zero when the latent mean norm does not increase, and this increase is assumed rather than measured on the OPT models.

  • The experimental validation is limited in scale and scope. The models are OPT models up to 1.3B parameters and the evaluation uses only perplexity, without zero shot task accuracy, so it is unclear whether the tolerance to degradation transition and the precision dependent preference hold for larger models or beyond perplexity. The key assumptions are also assessed only on a two layer ReLU MLP on MNIST because rerunning LLM pretraining is costly, which does not establish that they hold in LLM pretraining, the setting the theory is meant to explain.

Questions for authors
  • How do the parameter energy and the groupwise quantization scales of the averaged OPT checkpoints change across averaging coefficients, and do these changes track the PTQ perplexity differences? This would show whether the mechanism in the simplified GPTQ model is the one at work in GPTQ and AWQ.

  • Does the latent mean norm actually increase toward more recent saved weights in the OPT models trained with AdamW and weight decay, for example measured with a few additional data order runs at 125M? Can the precision threshold in Theorem 4.3 be estimated for these models and compared with the bit width at which the reversal is observed? Without such an estimate, what concrete rule does the framework give for choosing λ at a target precision?

  • Do the tolerance to degradation transition and the precision dependent preference hold for larger models and in zero shot accuracy on commonsense reasoning tasks, in addition to perplexity?

6 comments
AuthorsOpenReview ↗

Response to Official Review of Reviewer XgKb, Part (1/3)

The authors sincerely thank you for your valuable comments and suggestions. We have addressed all main points raised in your review. We have carefully revised and supplemented the paper accordingly.

  • We explain the use of a global quantization scale and clarify how the fixed PTQ calibration matrix is different from the training-loss Hessian (W1).
  • We clarify the role of spatial noise in the precision-dependent reversal and provide empirical evidence of latent-mean norm growth in OPT (W2).
  • We broaden the empirical evaluation with experiments on another model family and on zero-shot downstream tasks (W3, Q3).
  • We measure parameter energy, groupwise scales, reconstruction error, and perplexity across averaging kernels to assess the proposed mechanism under groupwise GPTQ (Q1).
  • We assess the key assumptions on OPT-125M with different data orders (Q2, W3).
  • We explain how Theorem 4.3 guides the choice of λ\lambda for real-world deployments (Q2).

Please see the detailed responses below.

Scale and Calibration Matrix Assumptions (W1)

Thank you for your question about our use of a global scale and a fixed calibration matrix to simplify the theoretical analysis. Analyzing quantization on a shared grid is a well established approach in PTQ theory [1,2]. We adopt this convention to keep our theoretical formulation consistent with prior work in the community and provide a common basis for comparison. We have also added empirical validation following your advice showing that the parameter energy and the quantization scales predicted by this simplified model is consistent with practical groupwise GPTQ; for details, please see the response to Parameter Energy and Quantization Scales (Q1) below.

Regarding the concern that a fixed calibration matrix cannot capture the differences in final Hessian traces shown in Figure 15 (corresponding to Figure 17 in the revised manuscript), we note that the measured Hessian and the calibration matrix correspond to different objectives. Figure 15 measures the cross-entropy Hessian of the FP model during LLM training, HCE(θ)=∇θ2LCE(θ).H_{\mathrm{CE}}(\theta)=\nabla_\theta^2\mathcal L_{\mathrm{CE}}(\theta). In contrast, our calibration matrix is defined by reference layer inputs XX during PTQ:

\qquad r(z)=\frac{1}{2n}\lVert z^\top X\rVert_2^2 =\frac12 z^\top Gz.$$ For fixed $X$, $G$ is the Hessian of this layer-output reconstruction loss. This fixed-input formulation is different from the Hessian in LLMs training and widely used in PTQ [1,3]. Thus, our theory uses the same Hessian $G$ defined by the input data across averaging kernels; it does not assume that their cross-entropy Hessians $H$ are identical. The lower Hessian trace of the averaged models is consistent with weight averaging placing the parameters in flatter regions of the pretraining loss landscape, as observed in SWA [4]. This empirical observation therefore does not contradict the fixed calibration matrix used in our PTQ analysis. ## How Spatial Noise Affects Precision Dependent Reversal (W2) Thank you for your valuable comment. You ask how spatial noise affects the precision-dependent reversal and whether the latent-mean norm increases in OPT, since the precision threshold could otherwise be zero. We would like to clarify that spatial noise directly affects the precision-dependent preference. At lower precision, the same increase in spatial noise leads to a larger quantization penalty. For example, consider two kernels with spatial noise factors $R_2>R_1$. Lower precision uses a coarser quantization grid. The contribution of the noise difference to the quantization error is $$\underbrace{\sigma^2(R_2-R_1)}_{\text{additional noise}} \times \underbrace{\frac{Pc_G}{(2^b-1)^2}}_{\text{quantization penalty of noise}}.$$ For the same pair of kernels, going from 4-bit to 2-bit quantization increases the quantization penalty of spatial noise by a factor of $(15/3)^2=25$ in our model. This makes the kernel that retains more noise relatively less favorable at lower precision. To address your concern about the latent-mean norm, we added an experiment on OPT-125M following your advice. Table 1 shows that the estimated latent-mean norm increases over the averaging window, supporting a nonzero precision threshold. *Table 1. Estimated latent-mean norm in the OPT-125M averaging window. At each step, we average the Q, K, V, FC1, and FC2 weights and biases in the last decoder layer across 16 branches and take the Euclidean norm.* | Optimizer step | 14358 | 14558 | 14758 | 14958 | 15158 | 15258 | |:---|---:|---:|---:|---:|---:|---:| | Estimated latent-mean norm | 170.119 | 170.327 | 170.540 | 170.756 | 170.978 | 171.090 |
AuthorsOpenReview ↗

Response to Official Review of Reviewer XgKb, Part (2/3)

More Empirical Evaluation (W3, Q3)

Thank you for suggesting that we broaden our empirical evaluation. We have expanded the empirical evaluation in two directions: validation on another model family and zero-shot downstream evaluation.

Cross-model Validation

We additionally pretrain GPT-2 models at three scales (124M, 355M, and 774M). These results have been added to Appendix C.3, “Cross-Model Validation,” and Figures 18–19 of the revised manuscript. Across all three scales, performance at 4-bit precision initially improves and then degrades as λ\lambda increases, while the λ\lambda–PPL correlations are negative at 4-bit precision and positive at 2-bit precision, consistent with the OPT results. For your convenient reference, we provide brief numerical summaries below.

Table 2. ΔNLL\Delta\mathrm{NLL} relative to LAWA for GPT-2 models under GPTQ on held-out RefinedWeb. Values are in units of 10−310^{-3}. Negative values indicate improvement; positive values indicate degradation.

Precisionλ\lambdaGPT-2 124MGPT-2 355MGPT-2 774M
4-bit0.006−2.50−4.01−4.92
4-bit1.0+34.57+37.08+22.28
2-bit0.006+48.42+59.26+55.48
2-bit1.0+184.42+356.62+128.68

Table 3. λ\lambda–PPL correlations for GPT-2 models under GPTQ. Each cell reports Pearson rr.

Evaluation dataPrecisionGPT-2 124MGPT-2 355MGPT-2 774M
RefinedWeb4-bit−0.712−0.843−0.948
RefinedWeb2-bit+0.873+0.754+0.370
FineWeb-Edu4-bit−0.957−0.909−0.937
FineWeb-Edu2-bit+0.870+0.746+0.371

Zero-shot Accuracy Validation

We have also added zero-shot evaluation on 6 downstream tasks for OPT-1.3B after 2-bit quantization, reported in Table 2 of the revised manuscript. LNWA outperforms Stable LR and both LR-decay baselines on all six tasks under both GPTQ and AWQ. These results show that LNWA’s robustness to low-precision quantization extends beyond perplexity to downstream task performance.

Table 4. Zero-shot accuracy (%; higher is better) of OPT-1.3B after 2-bit PTQ. Avg. is the mean over the six reported tasks. Best results within each PTQ method are in bold.

PTQMethodPIQAARC-EasyHellaSwagCOPAOBQASciQAvg.
GPTQStable LR60.1748.4829.3460.0015.0074.7047.95
GPTQLR decay (1%)60.1748.1528.7964.0016.8072.8048.45
GPTQLR decay (0%)59.3047.3528.1958.0015.4072.2046.74
GPTQLNWA63.2252.7430.4166.0019.0078.7051.68
AWQStable LR67.0360.1434.1367.0020.8079.6054.78
AWQLR decay (1%)66.6560.7734.7167.0021.2080.8055.19
AWQLR decay (0%)66.9260.8634.4662.0021.0080.0054.21
AWQLNWA68.2363.5135.4073.0023.0082.8057.66

Regarding model size, our experiments use scales comparable to those in related theoretical studies of LLM pretraining and weight averaging [5,6,7]. We chose this range to align our experimental setup with prior work and make it easier to compare findings across studies in the community. The consistent results across OPT models from 125M to 1.3B already support LNWA across model scales, while the additional GPT-2 experiments extend this evidence to another model family. We sincerely appreciate your suggestion and will further extend the experiments and analysis to larger models in future work.

AuthorsOpenReview ↗

Response to Official Review of Reviewer XgKb, Part (3/3)

Parameter Energy and Quantization Scales (Q1)

Thank you for asking how the parameter energy and groupwise quantization scales of the averaged OPT checkpoints change with the averaging coefficient and whether these changes track differences in PTQ perplexity. We have added a mechanism-validation experiment on OPT-125M under 2-bit GPTQ to the revised manuscript (Figure 5). Across the saved averaging kernels, we measure parameter energy E=N−1∑ℓ∥θℓ∥F2E=N^{-1}\sum_\ell\lVert \theta_\ell\rVert_F^2, groupwise scales summarized by S=N−1∑gngsg2S=N^{-1}\sum_g n_gs_g^2, output reconstruction MSE, and PTQ perplexity. Here, θℓ\theta_\ell denotes the FP weight matrices being quantized, sgs_g is the actual quantization scale of group gg, and ngn_g is its parameter count.

We provide a subset of the GPTQ results below for your convenient reference. Energy and MSE are reported as percentage increases relative to LAWA.

Table 5. Parameter energy, groupwise scales, output reconstruction MSE, and perplexity across averaging coefficients for OPT-125M under 2-bit GPTQ.

Measurementλ=0\lambda=0 (LAWA)0.0010.0010.010.010.10.111
Parameter energy increase (%)0.000.433.637.098.94
Groupwise scales SS (×10−2\times10^{-2})2.0362.0962.1192.1672.212
Output MSE increase (%)0.002.184.1110.9613.46
2-bit PPL73.9476.2675.8982.5789.92

As the averaging kernel favors more recent weights, parameter energy and groupwise scales increase overall, accompanied by larger reconstruction error and higher PTQ perplexity. This follows the relationship in our simplified model: increased parameter energy is associated with larger quantization scales and a higher reconstruction error. This relationship with actual groupwise scales and quantized weights connects the simplified mechanism to practical GPTQ.

Empirical Validation of Key Assumptions (Q2, W3)

Thank you for suggesting that we extend our qualitative assessment of the assumptions to OPT-125m model. We have also extended the assumption validation to OPT-125M in the revised manuscript (Figure 11). Using the original training configuration with AdamW and weight decay 0.10.1 following your advice. We analyze the Q, K, V, FC1, and FC2 weights and biases in the last decoder layer.

Figure 11(a) shows that temporal noise correlations decrease with the saving interval and become weak at larger intervals. Figure 11(b) shows that the first ten fitted directions capture only a small fraction of the held-out noise variance, consistent with spatial-noise dispersion. Figure 11(c) shows positive inner-product estimates for all increment pairs in each module. Figure 11(d) shows positive alignment with the window-start latent mean in most cases. These results provide empirical support for our key assumptions to LLM training.

The Rule of Choosing λ\lambda (Q2)

Thank you for asking how to choose λ\lambda and whether its exact value can be computed directly from Theorem 4.3. Theorem 4.3 provides a principled guidance for how to choose λ\lambda at a target precision rather than directly derives the specific experimental setting, reducing the need for a blind search.

In our experiments, the preference generally reverses between 2-bit and 4-bit precision. Practitioners often avoid 3-bit quantization because inference kernels offer limited support for it [8]. We therefore provide practical guidance on choosing λ\lambda for 2-bit and 4-bit deployments. We recommend starting with a small λ\lambda or LAWA (λ=0\lambda=0) at 2-bit precision to favor denoising, and exploring larger λ\lambda values at 4-bit precision and above to reduce temporal lag. This guidance agrees with the PTQ choices in Figure 6 (λ=0.02\lambda=0.02 at 4-bit precision and λ=0.004\lambda=0.004 at 2-bit precision).

[1] Frantar and Alistarh. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. NeurIPS, 2022.

[2] Chee et al. QuIP: 2-Bit Quantization of Large Language Models With Guarantees. NeurIPS, 2023.

[3] Frantar et al. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.

[4] Izmailov et al. Averaging Weights Leads to Wider Optima and Better Generalization. arXiv, 2018.

[5] Wen et al. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. ICLR, 2025.

[6] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. NeurIPS, 2024.

[7] Meterez et al. Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging. arXiv, 2026.

[8] Liu et al. ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. NeurIPS, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the AC

Dear Area Chair,

We sincerely thank you for your hard work and contributions to the community. As the rebuttal period comes to an end, we summarize the main concerns, our responses, and the reviewers’ feedback below for your review.

1. Reviewer XgKb

  • Applicability of the simplified PTQ model to groupwise PTQ: We clarified the calibration-matrix assumption and measured parameter energy, groupwise scales, reconstruction error, and perplexity under practical groupwise GPTQ. See Part 1 (W1); Part 3 (Q1).
  • Validity of the assumptions in LLM training: We added OPT experiments examining noise correlations, spatial dispersion, and progress alignment to validate our assumptions. See Part 1 (W2); Part 3 (Q2, W3).
  • Generality across models and downstream tasks: We added GPT-2 experiments at 124M, 355M, and 774M, together with zero-shot evaluations on six downstream tasks. See Part 2 (W3, Q3).

The reviewer stated that our rebuttal “fully resolves all the concerns raised in my review,” but requested newer model families and 7B/8B experiments. We therefore used published checkpoints of Olmo 3 7B and completed 72 PTQ evaluations at 4-bit and 2-bit precision. The results reproduce both theoretical predictions. See our follow-up “Scaling-up Validation,” Tables 1–2. Although the reviewer has not yet responded to our update, we believe this additional experiment can meet their expectations.

2. Reviewer t9vt

  • Statistical support for the experimental results: We repeated PTQ across four seeds and reported means, standard deviations, and error bars. The correlation signs remain consistent across all four seeds. See Part 2 (W2, Q3).
  • How λ\lambda is selected: We explained the theory-guided preference for smaller λ\lambda at 2-bit precision and larger values at 4-bit precision, with experimental choices. See Part 2 (Q1).
  • Whether the quantization mechanism holds with groupwise scales: We measured weight norms, actual groupwise scales, reconstruction errors, and perplexity across averaging kernels. See Part 3 (Q2, Q4).
  • Whether the assumptions hold in LLM training: We added OPT-125M experiments examining the noise and training-progress assumptions. See Part 3 (Q4).

The reviewer highlighted the importance of our work in the original review, writing, “The question is well chosen and practically relevant.”

3. Reviewer NPGQ

  • Sensitivity to temporal noise correlations: We clarified the noise assumptions and compared predicted and measured increases in retained noise on OPT-125M. See Part 1 (W1, Q1).
  • Connection between temporal lag and model quality: We added full-precision OPT-125M evaluations. PPL improves as lag decreases within the tolerance interval, then worsens at larger λ\lambda. See Part 2 (W2).
  • Selection of λ\lambda: We clarified how Theorem 4.3 guides kernel selection at different precisions and explained the experimental choices. See Part 2 (W3, Q2, Q3).
  • Generality beyond OPT: We added experiments at three GPT-2 scales and zero-shot evaluations on six downstream tasks. See Part 3 (W4).

We addressed all the concerns raised in their original review. The reviewer said our rebuttal “helped address my concerns and strengthen the empirical support for the proposed mechanism.” They remained positive about our contributions, writing, “I remain positive about its contributions.” We also clarified the experimental details in response to their two further questions.

Thank you again for reviewing our paper and the rebuttal discussion.

Best wishes,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.