- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how weight averaging affects post-training quantization and explains the effect through a trade-off between temporal lag and spatial noise. Based on this view, the authors propose LNWA and analyze how the preferred averaging strategy changes with quantization precision. The experiments show consistent trends across different averaging methods and bit-widths.
Critical strengths
-
The paper gives a clear explanation for the effect of weight averaging on PTQ by separating it into temporal lag and spatial noise.
-
The paper derives a family of averaging weights from the lag–noise trade-off and connects existing methods such as LAWA, WMA, and EMA within the same framework.
-
The observed preference shift between 2-bit and 4-bit quantization, as well as the comparison with learning-rate decay, provides consistent empirical evidence for the proposed lag–noise trade-off.
Critical weaknesses
-
The lag–noise decomposition is insightful, but the treatment of temporal correlation between nearby checkpoints could be explained more clearly. Since these checkpoints are collected from the same training trajectory, it would be useful to clarify under what assumptions the spatial-noise term can be characterized by , and how sensitive the analysis is to temporal correlations.
-
The paper uses the deviation from the latest latent state to characterize temporal lag, but the connection between this distance and actual training progress is not directly validated. The authors should show whether a larger temporal lag indeed corresponds to worse full-precision loss or model quality on the checkpoints used in the experiments.
-
Theorem 4.3 predicts a precision-dependent shift in the optimal lag–noise trade-off, but the experiments mainly demonstrate this trend qualitatively through the preference reversal between 2-bit and 4-bit quantization. It would strengthen the paper to compare the theoretically predicted optimum or critical precision with the empirical optimum observed in Figure 4.
-
The empirical evaluation is mainly limited to OPT models up to 1.3B parameters. The authors should consider evaluating larger or more recent LLMs to further verify the generality of the conclusions.
Questions for authors
-
How does the analysis account for the correlation between nearby checkpoints from the same training trajectory? In particular, under what conditions is the spatial-noise term a good approximation?
-
How is the LNWA hyperparameter selected in the experiments? Can it be determined from the theoretical quantities, or does it require a post-quantization validation sweep?
-
The paper argues that different bit-widths prefer different averaging strategies. Can the theory predict where this change happens, and does that prediction match the experimental results?
5 comments
Response to Official Review of Reviewer NPGQ, Part (2/3)
How Temporal Lag Relates to Full-Precision Model Quality (W2)
Thank you for asking how temporal lag relates to full-precision model quality. According to our theory, reducing temporal lag can improve full-precision model quality within the tolerance interval, as shown in Table 2a. Beyond the interval, the trend reverses because the cost of additional spatial noise outweighs the benefit of reducing temporal lag (Table 2b).
We evaluate OPT-125M in full precision on FineWeb-Edu. Temporal lag is . On the evaluated grid, FP PPL stays below the LAWA baseline through and first exceeds it at .
Table 2a. Temporal lag and full-precision perplexity within the tolerance interval, including the LAWA baseline.
| Kernel | Temporal lag | FP PPL | |
|---|---|---|---|
| LAWA | 4.5000 | 32.03 | |
| LNWA | 3.8626 | 31.90 | |
| LNWA | 3.3834 | 31.84 | |
| LNWA | 2.5000 | 31.79 |
Table 2b. Temporal lag and full-precision perplexity beyond the tolerance interval.
| Kernel | Temporal lag | FP PPL | |
|---|---|---|---|
| LNWA | 1.2500 | 32.08 | |
| LNWA | 1.0000 | 32.25 | |
| LNWA | 0.5556 | 32.82 | |
| Stable LR | 0.0000 | 35.58 |
The Rule of Choosing (W3, Q2, Q3)
Thank you for asking how to choose and whether its exact value can be computed directly from Theorem 4.3. Theorem 4.3 provides a principled guidance for how to choose at a target precision rather than directly derives the specific experimental setting, reducing the need for a blind search.
In our experiments, the preference generally reverses between 2-bit and 4-bit precision. Practitioners often avoid 3-bit quantization because inference kernels offer limited support for it [1]. We therefore provide practical guidance on choosing for 2-bit and 4-bit deployments. We recommend starting with a small or LAWA () at 2-bit precision to favor denoising, and exploring larger values at 4-bit precision and above to reduce temporal lag. This guidance agrees with the PTQ choices in Figure 6 ( at 4-bit precision and at 2-bit precision).
Response to Official Review of Reviewer NPGQ, Part (3/3)
More Empirical Evaluation (W4)
Thank you for suggesting that we broaden our empirical evaluation. We have expanded the empirical evaluation in two directions: validation on another model family and zero-shot downstream evaluation.
Cross-model Validation
We additionally pretrain GPT-2 models at three scales (124M, 355M, and 774M). These results have been added to Appendix C.3, “Cross-Model Validation,” and Figures 18–19 of the revised manuscript. Across all three scales, performance at 4-bit precision initially improves and then degrades as increases, while the –PPL correlations are negative at 4-bit precision and positive at 2-bit precision, consistent with the OPT results. For your convenient reference, we provide brief numerical summaries below.
Table 3. relative to LAWA for GPT-2 models under GPTQ on held-out RefinedWeb. Values are in units of . Negative values indicate improvement; positive values indicate degradation.
| Precision | GPT-2 124M | GPT-2 355M | GPT-2 774M | |
|---|---|---|---|---|
| 4-bit | 0.006 | −2.50 | −4.01 | −4.92 |
| 4-bit | 1.0 | +34.57 | +37.08 | +22.28 |
| 2-bit | 0.006 | +48.42 | +59.26 | +55.48 |
| 2-bit | 1.0 | +184.42 | +356.62 | +128.68 |
Table 4. –PPL correlations for GPT-2 models under GPTQ. Each cell reports Pearson over the six averaging kernels used in Figure 19.
| Evaluation data | Precision | GPT-2 124M | GPT-2 355M | GPT-2 774M |
|---|---|---|---|---|
| RefinedWeb | 4-bit | −0.712 | −0.843 | −0.948 |
| RefinedWeb | 2-bit | +0.873 | +0.754 | +0.370 |
| FineWeb-Edu | 4-bit | −0.957 | −0.909 | −0.937 |
| FineWeb-Edu | 2-bit | +0.870 | +0.746 | +0.371 |
Zero-shot Accuracy Validation
We have also added zero-shot evaluation on six downstream tasks for OPT-1.3B after 2-bit quantization, reported in Table 2 of the revised manuscript. LNWA outperforms Stable LR and both LR-decay baselines on all six tasks under both GPTQ and AWQ. These results show that LNWA’s robustness to low-precision quantization extends beyond perplexity to downstream task performance.
Table 5. Zero-shot accuracy (%; higher is better) of OPT-1.3B after 2-bit PTQ. Avg. is the mean over the six reported tasks. Best results within each PTQ method are in bold.
| PTQ | Method | PIQA | ARC-Easy | HellaSwag | COPA | OBQA | SciQ | Avg. |
|---|---|---|---|---|---|---|---|---|
| GPTQ | Stable LR | 60.17 | 48.48 | 29.34 | 60.00 | 15.00 | 74.70 | 47.95 |
| GPTQ | LR decay (1%) | 60.17 | 48.15 | 28.79 | 64.00 | 16.80 | 72.80 | 48.45 |
| GPTQ | LR decay (0%) | 59.30 | 47.35 | 28.19 | 58.00 | 15.40 | 72.20 | 46.74 |
| GPTQ | LNWA | 63.22 | 52.74 | 30.41 | 66.00 | 19.00 | 78.70 | 51.68 |
| AWQ | Stable LR | 67.03 | 60.14 | 34.13 | 67.00 | 20.80 | 79.60 | 54.78 |
| AWQ | LR decay (1%) | 66.65 | 60.77 | 34.71 | 67.00 | 21.20 | 80.80 | 55.19 |
| AWQ | LR decay (0%) | 66.92 | 60.86 | 34.46 | 62.00 | 21.00 | 80.00 | 54.21 |
| AWQ | LNWA | 68.23 | 63.51 | 35.40 | 73.00 | 23.00 | 82.80 | 57.66 |
Regarding model size, our experiments use scales comparable to those in related theoretical studies of LLM pretraining and weight averaging [2,3,4]. We chose this range to align our experimental setup with prior work and make it easier to compare findings across studies in the community. The consistent results across OPT models from 125M to 1.3B already support LNWA across model scales, while the additional GPT-2 experiments extend this evidence to another model family. We sincerely appreciate your suggestion and will further extend the experiments and analysis to larger models in future work.
[1] Liu et al. ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. NeurIPS, 2025.
[2] Wen et al. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. ICLR, 2025.
[3] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. NeurIPS, 2024.
[4] Meterez et al. Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging. arXiv, 2026.
Response to Official Review of Reviewer NPGQ, Part (1/3)
The authors sincerely thank you for your valuable comments and suggestions. We have addressed all main points raised in your review. We have carefully revised and supplemented the paper accordingly.
Please see the detailed responses below.
Spatial-Noise Approximation (W1, Q1)
Thank you for asking about our spatial noise assumption. We clarify the conditions of R(w) and present empirical evidence that the approximation is robust to the temporal correlations in practice.
To derive the spatial noise factor R(w), we start with the retained spatial noise ∑dwdϵd, where ϵd=θT−d−μT−d is the spatial noise in the saved weights at age d. With Γde=Cov(ϵd,ϵe∣F0), its total variance is
=\operatorname{Tr}\!\left(\sum_{d,e} w_d w_e\boldsymbol{\Gamma}_{de}\right) =\sum_d w_d^2\operatorname{Tr}(\boldsymbol{\Gamma}_{dd}) +2\sum_{d<e}w_dw_e\operatorname{Tr}(\boldsymbol{\Gamma}_{de}).$$ To simplify this variance, we use Assumption A.1, which specifies a common noise covariance $\boldsymbol{\Gamma}_{dd}=\boldsymbol{\Sigma}$ across retained weights and sets $\boldsymbol{\Gamma}_{de}=\boldsymbol{0}$ for $d\ne e$. The cross terms vanish, leaving $$V(\boldsymbol{w})=\operatorname{Tr}(\boldsymbol{\Sigma})\sum_d w_d^2 =\operatorname{Tr}(\boldsymbol{\Sigma})R(\boldsymbol{w}).$$ We consider this assumption reasonable for two reasons. First, LLMs are often pretrained for a single epoch, with each batch used once. Avoiding repeated data can substantially reduce noise correlations. Second, we save weights at large intervals, which can further reduce these correlations. We also test this assumption in the MNIST MLP and OPT-125M experiments, where the measured temporal noise correlations approach near zero as the saving interval increases (Figures 7 and 11(a) of the **revised manuscript**, respectively). To test whether the approximation is robust to temporal correlations in practice, we compare the predicted increase $R(\boldsymbol{w})-1/M$ with the measured increase over LAWA, as shown in Table 1. *Table 1. Theoretically predicted and experimentally measured increases in retained spatial noise relative to LAWA for OPT-125M.* | Kernel | $\lambda$ | Predicted noise increase | Measured noise increase | |:---|---:|---:|---:| | LAWA | $0$ | 0.000 | 0.000 | | LNWA | $0.02$ | 0.011 | 0.040 | | LNWA | $0.1$ | 0.100 | 0.176 | | LNWA | $0.4$ | 0.232 | 0.287 | For the LNWA kernels in Table 1, the measured noise increases exceed the predictions by up to $0.076$. The measured increases follow the predicted ordering in all six windows, indicating that this qualitative conclusion is insensitive to the temporal correlations observed in the experiment.