- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses rounding-boundary weight oscillation in low-bit quantization-aware pre-training (QAPT) and proposes CEWT, a post-update operation that constrains the empirical weight distribution. After each optimizer update, CEWT ranks the weights and maps them to fixed Gaussian quantiles, forcing each weight tensor to follow an approximately zero-mean Gaussian distribution. This prevents excessive weight accumulation near quantization boundaries and reduces optimization noise. Across multiple LLaMA/GPT architectures, quantizers, optimizers, and 1-bit/2-bit settings, CEWT generally improves pre-training perplexity with a reported geometric-mean training-time overhead of approximately 4%, without introducing an additional oscillation-suppression hyperparameter.
Critical strengths
- The paper reformulates oscillation suppression as a hard constraint on the empirical weight distribution rather than introducing an additional regularization term or weight-freezing rule.
- CEWT does not require tuning a regularization coefficient, interpolation ratio, or oscillation threshold.
- The evaluation includes LLaMA and GPT architectures, three quantizers, four hypersphere optimizers, 1-bit and 2-bit quantization, and model sizes ranging from approximately 94M to 610M parameters.
Critical weaknesses
-
The contributions of CEWT and the quantizer modifications are not fully separated. For example, QuEST and BBQ, the main CEWT configuration also removes the Hadamard transform and introduces TGCS. Therefore, the improvements in the main table cannot be attributed solely to the Gaussian projection. Adding a complete factorial ablation will be helpful.
-
The optimization dynamics induced by the rank-based projection need more analysis. After the CEWT projection, the numerical value of a weight is largely determined by its rank. If an optimizer update does not change the relative ordering of the weights, the projected weights may remain unchanged. The method therefore appears to optimize weight permutations rather than unconstrained continuous values.
-
The largest evaluated model contains 610M parameters, and the main language-model experiments use only C4. This does not establish that the method scales efficiently to modern multi-billion-parameter LLMs. Including at least one experiment on a 1B–7B parameter model will be helpful.
-
CEWT generally improves pre-training perplexity, but the HellaSwag, ARC, PiQA, and Winogrande results are mixed.
Questions for authors
Refer to weaknesses.
5 comments
Response 2 out of 2
- Weakness 3: Including at least one experiment on a 1B–7B parameter model will be helpful.
Response: We have limited access to shared hardware. A queuing system limits our task throughput to approximately 80% of an H100 GPU. We estimate that a 1B-scale experiment would consume 1.25 H100-months, which would take approximately 1.6 months given our throughput limit. Therefore, we are unable to provide results at the 1B-7B scale within the author-reviewer discussion period due to resource constraints. Our work focuses on quantization-aware pre-training (QAPT), and pre-training is known to be expensive. As noted in the paper, our existing results already required 1 RTX-5090-month and 1 H100-80GB-month. Two recent QAPT papers, BBQ and QuEST, provide QAPT results up to 300M and 1600M, respectively. Our 610M result is within this range. We also note that small language models can be very useful on edge devices with limited energy (e.g. mobile phones, smart watches, smart glasses, etc.) [1]
[1] MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Weakness 4: CEWT generally improves pre-training perplexity, but the HellaSwag, ARC, PiQA, and Winogrande results are mixed.
Response: Our primary evaluation criterion is perplexity evaluated on unseen validation splits, as this criterion aligns with the pre-training objective and tests generalization on unseen data. We acknowledge that improvements in next-token prediction do not necessarily translate into improvements in every downstream task. We note that in Table 13, CEWT outperforms baseline optimizers in 14 out of 19 comparisons for HellaSwag, 13 out of 19 comparisons for ARC-Easy, 9 out of 19 comparisons for ARC-Challenge, 12 out of 19 comparisons for PIQA, and 11 out of 19 comparisons for WinoGrande. For cross-dataset language modeling, CEWT improves perplexity in all 19 comparisons on Wiki and 16 of 19 comparisons on Pile 10k.
Dear Reviewer ALW9,
We are very grateful for your feedback and helpful suggestions. Would you be so kind as to let us know whether our response above has addressed your concerns?
Response 1 out of 2
We appreciate Reviewer ALW9 for reviewing our paper and for providing valuable feedback.
Response: We have added 10 rows labeled A, B, C, D, E, F, G, H, I, and J in Table 10, testing the effects of quantizer modifications, including removing Hadamard Transform and adding TGCS, without CEWT. Results suggest that without CEWT, these quantizer modifications alone are insufficient to improve performance. Rows 2 and 6 suggest that adding CEWT without TGCS already improves performance, while Rows 3 and 7 suggest TGCS can further improve performance, provided that HT is absent from both the quantizer and optimizer, or HT is present in both the quantizer and optimizer.
These results were obtained by running train_full_ablation.sh which is available in the supplemental material.
Response: We added Section O in the updated manuscript which contains our measurements for rank-change frequency (Figure 9) and projection displacement ratio (Figure 10). Since OpenReview does not allow us to include figures, we refer the reviewer to the updated manuscript for the figures. Figure 9 suggests that in every iteration, the vast majority of weights have their ranks/relative ordering updated (consistently above 98%) despite the CEWT projection operation. Figure 10 shows that the norm of projection displacement is small relative to the optimizer update (consistently less than 20%), suggesting CEWT projection is not strong enough to suppress beneficial weight movement.
Response: When applied to Frobenius norm hypersphere optimizers, CEWT is indeed optimizing for a permutation of a list of mn Gaussian quantiles. However, as the number of weights in a layer (e.g. mn) increases, a random permutation of this list gradually approaches true Gaussian noise, which is composed of continuous values. We discussed CEWT’s reliance on a large mn in the paper. In particular, Section 3 discusses the CEWT constraint (Eq. 3d) is the formal description of the zero-mean Gaussian prior “for a sufficiently large mn”. In Section 5 (Limitations), we discuss that CEWT does not work well for a small mn because CEWT “relies on the law of large numbers” and CEWT “forbids making small changes to parameters” when mn is small. In Section D.9 and Table 9, we experimentally find that for a toy example, the threshold for a “sufficiently large mn” is around 2^10. Our actual mn in Table 3 is between 2^18 and 2^22, and we observe for mn at these scales, CEWT works well, even outperforming the baseline hypersphere optimizers that are capable of optimizing unconstrained “continuous” values (which are often themselves stored in discrete/finite-precision data types such as fp32).