acceptodds
Under review as a conference paper at ICLR 2027

Quantization-Aware Pre-Training with Constrained Empirical Weight Distribution

Abstract

Quantization-Aware Pre-Training (QAPT) can increase the inference efficiency of DNNs, but a problematic behaviour known as rounding boundary weight oscillation can introduce detrimental noise into the training process and significantly reduce convergence speed. While existing methods can reduce this detrimental noise, they either introduce additional hyperparameters or memory overhead, or cannot consistently improve model accuracy. In this work, we propose optimization with **C**onstrained **E**mpirical **W**eight dis**T**ribution (CEWT), the first hyperparameter-free memory-overhead-free oscillation suppression method that consistently improves QAPT performance: an optimizer post-update step that projects weights to the nearest point in weight space whose empirical distribution (histogram) matches a zero-mean Gaussian. Our key insight is many quantizers are designed with the implicit assumption that the to-be-quantized data are permutations of samples from a zero-mean Gaussian, and this assumption is not true during QAPT. By enforcing the zero-mean Gaussian prior as a hard constraint, CEWT can suppress this detrimental noise. Empirical results on various combinations of SOTA quantizers and hypersphere optimizers suggest, that with a geomean increase of 4% in training time, CEWT can consistently reduce the pre-training perplexity (by an average of 2.5 and up to 21 points) of low-precision (down to 1-bit activations and weights and up to 610M parameters) LLaMA/GPT models without introducing any hyperparameters or storage overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, QAT, Efficient ML, Edge Inference, QAPT

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ALW9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses rounding-boundary weight oscillation in low-bit quantization-aware pre-training (QAPT) and proposes CEWT, a post-update operation that constrains the empirical weight distribution. After each optimizer update, CEWT ranks the weights and maps them to fixed Gaussian quantiles, forcing each weight tensor to follow an approximately zero-mean Gaussian distribution. This prevents excessive weight accumulation near quantization boundaries and reduces optimization noise. Across multiple LLaMA/GPT architectures, quantizers, optimizers, and 1-bit/2-bit settings, CEWT generally improves pre-training perplexity with a reported geometric-mean training-time overhead of approximately 4%, without introducing an additional oscillation-suppression hyperparameter.

Critical strengths
  1. The paper reformulates oscillation suppression as a hard constraint on the empirical weight distribution rather than introducing an additional regularization term or weight-freezing rule.
  2. CEWT does not require tuning a regularization coefficient, interpolation ratio, or oscillation threshold.
  3. The evaluation includes LLaMA and GPT architectures, three quantizers, four hypersphere optimizers, 1-bit and 2-bit quantization, and model sizes ranging from approximately 94M to 610M parameters.
Critical weaknesses
  1. The contributions of CEWT and the quantizer modifications are not fully separated. For example, QuEST and BBQ, the main CEWT configuration also removes the Hadamard transform and introduces TGCS. Therefore, the improvements in the main table cannot be attributed solely to the Gaussian projection. Adding a complete factorial ablation will be helpful.

  2. The optimization dynamics induced by the rank-based projection need more analysis. After the CEWT projection, the numerical value of a weight is largely determined by its rank. If an optimizer update does not change the relative ordering of the weights, the projected weights may remain unchanged. The method therefore appears to optimize weight permutations rather than unconstrained continuous values.

  3. The largest evaluated model contains 610M parameters, and the main language-model experiments use only C4. This does not establish that the method scales efficiently to modern multi-billion-parameter LLMs. Including at least one experiment on a 1B–7B parameter model will be helpful.

  4. CEWT generally improves pre-training perplexity, but the HellaSwag, ARC, PiQA, and Winogrande results are mixed.

Questions for authors

Refer to weaknesses.

5 comments
AuthorsOpenReview ↗

Response 1 out of 2

We appreciate Reviewer ALW9 for reviewing our paper and for providing valuable feedback.

  • Weakness 1: The contributions of CEWT and the quantizer modifications are not fully separated… Adding a complete factorial ablation will be helpful.

Response: We have added 10 rows labeled A, B, C, D, E, F, G, H, I, and J in Table 10, testing the effects of quantizer modifications, including removing Hadamard Transform and adding TGCS, without CEWT. Results suggest that without CEWT, these quantizer modifications alone are insufficient to improve performance. Rows 2 and 6 suggest that adding CEWT without TGCS already improves performance, while Rows 3 and 7 suggest TGCS can further improve performance, provided that HT is absent from both the quantizer and optimizer, or HT is present in both the quantizer and optimizer.

RowWeight QuantizerHTTGCSOptimizerPerplexityRBM
1QuESTTFAdamH32.350.095
AQuESTTTAdamH32.370.106
BQuESTFTAdamH32.460.099
CQuESTFFAdamH32.420.079
2QuESTFFAdam-CEWT31.530.071
3QuESTFTAdam-CEWT30.700.056
DQuESTTTAdam-CEWT32.290.099
EQuESTTFAdam-CEWT32.470.096
4QuESTTTAdam-CEWT-HT30.810.056
5BBQTFAdamH31.780.022
FBBQTTAdamH31.780.029
GBBQFTAdamH31.640.038
HBBQFFAdamH31.610.029
6BBQFFAdam-CEWT30.690.015
7BBQFTAdam-CEWT30.130.010
IBBQTTAdam-CEWT31.760.025
JBBQTFAdam-CEWT31.740.023
8BBQTTAdam-CEWT-HT30.290.010

These results were obtained by running train_full_ablation.sh which is available in the supplemental material.

  • Weakness 2: If an optimizer update does not change the relative ordering of the weights, the projected weights may remain unchanged.

Response: We added Section O in the updated manuscript which contains our measurements for rank-change frequency (Figure 9) and projection displacement ratio (Figure 10). Since OpenReview does not allow us to include figures, we refer the reviewer to the updated manuscript for the figures. Figure 9 suggests that in every iteration, the vast majority of weights have their ranks/relative ordering updated (consistently above 98%) despite the CEWT projection operation. Figure 10 shows that the norm of projection displacement is small relative to the optimizer update (consistently less than 20%), suggesting CEWT projection is not strong enough to suppress beneficial weight movement.

  • Weakness 2: The method therefore appears to optimize weight permutations rather than unconstrained continuous values.

Response: When applied to Frobenius norm hypersphere optimizers, CEWT is indeed optimizing for a permutation of a list of mn Gaussian quantiles. However, as the number of weights in a layer (e.g. mn) increases, a random permutation of this list gradually approaches true Gaussian noise, which is composed of continuous values. We discussed CEWT’s reliance on a large mn in the paper. In particular, Section 3 discusses the CEWT constraint (Eq. 3d) is the formal description of the zero-mean Gaussian prior “for a sufficiently large mn”. In Section 5 (Limitations), we discuss that CEWT does not work well for a small mn because CEWT “relies on the law of large numbers” and CEWT “forbids making small changes to parameters” when mn is small. In Section D.9 and Table 9, we experimentally find that for a toy example, the threshold for a “sufficiently large mn” is around 2^10. Our actual mn in Table 3 is between 2^18 and 2^22, and we observe for mn at these scales, CEWT works well, even outperforming the baseline hypersphere optimizers that are capable of optimizing unconstrained “continuous” values (which are often themselves stored in discrete/finite-precision data types such as fp32).

AuthorsOpenReview ↗

Response 2 out of 2

  • Weakness 3: Including at least one experiment on a 1B–7B parameter model will be helpful.

Response: We have limited access to shared hardware. A queuing system limits our task throughput to approximately 80% of an H100 GPU. We estimate that a 1B-scale experiment would consume 1.25 H100-months, which would take approximately 1.6 months given our throughput limit. Therefore, we are unable to provide results at the 1B-7B scale within the author-reviewer discussion period due to resource constraints. Our work focuses on quantization-aware pre-training (QAPT), and pre-training is known to be expensive. As noted in the paper, our existing results already required 1 RTX-5090-month and 1 H100-80GB-month. Two recent QAPT papers, BBQ and QuEST, provide QAPT results up to 300M and 1600M, respectively. Our 610M result is within this range. We also note that small language models can be very useful on edge devices with limited energy (e.g. mobile phones, smart watches, smart glasses, etc.) [1]

[1] MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

  • Weakness 4: CEWT generally improves pre-training perplexity, but the HellaSwag, ARC, PiQA, and Winogrande results are mixed.

Response: Our primary evaluation criterion is perplexity evaluated on unseen validation splits, as this criterion aligns with the pre-training objective and tests generalization on unseen data. We acknowledge that improvements in next-token prediction do not necessarily translate into improvements in every downstream task. We note that in Table 13, CEWT outperforms baseline optimizers in 14 out of 19 comparisons for HellaSwag, 13 out of 19 comparisons for ARC-Easy, 9 out of 19 comparisons for ARC-Challenge, 12 out of 19 comparisons for PIQA, and 11 out of 19 comparisons for WinoGrande. For cross-dataset language modeling, CEWT improves perplexity in all 19 comparisons on Wiki and 16 of 19 comparisons on Pile 10k.

Dear Reviewer ALW9,

We are very grateful for your feedback and helpful suggestions. Would you be so kind as to let us know whether our response above has addressed your concerns?

Reviewer_1EoNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CEWT targets rounding-boundary weight oscillation in 1-bit and 2-bit quantization-aware pre-training of LLMs. After every hypersphere-optimizer step, it replaces each layer's weights by their ranks mapped through the inverse Gaussian CDF, so the weight histogram is exactly Gaussian, which is what quantizers like QuEST and BBQ assume.

Critical strengths
  1. The proposed method is simple, cheap, and effective. The method is a one-line post-step (argsort, erfinv, rescale) with no new hyperparameters, no extra memory, and about 4% wall-time overhead (Table 14).

  2. The performance is consistent across a wide grid of parameters. Perplexity drops in 98 of 99 combinations of 4 optimizers, 3 quantizers, 2 precisions, 2 architectures, and sizes 95M to 610M (Table 3), with ablations on Hadamard removal, TGCS, and the norm radius R (Tables 4, 10).

Critical weaknesses
  1. Insufficient comparison against prior oscillation-suppression methods on real networks. Beyond the toy problem, periodic interpolation, dampening loss, and PQN are compared only on a single configuration (LLaMA-95M, AdamH, appendix H and I), and iterative freezing and CGA are never run on a deep network. Table 3's 99 settings compare only against plain STE. The authors should report the competing methods on actual neural networks. Otherwise, it is impossible to assess the superiority of the proposed method.

  2. No AdamW baseline. QuEST and BBQ, the claimed SOTA, are published with AdamW, but every baseline in Table 3 is a hypersphere optimizer. The paper therefore never shows that Adam-CEWT beats the published SOTA recipe, only its own hypersphere re-implementation of it.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response 1 out of 2

We sincerely thank Review 1EoN for reviewing our paper and for providing valuable feedback. We are glad you find our method simple, cheap and effective.

  • Weakness 1: periodic interpolation, dampening loss… are compared only on a single configuration (LLaMA-95M, AdamH).

Response: we have added Table A in Section H, which compares CEWT against periodic interpolation (PI) and dampening loss (DL) on LLaMA-125M with AdamH. For your convenience, we have also included Table A below. Results show that CEWT is within 0.44 perplexity of PI with the best hyperparameter, that CEWT outperforms DL with the best hyperparameter, and that CEWT significantly outperforms PI and DL with bad hyperparameters.

HyperparametersAdamHAdam-CEWTAdamH + PIAdamH + DL
No HP27.9626.52--
Good HP--26.08 (α = 1e-3)26.87 (α = 1e-5)
Bad HP--29.39 (α = 1e-2)31.84 (α = 1e-3)

We note that we choose a good and bad hyperparameter based on the hyperparameter sweep in Section H (available in the original submission), which consumed 48 RTX-5090-hours of training time, whereas CEWT is effective on the first training run (without the need for any prior knowledge) as it is hyperparameter-free. CEWT supports use cases where hyperparameter tuning is infeasible---for example, when practitioners can only afford a single training run.

  • Weakness 1: PQN … (is) compared only on a single configuration (LLaMA-95M, AdamH).

Response: We added 5 additional comparisons to Table 11 evaluated on LLaMA-125M. For your convenience, we also include Table 11 here. While PQN excels in reducing rounding boundary mass, CEWT outperforms PQN in terms of pre-training perplexity on all 5 tested configurations. We attribute this performance gap to the biasedness of the PQN: rounding error is deterministic for a deterministic input, whereas PQN simulates deterministic rounding error with stochastic noise; as a result, making progress in the expected PQN objective is not guaranteed to making progress in the real objective; we demonstrated this biasedness of PQN for a linear-regression-with-quantized-weights problem in Section D.3 and 4.1. At 3-bit quantization, when rounding boundary oscillation noise is weaker, PQN underperforms compared to the STE whereas CEWT outperforms both the STE and PQN.

ModelQuantizerBits W/AAdamH + STE PPLAdamH + STE RBMAdamH + PQN PPLAdamH + PQN RBMAdam-CEWT PPLAdam-CEWT RBM
LLaMA-95MQuEST152.850.24050.610.00248.020.119
LLaMA-95MBBQ151.060.03449.540.00146.120.010
LLaMA-95MQuEST232.300.09532.140.00730.670.056
LLaMA-95MBBQ231.800.02331.750.00530.180.010
LLaMA-95MQuEST327.270.04327.310.01026.490.029
LLaMA-95MBBQ327.350.01727.640.00826.670.010
LLaMA-125MQuEST145.480.23444.380.00241.720.119
LLaMA-125MBBQ144.510.02942.740.00140.600.010
LLaMA-125MQuEST227.960.09027.770.00726.520.056
LLaMA-125MBBQ227.420.02027.350.00526.130.010
LLaMA-125MBBQ323.680.01523.780.00823.180.010

We also note that in the original submission, Table 11 compares CEWT against PQN and STE (with LLaMA-95M) on the cross product of three precisions and two quantizers, which is 6 configurations, if a configuration is defined as the combination of architecture, size, quantizer, and precision.

AuthorsOpenReview ↗

Response 2 out of 2

  • Weakness 1: iterative freezing and CGA are never run on a deep network

Response: we added Section P, Table 18, and Table 19, which compare CEWT against iterative freezing (IF) and CGA. We included the two tables here for your convenience. Results show CEWT can outperform IF and CGA. We acknowledge that we did not extensively sweep the hyperparameters of IF and CGA due to the limited time of the rebuttal period. We additionally note that both IF and CGA include two hyperparameters, and CEWT has none. As noted above, CEWT supports use cases where hyperparameter tuning is infeasible---for example, when practitioners can only afford a single training run.

MetricAdamWAdamW + IFAdam-CEWT
Perplexity33.5832.0030.70
MetricAdamHAdamH + CGAAdam-CEWT
Perplexity32.3032.2430.67
  • No AdamW baseline.

Response: we added Section Q and Table 20, which we provide below for your convenience. Results show that AdamH generally outperforms, or performs on par with, its baseline optimizer AdamW, while Adam-CEWT consistently outperforms both AdamW and AdamH. We chose AdamH as the baseline in Table 3 because AdamH and Adam-CEWT differ by only one line (Equation 4), and the performance improvement therefore reflects the effects of that one line. In contrast, Adam-CEWT and AdamW differ in multiple ways. Examples include that Adam-CEWT does not use weight decay, and has a different learning rate than AdamW.

ModelQuantizerBitsAdamWAdamHAdam-CEWT
LLaMA-95mQuEST233.5032.3030.67
LLaMA-95mBBQ234.0031.8030.18
LLaMA-95mQuEST152.6452.8548.02
LLaMA-95mBBQ151.2251.0646.12
LLaMA-125mQuEST146.1845.4841.72
LLaMA-125mBBQ144.5744.5140.60

I really appreciate the author's response and hard work; the experiments seem very thorough.

My lingering concern is with the exp results on IF and CGA on DNN, which, as the authors mentioned, were not run with optimal hyperparameters, and I appreciate the honesty.

I am leaning more towards acceptance post-rebuttal, but will keep my score for now and discuss with other reviewers to decide whether the current changes suffice for acceptance.

Should the paper be accepted, it would be very nice to compare the proposed approach against the baselines with near-optimal hyperparam settings and scale this experiment to other models as well. One option might be to simply demonstrate the superiority of the proposed method on smaller networks that are easier to train (e.g., CNNs or ViTs).

Reviewer_C75VOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Constrained Empirical Weight Distribution (CEWT) to suppress rounding-boundary weight oscillations that hinder convergence in extremely low-precision quantization-aware pre-training (QAPT). CEWT enforces a zero-mean Gaussian weight distribution through a hard constraint, using rank-based projection within hypersphere optimizers. The method also removes Hadamard transforms and introduces Tensor-wise Quantization Grid with Channel-wise Scaling (TGCS) to improve quantization compatibility. Experiments on LLaMA and GPT models up to 610M parameters, using multiple quantizers and optimizers, demonstrate improved pre-training perplexity across most configurations with only 4% average training-time overhead. Additional experiments investigate oscillation suppression, zero-shot performance, and applicability to vision models.

Critical strengths

S1. A well-motivated and practically relevant problem. The paper addresses rounding-boundary weight oscillation in low-bit QAT, which can impair convergence. Its discussion of existing methods clearly motivates the need for an effective solution without additional hyperparameter tuning.

S2. A simple and technically interesting optimization approach. CEWT introduces a deterministic, rank-based Gaussian projection that preserves weight ordering without additional hyperparameters or persistent optimizer states. Its simplicity and compatibility with multiple hypersphere optimizers are appealing.

S3. Extensive evaluation across multiple low-precision configurations. Experiments cover different architectures, model sizes, quantizers, optimizers, and bit widths. Substantial perplexity improvements, particularly in 1-bit settings, and comparisons with established oscillation-suppression methods support the method's effectiveness.

S4. Useful implementation analysis and ablation studies. The analysis of TGCS and Hadamard transforms demonstrates practical implementation considerations. Additional experiments on model size and training duration provide insights into CEWT's applicability and limitations.

Critical weaknesses

W1. Insufficient justification for the Gaussian distribution. Section 3.1; Equations 3–5. The paper motivates Gaussian weight constraints primarily through existing quantizers' distributional assumptions, without demonstrating their superiority for oscillation suppression. Comparisons with alternative distributions (e.g., Laplace, logistic, or uniform) would help establish whether the improvements arise from Gaussianity or simply imposing a fixed distribution.

W2. Incomplete ablation of CEWT and quantizer modifications. Sections 3.2–3.3; Appendix E, Table 10. The proposed approach combines Gaussian projection, Hadamard transform removal, and TGCS quantization. However, Table 10 lacks important controls evaluating the modified quantizers without CEWT. Consequently, the individual contribution of CEWT to the reported improvements remains unclear.

W3. Inconsistent performance improvements across configurations. Section 4.2; Table 3; Appendices K and M. Despite claims of consistent improvements, CEWT occasionally degrades performance. For example, Table 3 reports increased perplexity (51.23 → 52.79) for LLaMA-95M with LSQ-1 and SSO. Zero-shot results are mixed, while DeiT-Base accuracy decreases from 39.80% to 39.24% in one configuration. These exceptions suggest that the consistency claims should be qualified.

W4. Insufficient evidence linking oscillation suppression to performance gains. Section 4.2; Figure 5. CEWT directly constrains weight distributions, making reduced Rounding Boundary Mass (RBM) partly an expected consequence rather than independent evidence of suppressed oscillations. Improvements may also arise from altered quantization errors or optimization dynamics. More systematic measurements of quantized-bin transitions and convergence behavior would strengthen the causal explanation.

W5. Limited analysis of rank-based projection dynamics. Section 3.1; Equation 4. CEWT's projection depends solely on weight rankings. Updates that preserve rankings may therefore be discarded, while rank changes introduce discrete adjustments. The paper does not adequately examine rank-change frequency, projection distortion, or their implications for convergence stability.

W6. Limited scalability and statistical validation. Sections 4–5 Experiments are limited to models of at most 610M parameters, while per-step sorting introduces computational and temporary memory costs. Although the reported average overhead is 4%, some configurations show substantially higher costs (e.g., 13.59 → 19.75 hours). Additionally, the main results lack multi-seed statistics, making the reliability of smaller improvements uncertain.

Questions for authors

My main concerns are the insufficient justification of the Gaussian constraint, incomplete separation of the proposed optimizer projection from quantizer modifications, and limited evidence supporting the claimed consistency and causal mechanism of the improvements. Addressing these issues, especially through more comprehensive ablations and analysis of the optimization dynamics, would significantly strengthen the paper's technical contribution and empirical credibility.

4 comments
AuthorsOpenReview ↗

Response 1 out of 2

We are grateful to Review C75V for reviewing our paper and for providing valuable feedback. We are glad you find our method simple and technically interesting.

  • W1: Comparisons with alternative distributions (e.g., Laplace, logistic, or uniform) would help establish whether the improvements arise from Gaussianity or simply imposing a fixed distribution.

Response: We have included these comparisons in Section N and Table 17 in the updated manuscript. We additionally provide a copy of Table 17 here for your convenience. Experiments are conducted on LLaMA-95M with the QuEST quantizer pre-trained on 3 billion tokens. Results show that Gaussianity outperforms other distributions, although imposing a fixed distribution also reduces perplexity compared to the baseline hypersphere optimizer.

MetricAdamHAdam-CEWT (Gaussian)Adam-CEWT (Laplace)Adam-CEWT (Logistic)Adam-CEWT (Uniform)
PPL32.3030.6731.1930.7930.80

We want to clarify that we do not claim that Gaussian is “the optimal” distribution. CEWT is designed to suppress the irregularly high density spikes that occur due to rounding boundary weight oscillation, and constraining the weight histogram to match any scalar distribution whose PDF is smooth and does not have visible spikes can, by design, suppress density spikes. Table 17 supports this: Gaussian, Logistic, and Uniform have smooth PDFs within their support, and all three improves performance by at least 1.5 perplexity compared to the baseline hypersphere optimizer; In contrast, the PDF of Laplace has a visible sharp spike near its center, which is mapped to a rounding boundary; as a result, the performance improvement of Laplace is limited (1.11 perplexity) compared to the other smooth distributions (Gaussian, Logistic, and Uniform).

CEWT chooses Gaussian as its default distribution due to the following two motivations. First, the weight distribution of round-boundary-oscillation-free models, specifically those trained without any quantization, tend to look like a zero-mean Gaussian. Even when quantization is applied, pre-clipping weight distributions still resemble a bell (e.g. Figure 1a and 1c) but now with the addition of rounding boundary density spikes. By constraining the pre-clipping weight distribution to match a Gaussian, CEWT preserves the bell shape while simultaneously removing the density spikes. Second, the zero-mean Gaussian prior is built into the design of many quantizers. For example, QuEST is designed to minimize the mean-squared quantization error for zero-mean gaussian data, while we observe (in Figure 1a) that in practice QuEST’s pre-clipping distribution does not match a zero-mean Gaussian. To bridge this gap between assumption and reality, CEWT enforces pre-clipping weight distribution to match a zero-mean Gaussian. Table 17 shows Gaussian has minor performance advantages (around 0.12 perplexity) over the other two smooth distributions, Logistic and Uniform, and we attribute this performance advantage to the zero-mean Gaussian prior built into the design of QuEST.

  • W2: Table 10 lacks important controls evaluating the modified quantizers without CEWT.

Response: We have added 10 rows labeled A, B, C, D, E, F, G, H, I, and J in Table 10, testing the effects of quantizer modifications, including removing Hadamard Transform and adding TGCS, without CEWT. Results suggest that without CEWT, these quantizer modifications alone are insufficient to improve performance. Rows 2 and 6 suggest that adding CEWT without TGCS already improves performance, while Rows 3 and 7 suggest TGCS can further improve performance, provided that HT is absent from both the quantizer and optimizer, or HT is present in both the quantizer and optimizer.

RowWeight QuantizerHTTGCSOptimizerPerplexityRBM
1QuESTTFAdamH32.350.095
AQuESTTTAdamH32.370.106
BQuESTFTAdamH32.460.099
CQuESTFFAdamH32.420.079
2QuESTFFAdam-CEWT31.530.071
3QuESTFTAdam-CEWT30.700.056
DQuESTTTAdam-CEWT32.290.099
EQuESTTFAdam-CEWT32.470.096
4QuESTTTAdam-CEWT-HT30.810.056
5BBQTFAdamH31.780.022
FBBQTTAdamH31.780.029
GBBQFTAdamH31.640.038
HBBQFFAdamH31.610.029
6BBQFFAdam-CEWT30.690.015
7BBQFTAdam-CEWT30.130.010
IBBQTTAdam-CEWT31.760.025
JBBQTFAdam-CEWT31.740.023
8BBQTTAdam-CEWT-HT30.290.010
AuthorsOpenReview ↗

Response 2 out of 2

  • W3: These exceptions suggest that the consistency claims should be qualified.

Response: Thank you for pointing out the exceptions. We have revised the paper, which now states that CEWT “reduces QAPT perplexity on a broad range of evaluated configurations” to qualify the consistency claims and avoid misleading our readers.

  • W4: More systematic measurements of quantized-bin transitions and convergence behavior would strengthen the causal explanation.

Response: The second column of Figure 5 contains our measurement of EMA Oscillation Frequency, which is a direct measurement of how often weights oscillate/transition between neighbouring quantization bins. Figure 5 shows a correlation between rounding boundary mass (column 1), oscillation frequency (column 2), and perplexity (column 4).

  • W5: Updates that preserve rankings may therefore be discarded, while rank changes introduce discrete adjustments. The paper does not adequately examine rank-change frequency, projection distortion, or their implications for convergence stability.

Response: We added Section O in the updated manuscript which contains our measurements for rank-change frequency (Figure 9) and projection displacement ratio (Figure 10). Since OpenReview does not allow us to include figures, we refer Reviewer C75V to the updated manuscript for the figures. Figure 9 suggests that in every iteration, the vast majority of weights have their ranks updated (consistently above 98%) despite the CEWT projection operation. Figure 10 shows that the norm of projection displacement is small relative to the optimizer update (consistently less than 20%), suggesting CEWT projection is not strong enough to suppress beneficial weight movement.

  • W6: Experiments are limited to models of at most 610M parameters.

Response: We are unable to provide results at a larger scale due to resource constraints. Our work focuses on quantization-aware pre-training (QAPT), and pre-training is known to be expensive. As noted in the paper, our existing results already required 1 RTX-5090-month and 1 H100-80GB-month. Two recent QAPT papers, BBQ and QuEST, provide QAPT results up to 300M and 1600M, respectively. Our 610M result is within this range. We also note that small language models can be very useful on edge devices with limited energy (e.g. mobile phones, smart watches, smart glasses, etc.) [1]

[1] MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

  • W6: per-step sorting introduces computational and temporary memory costs. Although the reported average overhead is 4%, some configurations show substantially higher costs (e.g., 13.59 → 19.75 hours).

Response: We note our method CEWT is hyperparameter-free and matches the performance of existing methods like Periodic Interpolation and Dampening Loss without the need for hyperparameter tuning, which can require training multiple times. In addition, as future work, better engineering tweaks, such as operator fusion and better use of parallelism, can further reduce the overhead.

As noted in Section L, our experiments were conducted on shared hardware and the execution time has a large variance which we cannot control. As a result, we report the geomean increase to mitigate the effects of variance.

  • W6: Additionally, the main results lack multi-seed statistics, making the reliability of smaller improvements uncertain.

Response: We added Section R and Table 21, which reports the perplexity improvement of LLaMA-95M pre-trained on 3 billion C4 tokens, across three seeds. Results suggest the mean perplexity reduction (1.54) is much larger than the reduction’s standard deviation (0.11), suggesting the improvement of CEWT is relatively insensitive to the seed. We are unable to provide multi-seed statistics for every configuration in Table 3 (which would consume several GPU-months of resources).

Dear Reviewer C75V,

We are very grateful for your feedback and helpful suggestions. Would you be so kind as to let us know whether our response above has addressed your concerns?

Dear Reviewer C75V,

  • W1: Comparisons with alternative distributions (e.g., Laplace, logistic, or uniform) would help establish whether the improvements arise from Gaussianity or simply imposing a fixed distribution.

Additional Response: We have updated Table 17, whose previous version is available in our response to W1 above, by rerunning the experiments and measuring RBM and EMA Oscillation Frequency, on top of perplexity. EMA Oscillation Frequency provides the most direct evidence that Laplace distribution prior suffers more from rounding boundary oscillation noise (compared to Gaussian, Logistic, and Uniform), which we attribute to the visible spike located at the center of the PDF of the Laplace distribution. The other conclusions we stated above still apply. First, constraining the weight distribution to match any distribution whose PDF is density-spike-free can, by design, suppress rounding boundary density spikes and rounding boundary oscillation frequency compared to the baseline hypersphere optimizer. Second, Table 17 shows Gaussian outperforms other distributions, which we attribute to the zero-mean Gaussian prior built into the design of QuEST.

OptimizerPPLRBMEMA Oscillation Frequency (Unit: 1e-4)
AdamH32.310.09536.34
Adam-CEWT (Gaussian)30.670.0567.94
Adam-CEWT (Laplace)31.180.0719.40
Adam-CEWT (Logistic)30.770.0628.00
Adam-CEWT (Uniform)30.770.0097.75
  • W4: CEWT directly constrains weight distributions, making reduced Rounding Boundary Mass (RBM) partly an expected consequence rather than independent evidence of suppressed oscillations… More systematic measurements of quantized-bin transitions and convergence behavior would strengthen the causal explanation.

Additional Response: Our updated Table 17 contains direct measurement of EMA Oscillation Frequency, which serves as an additional piece of evidence to the second column of Figure 5, which we mentioned above in our response to W4. Table 17 shows CEWT’s distribution constraint reduces EMA Oscillation Frequency and reduces perplexity compared to the baseline hypersphere optimizer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.