acceptodds
Withdrawn from ICLR 2027

Generalization behavior of OPTQ and the role of regularization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large neural networks can be compressed by rounding or “quantizing” their weights to numbers that admit representations with fewer bits. One algorithm for quantization, OPTQ, progressively quantizes the weights of a neural network so that the squared quantization error on a specified calibration dataset is as small as possible. We study the performance of OPTQ and a variant algorithm, stochastic OPTQ, in a generalization setting and derive bounds for the expected squared error accrued by the algorithm when a test point is drawn from a fixed distribution. We prove two results. One result relates the generalization error to the error on a calibration dataset comprising independent samples from the same distribution as the test distribution. The other result bounds the generalization error of stochastic OPTQ for all sufficiently nice distributions, regardless of the calibration dataset. In both of these results, the regularization term plays an important role. We use insights from these results to make a new recommendation for the choice of and see that this choice of preforms favorably in experiments when compared to prior recommendations in the literature.

Primary area:
learning theory
Keywords:
OPTQ, quantization, generalization, regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8JUoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the generalization behavior of OPTQ, a data-dependent post-training quantization algorithm, and its stochastic variant. The central question is how well quantization error measured on a finite calibration set predicts error on unseen inputs, and how regularization affects this relationship. The paper makes two main theoretical contributions. First, it establishes a regularized empirical-to-population comparison bound that applies uniformly to all error vectors, including those depending on the calibration data, and specializes this result to OPTQ. Second, it derives bounds on positive-semidefinite quadratic forms of the quantization error for stochastic OPTQ, yielding distribution-agnostic guarantees for test inputs with finite second moments. Based on these results, the paper proposes a regularization parameter scaling with the squared Frobenius norm of the calibration matrix, the calibration-set size, and the weight dimension. The empirical evaluation compares this recommendation against existing regularization heuristics on three synthetic input distributions: uniform Hadamard, non-uniform Hadamard, and a ReLU neural-network distribution. The reported results suggest that the proposed scaling, particularly with a scaling constant of one, reduces generalization error in low-sample regimes while retaining competitive performance when more calibration samples are available. Overall, the paper provides a theoretically motivated analysis of regularization in OPTQ and supports its proposed parameter-selection heuristic with controlled synthetic experiments.

Critical strengths

1: A meaningful theoretical treatment of generalization in data-dependent quantization. The paper addresses an important gap between minimizing quantization error on calibration data and controlling error on unseen inputs. Theorem 3.1 establishes a uniform comparison between regularized empirical error and population error, including for error vectors that depend on the calibration sample. This is consequential because data-dependent quantization can exploit spurious correlations in small calibration sets, and a uniform bound directly addresses this difficulty rather than relying on a fixed, data-independent error vector. 2: A complementary guarantee for stochastic OPTQ and a practical regularization heuristic. Theorem 3.3 bounds general positive-semidefinite quadratic forms of the quantization error, with a high-probability guarantee over stochastic rounding. Its corollaries extend the analysis to test distributions that need not match the calibration distribution. The paper then uses these insights to motivate a parameter-selection heuristic and demonstrates consistent improvements over existing recommendations in the low-sample regime of its synthetic experiments. The combination of theoretical analysis and an actionable recommendation gives the work value beyond a purely abstract characterization.

Critical weaknesses

1: The empirical validation is too narrow to establish practical relevance to modern neural-network quantization. All experiments use synthetic distributions in a 64-dimensional setting. Although these controlled experiments illustrate the proposed mechanism, they do not establish that the recommended regularization scaling improves quantization quality for actual pretrained language models. The paper does not report evaluations on real model weights and activation calibration data, standard language-model benchmarks, or practical low-bit quantization settings. Consequently, it remains unclear whether the proposed heuristic provides meaningful improvements in realistic deployment scenarios or whether its benefits are specific to the selected synthetic distributions. 2: The proposed parameter-selection rule is not established as a generally reliable practical choice. The theoretical analysis motivates a range for the regularization parameter, but the simplified scaling in Equation (35) relies on heuristic assumptions relating the maximum row norm to the expected input norm and the empirical second moment to its population counterpart. The scaling constant is then selected empirically. The experiments cover only three synthetic distributions, and the reported evidence does not establish robustness to substantially different distributional structures or calibration regimes. Moreover, the analysis assumes bounded input norms for the empirical-to-population guarantee, while the distribution-agnostic stochastic result requires a separate interpretation through the finite second moment of the test input. The paper would benefit from a clearer account of which assumptions support each guarantee and how reliably the proposed heuristic satisfies their practical requirements.

Questions for authors

Clarification question 1 — Practical validation: Have the authors evaluated the proposed regularization rule on any pretrained language models using real calibration activations and standard language-model evaluation metrics? If such experiments exist, how does the proposed rule compare with the existing baselines in terms of perplexity or downstream performance at matched quantization bit-widths? If not, the practical scope of the current conclusions should be explicitly qualified. Clarification question 2 — Theoretical justification of the heuristic: Equation (35) is motivated by bounds that depend on assumptions about input-norm bounds and empirical second moments, followed by a heuristic simplification and empirical selection of the scaling constant. Can the authors clarify the precise conditions under which this simplified rule is expected to satisfy the requirements of the preceding theorems, and whether its scaling remains appropriate when calibration activations are heavy-tailed or have highly anisotropic covariance? Suggestions for future work: Evaluate the proposed rule on real LLMs across several model sizes, calibration-set sizes, and low-bit quantization settings. It would also be useful to investigate adaptive selection of the regularization parameter and to characterize when the proposed scaling is preferable to existing heuristics under realistic activation distributions.

Reviewer_rc4gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives theoretical generalisation bounds for the weight quantisation algorithm OPTQ. Generalisation in this case referring narrowly to how the quantisation error at the output of a layer generalises from a finite calibration data to test data. Based on their theoretical results, the authors then derive a rule for scaling the regularisation strength λ\lambda with the calibration set size. They then perform small scale synthetic experiments, comparing their approach to existing methods for setting λ\lambda.

Critical strengths
  1. The writing and presentation is generally clear and easy to read (although the density of mathematics is a little tough for someone like me who is not a theoretical ML researcher), providing a good introduction to the algorithm of OPTQ, which I appreciated as someone who is generally familiar with quantisation but not this specific algorithm.
  2. The theory appears to be correct, and does indeed address a gap in the characterisation of the behaviour of OPTQ (or post-training quantisation in general), i.e. that of generalisation from the calibration dataset to actual deployment.
Critical weaknesses
  1. They disregard numerical instability. The approach ignores one practical reason for λ\lambda, which is to guard against numerical instability during matrix inversion by introducing a spectral floor on the eigenvalues of XTXm\tfrac{X^TX}{m}. The proposed scaling would drive this floor to zero as it is sublinear in the size of the calibration set mm, making the approach vulnerable to numerical instability induced by poorly conditioned activation data.
  2. The approach is poorly motivated from a practical perspective and poorly justified by the experiments. The practical setting of OPTQ is the post-training quantisation of the weights of a large language model. The authors' experiments only demonstrate that under synthetic scenarios, existing approaches may not apply sufficient regularisation when the calibration set is small. Furthermore, they do so not on the actual reconstruction objective, which is what all the theory is focused on, but a different measure for no discernable reason. They do not actually provide a compelling reason for using their approach in a practical scenario (or at least I am struggling to see it). Is the point that you can set λ\lambda on a small calibration dataset and then scale it up for a larger one? The experiments show that performance for most methods converges for more calibration data, meaning the proposed approach doesn't seem to have an advantage. Besides, there are no experiments on real trained LLMs or even any real neural network activations, never mind downstream tasks, to provide any evidence that their approach has practical value. OPTQ LLM experiments are completely accessible to AI researchers in academia for smaller 8-9B models, so I don't feel like the authors can excuse this.
Questions for authors

Could the authors explain why they used the error measure they did in the experiments rather than the reconstruction error when the latter is both easy to evaluate and more relevant to their theory? Also why didn't they empirically evaluate the looseness of the bound they derived?

Reviewer_BpM7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the rounding quantization error of OPTQ on a calibration set relates to its error on unseen inputs from the same distribution or any distribution, controlled by a ridge regularization parameter lambda.

Critical strengths
  • The paper is extraordinarily well motivated and explained, including relevant background on quantization, OPTQ, and highlighting the central issue of separating calibration error from generalization error.
  • The theorem 3.3 is clear, and at this point in the paper clearly explained with all pieces before it.
  • A concreate way to pick the regularization parameter is practical and useful. I really enjoyed reading this paper and following the arugments.
Critical weaknesses
  • The theorem 3.1 applies to any quantizer, and is broadly useful beyond just OPTQ, but the result is fairly standard. Its worth noting that the contribution is applying the ridge regularization to quantization.
  • Based on the term in the square root in Theorem 3.1 and Corollary 3.2, the bound seems to require lambda > 2 R^2 log(N/epsilon). It is unclear if the demonstrated lambdas satisfy this. Can you clarify? Its unclear whether the settings shown in Figure 2 and Figure 3 satisfy these conditions, as well as Eq 31/34/35.
  • The numerator in paranthesis of Eq 22 is >= 1. I'm not sure this shows that OPTQ generalizes better than RTN. I would temper this and some of the other claims in the paper. For example R^2/E[||x||]^2 can be very large in modern models due to outliers and infrequently used tokens.
  • If you are to show experiments it would be better to compare with RTN or other baselines (Figures 1-3).
  • I found a few typos, and some notational slips. Its worthwhile to check your proofs carefully again.
Questions for authors
  • How does the rule perform on real layers of an LLM, measured by perplexity or task accuracy? (e.g. an open weights model)
  • Please clarify if lambda_2 is supposed to be scale invariant, because it appears to need to scale with X^T X
  • Please comment on tuning of lambda_1
  • Do the proofs offer an improvement over prior bounds? (please state them if so)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.