acceptodds
Withdrawn from ICLR 2027

Data-Free Transformer Quantization Using Parameter-Space Symmetry

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformer models are widely used in many learning tasks but incur large memory and compute costs, limiting their deployability. Post-Training Quantization (PTQ) is a promising solution but can lead to significant performance degradation. Many PTQ methods estimate weight and activation distributions with calibration data to account for outliers and maintain quantized performance. We propose a data-free approach to improve quantization by exploiting parameter space symmetries. We address outliers and high variability in weights by finding a transformation of the model weights that minimizes quantization error variance. Our approach is light-weight, data-free, and can be integrated as a pre-processing step within other PTQ methods. We evaluate our approach by testing quantized large language models on several benchmark tasks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
quantization, parameter-symmetry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rRNUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Transformer quantization method that exploits parameter-space symmetry in attention. It jointly transforms Query and Key weights while preserving attention scores, and uses an analytical estimate of quantization noise to guide optimization without calibration samples.

Critical strengths
  1. The joint Query–Key transformation preserves attention scores while changing weight distributions, providing a principled way to improve quantization without altering the full-precision model.

  2. The method requires no calibration samples and can be combined with existing quantization methods, including AWQ and bitsandbytes, making it easy to integrate into existing quantization pipelines.

Critical weaknesses
  1. The motivation for data-free quantization is not sufficiently established. The paper does not clearly explain when avoiding calibration data is necessary or what practical limitations of existing calibration-based methods it addresses. Since the method analytically estimates quantization noise without calibration samples, “calibration-free” may also better describe its setting and distinguish it from approaches that synthesize calibration data.

  2. The experimental comparisons with existing methods are limited. Prior work, including SmoothQuant (Xiao et al., 2023) and OmniQuant (Shao et al., 2024), has explored function-preserving transformations for quantization, including transformations within attention. Comparisons with these methods and prior Q–K reparameterization approaches would help clarify the practical advantages of the proposed method.

  3. The analytical noise estimate relies on assumptions about quantization errors and activation distributions, but its agreement with actual quantization errors is not fully clear. Although Fig. 1(b) shows that the estimated noise decreases during optimization, its trend does not fully align with the empirical noise, particularly in later iterations.

  4. The experiments mainly cover BERT and relatively small OPT models, and the gains vary across tasks and quantization settings. Evaluating more model scales and bit-widths would help clarify how well the method generalizes, particularly under low-bit quantization.

Questions for authors
  1. To what extent does the analytical noise estimate reflect actual quantization errors across different transformations and quantization settings?

  2. How does the proposed Q–K transformation compare with existing Q–K reparameterization methods in terms of quantization performance?

Reviewer_BR9YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a data-free weight quantization method. The authors define the quantization error of the attention scores in terms of the quantization error of the query and keys weights. They define the error of the weights as uniform and the activation as gaussian with the layer norm learned parameters as the distribution moments. The authors show a closed form solution for the quantization error of the attention scores under these assumptions. To reduce the quantization error the authors propose to right multiply the query embedding with an orthogonal matrix and left multiply the keys embedding with the inverse of the matrix. Then the authors propose to minimize the expected error wrt the orthogonal matrix. The authors evaluate their method on OPT-350M and OPT-1.3B on language generation and zero-shot tasks.

Critical strengths

None.

Critical weaknesses

This paper does not meet the standard for publication at ICLR due to substantial weaknesses in its writing, methodology, novelty, and experimental evaluation.

  • Writing: The paper contains missing in-text references (e.g., L25–30), inconsistent notation (e.g., WVW_V and WvW_v in Sec. 3.1), and incorrect cross-references (e.g., L202 should point to Appendix 1 but instead links to Sec. 3.2). Figures are unnumbered, and an AI use statement is missing. Citation formatting is sometimes incorrect (e.g., L345), and some bibliographic entries are inaccurate: AWQ appeared at MLSys 2024, rather than as a 2026 arXiv paper, and the bitsandbytes citation is incorrect.
  • Methodology: The method applies only to models with LayerNorm. Furthermore, it assumes that activations are Gaussian-distributed, using the learned LayerNorm parameters as their moments. This assumption requires justification. Although studied in a different model, [1] provides evidence that activation distributions are generally not Gaussian.
  • Limited novelty: Numerous papers already optimize orthogonal matrices and, more broadly, invertible transformations (see, for example, [2] and the related work discussed therein). The paper does not adequately address these connections or clarify its novelty relative to prior work.
  • Experiments: The experimental comparison is insufficient. The evaluation is limited to small models, omits relevant baselines such as GPTQ, and does not include zero-shot downstream tasks beyond perplexity evaluation, as is common in the literature.

[1] Son, D., Choi, E., & Yoo, S. (2026). NSNQuant: A double normalization approach for calibration-free low-bit vector quantization of KV cache. Advances in Neural Information Processing Systems, 38, 43124–43159.
[2] Ofir Gordon, Lior Dikstein, Arnon Netzer, Idan Achituve, and Hai Victor Habi. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:36315–36346, 2026.

Questions for authors

.

Reviewer_A6W3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies data-free post-training quantization in terms of parameter-space symmetries in attention layers. The author apply orthogonal transformations to the query and key and meanwhile perserve their product. Experiments on BERT and OPT show that the proposed method can somehow improve the quantized model performance compared to some baseline methods like AWQ.

Critical strengths
  1. The proposed method is lightweight.
  2. The paper provides quantization-noise objective analyses.
  3. The proposed method is evaludated by experiments.
Critical weaknesses
  1. Error in the theoretical analysis. The XX is missing in Proposition 4.2 and Appendix, which seems to be a critical error.
  2. Limited novelty. The paper lacks the comparison with QuaRot, SpinQuant, DuQuant and FlatQuant. These methods have already explored orthogonal rotation method for post-training quantization, which seems to be similar to the main idea of this paper.
  3. Limited experiments. The paper only evaluates the proposed method on BERT and OPT, and the ppl on opt-1.3B under w8a8 settings is extremely high (4193.2). Also, comparisons with stronger baseline methods are missing in the experiments.
Questions for authors

See weakness 1-3.

Comments on the paper

1 comment

We thank the reviewers for their feedback and insightful comments. We are especially grateful for the additional citations and baseline recommendations. Reviewers point out that many of these baselines also explore orthogonal rotations for improving quantization but to our knowledge we are the only fully data-free approach. We will withdraw the paper at this time and work on addressing these concerns and clarifying the unique aspects of our approach in a future revision of the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.