acceptodds
Under review as a conference paper at ICLR 2027

LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation

Abstract

Post-training quantization (PTQ) enables effective model compression with minimal accuracy loss. While most weight-only PTQ methods target the challenging sub-3-bit regime and often require fine-tuning, we propose LoPRo—a fine-tuning-free algorithm that improves residual matrix quantization via block-wise permutation and Walsh-Hadamard transforms to align columns of similar importance. We also introduce a mixed-precision, fast low-rank decomposition based on rank-1 sketching (R1SVD) to reduce quantization costs. Experiments show LoPRo outperforms existing fine-tuning-free PTQ methods at 2- and 3-bit (with its vector variant LoPRo delivering the best 2-bit results), matching fine-tuned accuracy while achieving up to 5 quantization speedup over GPTVQ. On Mixtral-8x7B, it finishes quantization in under 2.5 hours, reducing perplexity by 0.75 over MoEQuant++ (3.17 vs. 3.00 avg. bits) and boosting BoolQ accuracy by 5.4 points. LoPRo also achieves higher accuracy than other low-rank methods at much lower ranks, with low latency overhead (10.3% at batch 1, lower for larger batches).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, low-rank, rotation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yCdZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoPRo, a fine-tuning-free post-training low-bit weight quantization method for LLMs that combines low-rank decomposition with importance-aware partial rotation. It identifies a conflict between low-rank decomposition and global rotation caused by a negative relationship between Hessian-based importance and residual error, and addresses it by protecting important columns while applying block-wise Walsh–Hadamard rotations to the rest. The method also introduces R1SVD for efficient low-rank approximation and supports a vector-quantization variant. Experiments on multiple LLM families show competitive low-bit performance.

Critical strengths
  1. The paper identifies an interaction between low-rank decomposition and global rotation, motivating a partial-rotation strategy rather than applying global transformations indiscriminately.
  2. The importance-aware column ordering provides an intuitive strategy for preserving important residual components, while R1SVD reduces the computational cost of low-rank decomposition.
  3. The experiments cover multiple LLM families across different quantization settings. The results show promising performance compared with existing post-training quantization baselines, alongside fast quantization and modest additional memory overhead.
Critical weaknesses
  1. The theoretical justification relies on a diagonal Hessian approximation and additional simplifying assumptions, which may limit the generality of the analysis under practical neural network conditions.
  2. The experiments do not fully isolate the individual contributions of importance-aware column ordering and partial rotation, making it difficult to attribute the observed gains to specific design choices.
  3. The improvements vary across models and settings. It remains unclear whether the method provides consistent advantages when effective bit-widths and total storage overheads are strictly matched against strong baselines.
Questions for authors
  1. Could the authors provide additional ablations comparing importance-aware ordering with random ordering and no ordering, while keeping the rotation strategy and quantizer fixed?
  2. How consistent is the relationship between Hessian-based importance and residual quantization error across different layers and architectures? How sensitive is LoPRo to deviations from this relationship?
  3. How does LoPRo compare with vector-quantization and low-rank baselines when the total memory footprint is strictly matched, including the overhead introduced by low-rank components and vector-quantization codebooks?
  4. The paper reports inference latency overhead at batch size 1. Could the authors clarify how this overhead is distributed between the low-rank branch and rotation-related operations, and how it changes with batch size?
Reviewer_RbaaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LoPRo, a low-bit post-training quantization method that requires no fine-tuning. The paper first points out that activation-perceptual low-rank decomposition results in small residual quantization steps for important channels, and global rotation may disrupt this structure, leading to increased quantization error. To address this issue, the paper proposes a column sorting strategy based on channel importance and residual size, employing block-based partial rotation to protect important channels while improving the quantization distribution of other residuals. Empirical results show that the LoPRo method achieves competitive quantization accuracy while maintaining low additional memory overhead and inference latency.

Critical strengths
  1. The paper identifies a potential conflict between low-rank residuals and global rotations, and analyzed it from the perspective of the relationship between channel importance and quantization error.
  2. The proposed permutation-based partial rotation effectively improves residual quantization and supports both scalar and vector quantization.
  3. R1SVD effectively reduces decomposition time and storage overhead, making the method competitive even at lower Ranks.
  4. The experimental verification is comprehensive, covering multiple models, multiple methods, and multiple verification metrics.
Critical weaknesses
  1. Proposition 1 relies on a diagonal proxy loss and an incoherence assumption (A3) whose validity on real LLM residuals is not quantitatively verified. This limits the applicability of the theoretical guarantees to actual quantization errors.The proposed ranking score does not necessarily ensure homogeneous Hessian importance and quantization step sizes within blocks, as required by the theoretical analysis.
  2. The ranking score in Eq. (8) does not necessarily yield blocks with similar Hessian importance and quantization step sizes. The connection between the proposed ranking strategy and these assumptions needs further justification.
Questions for authors
  1. Proposition 1 relies on separate within-block bounds on Hessian importance and quantization step sizes. However, the ranking score in Eq. (8) is based on their importance-to-residual ratio, which does not necessarily ensure homogeneity in either quantity. Could the authors clarify how this ranking strategy relates to the assumptions of Proposition 1.
Reviewer_Gkm9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LoPRo, a fine-tuning-free framework for ultra-low-bit weight quantization of large language models. The authors observe that applying a full Hadamard rotation to the residual after activation-aware scaling and low-rank decomposition can reduce quantization accuracy. They explain this effect through a proxy loss based on the negative coupling between Hessian importance and residual quantization step sizes. LoPRo reorders columns by importance, protects the leading columns, and applies blockwise Hadamard rotations to the remaining columns. The paper also introduces R1SVD, a mixed-precision low-rank decomposition method based on rank-1 randomized sketching, to reduce decomposition time and storage costs. Experiments on LLaMA, Mixtral, and Qwen models show competitive accuracy and quantization efficiency in several 2-bit and 3-bit settings. The main contribution is identifying a problem with directly combining low-rank decomposition and rotation, and providing a structured solution that balances accuracy and computational cost.

Critical strengths
  1. Identifies and demonstrates performance degradation when full Hadamard rotation is directly applied after low-rank decomposition.

  2. Connects the method to this observation through importance-based permutation, protection of key columns, and blockwise rotation of the remaining columns.

  3. R1SVD provides practical speed benefits, while mixed-precision storage reduces low-rank factor overhead.

  4. Experiments cover dense and MoE models, scalar and vector quantization, and optional fine-tuning.

  5. The appendix provides substantial ablations on rank, decomposition methods, iteration count, block size, and calibration-data source.

Critical weaknesses
  1. The use of A3 to establish a nonnegative rotation gain, and the treatment of off-diagonal Hessian terms, require further justification.

  2. Some claims go beyond what the proxy-loss analysis directly supports.

  3. Latency and throughput changes are reported inconsistently, and the full deployment overhead of the main 2-bit and 3-bit settings remains unclear.

  4. R1SVD is mainly compared with full SVD, with limited comparisons against efficient randomized low-rank methods.

Questions for authors
  1. A3 bounds ϵmax⁡′\epsilon'_{\max} by κ0ϵˉB\kappa_0\bar{\epsilon}_B, but with κ0>1\kappa_0>1, this alone does not establish Δinc≥0\Delta_{\mathrm{inc}}\geq0 in Equation (20). What additional conditions justify this step, and how is “outlier-dominated” defined quantitatively?

  2. A1 and A2 do not directly control the dropped off-diagonal Hessian contribution, and the main text and appendix treat mϵ2/12m\epsilon^2/12 differently. How are the omitted terms bounded, and under what consistent quantization-error assumptions does the argument hold?

  3. The W4A16 baseline leaves the applicability of the efficiency results to the main 2-bit and 3-bit configurations unclear. Which LoPRo weight formats were tested, and do the measurements include permutation, FWHT, dequantization, and the low-rank branch for both scalar and vector quantization?

  4. Table 8 demonstrates faster decomposition than full SVD, but does not isolate the benefits over efficient randomized alternatives. How does R1SVD compare with optimized RSVD or truncated methods at matched ranks and similar iteration budgets, and how much of its benefit comes from rank-1 iterations versus mixed precision?

  5. Some comparisons use different average bitwidths, QTIP performs better on LLaMA-2 13B, and the Qwen results lack external quantization baselines. These limit the support for broad performance claims. Which model, bitwidth, and metric combinations establish an advantage at comparable storage budgets, distinguishing LoPRo, LoPRo_v, and their fine-tuned versions?

Comments on the paper

2 comments

Dear Reviewers,

I would like to kindly remind everyone to review the authors' responses and provide your feedback.

If you have any further questions or concerns regarding the authors' responses, please feel free to raise them early, so that the authors have sufficient time to address them before the rebuttal period ends.

Thank you all very much for your time and effort! And, have a nice weekend. :)

Best regards,

AC

Dear Reviewers,

I would like to kindly remind everyone to review the authors' responses and provide your feedback.

If you have any further questions or concerns regarding the authors' responses, please feel free to raise them early, so that the authors have sufficient time to address them before the rebuttal period ends.

Thank you all very much for your time and effort! And, have a nice weekend. :)

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.