acceptodds
Under review as a conference paper at ICLR 2027

Curvature Is a Budget: Coordinates Before Resolution in Second-Order Quantization

Abstract

A weight rotation preserves the full-precision model but changes the Hessian that measures quantization error. Multiple public repositories contain code paths that pair transformed weights with untransformed curvature: we reproduce this error in ModelTC/LightCompress and identify the corresponding metric-collection pattern in spcl/QuaRot's export source. Users can unknowingly trigger this silent failure by omitting one setting in LightCompress: the run completes without a frame warning while Llama-3.1-8B perplexity rises from 6.525 to 7.456, worse than rotated rounding at 7.283. In a controlled four-bit panel, stale dense curvature makes GPTQ worse than round-to-nearest on five of seven checkpoints, while aligned block-32 curvature beats stale dense curvature on all seven. At three bits, the median stale-to-aligned dense-metric perplexity ratio reaches 25.31. We derive the exact output error of a complete GPTQ sweep under a mismatched metric and characterize when block projection commutes with coordinate transport, enabling exact transport within compatible block representations using O(db) stored coefficients. Curvature must be aligned before it is budgeted.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
post-training quantization, second-order optimization, coordinate transformations, Hessian curvature, GPTQ, large language models, model compression, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9MMfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Many of post-training quantization methods apply the (randomized) Hadamard transform HH to the weights to mitigate outliers and then GPTQ[1] minimization problem is solved: $ || Q(W H) H^{-1} X - W X ||_F^2 \rightarrow \min$. Here WW is the weight matrix and XX are input activations of a linear layer. Equivalently, the objective is the classic GPTQ minimization with a change of variables to W′=WH,X′=H−1XW'=WH, X'=H^{-1}X.

This quantizes the weights offline. During inference activations are transformed online as X←QRTN(H−1X)X \gets Q_{RTN} (H^{-1} X).

The paper investigates a failure mode when activations supplied to GPTQ are kept unchanged (XX instead of H−1XH^{-1} X), so the minimization becomes $ || Q(W H) X - W H X ||_F^2 \rightarrow \min$. Naturally, without the paired inverse Hadamard transform applied to the activations passed to GPTQ the minimization objective shifts and model quality degrades.

The problem the paper is solving is not substantial. As the authors point out "the failure is one configuration line away". Then the solution is one line away also. Concretely, the fix is to ensure that the inverse Hadamard transform is being applied to activations passed to GPTQ offline and to input activations online (online rotations). In existing codebases this is done by setting an argument online_rotate=True (an example from ModelTC[2]).

Therefore I believe the main question of the paper is 1) not significant and 2) already solved.

Experimental results I find doubtful, because authors report that for Qwen2.5-7B RTN performs better than GPTQ (both without any transform). Theoretical results provided by the authors I find not very relevant, since the authors themselves describe a clear fix to the failure mode by changing a configuration.

[1] https://arxiv.org/abs/2210.17323

[2] https://github.com/ModelTC/LightCompress/blob/main/docs/en/source/practice/quarot_gptq.md

Critical strengths
  1. Interaction between different quantization related techniques is an important question to study. Applying rotation transforms before quantization introduces an additional error: Q(WH)Q(H−1X)≠Q(W)HH−1Q(X)=Q(W)Q(X)Q_{} (W H) Q(H^{-1} X) \neq Q_{} (W) H H^{-1} Q(X) = Q_{} (W) Q(X).
Critical weaknesses
  1. Contribution of the paper is marginal. A simple solution already exists: pass the online_rotate flag equal to True. Defaulting it to True is an engineering fix but it is overall not a scientific problem. Constructing transforms that commute with quantization I believe is excessive in this case.

  2. Experimental results provided do not appear to be reliable. In Table 1 authors show that for 2 models under no transform RTN quantization outperforms GPTQ, which I find hard to believe.

Questions for authors
  1. How would you explain that for Qwen2.5-7B and Llama-3.1-8B under no transform PPL with RTN is lower than with GPTQ? (Table 1)
Reviewer_BpUgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies coordinate mismatch between transformed weights and the curvature supplied to second-order quantization. It derives the actual layer output error of a complete GPTQ sweep under a mismatched metric and characterizes when block truncation commutes with coordinate transport. Controlled experiments demonstrate that stale curvature can substantially degrade quantization, and a public-code configuration case illustrates a practical risk.

Critical strengths
  1. The paper systematically distinguishes the objective used by GPTQ from the actual layer output error. Its analysis of when compressed curvature representations can be transported consistently provides useful guidance for designing quantization pipelines under storage constraints.

  2. The controlled experiments isolate metric alignment while holding transformed weights and quantization settings fixed. The finding that aligned block curvature can outperform stale dense curvature demonstrates the potential severity of mismatch. The reproduced LightCompress configuration failure establishes a concrete implementation risk beyond synthetic comparisons.

Critical weaknesses
  1. The conceptual and theoretical novelty appears limited. Consistently transforming the Hessian is a standard coordinate-change requirement already respected by methods such as QuIP and QuIP#. The complete-sweep error identity extends an existing residual factorization to a mismatched metric, while the block-compatibility criterion follows from standard projection and invariant-subspace properties. Although useful, the incremental theoretical contribution is not sufficiently compelling to support acceptance.

  2. The practical prevalence and broader impact of the failure remain insufficiently established. The LightCompress experiment demonstrates a reachable inconsistent configuration, but the official combined example enables the appropriate setting; the QuaRot finding concerns a specific export path rather than establishing a general flaw in the method. Repeating mismatch experiments across models establishes potential severity, not prevalence in actual workflows. The large improvements over stale-curvature baselines therefore primarily demonstrate correction of an inconsistent setup, rather than an advance over correctly implemented rotation-based quantization.

Questions for authors
  1. Beyond the identified configuration and export-path cases, what evidence establishes that coordinate mismatch occurs in commonly used workflows or affects published results? Please distinguish potential misuse, executable implementation defects, and confirmed effects on existing evaluations.

  2. What is the principal nontrivial theoretical advance beyond the known coordinate transformation rule, existing GPTQ residual factorizations, and standard projection-commutation properties? Please clarify what new design capability or prediction follows beyond ensuring correctness.

Reviewer_nv3wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies and studies the problem of mismatch in coordinate frame between the transformed weights and the metric used to quantize these weights. The problem is studied both mathematically and empirically, investigating the impact of the mismatch in the GPTQ quantizer, and considering the standard block cache strategy to reduce storage.

Critical strengths

Identifying the mismatch in coordinate frame is in itself an interesting finding.

The mathematical analysis is interesting and clearly exposed.

The experiments highlight the empirical validity of the observed problem.

Critical weaknesses
  1. Clarity: The paper is in general clear, but the introduction uses terms that are not truly obvious at first. For example, at Line 036, “live in different stages”; the discussion at Lines 050-052. I recommend the authors to proof-read and clarify the message in the introduction.

  2. Methodology:

2.1: In Corollary 3.2, how realistic is the assumption that H is positive definite? This essentially assumes convexity.

2.2: The mathematical analysis focuses on weight quantization. How about activation quantization?

2.3: The mathematical analysis focuses on linear (fully-connected) layers. How about other types of layers, in particular attention ones?

  1. Experiments: Many of the numbers for “earlier coordinates” in Tables 16 and 17 look terrible. It therefore seems surprising that this was not noticed before, no?
Questions for authors

See Weaknesses above

Comments on the paper

1 comment

Dear Reviewers, Area Chairs,

If there are any other concerns or comments, we are happy to clarify and answer specific questions, including any additional concerns you may have after our rebuttal. We really enjoyed some of the rebuttal exchanges and look forward to updating the paper with the suggestions and comments.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.