acceptodds
Under review as a conference paper at ICLR 2027

OPTQ: Quantization Error is Relocated, Not Eliminated

Abstract

Post-Training Quantization (PTQ) inherently cannot eliminate the quantization error it generates; it can only determine where to distribute this error. The magnitude of the residual injected during rounding to a fixed grid is strictly determined by the grid itself, remaining identical across any algorithm. Consequently, what a quantizer actually controls is not the magnitude of the error, but the allocation of the error across channels, basis directions, and time. This paper formalizes this perspective. For an arbitrary linear operator receiving a quantized tensor, the distortion surviving to the output is dominated by the operator's spectrum. Thus, a good quantizer is one that steers the error toward the directions that the operator attenuates. This framework not only designates operators for each axis but also, equally importantly, predicts which axes are not worth optimizing. For instance, the time axis of the Value cache is dismissed prior to any empirical experiments, solely based on the observation that the measured lag-1 attention autocorrelation is approximately 0.06. By instantiating this framework, we derive a calibration-free sequential quantizer for activations, where the downstream operator is not estimated from data but exactly determined by the layer's weights. For the Key cache, we formulate an operator combining channel mean subtraction and per-head Hadamard rotation. Empirically, under a W4A4KV4 setting, our training-free configuration achieves a perplexity of 5.85 on Llama-2-7B, outperforming recent methods that rely on learned transformations (QuaRot, FlatQuant, SpinQuant, and OSTQuant). Incorporating transformations learned to pass through the relocation operator establishes a new state-of-the-art, yielding a perplexity of 5.714 on Llama-2-7B and 6.899 on Llama-3-8B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_W5HMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that rounding to a fixed grid introduces a fixed amount of error, so quantizers cannot reduce the error, only relocate it. It measures distortion via the weight (activation) "operator" that acts on the activation (weight) quantized tensor, and introduces OptQuant, a quantizer whose role is to push error in the direction the operator attenuates via a GPTQ-style sequential error feedback. Such feedback is applied independently to:

  1. activations, with no need of calibration data, using non-uniform Gaussian levels (via Lloyd-Max).
  2. weights, using GPTQ (needs calibration data), with block-Hadamard rotation and non-uniform Gaussian levels. K and V use a custom quantization strategy. In addition, learned rotations for the input activations of o_proj and down_proj is also proposed (ROTA-STE).

Results are reported on several models of the Llama family (1B-70B), with additional results on Qwen and EXAONE in the appendices. Evaluation covers perplexity (Wikitext / C4) and accuracy on several conventional zero-shot tasks. Additional measurements (more limited) on long-context and reasoning tasks are also reported.

OptQuant (typically inclusive of ROTA-STE) showing improved performance against various competing quantization methodologies at 4 bits (W4A4KV4). It also achieves large margins at W3A3KV3 and at 2-bit weight-only, and wins on LongBench with 3-bit KV caches.

Critical strengths

The main strength is providing a unified view of several quantization approaches (GPTQ, SmoothQuant/AWQ, rotations, ...), showing that error feedback (GPTQ-style) moves the error, while rotation shrinks it. The proposed OptQuant achieves remarkable performance improvement, especially under aggressive quantization regimes (3 bits).

Critical weaknesses

The activation quantization process requires a massive D x D FP32 (or FP16) matrix per linear module, wiping out any compression gain. The authors acknowledge this limitation in the appendix B.3, but the main text does not discuss it. In addition, activations and KV cache use non-uniform levels (from Lloyd–Max) which can't be accelerated in conventional INT4 HW. Efficiency measurements are limited to overhead ratios for the activation pre-processing relative to a layer GEMM (Table 13a, whose GEMM operations, precision, and layer are unspecified, and which does not appear to include the relocation matmul) and a fused-kernel study (Table 13b). There is no low-bit GEMM, no end-to-end latency or throughput measurement. Given these two drawbacks (per-token D x D compensation step and non-uniform activation levels), I expect the activation relocation scheme (arguably the key contribution of this paper) to be impractical on real HW as presented.

It should be also noted that all competing algorithms (to which OptQuant is compared) utilize evenly-spaced quantization levels for activations, hence the direct comparison (Table 1, for example) is not entirely fair.

Questions for authors

As per "Critical Weaknesses", I would ask the authors for measured decode/prefill latency and memory including the compensation factors. W4A4KV4 results with uniform INT4 activation/KV grids would also be helpful for a direct comparison against presented baselines.

Reviewer_8w2qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents OptQuant, a post-training quantization (PTQ) framework that treats quantization as an error-relocation problem rather than solely an error-minimization problem. It argues that, once the grid and scale are fixed, quantization error is not eliminated but relocated, and that downstream distortion depends on how the error aligns with the consuming operator’s spectrum. Based on this principle, OptQuant combines sequential error compensation, Hadamard transformations, Gaussian companding, and KV-cache quantization techniques. It also introduces ROTA-STE to learn transformations through the relocation operator. Experiments across multiple LLMs report improved perplexity and zero-shot accuracy, particularly under aggressive W3A3KV3 quantization.

Critical strengths

(1) An unified conceptual framing that places GPTQ, scaling methods and rotation methods under one objective.

(2) The eigenbasis analysis shows that relocating quantization error toward less sensitive directions can reduce downstream distortion, even when the raw quantization error increases.

(3) OptQuant demonstrates strong perplexity and zero-shot accuracy results across multiple LLMs, particularly under aggressive W3A3KV3 quantization.

(4) The paper evaluates different model architectures, bitwidth configurations, and benchmarks, including weight-only quantization, joint weight-activation-KV quantization, and long-context tasks

Critical weaknesses

(1) The paper emphasizes that its calibration-free/tuning-free configuration outperforms learned methods. However, it does not provide a controlled comparison with calibration-based activation quantizers.

(2) Furthermore, the overall pipeline still uses calibration data for GPTQ and K-cache mean estimation, while the best reported result includes learned ROTA-STE transformations.

(3) The ablation study mainly shows improvements from combining different quantization techniques. It does not clearly show how much improvement comes specifically from error relocation rather than Hadamard transformations, Gaussian companding, or ROTA-STE.

(4) The theoretical analysis assumes zero-mean, equal-variance, and uncorrelated rounding errors, but the paper does not verify these assumptions experimentally. Moreover, the proposed implementation uses nonuniform quantization and an approximate relocation method that compensates errors between blocks rather than after every quantized value. It is unclear whether the theoretical guarantees still hold for the method evaluated in the experiments.

(5) The paper does not include a dedicated related work section. Existing methods are discussed throughout the paper, but a more systematic comparison would help clarify the novelty relative to GPTQ, scaling methods, and rotation-based approaches. Several existing methods are mentioned in the introduction without citations at their first occurrence. The abstract also contains broken words, such as “direc- tions” and “tem- poral,” and parts of the paper are difficult to follow.

(6) The relocation operator requires additional compensation matrices, which substantially increase memory requirements and may limit practical deployment and scalability, particularly for large LLMs.

(7) Reproducibility. No code or reproducibility statement is provided. Given the number of custom components and modified baselines and several claims without supporting tables (e.g., Fast Mode’s ≈95% retention), releasing code would substantially help verify the results.

Questions for authors

(1) How does the activation quantizer compare with calibration-based activation quantizers under the same settings?

(2) How often does clipping occur after relocation and how does the method perform with uniform grids instead of Gaussian companding?

(3) Do the claims in Observation 1, Proposition 1 still hold for Fast Mode, which performs error relocation between blocks rather than after every quantized value?

Reviewer_oTTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This works introduces OPTQuant: a collection of strategies to address post training quantization from a number of different angles. In particular, the paper introduces a "error relocation" strategy for activation quantization, a centering and incoherence processing-based strategy for KV cache compression and a Gaussian companding strategy for weight quantization. The paper demonstrate the effectiveness of the proposed method on a wide range of tasks and setting against state of the art methods in the literature.

Critical strengths
  • The proposed OptQuant provides substantial performance improvements when compared to recent methods in literature
  • To the best of my knowledge, the paper is the first to introduce a dynamic error correction strategy for activation quantization.
  • Figure 1 provides a very useful overview of the method, substantially aiding the reader in understanding the various method components.
Critical weaknesses

Main concerns

  • The paper introduces a wide range of techniques and reports results for a many tasks and models. However, the main text reports little to no ablation studies to assess the effectiveness of each component. Overall the intuition regarding the impact of each choice is lacking.
  • The proposed OptQuant introduces a number of extra operations that need to be performed at inference time. For example, layers such as the down_proj require 2 rotations (RR and RTR^T), a error-relocation operation, a Hadamard transform and a Lloyd-Max dequantization. However, no runtime estimate is provided making the comparison with other method very difficult.
  • The paper mentions that one of the main contribution is the identification of the "axis that are not worth pursuing". However, very little content is provided in the main paper, with no illustration or in-depth study.
  • Sections 3.1-3.6 are very compressed and quite difficult to follow despite their central role in the paper.

Other issues

  • One of the main claims of the paper is that the "Quantization error is relocated, not eliminated". Under a fixed quantization grid, this statements holds true, however OptQuant and many other methods in literature use transforms preceding quantization. These linear operators can effectively change the size and shape of the quantization grid since the cell-size is determined by the activation (and weight) scale after the transformation. Furthermore, the proposed Lloyd-Max Gaussian companding strategy used for weight quantization also changes the grid-cells shape, reducing (and not relocating) quantization error.
  • The description of each one of the strategies introduced in the paper is very compressed with little to no illustration and intuition, making the method section feel very scattered and hard to follow.
  • The introduction and other sections often refer to quantities (e.g. lag-1 autocorrelation) that are not yet defined or sufficiently explained to provide much intuition
  • The experimental section focuses on reporting numerical gains instead of providing intuition or informative analysis.
  • The abstract contains some minor formatting issues.
Questions for authors
  1. Can the authors clarify how the activation error relocation strategy in section 3.2 is applied in practice and provide simple estimates of the overhead it introduces vs downstream performance benefits for a few bit-widths? Does the trade-off results in a strictly better solution then just having a finer quantization grid for the activations?

  2. How much does each on of the introduced components contribute to the model performance? The author report one numerical reference for each component in the main text but a detailed ablation would prove more impactful.

  3. Can the authors clarify the computational and downstream performance impact of the Gaussian companding weight quantization strategy? Why is it applied only to the down_proj layers? What is the expected computational overhead?

  4. Does OptQuant and the observations provided in the main text hold for different model families such as more recent Qwen architectures?

2 comments

Individual Response

Thank you for the constructive review. We answer the weaknesses (W) and questions (Q) in order.

W1. The revision moves a component-wise ablation into the main text (Q2), with 8B and MoE results in the appendix.

W2. Per linear layer: a block-Hadamard on the input (fast Walsh-Hadamard, inverse folded into the weights), one relocation pass, rounding, and the INT4 GEMM; the o/down transform exists only in the optional learned variant. We withdraw the 16.5% overhead in Table 13a (measured at d=4096 with 32k tokens; not representative). Measured relocation cost vs. the layer's FP16 GEMM on B200: 3.9–6.7× at 8k-token prefill and 9–30× at batch-1 decode (FLOP lower bound 0.5–1.75×); the compensation factor, stored losslessly in INT4 (128-wide groups), takes 1.1–1.5× the INT4 weight memory. These costs move into the main text.

W3. Moved into the main text. The temporal (Σ-Δ) gain on the V cache is bounded by the lag-1 autocorrelation of attention rows (Prop. 3); the measured ≈0.05 predicts near-zero headroom, as a direct implementation confirms. We add a second case: optimizing the transform to minimize the sum of relocation pivots reaches the AM-GM bound, yet raises kurtosis (3.0 → 3.9–5.1) and increases realized distortion by 1.3–2.7 dB.

W4, W6, W7, W9. Presentation. Section 3 is expanded by about one page (bit-ladder and long-context tables move to the appendix); each subsection opens with the error mode it targets and its intuition, referring to Figure 1; the lag-1 autocorrelation and operator metric are defined in Section 2 before use; the abstract is fixed.

W5. Agreed: the claim holds for a fixed grid, and the revision says so. Transforms, scales and companding set the grid and the residual's magnitude; relocation then decides its placement. Q2 quantifies both.

W8. We add analysis from the new experiments: which component helps at which bit-width (Q2), relocation versus finer scales (Q1), when companding matters (Q3), and where the method is costly (W2).

Q1. Per token, after the Hadamard, activations are processed in 128-channel blocks: each block is rounded and its error, propagated through a factor U precomputed offline from the weights (H = WᵀW), is subtracted from the channels not yet quantized. Llama-2-7B W4A4KV4, without → with relocation: per-token scales 6.06 → 5.82, 128-group 5.80 → 5.70 (8B: 7.93 → 7.12, 7.10 → 6.80). Without relocation, A5 gives 5.74 (per-token) / 5.65 (128-group) and A6 5.64 (per-token), so relocation is worth about 0.7 bit. Given its current cost (W2), a wider grid is cheaper where A5/A6 hardware exists; on INT4-only hardware relocation recovers part of that precision.

Q2. Llama-2-7B:

Weights (weight-only, group 128), W4 / W3 / W2: RTN 5.78 / 8.22 / 1.2e5 → +relocation (GPTQ) 5.63 / 7.53 / div. → +block-Hadamard 5.59 / 6.60 / 1.8e5 → +Lloyd-Max 5.56 / 5.86 / 8.46. Activations (W4A4KV4), without → with relocation: uniform per-token 6.06 → 5.82; uniform 128-group 5.80 → 5.70; Lloyd-Max 128-group 5.79 → 5.69. Relocation is the largest weight-path step at W4; Hadamard and Lloyd-Max become essential at W3/W2. On activations, relocation adds 0.10–0.24 PPL (8B: 0.22–0.81) and finer group scales a comparable amount, which our ablation row L6 had attributed to companding; we correct this. The learned transform is optional: the training-free configuration now matches it in PPL on all five models.

Q3. It is not specific to down_proj: Lloyd-Max levels are used for all weights. Quantization is offline; dequantization is a 16-entry lookup as in NF4/LUT-GEMM kernels, with no extra GEMM. Impact (7B weight-only): W4 5.59 → 5.56, W3 6.60 → 5.86, W2 1.8e5 → 8.46. For activations it is optional: with 128-group scales uniform INT4 is within 0.1 PPL at W4 (7B to 13B), while at 3 bits it still matters (8B W3A3KV3: 11.67 vs. 8.99).

Q4. The appendix already includes Qwen2.5-7B/14B and Qwen3-8B. New MoE results (WikiText-2 PPL; weight-only group 128, GPTQ / ours; W4A4 with FP16 KV, RTN / ours):

FP16W4W3W2W4A4
Qwen1.5-MoE-A2.7B7.227.48 / 7.348.89 / 7.67div. / 10.3825.7 / 7.44
Qwen3-30B-A3B6.096.34 / 6.237.57 / 6.576498 / 8.378.43 / 6.34
Mixtral-8x7B3.844.10 / 3.945.37 / 4.219205 / 5.44not run

The ordering holds on C4 and zero-shot accuracy (Mixtral W3: 63.9 vs. 69.6; FP16 70.6). Quantizing only the routed experts to W2 (attention and shared experts kept at higher precision) gives 7.90 / 7.30 / 4.97 vs. GPTQ's 16.0 / 24.4 / 40.4.

Reviewer_oTTAOpenReview ↗

Response to the Authors

I thank the authors for their clarification and updates to the paper, which contribute to increasing the readability of the submission.

However, after reading the main weakness identified by reviewer W5HM and the author response, I realize that the critical issue of the additional memory and compute required for the activation error relocation remains not sufficiently addressed. Furthermore, the claims of error relocation in the paper, and visualized in Figure 2 (appendix) need to be carefully tuned and supported by additional measurements since the statements do not seem to hold well for down_proj and k_proj layers. In general, providing a clear runtime and memory analysis for the proposed method is crucial to justify the gains against other methods in literature that do not require such a overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.