acceptodds
Under review as a conference paper at ICLR 2027

OffQ: Taming Structured Outliers in LLM Quantization by Offsetting

Abstract

Low-bit quantization is now a standard approach for accelerating large language model (LLM) inference, as it substantially reduces memory usage and compute cost. However, structural activation outliers remain a major obstacle to effective quantization. In this paper, we introduce OffQ, a post-training quantization method that mitigates this issue through a novel offsetting mechanism. Specifically, OffQ first identifies a low-dimensional outlier subspace in the activations using a tailored top-1 PCA, and then concentrates high-magnitude activations into a single channel by rotation. It then absorbs this concentrated outlier energy into a shared offset, reducing the variance of the activations and making them more amenable to uniform-grid, uniform-precision quantization. This strategy enables accurate W4A4KV4 quantization of LLMs while preserving efficient low-bit execution. Experiments across diverse LLM architectures and benchmarks show that OffQ consistently outperforms state-of-the-art baselines, improving accuracy while maintaining low-bit efficiency. Code will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM quantization, singular defect, W4A4KV4

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QpSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OffQ, a post-training quantization (PTQ) method for the W4A4KV4 setting. Its central observation is that activation outliers in LLMs are structured: they can be modeled as x=μ+ϵx=\mu+\epsilon, where μ\mu is a high-norm constant direction shared across layers, token positions, and sequence locations (e.g., "singular defects"/massive activations, or attention-bias-induced outliers in Qwen 2.5). Building on this, OffQ introduces an "Outlier2offset" equivalent transform: (1) a Top-1 PCA identifies the low-dimensional outlier subspace and rotates activations so outlier energy concentrates into as few channels as possible; (2) channels are reordered and partitioned into groups, with one outlier channel per group; (3) a group-wise Hadamard rotation converts the outlier energy into a constant per-group offset absorbed into the zero-point of standard asymmetric quantization. All transforms except three online Hadamard rotations are fused into weight matrices. Experiments cover Llama 2/3/3.2 and Qwen 2.5 (1B–72B) under W4A4KV4, reporting WikiText perplexity and zero-shot accuracy on eight commonsense tasks, plus vLLM end-to-end latency (1.5–1.7× prefill, 2.0–2.4× decode speedup vs. 16-bit) at an average effective activation bit-width of 4.25.

Critical strengths
  1. The "Outlier2offset" formulation is a genuinely new equivalent transform, and it is elegantly motivated. Prior rotation-based PTQ (QuaRot, SpinQuant, DuQuant, etc.) aims to spread outlier energy into a Gaussian-like distribution. OffQ instead concentrates the structured outlier into a single channel and converts it into a per-group constant absorbed by the asymmetric zero-point. The "shooting at the same target" argument in Section 3 gives a clean optimality condition (t1=⋯=tDt_1=\dots=t_D), and the observation that any orthogonal matrix with a constant first row satisfies it is a nice touch, since it makes the technique composable with learned rotations. This is the main reason the paper is worth discussing.

  2. The empirical results under a hard setting are broad and consistent. W4A4KV4 with uniform grids and uniform precision is a demanding, deployment-relevant setting, and the paper evaluates ten models across two families. The gains over QuaRot/SpinQuant/DFRot/KurTail/OSTQuant are consistent, and the fact that OffQ beats ResQ (which spends 4.5 effective bits on mixed precision) while using 4.25 bits is a meaningful data point. The kernel-level integration into vLLM with zero-point support added to the humming INT4 path, plus measured TTFT/TPOT, strengthens the practical claim considerably.

Critical weaknesses
  1. The cost–benefit justification for Top-1 PCA is not established, and the key ablation is missing. Per-channel magnitude detection is an O(D)O(D) single pass and maps directly onto the first-principles fact that activation magnitude drives the output O=WXO=WX. Top-1 PCA instead requires sampling tokens, computing a covariance, and performing an eigendecomposition. PCA's theoretical advantage is that it accounts for the multi-channel covariance structure; but by selecting only the single L∞L^\infty-largest token per sequence, the paper collapses that global view onto an extremely small sample, arguably discarding the very advantage that motivates using PCA. The ablation in Table 2 only compares Top-1 PCA against standard PCA over all tokens (PPL 6.98 vs. 8.27), which shows that normal tokens dilute the leading eigenvector — but it does not compare against the cheaper, more first-principles alternative of per-channel magnitude selection combined with the same rotation/offsetting pipeline. Without that comparison, the paper does not answer whether the accuracy gain of Top-1 PCA justifies its extra cost, nor whether the gain comes from PCA per se or simply from focusing on extreme tokens (which magnitude-based selection would also do). This is consequential because the entire "Step 1" of the pipeline, and the claimed novelty of the outlier-subspace identification, rests on this choice.

  2. The rotation itself is not fundamentally different from mature methods; the contribution is a change of rotation objective, and the independent cost of the shared offset is never isolated. Rotation as a quantization preprocessing step is well established. OffQ changes the objective from "disperse" to "concentrate + convert to offset." That is a legitimate new equivalent transform, but the paper's framing somewhat overstates its distance from prior work. More importantly, the design has a real, unmeasured accuracy cost: an independent per-channel bias (i.e., retaining outlier channels at higher precision, as ResQ does) is theoretically strictly more expressive than a shared per-group offset, because it can represent each outlier channel exactly rather than forcing GG outlier channels to share one constant within a group. OffQ substitutes a shared offset for independent high-precision retention, yet the paper only measures this substitution indirectly through end-to-end PPL. The fact that OffQ at 4.25 bits beats ResQ at 4.5 bits shows that other design choices (Top-1 PCA + sorted grouping) outweigh the loss from sharing, but it does not show that the shared offset itself has smaller quantization error than independent retention. No ablation separates the isolated accuracy cost of "shared offset vs. independent high-precision outlier channels." This matters because a reader cannot tell how much of OffQ's headroom is being left on the table by the offset-sharing design, nor whether the method would remain competitive if the offset were replaced by exact retention at equal bit-width.

Questions for authors
  1. Please provide the missing ablation separating the offset-sharing cost: compare (a) OffQ with GG-group shared offsets, (b) OffQ with the top-GG outlier channels retained at higher precision (independent per-channel bias) instead of being absorbed into zero-points, at matched effective bit-width, and (c) a hybrid. This would quantify how much accuracy is lost specifically to offset sharing, independent of Top-1 PCA and grouping.

  2. How is the number of groups GG (or group size) selected in practice, and is it chosen per model or fixed? Figure 3 shows a clear accuracy/storage trade-off; is there a principled criterion, or is 128 chosen empirically? Also, how sensitive is OffQ to the choice of the L∞L^\infty token selection when a sequence contains multiple competing outlier tokens?

  3. For the Qwen 2.5 bias-induced outliers, U3U_3 is computed per-layer, per-head via Top-1 PCA. Does the number of such bias-outlier channels ever exceed the number of groups GG, in which case some outliers cannot be absorbed? How does OffQ behave in that regime?

5 comments

I will keep my score.

We would like to thank Reviewer QpSr for the constructive and valuable comments. Below we answer the questions in detail. Table numbers continue from the revised paper and across our responses for easier reference.

Per-channel magnitude detection is an O(D) single pass, while Top-1 PCA requires sampling tokens, computing a covariance, and an eigendecomposition.

Per-channel magnitude detection and Top-1 PCA have similar costs, because the calibration bottleneck is network inference; token sampling, covariance, and eigendecomposition are negligible. In practice, calibrating Llama3-8B takes 76 seconds on an H100 GPU with either per-channel magnitude detection or Top-1 PCA.

Does the number of Qwen 2.5 bias-outlier channels ever exceed the number of groups (G)? How does OffQ behave in that regime?

Yes, the number of outlier channels can exceed the number of groups (each head forms one group), as shown in Fig. 4(b) in Appendix A. OffQ still works well, as shown by its competitive performance in Tab. 1, because it concentrates all outlier channels within a head into a single channel and then absorbs it.

Compare against per-channel magnitude selection combined with the same rotation/offsetting pipeline.

The core of OffQ is identifying the structured outliers and converting them into offsets. Per-channel magnitude selection is a less general variant of OffQ: it replaces Top-1 PCA in the identification step and relies on the outlier channels being sparse. It works in that case, e.g., on Llama (Tab. 16), but fails when the outliers span more channels than groups, as shown by the severe degradation on Qwen2.5-1.5B/3B/7B (Tab. 16). Top-1 PCA is robust in both cases: it finds the outlier direction even when it spans many channels, and concentrates it into one channel. We have added a new discussion of this design choice (per-channel magnitude vs. Top-1 PCA) to Appendix B of the revised paper.

Table 16. WikiText2 PPL of the OffQ variant using per-channel magnitude selection on Llama (L) and Qwen (Q)

OffQ variantL3-8BL2-7BL2-13BL3.2-3BL3.2-1BL3-70BQ2.5-1.5BQ2.5-3BQ2.5-7BQ2.5-14BQ2.5-32BQ2.5-72B
Per-channel magnitude7.045.785.098.8912.483.869111.6113.134432.136.095.574.30
Top-1 PCA6.985.775.118.7812.323.8811.358.987.666.075.524.29

Retaining outlier channels at higher precision is strictly more expressive than a shared per-group offset

Converting an outlier channel into an offset incurs no accuracy cost. The group-wise Hadamard maps the outlier channel to a constant added to every element of the group. This constant does not change the range of the group, so the remaining values are quantized exactly as if the outlier were absent. Meanwhile, the constant is stored in the BF16 zero-point, so all the information in the outlier channel is represented by the offset.

Ablation: (a) shared offsets, (b) top-G outlier channels retained at higher precision, at matched effective bit-width, and (c) a hybrid.

We implemented a variant of OffQ that keeps the top-G outlier channels in high precision. The same 32 outlier channels identified by Top-1 PCA are kept in BF16 instead of being converted into offsets. For the remaining 4-bit activation channels, we first apply a random rotation to flatten them and then partition them with a group size of 127, so that the number of groups is the same as in OffQ. The results in Tab. 17 show that the high-precision outlier-channel variant has a much higher PPL (7.92 vs. 6.98), because the BF16 outlier channels do not work well with INT4 weights. It matches OffQ (6.96 vs. 6.98) only when the corresponding 32 weight columns are also kept in BF16 (as in ResQ), which is not a fair comparison for OffQ, since OffQ quantizes all weights to 4 bits.

Regarding the hybrid setting (c), could you please elaborate on it? Thank you!

Table 17. Llama3-8B with high-precision outlier channels for OffQ

MethodActivationWeightWikiText2 PPL
OffQ variant32 BF16 channels; group size 127 for 4-bit channels4-bit7.92
Same as above32 BF16 columns; the rest 4-bit6.96
OffQ4-bit, group size 1284-bit6.98

How is the group size selected, and is it chosen per model or fixed?

The group size of 128 is chosen empirically based on Fig. 3: more groups absorb more outlier channels but store more scales and zero-points; 128 (4.25 bits on average) balances the two. We use a fixed group size of 128 for all models in our experiments, and it works well.

How sensitive is the L∞L^\infty token selection to multiple competing outlier tokens?

OffQ is very robust in this respect, as shown by the various model sizes and model families in our experiments. According to the study of Wang et al. (2025), all singular-defect tokens share the same direction, regardless of their positions in the sequence. Therefore, the method works even when a sequence contains multiple competing outlier tokens.

Dear Reviewer QpSr,

Thank you for your message. As it was posted before our response, we would like to kindly invite you to read our rebuttal, which includes the requested ablations (Tabs. 16–17). We would be grateful if you could reconsider your evaluation in light of these results, and we are happy to address any remaining concerns.

Reviewer_4QpVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a novel rotation method that introduces the top-one PCA. followed by a random Hadamard transformation. Then the suggested method is extended to group quantization setting with ordering of channels that the g^{th} largest outlier will be in group g, making it a permutation on top of the PCA. Finally the paper presents results on Llama and Qwen 2.5.

Critical strengths

(1) The idea of top-1 PCA base rotation is very nice and might be very useful beyond the scope of this work.

(2) Simple method: the paper presents a simple method which doesn't require an optimization. The idea of using top-1 PCA looks interesting and helpful. Note that this is base on the analysis and results of Table 3.

(3) The related works are well organized and rightly position the paper.

Critical weaknesses

(1) Experimental results: Table 1 does not present an equal compression as stated by the authors (line 349 "OffQ uses per-group asymmetric quantization with a group size of 128, whereas the baselines follow their official implementations and use per-token asymmetric quantization"). Only Table 3 present a fair comparison but it is limited to PPL and two models. Moreover, the paper only uses two models Qwen 2.5 and Llama which are relatively old.

(2) Related works: A key missing part of the related works is the microscaling data format which utilizes group quantization. As this paper addresses group quantization adding references to microscaling work [R.1, R.2, R.3] would be helpful since these works have already introduced block Hadamard transformations, which are used in this work and even learned affine transformations for group quantization. Moreover, as the authors already suggested this as future work in line 471 this may increase the importance and contribution of this works.

(3) Structure limitation: A major limitation of this work is the fact that in order to fold the matrix it requires a specific architecture structure as shown in Figure 2. However, in newer models typically there are additional RMS norm layers with scale per channel that prevent the matrix folding into the weights.

Questions for authors

(1) Are the benefits of the suggested approach are limited to group quantization or are they applicable to non group quantization ? mostly the top-1 PCA

(2) Can the authors create a comprable set of experiments under the same quantization setup for equal compression ? as it is hard to judge the benefit of the suggested method?

(3) Can the authors discuss the structure limitation of the suggested method as in new model the matrices maybe not be folded. A notable example of such a model is Gemma4 which uses RMS norm layers (with per channels scales ) at the outputs of FNN and Attention blocks and preventing the folding.

(4) Is max token PCA is the "best" approach? or could other selection methods may help?

3 comments

We thank Reviewer 4QpV for the valuable comments and suggestions. Below we answer the questions in detail. Table numbers continue from the revised paper and across our responses for easier reference.

Are the benefits applicable to non-group quantization?

Yes, although OffQ works best with per-group quantization. With one group per token, Top-1 PCA still concentrates the outlier into one channel, which is absorbed as the per-token offset. As shown in Tab. 11, compared with the Tab. 1 baselines, which use the same per-token quantization (i.e., equal compression), it outperforms QuaRot on all 12 models and is comparable with SpinQuant and DFRot, with the largest gains on Qwen2.5 (e.g., 6600 → 13.24 on 1.5B).

Table 11. WikiText2 PPL of OffQ without group quantization on Llama (L) and Qwen (Q)

MethodL3-8BL2-7BL2-13BL3.2-3BL3.2-1BL3-70BQ2.5-1.5BQ2.5-3BQ2.5-7BQ2.5-14BQ2.5-32BQ2.5-72B
OffQ, non-group7.386.015.319.4513.615.1113.249.848.206.645.894.70

Experiments under the same quantization setup for equal compression

We also equip all runnable baselines with the same per-group (size 128) asymmetric activation quantization, extending Tab. 3 to all models with PPL/0-shot$^8$ (Tabs. 12–13). Other Tab. 1 methods are omitted because their code does not support per-group quantization or fails to run in this setting. OffQ achieves the best PPL on most models; OSTQuant has slightly better PPL on Llama3.2-1B/3B. Note that OSTQuant, SpinQuant, and DFRot are learning-based and expensive: on Llama3-70B, OSTQuant trains for 17.8 h on 8 H100s, 1100$\times$ the GPU hours of OffQ's 7.3-min calibration on one H100.

Table 12. Llama, per-group activation quantization

Method3-8B2-7B2-13B3.2-3B3.2-1B3-70B
QuaRot7.40/64.046.02/61.925.32/64.209.39/58.3113.03/49.315.30/69.07
SpinQuant7.22/63.605.87/61.635.24/65.289.06/58.7513.09/50.155.15/68.70
DFRot7.26/63.525.95/60.945.23/63.939.21/58.7813.02/49.734.53/70.44
OSTQuant7.15/65.145.87/61.615.16/64.988.75/58.7111.91/51.404.15/70.71
OffQ6.98/65.495.77/61.995.11/65.258.78/60.8012.32/50.913.88/70.63

Table 13. Qwen2.5, per-group activation quantization

Method1.5B3B7B14B32B72B
QuaRot5864.74/39.3359.99/49.493417.89/38.996.33/68.775.67/69.514.46/72.44
OffQ11.35/57.538.98/61.477.66/66.166.07/69.205.52/69.594.29/72.68

Structural limitation in newer models such as Gemma4, whose post-norms prevent folding

OffQ needs only simple extensions for Gemma4.

  1. Outlier pattern. We extend the structured outlier in Eq. (1) from a vector to a subspace. In Gemma4-12B, outlier channels are shared only across adjacent layers (e.g., outlier channel 734 in layers 13–23, 2467 in 24–40, 1276 in 42–46), but overall they lie in a small sparse subset, which per-group quantization can handle.
  2. Non-fusable rotation. Since these outliers are sparse, the simpler absmax variant of OffQ suffices: we identify them with running absmax statistics before the q/k/v and gate/up projections, and use a permutation instead of the Top-1 PCA rotation. With group size 128 and hidden size 3840, 30 outlier channels can be absorbed. Unlike a rotation, a permutation passes through norm layers. The 128$\times$128 Hadamard is fused into the quantization kernel, so the runtime overhead is negligible.

As shown in Tabs. 14–15, QuaRot needs costly non-fusable rotations and still fails on W4A4KV4 Gemma4-12B, while OffQ reaches 9.66 PPL (BF16: 8.53) and retains 97.4% of the BF16 0-shot accuracy. This also shows OffQ remains effective on a recent model family, addressing the concern about older models.

Table 14. W4A4KV4 Gemma4-12B, WikiText2 PPL

MethodPPL
BF168.53
QuaRot1.8e9
OffQ9.66

Table 15. W4A4KV4 Gemma4-12B, 0-shot accuracy

Methodarc_carc_eboolqhellasobqapiqasiqawinoAvg
BF1661.6985.9884.6281.1146.0082.9253.7974.2771.30
OffQ59.8183.9683.5579.0646.0079.6552.1571.2769.43

Is max-token PCA the "best" approach?

Top-1 PCA is the robust default: it identifies structured outliers whether they are sparse or spread over many channels (e.g., absmax fails in Qwen2.5 due to dense outlier channels in bias terms; see Tab. 16). When the outliers are sparse, absmax selection is a simpler variant, which we use for Gemma4 to avoid the non-fusable online rotation. We have added a new discussion of this design choice (per-channel magnitude vs. Top-1 PCA) to Appendix B of the revised paper.

Adding references to microscaling work [R.1, R.2, R.3] would be helpful

Thank you. We are glad to add them to the related work. Could you kindly provide the details of these references?

Thanks the authors for detailed response, they have address all of my concretions. Importenetly the extending of Table 3 which provide a deatiled compression under the same setup. About the structure limitations, as it requires a new kernel with Hadamard transformation I suggest to highlight this point and also cite [R.1] as they shown that it is indeed negligible. I will raise my score

Also, I am sorry for missing the references:

[R.1] Egiazarian, Vage, et al. "Bridging the gap between promise and performance for microscaling FP4 quantization." International Conference on Learning Representations. Vol. 2026. 2026.

[R.2] Gordon, Ofir, et al. "LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs." Forty-third International Conference on Machine Learning, ICML 2026.

[R.3] Shao, Yuantian, et al. "Block rotation is all you need for mxfp4 quantization." arXiv preprint arXiv:2511.04214 (2025).

Thank you very much for the positive feedback and for providing the references.

Following your suggestion, we have cited [R.1–R.3] in the revised paper (L471). We have also added the Gemma 4 extension to a new Appendix E, highlighting that it requires a quantization kernel with the fused Hadamard rotation, and citing [R.1], which shows that its overhead is negligible.

Reviewer_XvaSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OffQ, a rotation-based LLM quantization method that incorporates Top-1 PCA to mitigate activation outliers.

Critical strengths
  1. The paper is well written and easy to follow, with a clear discussion of related work.
  2. The experiments are comprehensive, covering multiple models and representative baselines.
Critical weaknesses
  1. The permutation strategy has already been explored in DuQuant, which somewhat limits the novelty of the proposed method.
  2. Figure 1 could be improved to more clearly illustrate the distribution of activation outliers.
  3. The discussion in Appendix C appears inaccurate, as both DuQuant and FlatQuant quantize all activations, including query, key, and value activations.
Questions for authors

How does OffQ perform under MXFP4/NVFP4 quantization compared with recent methods such as MR-GPTQ, Micro, and DuQuant++?

1 comment

We would like to thank Reviewer XvaS for the encouraging and insightful comments. We have updated the manuscript based on the feedback. Below we provide detailed answers to the questions. Table numbers continue from the revised paper and across our responses for easier reference.

How does OffQ perform under MXFP4/NVFP4 quantization compared with recent methods such as MR-GPTQ, Micro, and DuQuant++?

We implemented OffQ under MXFP4 quantization following the setup in the official codebase of DuQuant++, and evaluated it on Llama3-8B and Llama3.2-3B. MXFP4 already provides a shared scale per block of 32 elements; OffQ only adds one BF16 zero-point (offset) per 128 activation elements to absorb the outlier channel. This adds 0.125 bits per activation element, and no weight bits. The results are shown in Tab. 10 below: OffQ achieves competitive results on both models (6.76 on Llama3-8B and 8.47 on Llama3.2-3B). This indicates that the offsetting technique carries over to microscaling formats, which we listed as future work in Sec. 5.

Table 10. WikiText2 PPL under MXFP4 quantization

ModelMethodWikiText2 PPL
Llama3-8BFP166.14
MR-GPTQ7.29
DuQuant++6.88
OffQ6.76
Llama3.2-3BFP167.81
MR-GPTQ8.79
DuQuant++8.63
OffQ8.47

The permutation strategy has already been explored in DuQuant, which somewhat limits the novelty of the proposed method.

Thank you for the comment. We agree that channel permutation is not new, and we explicitly credit DuQuant and RPTQ for it (L244). DuQuant's insight that permutation helps distribute outliers across blocks is an important one, and OffQ builds on it with a different objective:

  • DuQuant uses a zigzag permutation to balance outliers across blocks, so that the block rotation can smooth each outlier into a flatter distribution.
  • OffQ places exactly one outlier channel (after Top-1 PCA concentration) into each group, so that the group-wise Hadamard rotation maps it onto the constant direction and turns it into a per-group offset, which the asymmetric zero-point absorbs exactly. Because of this different objective, sorted assignment of the remaining channels fits our pipeline slightly better (Tab. 2).

In OffQ, the permutation is therefore a supporting step. Our main contributions are the concentrate-then-offset view of outliers:

  1. the formulation of structured outliers as a shared high-norm direction (Eq. 1),
  2. the derivation of the optimality condition for the rotation under asymmetric quantization (t1=⋯=tDt_1 = \dots = t_D),
  3. a practical construction of the optimal rotation (Top-1 PCA + sorted grouping + group-wise Hadamard), and
  4. extensive validation on 10+ models with vLLM kernel integration.

We believe these ideas are complementary to DuQuant and bring new insights to the community.

Figure 1 could be improved to more clearly illustrate the distribution of activation outliers.

Thank you for the suggestion. We have added a 3D visualization of the activation distribution of Llama3-8B before and after OffQ in Appendix A, Fig. 4(c). It shows the inputs to the q/k/v projections in layer 15: before OffQ, the activations have sparse outlier channels; after OffQ, the distribution is flat within each group.

The discussion in Appendix C appears inaccurate, as both DuQuant and FlatQuant quantize all activations, including query, key, and value activations.

Thank you very much for the correction. After inspecting their codebases, we confirm that both DuQuant and FlatQuant quantize the KV cache, i.e., they use the W4A4KV4 setting, although it is labeled as W4A4 in their papers. We have removed the incorrect statement from Appendix C.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.