- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes OffQ, a post-training quantization (PTQ) method for the W4A4KV4 setting. Its central observation is that activation outliers in LLMs are structured: they can be modeled as , where is a high-norm constant direction shared across layers, token positions, and sequence locations (e.g., "singular defects"/massive activations, or attention-bias-induced outliers in Qwen 2.5). Building on this, OffQ introduces an "Outlier2offset" equivalent transform: (1) a Top-1 PCA identifies the low-dimensional outlier subspace and rotates activations so outlier energy concentrates into as few channels as possible; (2) channels are reordered and partitioned into groups, with one outlier channel per group; (3) a group-wise Hadamard rotation converts the outlier energy into a constant per-group offset absorbed into the zero-point of standard asymmetric quantization. All transforms except three online Hadamard rotations are fused into weight matrices. Experiments cover Llama 2/3/3.2 and Qwen 2.5 (1B–72B) under W4A4KV4, reporting WikiText perplexity and zero-shot accuracy on eight commonsense tasks, plus vLLM end-to-end latency (1.5–1.7× prefill, 2.0–2.4× decode speedup vs. 16-bit) at an average effective activation bit-width of 4.25.
Critical strengths
-
The "Outlier2offset" formulation is a genuinely new equivalent transform, and it is elegantly motivated. Prior rotation-based PTQ (QuaRot, SpinQuant, DuQuant, etc.) aims to spread outlier energy into a Gaussian-like distribution. OffQ instead concentrates the structured outlier into a single channel and converts it into a per-group constant absorbed by the asymmetric zero-point. The "shooting at the same target" argument in Section 3 gives a clean optimality condition (), and the observation that any orthogonal matrix with a constant first row satisfies it is a nice touch, since it makes the technique composable with learned rotations. This is the main reason the paper is worth discussing.
-
The empirical results under a hard setting are broad and consistent. W4A4KV4 with uniform grids and uniform precision is a demanding, deployment-relevant setting, and the paper evaluates ten models across two families. The gains over QuaRot/SpinQuant/DFRot/KurTail/OSTQuant are consistent, and the fact that OffQ beats ResQ (which spends 4.5 effective bits on mixed precision) while using 4.25 bits is a meaningful data point. The kernel-level integration into vLLM with zero-point support added to the humming INT4 path, plus measured TTFT/TPOT, strengthens the practical claim considerably.
Critical weaknesses
-
The cost–benefit justification for Top-1 PCA is not established, and the key ablation is missing. Per-channel magnitude detection is an single pass and maps directly onto the first-principles fact that activation magnitude drives the output . Top-1 PCA instead requires sampling tokens, computing a covariance, and performing an eigendecomposition. PCA's theoretical advantage is that it accounts for the multi-channel covariance structure; but by selecting only the single -largest token per sequence, the paper collapses that global view onto an extremely small sample, arguably discarding the very advantage that motivates using PCA. The ablation in Table 2 only compares Top-1 PCA against standard PCA over all tokens (PPL 6.98 vs. 8.27), which shows that normal tokens dilute the leading eigenvector — but it does not compare against the cheaper, more first-principles alternative of per-channel magnitude selection combined with the same rotation/offsetting pipeline. Without that comparison, the paper does not answer whether the accuracy gain of Top-1 PCA justifies its extra cost, nor whether the gain comes from PCA per se or simply from focusing on extreme tokens (which magnitude-based selection would also do). This is consequential because the entire "Step 1" of the pipeline, and the claimed novelty of the outlier-subspace identification, rests on this choice.
-
The rotation itself is not fundamentally different from mature methods; the contribution is a change of rotation objective, and the independent cost of the shared offset is never isolated. Rotation as a quantization preprocessing step is well established. OffQ changes the objective from "disperse" to "concentrate + convert to offset." That is a legitimate new equivalent transform, but the paper's framing somewhat overstates its distance from prior work. More importantly, the design has a real, unmeasured accuracy cost: an independent per-channel bias (i.e., retaining outlier channels at higher precision, as ResQ does) is theoretically strictly more expressive than a shared per-group offset, because it can represent each outlier channel exactly rather than forcing outlier channels to share one constant within a group. OffQ substitutes a shared offset for independent high-precision retention, yet the paper only measures this substitution indirectly through end-to-end PPL. The fact that OffQ at 4.25 bits beats ResQ at 4.5 bits shows that other design choices (Top-1 PCA + sorted grouping) outweigh the loss from sharing, but it does not show that the shared offset itself has smaller quantization error than independent retention. No ablation separates the isolated accuracy cost of "shared offset vs. independent high-precision outlier channels." This matters because a reader cannot tell how much of OffQ's headroom is being left on the table by the offset-sharing design, nor whether the method would remain competitive if the offset were replaced by exact retention at equal bit-width.
Questions for authors
-
Please provide the missing ablation separating the offset-sharing cost: compare (a) OffQ with -group shared offsets, (b) OffQ with the top- outlier channels retained at higher precision (independent per-channel bias) instead of being absorbed into zero-points, at matched effective bit-width, and (c) a hybrid. This would quantify how much accuracy is lost specifically to offset sharing, independent of Top-1 PCA and grouping.
-
How is the number of groups (or group size) selected in practice, and is it chosen per model or fixed? Figure 3 shows a clear accuracy/storage trade-off; is there a principled criterion, or is 128 chosen empirically? Also, how sensitive is OffQ to the choice of the token selection when a sequence contains multiple competing outlier tokens?
-
For the Qwen 2.5 bias-induced outliers, is computed per-layer, per-head via Top-1 PCA. Does the number of such bias-outlier channels ever exceed the number of groups , in which case some outliers cannot be absorbed? How does OffQ behave in that regime?
5 comments
We would like to thank Reviewer QpSr for the constructive and valuable comments. Below we answer the questions in detail. Table numbers continue from the revised paper and across our responses for easier reference.
Per-channel magnitude detection is an O(D) single pass, while Top-1 PCA requires sampling tokens, computing a covariance, and an eigendecomposition.
Per-channel magnitude detection and Top-1 PCA have similar costs, because the calibration bottleneck is network inference; token sampling, covariance, and eigendecomposition are negligible. In practice, calibrating Llama3-8B takes 76 seconds on an H100 GPU with either per-channel magnitude detection or Top-1 PCA.
Does the number of Qwen 2.5 bias-outlier channels ever exceed the number of groups (G)? How does OffQ behave in that regime?
Yes, the number of outlier channels can exceed the number of groups (each head forms one group), as shown in Fig. 4(b) in Appendix A. OffQ still works well, as shown by its competitive performance in Tab. 1, because it concentrates all outlier channels within a head into a single channel and then absorbs it.
Compare against per-channel magnitude selection combined with the same rotation/offsetting pipeline.
The core of OffQ is identifying the structured outliers and converting them into offsets. Per-channel magnitude selection is a less general variant of OffQ: it replaces Top-1 PCA in the identification step and relies on the outlier channels being sparse. It works in that case, e.g., on Llama (Tab. 16), but fails when the outliers span more channels than groups, as shown by the severe degradation on Qwen2.5-1.5B/3B/7B (Tab. 16). Top-1 PCA is robust in both cases: it finds the outlier direction even when it spans many channels, and concentrates it into one channel. We have added a new discussion of this design choice (per-channel magnitude vs. Top-1 PCA) to Appendix B of the revised paper.
Table 16. WikiText2 PPL of the OffQ variant using per-channel magnitude selection on Llama (L) and Qwen (Q)
| OffQ variant | L3-8B | L2-7B | L2-13B | L3.2-3B | L3.2-1B | L3-70B | Q2.5-1.5B | Q2.5-3B | Q2.5-7B | Q2.5-14B | Q2.5-32B | Q2.5-72B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Per-channel magnitude | 7.04 | 5.78 | 5.09 | 8.89 | 12.48 | 3.86 | 9111.61 | 13.13 | 4432.13 | 6.09 | 5.57 | 4.30 |
| Top-1 PCA | 6.98 | 5.77 | 5.11 | 8.78 | 12.32 | 3.88 | 11.35 | 8.98 | 7.66 | 6.07 | 5.52 | 4.29 |
Retaining outlier channels at higher precision is strictly more expressive than a shared per-group offset
Converting an outlier channel into an offset incurs no accuracy cost. The group-wise Hadamard maps the outlier channel to a constant added to every element of the group. This constant does not change the range of the group, so the remaining values are quantized exactly as if the outlier were absent. Meanwhile, the constant is stored in the BF16 zero-point, so all the information in the outlier channel is represented by the offset.
Ablation: (a) shared offsets, (b) top-G outlier channels retained at higher precision, at matched effective bit-width, and (c) a hybrid.
We implemented a variant of OffQ that keeps the top-G outlier channels in high precision. The same 32 outlier channels identified by Top-1 PCA are kept in BF16 instead of being converted into offsets. For the remaining 4-bit activation channels, we first apply a random rotation to flatten them and then partition them with a group size of 127, so that the number of groups is the same as in OffQ. The results in Tab. 17 show that the high-precision outlier-channel variant has a much higher PPL (7.92 vs. 6.98), because the BF16 outlier channels do not work well with INT4 weights. It matches OffQ (6.96 vs. 6.98) only when the corresponding 32 weight columns are also kept in BF16 (as in ResQ), which is not a fair comparison for OffQ, since OffQ quantizes all weights to 4 bits.
Regarding the hybrid setting (c), could you please elaborate on it? Thank you!
Table 17. Llama3-8B with high-precision outlier channels for OffQ
| Method | Activation | Weight | WikiText2 PPL |
|---|---|---|---|
| OffQ variant | 32 BF16 channels; group size 127 for 4-bit channels | 4-bit | 7.92 |
| Same as above | 32 BF16 columns; the rest 4-bit | 6.96 | |
| OffQ | 4-bit, group size 128 | 4-bit | 6.98 |
How is the group size selected, and is it chosen per model or fixed?
The group size of 128 is chosen empirically based on Fig. 3: more groups absorb more outlier channels but store more scales and zero-points; 128 (4.25 bits on average) balances the two. We use a fixed group size of 128 for all models in our experiments, and it works well.
How sensitive is the token selection to multiple competing outlier tokens?
OffQ is very robust in this respect, as shown by the various model sizes and model families in our experiments. According to the study of Wang et al. (2025), all singular-defect tokens share the same direction, regardless of their positions in the sequence. Therefore, the method works even when a sequence contains multiple competing outlier tokens.
Dear Reviewer QpSr,
Thank you for your message. As it was posted before our response, we would like to kindly invite you to read our rebuttal, which includes the requested ablations (Tabs. 16–17). We would be grateful if you could reconsider your evaluation in light of these results, and we are happy to address any remaining concerns.
I will keep my score.