- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper develops a theoretical framework for understanding the interaction between weight compensation and output-preserving transformations in low-bit weight-activation quantization of LLMs. The key idea is to exactly decompose the local reconstruction error into an activation-guided weight compensation (AGWC) term and an orthogonal residual. For a fixed transformation, the former can be reduced through weight optimization, while the latter cannot be eliminated by modifying the quantized weights and therefore motivates transformation design.
Critical strengths
- It provises a useful conceptual decomposition of weight-activation quantization error.
- The motivation is clear and well presented.
- The empirical comparison is reasonably controlled.
Critical weaknesses
- The theoretical analysis assumes no-clipping, dynamic per-token symmetric uniform activation quantization. In contrast, the experiments use activation clipping with a ratio of 0.9 and K/V clipping of 0.95, while K/V quantization is asymmetric. This does not invalidate the analysis, but it weakens the claim that the derived bounds directly explain the actual experimental pipeline. It would substantially strengthen the paper to either extend the argument to clipped quantizers or empirically demonstrate that the theoretically motivated quantities remain predictive under the practical setting.
- The paper demonstrates that the final recipe improves perplexity, but this is not the same as demonstrating that the proposed theoretical mechanism explains the improvement, such as whether J_co actually decreases substantially after random sign modulation; whether the predicted O(N2_co) vs. approximately O(N_co) behavior can be observed across layers/models; whether J_co or the derived bound correlates with layer reconstruction error or final perplexity; and so on.
- Theorem 2 bounds the minimum J_co among independently sampled sign configurations. However, in practice the configurations are selected using validation perplexity rather than J_co. Moreover, one configuration jointly determines multiple rotation types, sign vectors are shared across layers within each category, and the final model-level criterion aggregates the behavior of many layers. Hence, the theorem does not directly imply that the implemented model-level SPA procedure will attain the predicted improvement. The empirical results suggest that candidate search is useful, but the theoretical connection would be more convincing if the paper showed that low validation perplexity candidates also correspond to lower predicted residual quantities.
- The experimental scope is somewhat limited for an ICLR 2027 LLM quantization paper. Although eight models are tested, they are limited to Llama variants and one Mistral model, with model sizes ranging only from 1B to 13B. Moreover, the paper should include comparisons with more advanced methods.
Questions for authors
- How tight are the proposed residual bounds in actual LLM layers?
- Can the authors directly validate the constructive-interference hypothesis?
7 comments
Response to Reviewer nEy4 — 2/5
W4
The experimental scope is somewhat limited for an ICLR 2027 LLM quantization paper. Although eight models are tested, they are limited to Llama variants and one Mistral model, with model sizes ranging only from 1B to 13B. Moreover, the paper should include comparisons with more advanced methods.
A-W4
Regarding model coverage, additional experiments on Llama-30B, Qwen3-8B [1], and Qwen3-32B support the effectiveness of L2-SmoothRot on the Qwen3 family and at the 30B scale. Regarding comparisons with advanced methods, we compared L2-SmoothRot with OSTQuant [2] and FlatQuant [3], both of which learn their transforms via backpropagation, on Llama3-8B, and L2-SmoothRot outperformed OSTQuant but trailed FlatQuant in PPL and mean zero-shot accuracy, with calibration time and peak GPU memory between the two.
Expanded model coverage. We added W4A4KV4 results for Llama-30B, Qwen3-8B, and Qwen3-32B, reporting WikiText-2 PPL and six-task mean zero-shot accuracy (%). We updated Tables 1 and 3 in the revised manuscript. L2-SmoothRot achieves the lowest perplexity on both 30B-class models and the highest mean accuracy on both Qwen3 models. SpinQuant attains lower perplexity on Qwen3-8B. These results support the effectiveness of L2-SmoothRot on the Qwen3 family and at the 30B scale.
| Model | QuaRot | SpinQuant | L2-SmoothRot |
|---|---|---|---|
| PPL ↓ / Acc. ↑ | PPL ↓ / Acc. ↑ | PPL ↓ / Acc. ↑ | |
| Llama-30B | 4.51 / 73.43 | 4.57 / 72.85 | 4.49 / 73.11 |
| Qwen3-8B | 11.09 / 62.90 | 10.57 / 63.66 | 10.91 / 64.15 |
| Qwen3-32B | 8.57 / 70.23 | 8.31 / 70.92 | 8.26 / 72.08 |
Comparison with advanced methods. On Llama3-8B under W4A4KV4, we evaluated OSTQuant and FlatQuant alongside L2-SmoothRot. L2-SmoothRot achieved lower WikiText-2 perplexity and higher mean zero-shot accuracy than OSTQuant. FlatQuant achieved the best results on both metrics. L2-SmoothRot fell between OSTQuant and FlatQuant in both transform calibration time and peak GPU memory. Backpropagation-free L2-SmoothRot achieved competitive results with these methods and required less calibration time than FlatQuant.
| Method | Backprop | PPL ↓ | Acc. ↑ | Calib. time (min) | Calib. peak mem. (GiB) |
|---|---|---|---|---|---|
| OSTQuant | Yes | 7.32 | 67.74 | 24.6 | 35.49 |
| FlatQuant | Yes | 7.01 | 69.04 | 150.9 | 7.11 |
| L2-SmoothRot | No | 7.20 | 68.10 | 56.1 | 8.21 |
Manuscript revision
Based on these additional experiments, we updated Table 1 (p. 8) in the main text and Table 3 (p. 22) in Appendix D to include results for Llama-30B, Qwen3-8B, and Qwen3-32B.
[1] Yang et al., Qwen3 Technical Report.
[2] Hu et al., OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting.
[3] Sun et al., FlatQuant: Flatness Matters for LLM Quantization.
Response to Reviewer nEy4 — 3/5
Q1
How tight are the proposed residual bounds in actual LLM layers?
A-Q1
We measured the orthogonal residual and its Theorem 1 bound in selected FFN down-projection layers of Llama3-8B and found that the bounds are conservative. We used scaling and 4-bit RTN on 128 WikiText-2 validation samples.
To understand why, we evaluated the intermediate quantities in the proof. The spectral-norm inequality produced the largest increase: approximately 55× in layer 16, compared with 1–3× for each remaining relaxation. This step uses the maximum weight amplification, which can substantially exceed the amplification actually experienced by activation errors.
The table reports per-token values averaged over 10 random-sign Hadamard configurations.
| Layer | Residual | Bound |
|---|---|---|
| 0 | ||
| 8 | ||
| 16 | ||
| 24 | ||
| 31 |
Q2
Can the authors directly validate the constructive-interference hypothesis?
A-Q2
Yes. Constructive interference occurs when same-sign contributions from multiple outlier channels add up at one Hadamard output coordinate, creating an activation peak. In the synthetic examples, fixed Hadamard produced the larger peak in the worst case and a single random sign draw did so in the ordinary case, with best-of-10 (corresponding to our SPA) suppressing both peaks. In Llama3-8B's FFN down-projection inputs, fixed Hadamard produced the larger peak in the layer-29 example and a single random sign draw did so in the layer-28 example, with best-of-10 again suppressing both peaks.
We added Figs. 3, 4, 6, and 7 to Appendix F of the revised manuscript to visualize constructive interference and peak suppression in these examples.
The table below reports the peak CO activation value, defined as the maximum across channels of the mean absolute activation over tokens.
| Input | Before rotation | Fixed Hadamard | Random signs | Best of 10 |
|---|---|---|---|---|
| Synthetic worst case | 1.0000 | 0.5345 | 0.2840 | 0.2506 |
| Synthetic ordinary case | 1.0000 | 0.2673 | 0.3341 | 0.2506 |
| Llama3-8B, layer 28 | 0.6130 | 0.0731 | 0.0810 | 0.0690 |
| Llama3-8B, layer 29 | 1.8155 | 0.0880 | 0.0744 | 0.0778 |
W2
The paper demonstrates that the final recipe improves perplexity, but this is not the same as demonstrating that the proposed theoretical mechanism explains the improvement, such as whether J_co actually decreases substantially after random sign modulation; whether the predicted O(N2_co) vs. approximately O(N_co) behavior can be observed across layers/models; whether J_co or the derived bound correlates with layer reconstruction error or final perplexity; and so on.
A-W2 (1/2)
We conducted additional experiments and found that random sign modulation actually reduced relative to fixed Hadamard rotation. Regarding the dependence on , we validated the quadratic growth of under fixed rotation and its approximately linear growth under random signs on synthetic inputs. Regarding the relationship of and to PPL, our rationale is that reducing these quantities can suppress activation peaks and reduce local quantization error, potentially improving PPL.
decrement. We measured on two synthetic inputs with 64 CO channels and at Llama3-8B's FFN down-projection inputs on 128 WikiText-2 samples. Fixed Hadamard uses no sign flips, random signs uses one random sign configuration, and best-of-10 mirrors SPA's search over 10 configurations using for selection in this analysis. The table reports relative to fixed Hadamard rotation. Random signs reduce in two examples, and best-of-10 reduces it relative to fixed rotation in all four, including the two where the random draw increases it.
The table below reports relative to fixed Hadamard rotation (normalized to 1).
| Input | Fixed Hadamard | Random signs | Best of 10 |
|---|---|---|---|
| Synthetic worst case | 1.000 | 0.282 | 0.220 |
| Synthetic ordinary case | 1.000 | 1.563 | 0.879 |
| Llama3-8B, layer 28 | 1.000 | 1.043 | 0.945 |
| Llama3-8B, layer 29 | 1.000 | 0.818 | 0.856 |
Response to Reviewer nEy4 — 1/5
Thank you for your thorough assessment of our manuscript and for identifying specific ways to strengthen it. Your comments on the theoretical assumptions, empirical validation, SPA candidate selection, and experimental coverage have guided the revisions presented in our responses below.
We have revised the manuscript in response to the reviewers’ insightful comments and questions. The changes are highlighted in the updated PDF.
W1
A-W1
We can extend our theoretical analysis to clipped symmetric quantization and asymmetric quantization. We have added these extensions and their proofs as Lemmas 1 and 2 in Appendix B.2 of the revised manuscript.
These extensions replace the activation quantization error bound in Eq. (34) of the proof of Theorem 1 (Appendix B.1). Specifically, with qA=2bA−1−1, qA′=2bA−1, and clipping ratio ρ∈(0,1], the three settings give
\leq d\sum_{t=1}^{T}\\|(\mathbf Z)_{t,:}\\|_\infty^2 \times \begin{cases} \dfrac{1}{4q_A^2}, & \text{no-clipping symmetric},\\\\[6pt] \max\left\lbrace\dfrac{\rho}{2q_A},\\,1-\rho\right\rbrace^{2}, & \text{clipped symmetric (Lemma 1)},\\\\[6pt] \dfrac{1}{(q_A')^2}, & \text{asymmetric (Lemma 2)}. \end{cases}$$ The clipped asymmetric case can also be handled by combining the arguments of Lemmas 1 and 2. Substituting the corresponding bound for Eq. (34), the remainder of the proof proceeds unchanged. The residual bound therefore retains the same dependence on $\mathcal J_{\mathrm{co}}$ and $\mathcal J_{\mathrm{reg}}$, with only the quantizer-dependent coefficient modified. The subsequent analyses of these quantities remain applicable. ### Manuscript revision In response to your comment, we added the extensions to clipped symmetric and asymmetric quantization, together with their proofs, as Lemmas 1 and 2 in Appendix B.2 (pp. 17-18) of the revised manuscript.