- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ThinQuant, a scalable method for learning rotations for low-bit LLM quantization. Its key idea is to compress the calibration set by selecting geometrically important activation vectors that frequently appear as extreme points under random projections. The resulting low-rank calibration subspace further enables an exact reduction from optimizing a full d × d rotation matrix to a thin d × r matrix on the Stiefel manifold. Combined with an ADMM solver, ThinQuant substantially reduces calibration time, memory, and storage while maintaining competitive W4A4(KV4) performance, and scales rotation learning to LLaMA-3.1-405B on a single H200 GPU.
Critical strengths
- Compressing the calibration set through a geometry-aware selection procedure is interesting and potentially broadly useful.
- The connection between convex-hull geometry and worst-case activation magnitudes provides a principled motivation for the selection method.
- The exact subspace reduction is technically elegant and significantly reduces the dimensionality of rotation optimization.
- ThinQuant demonstrates strong practical scalability, especially on LLaMA-3-70B and LLaMA-3.1-405B.
- The ablations show that the proposed selection strategy is substantially better than random sampling or selecting activations solely by their norms.
Critical weaknesses
- The evaluation is largely restricted to W4A4 and W4A4KV4 rotation-based quantization. This makes it difficult to determine whether calibration-set compression is a generally useful technique or primarily tailored to the specific rotation objective studied here.
- The paper does not evaluate whether the selected calibration subset benefits weight-only quantization methods, such as EfficientQAT, where the optimization objective and relevant calibration statistics may differ.
- The evaluated model families and downstream tasks are relatively limited. In particular, the paper lacks experiments on newer model architectures and modern reasoning benchmarks involving mathematics, coding, or long chain-of-thought generation.
- Since aggressive activation quantization can disproportionately degrade reasoning capabilities, perplexity and conventional zero-shot benchmarks may not adequately characterize the practical quality of W4A4 models.
Questions for authors
- Have the authors tested the selected calibration subsets with weight-only methods such as EfficientQAT? If not, what prevents the geometric selection criterion from being adapted to their reconstruction or QAT objectives?
- How does ThinQuant perform on newer model families and reasoning-focused benchmarks, such as mathematical reasoning, code generation, and long-form chain-of-thought tasks?
- Does the selected subset remain effective when the quantization objective changes, or must activation selection be redesigned for each quantization method?
Overall, I find calibration-set compression promising. If the authors can demonstrate that it generalizes across broader quantization methods, newer models, and reasoning-intensive evaluations, I would view this as a substantially stronger contribution and would be inclined to raise my score.
5 comments
Applicability to other loss functions or objectives
We thank the reviewer for this interesting question. While in the paper we use an kurtosis loss, we believe our selection approach should be applicable to any approximation of the .
We looked around the literature for alternative objectives and identified the LogSumExp heuristic introduced by the authors of GaugeQuant [1], which provides a smooth approximation to the and aims to suppress activation outliers. We run our same rotation learning pipeline but change the loss function to a LogSumExp heuristic introduced by the authors of [1] and use their default optimizer to reduce the objective.
In the table below, we compare random selection with this objective against our geometry-aware selection procedure. Our results demonstrate that the learned rotation with our selection is much better than a random untrained rotation matrix. Additionally, randomly selecting very few activations leads to substantial degradation in performance, even compared to the untrained rotation.
| Configuration | Geometric selection (trained) | Untrained rotation | Random selection (trained) |
|---|---|---|---|
| W4A4KV4 | 7.89 | 8.30 | 42.68 |
| W4A4KV16 | 7.72 | 8.14 | 38.13 |
| Comparison of geometric selection, untrained rotation, and random selection using the LogSumExp loss function |
To better understand this behavior, we examine how the learned rotations generalize across different calibration sets. Specifically, we consider two sets of 130 activations. One selected randomly ("Random 130") and one selected using our geometry-aware procedure ("Geometric 130"). We train a rotation separately on each set and evaluate the resulting LogSumExp loss on both sets. Here we assess whether optimizing the loss on one set also reduces the loss on the other.
| Rotation trained on | Random 130 (evaluation) ↓ | Geometric 130 (evaluation) ↓ |
|---|---|---|
| No training | 0.93 | 1.03 |
| Random 130 | 0.91 | 1.33 |
| Geometric 130 (ours) | 0.93 | 0.93 |
| LogSumExp loss for different rotation training and evaluation sets. Each row specifies the activations used to train the rotation, while each column specifies the activations used to evaluate the loss. |
The results show that training on "Random 130" slightly reduces the loss on the same set (0.93 → 0.91), but substantially increases the loss on "Geometric 130" (1.03 → 1.33). This suggests that the rotation overfits to the randomly selected activations and fails to generalize to the more challenging geometric set. In contrast, training on "Geometric 130" reduces the loss on the geometric set (1.03 → 0.93) while maintaining the same loss on the random set (0.93). This is consistent with our geometric selection procedure capturing more challenging activations.
New Architectures & Reasoning Tasks
We thank the reviewer for these suggestions. We introduce an additional model, Gemma-4-31B, evaluated on the same benchmarks used in the paper. We then address the questions about model architectures and reasoning capabilities jointly by evaluating Qwen3-8B on general knowledge and mathematical reasoning benchmarks.
Gemma-4-31B
| Metric | Dense (BF16) | SpinQuant | DartQuant | ThinQuant |
|---|---|---|---|---|
| WikiText-2 PPL ↓ | 6.38 | 9.39 | 13.28 | 9.10 |
| C4 PPL ↓ | 10.12 | 14.12 | 20.19 | 13.96 |
| Nine-task average (%) ↑ | 51.42 | 41.24 | 39.30 | 41.69 |
| Rotation calibration time (min) ↓ | — | 171.20 | 18.23 | 2.56 |
ThinQuant achieves the best quantized result on all three quality metrics, while completing rotation calibration in 2.56 minutes, approximately 7× faster than DartQuant and 67× faster than SpinQuant. We apply ThinQuant without changing its selection or activation-budget, providing evidence that its benefits extend beyond the Llama family.
To evaluate both another non-Llama architecture and reasoning capabilities, we additionally evaluate Qwen3-8B on MMLU, GSM8K, MATH500, and AIME 2026.
For calibration, we use reasoning trajectories generated by the full-precision target model, motivated by prior work on reasoning-aware quantization [1] and pruning [2]. These studies highlight the importance of calibration data that reflects the activations encountered during reasoning. We prompt Qwen3-8B with 256 NuminaMath-1.5 training problems, collect its generated reasoning trajectories, and sample 128 sequences of 4,096 tokens, using the same calibration windows across methods. ThinQuant applies its geometric activation selection to these sequences to learn the rotations. Evaluations use W4A4KV4, with symmetric GPTQ weight quantization and asymmetric activation and KV-cache quantization, as in the paper.
MATH500 and AIME results report the mean and sample standard deviation across three generation seeds, keeping the models fixed across tasks.
| Metric | Dense (BF16) | SpinQuant | DartQuant | ThinQuant |
|---|---|---|---|---|
| MMLU, 5-shot (%) ↑ | 75.17 | 65.90 | 65.43 | 65.66 |
| GSM8K (%) ↑ | 95.45 | 94.47 | 93.56 | 94.01 |
| MATH500 (%) ↑ | 97.20 ± 0.20 | 93.47 ± 0.50 | 93.67 ± 0.23 | 92.93 ± 0.42 |
| AIME 2026 (%) ↑ | 63.33 ± 5.77 | 47.78 ± 3.85 | 46.67 ± 3.33 | 51.11 ± 3.85 |
| Rotation calibration time (min) ↓ | — | 54.85 | 72.18 | 5.14 |
| Calibration disk storage (GiB) ↓ | — | 0.065 | 432.176 | 0.194 |
We find that ThinQuant remains highly competitive on reasoning tasks, while reducing rotation-calibration time by approximately 11× relative to SpinQuant and 14× relative to DartQuant. On AIME 2026, which is the most challenging reasoning benchmark in this evaluation, ThinQuant achieves the highest mean accuracy among quantized methods.
[1] Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models. COLM 2025. https://arxiv.org/abs/2504.04823
[2] Ryan Lucas, Kayhan Behdin, Zhipeng Wang, Qingquan Song, Shao Tang, and Rahul Mazumder. Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction. ICLR 2026. https://arxiv.org/abs/2509.12464
We thank the reviewer for their insightful comments. It prompted us to considering the generality of the approach and scope of our work.
Scope of the original submission
We agree that our selection procedure is tailored to the rotation objective, and in the original submission we do not claim that it is a general technique for calibration-set compression. Our focus on rotations was deliberate and in our opinion rotation learning is a very important problem. As an example, the table below shows the quantization performance of LLaMA-2-70B at W4A4KV4 with and without rotation, but with the same weight quantizer (GPTQ). Without rotation, we find the perplexity can reach 106, which is consistent with prior findings e.g. from SpinQuant.
One reason for why we focus on activation quantization (and rotations specifically) is that weights can be quantized offline, and dedicated algorithms such as GPTQ and EfficientQAT are near-lossless for weight-only quantization at 4 bits. We deliberately focus on activation quantization, since this tends to be the more difficult setting for the reason that (unlike weights) activations must be quantized online with RTN, so no expensive optimization is possible. The error of RTN is governed by the largest coordinate of the activations, which is dominated by a small number of outlier channels. Rotations have proven to be a very effective way of addressing this issue.
A learned rotation is therefore what makes joint weight and activation quantization work at all, and the calibration of the rotation is currently one of the most expensive parts of a weight and activation quantization pipeline; 17.2 GPU-hours for SpinQuant on LLama-2-70B. This is the cost our paper addresses, and we view it as a valid scope.
Applicability to weight-only quantization
Nevertheless, we tested whether our selection procedure is useful for weight-only quantization using Qwen3-8B. We use GPTQ at W4A16KV16, without any rotations, and compare the full calibration set against subsets obtained using our geometric selection algorithm.
The full baseline uses 128 WikiText-2 training sequences of 2,048 tokens, totaling 262,144 calibration tokens. We tried instead doing GPTQ on only our selected activations from Algorithm 1, with budgets from B=2L+1=73 to 64B=4,672.
To our surpise, with only 73 selected tokens GPTQ closely approaches the full-calibration baseline. At budgets of 292 tokens and above, both perplexities actually improve over the full baseline in this experiment. These results provide evidence that our geometric selection can also be useful for a weight-reconstruction method, despite being motivated by the rotation objective.
The savings in GPTQ runtime are however more modest than the reduction in tokens. This is because the main quantity used for weight reconstruction in GPTQ is X⊤X, which is always a d×d matrix regardless of the number of calibration tokens. This matrix can be accumulated in batches without keeping all of X in GPU memory. Reducing the calibration budget therefore mainly helps by reducing the time needed to form X⊤X.
Our rotation objective instead depends on the largest activation coordinates and cannot be summarized by X⊤X. This is where our selection procedure helps.