acceptodds
Under review as a conference paper at ICLR 2027

ThinQuant: Scalable Rotation Learning For Weight And Activation Quantization

Abstract

Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based procedures such as SpinQuant and computationally friendlier gradient-free approaches such as DartQuant, but both remain hard to scale to the largest architectures. To address the computational bottlenecks in gradient-free rotation learning, we introduce two ideas for efficiency, (i) a data selection procedure which reduces the required number of calibration data points, and (ii) an exact reduction of the associated optimization on this reduced calibration set. Our data selection procedure exploits the geometric structure of the convex hull of the activations. Using this idea, we show that a carefully selected calibration set with several orders of magnitude fewer activations than state-of-the-art rotation-based methods can match their performance in low-bit quantization settings. Under this extreme data efficiency, the selected activations span an -dimensional subspace with , making optimization over a rotation equivalent to optimizing a matrix on the Stiefel manifold. We solve this reduced problem using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant. For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes. Unlike SpinQuant and DartQuant, ThinQuant also scales to Llama-3.1-405B on a single H200 GPU, completing rotation calibration in just over 2 hours and achieving WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT.

Primary area:
optimization
Keywords:
Quantization, Rotation Learning, Manifold Optimization, Model Compression, Efficient ML

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZPJUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ThinQuant, a scalable method for learning rotations for low-bit LLM quantization. Its key idea is to compress the calibration set by selecting geometrically important activation vectors that frequently appear as extreme points under random projections. The resulting low-rank calibration subspace further enables an exact reduction from optimizing a full d × d rotation matrix to a thin d × r matrix on the Stiefel manifold. Combined with an ADMM solver, ThinQuant substantially reduces calibration time, memory, and storage while maintaining competitive W4A4(KV4) performance, and scales rotation learning to LLaMA-3.1-405B on a single H200 GPU.

Critical strengths
  1. Compressing the calibration set through a geometry-aware selection procedure is interesting and potentially broadly useful.
  2. The connection between convex-hull geometry and worst-case activation magnitudes provides a principled motivation for the selection method.
  3. The exact subspace reduction is technically elegant and significantly reduces the dimensionality of rotation optimization.
  4. ThinQuant demonstrates strong practical scalability, especially on LLaMA-3-70B and LLaMA-3.1-405B.
  5. The ablations show that the proposed selection strategy is substantially better than random sampling or selecting activations solely by their norms.
Critical weaknesses
  1. The evaluation is largely restricted to W4A4 and W4A4KV4 rotation-based quantization. This makes it difficult to determine whether calibration-set compression is a generally useful technique or primarily tailored to the specific rotation objective studied here.
  2. The paper does not evaluate whether the selected calibration subset benefits weight-only quantization methods, such as EfficientQAT, where the optimization objective and relevant calibration statistics may differ.
  3. The evaluated model families and downstream tasks are relatively limited. In particular, the paper lacks experiments on newer model architectures and modern reasoning benchmarks involving mathematics, coding, or long chain-of-thought generation.
  4. Since aggressive activation quantization can disproportionately degrade reasoning capabilities, perplexity and conventional zero-shot benchmarks may not adequately characterize the practical quality of W4A4 models.
Questions for authors
  1. Have the authors tested the selected calibration subsets with weight-only methods such as EfficientQAT? If not, what prevents the geometric selection criterion from being adapted to their reconstruction or QAT objectives?
  2. How does ThinQuant perform on newer model families and reasoning-focused benchmarks, such as mathematical reasoning, code generation, and long-form chain-of-thought tasks?
  3. Does the selected subset remain effective when the quantization objective changes, or must activation selection be redesigned for each quantization method?

Overall, I find calibration-set compression promising. If the authors can demonstrate that it generalizes across broader quantization methods, newer models, and reasoning-intensive evaluations, I would view this as a substantially stronger contribution and would be inclined to raise my score.

5 comments

We thank the reviewer for their insightful comments. It prompted us to considering the generality of the approach and scope of our work.

Scope of the original submission
We agree that our selection procedure is tailored to the rotation objective, and in the original submission we do not claim that it is a general technique for calibration-set compression. Our focus on rotations was deliberate and in our opinion rotation learning is a very important problem. As an example, the table below shows the quantization performance of LLaMA-2-70B at W4A4KV4 with and without rotation, but with the same weight quantizer (GPTQ). Without rotation, we find the perplexity can reach 10610^6, which is consistent with prior findings e.g. from SpinQuant.

MethodWiki2-PPL ↓Zero-shot accuracy ↑Rotation calibration time
No rotation≈ 10⁶35.6%--
SpinQuant3.8567.62%17.2 hrs
ThinQuant3.9166.92%11 min 42 s

One reason for why we focus on activation quantization (and rotations specifically) is that weights can be quantized offline, and dedicated algorithms such as GPTQ and EfficientQAT are near-lossless for weight-only quantization at 4 bits. We deliberately focus on activation quantization, since this tends to be the more difficult setting for the reason that (unlike weights) activations must be quantized online with RTN, so no expensive optimization is possible. The error of RTN is governed by the largest coordinate of the activations, which is dominated by a small number of outlier channels. Rotations have proven to be a very effective way of addressing this issue.

A learned rotation is therefore what makes joint weight and activation quantization work at all, and the calibration of the rotation is currently one of the most expensive parts of a weight and activation quantization pipeline; 17.2 GPU-hours for SpinQuant on LLama-2-70B. This is the cost our paper addresses, and we view it as a valid scope.

Applicability to weight-only quantization

Nevertheless, we tested whether our selection procedure is useful for weight-only quantization using Qwen3-8B. We use GPTQ at W4A16KV16, without any rotations, and compare the full calibration set against subsets obtained using our geometric selection algorithm.

The full baseline uses 128 WikiText-2 training sequences of 2,048 tokens, totaling 262,144 calibration tokens. We tried instead doing GPTQ on only our selected activations from Algorithm 1, with budgets from B=2L+1=73B=2L+1=73 to 64B=4,67264B=4,672.

Calibration budgetSelected tokensWikiText-2 PPL ↓C4 PPL ↓GPTQ time (s)
Full baseline262,14411.31617.929364.6
B7311.37018.173272.6
2B14611.31218.031274.9
4B29211.08517.893279.5
8B58410.97117.601277.2
16B1,16811.01617.515282.6
32B2,33611.00117.627278.9
64B4,67210.77917.547279.8

To our surpise, with only 73 selected tokens GPTQ closely approaches the full-calibration baseline. At budgets of 292 tokens and above, both perplexities actually improve over the full baseline in this experiment. These results provide evidence that our geometric selection can also be useful for a weight-reconstruction method, despite being motivated by the rotation objective.

The savings in GPTQ runtime are however more modest than the reduction in tokens. This is because the main quantity used for weight reconstruction in GPTQ is X⊤XX^\top X, which is always a d×dd \times d matrix regardless of the number of calibration tokens. This matrix can be accumulated in batches without keeping all of XX in GPU memory. Reducing the calibration budget therefore mainly helps by reducing the time needed to form X⊤XX^\top X.

Our rotation objective instead depends on the largest activation coordinates and cannot be summarized by X⊤XX^\top X. This is where our selection procedure helps.

Applicability to other loss functions or objectives

We thank the reviewer for this interesting question. While in the paper we use an ℓ4\ell_4 kurtosis loss, we believe our selection approach should be applicable to any approximation of the ℓ∞\ell_\infty.

We looked around the literature for alternative objectives and identified the LogSumExp heuristic introduced by the authors of GaugeQuant [1], which provides a smooth approximation to the ℓ∞\ell_\infty and aims to suppress activation outliers. We run our same rotation learning pipeline but change the loss function to a LogSumExp heuristic introduced by the authors of [1] and use their default optimizer to reduce the objective.

In the table below, we compare random selection with this objective against our geometry-aware selection procedure. Our results demonstrate that the learned rotation with our selection is much better than a random untrained rotation matrix. Additionally, randomly selecting very few activations leads to substantial degradation in performance, even compared to the untrained rotation.

ConfigurationGeometric selection (trained)Untrained rotationRandom selection (trained)
W4A4KV47.898.3042.68
W4A4KV167.728.1438.13
Comparison of geometric selection, untrained rotation, and random selection using the LogSumExp loss function

To better understand this behavior, we examine how the learned rotations generalize across different calibration sets. Specifically, we consider two sets of 130 activations. One selected randomly ("Random 130") and one selected using our geometry-aware procedure ("Geometric 130"). We train a rotation separately on each set and evaluate the resulting LogSumExp loss on both sets. Here we assess whether optimizing the loss on one set also reduces the loss on the other.

Rotation trained onRandom 130 (evaluation) ↓Geometric 130 (evaluation) ↓
No training0.931.03
Random 1300.911.33
Geometric 130 (ours)0.930.93
LogSumExp loss for different rotation training and evaluation sets. Each row specifies the activations used to train the rotation, while each column specifies the activations used to evaluate the loss.

The results show that training on "Random 130" slightly reduces the loss on the same set (0.93 → 0.91), but substantially increases the loss on "Geometric 130" (1.03 → 1.33). This suggests that the rotation overfits to the randomly selected activations and fails to generalize to the more challenging geometric set. In contrast, training on "Geometric 130" reduces the loss on the geometric set (1.03 → 0.93) while maintaining the same loss on the random set (0.93). This is consistent with our geometric selection procedure capturing more challenging activations.

New Architectures & Reasoning Tasks

We thank the reviewer for these suggestions. We introduce an additional model, Gemma-4-31B, evaluated on the same benchmarks used in the paper. We then address the questions about model architectures and reasoning capabilities jointly by evaluating Qwen3-8B on general knowledge and mathematical reasoning benchmarks.

Gemma-4-31B

MetricDense (BF16)SpinQuantDartQuantThinQuant
WikiText-2 PPL ↓6.389.3913.289.10
C4 PPL ↓10.1214.1220.1913.96
Nine-task average (%) ↑51.4241.2439.3041.69
Rotation calibration time (min) ↓—171.2018.232.56

ThinQuant achieves the best quantized result on all three quality metrics, while completing rotation calibration in 2.56 minutes, approximately 7× faster than DartQuant and 67× faster than SpinQuant. We apply ThinQuant without changing its selection or activation-budget, providing evidence that its benefits extend beyond the Llama family.

To evaluate both another non-Llama architecture and reasoning capabilities, we additionally evaluate Qwen3-8B on MMLU, GSM8K, MATH500, and AIME 2026.

For calibration, we use reasoning trajectories generated by the full-precision target model, motivated by prior work on reasoning-aware quantization [1] and pruning [2]. These studies highlight the importance of calibration data that reflects the activations encountered during reasoning. We prompt Qwen3-8B with 256 NuminaMath-1.5 training problems, collect its generated reasoning trajectories, and sample 128 sequences of 4,096 tokens, using the same calibration windows across methods. ThinQuant applies its geometric activation selection to these sequences to learn the rotations. Evaluations use W4A4KV4, with symmetric GPTQ weight quantization and asymmetric activation and KV-cache quantization, as in the paper.

MATH500 and AIME results report the mean and sample standard deviation across three generation seeds, keeping the models fixed across tasks.

MetricDense (BF16)SpinQuantDartQuantThinQuant
MMLU, 5-shot (%) ↑75.1765.9065.4365.66
GSM8K (%) ↑95.4594.4793.5694.01
MATH500 (%) ↑97.20 ± 0.2093.47 ± 0.5093.67 ± 0.2392.93 ± 0.42
AIME 2026 (%) ↑63.33 ± 5.7747.78 ± 3.8546.67 ± 3.3351.11 ± 3.85
Rotation calibration time (min) ↓—54.8572.185.14
Calibration disk storage (GiB) ↓—0.065432.1760.194

We find that ThinQuant remains highly competitive on reasoning tasks, while reducing rotation-calibration time by approximately 11× relative to SpinQuant and 14× relative to DartQuant. On AIME 2026, which is the most challenging reasoning benchmark in this evaluation, ThinQuant achieves the highest mean accuracy among quantized methods.

[1] Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models. COLM 2025. https://arxiv.org/abs/2504.04823

[2] Ryan Lucas, Kayhan Behdin, Zhipeng Wang, Qingquan Song, Shao Tang, and Rahul Mazumder. Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction. ICLR 2026. https://arxiv.org/abs/2509.12464

Reviewer_DDtoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ThinQuant which is a scalable framework for learning the rotations used in quantization of LLMs. It keeps only the activations that most often attain the largest projection along random Gaussian directions and bounds the resulting objective gap in Theorem 1. Since these activations span a subspace of rank r < d the rotation optimization reduces exactly to a d × r Stiefel manifold problem solved with ADMM. ThinQuant matches or improves on the perplexity of DartQuant at a much lower calibration cost and scales up to LLaMA-3.1-405B.

Critical strengths

Previous rotation-learning methods optimize against a large amount of calibration data which leads to high computational and memory costs. ThinQuant selects a small subset of activations instead and gains highly on cost and memory efficiency.

  1. Data perspective. The paper simplifies the problem to a small subset that preserves the largest outliers of all activations so that F_I(R) = max ‖Rxᵢ‖∞ ≈ F(R). This subset comes from a geometric activation-selection procedure. The selection counts give the empirical Monte Carlo estimate q̂ = 1 − (1/M) Σ eᵢ of the probability that the selected set misses the maximizer along a random direction where the sum runs over the selected activations. Theorem 1 bounds the objective gap in terms of this probability. The selection procedure and its mathematical justification are therefore well connected.

  2. Optimization perspective. The paper further exploits the low rank of the selected set to reduce the optimization dimension from d × d to d × r. This turns the full rotation problem into an exactly equivalent problem on the much smaller Stiefel manifold St(d,r).

ThinQuant completes the entire rotation calibration of LLaMA-3-70B in under 12 minutes and achieves a WikiText-2 perplexity of 5.63. It is substantially faster than both DartQuant and SpinQuant and requires far less storage and memory than DartQuant. It achieves these savings while keeping perplexity comparable to DartQuant and at most 0.58 higher than SpinQuant. The method also scales to LLaMA-3.1-405B on a single H200 GPU which makes learned rotations practical even for the largest models.

Critical weaknesses
  1. The writing is hard to read. The high-level message is not clear and the overall organization needs improvement. Moving the proof of Theorem 2 and the proof outline of Theorem 1 to the appendix would improve readability. The experiment section spans less than two pages. The main text contains no analysis and the only ablations appear in Appendix C.

  2. The experiments cover only W4A4 with a 4-bit or 16-bit KV cache and only LLaMA family models. Results for other bit widths and other model families are missing. The method also relies on many heuristics. Figure 3 reaches zero miss probability at B = 1280 on LLaMA-3-8B and LLaMA-3.1-405B while the experiments use B = 2L+1 which is 161 for LLaMA-3-70B. The paper lacks ablations that compare performance and runtime across such choices.

Questions for authors
  1. Did the authors run experiments on the hyperparameter B? The best B can differ across models. How do the authors find the trade-off point between cost and quality? What is the basis for B = 2L+1 beyond admitting one activation per projection in every layer?

  2. For a random Gaussian direction g the projection gᵀx has variance ‖x‖₂². In high dimensions Algorithm 1 therefore favors activations with a large norm. Does the selected set differ in practice from simply taking the top-B activations by norm?

  3. Algorithm 1 samples directions uniformly on the hypersphere. If the activations concentrate in a low-dimensional subspace then a few dominant activations become the maximizer for most directions and most samples find the same extreme points again. This dominant subspace can be found with an SVD of the activations. Sampling more directions in its null space would then find the remaining extreme points more efficiently under the same budget B. Did the authors consider such a sampling scheme?

6 comments

We thank the reviewer for their positive assessment of the paper and their constructive feedback, particularly on the writing of the paper and in strengthening the experiments.

Paper readability
We sincerely thank the reviewer for providing insights on improving readability. Following the reviewer's suggestion, we have moved the proof of Theorem 2 and the proof outline of Theorem 1 to the Appendix and moved the ablation study to the main paper. In the submitted version, we prioritized giving more motivation and details for the algorithm derivation with less focus on experiments due to page limits. Our high-level message is:

(i) Data-efficiency is extremely important to reduce the runtime and cost of rotation learning. On LLaMA-3 70B, ThinQuant reduces run-time from 17 hours for SpinQuant to less than 12 minutes) and disk storage (960 GB for DartQuant to 0.51 GB for ThinQuant).

(ii) Randomly selecting a few data points for data-efficiency gives extremely poor results (29.51 WT-2 ppl for random data selection against 5.63 WT-2 ppl using our procedure on LLaMA-3 70B).

(iii) We design a dedicated geometry-aware data-selection algorithm that produces the gains reported in (i) while overcoming the challenges of a random subset selection of data points discussed in (ii). We dedicate the entirety of subsection 3.1 to give details and justifications of why this procedure works in practice.

(iv) We note an additional hidden benefits in extreme data-efficiency (defined as a setting where the number of selected data points rr is smaller than the hidden dimension of a Language model dd) which is discussed in Theorem 2. At a high-level, under extreme data-efficiency where r<dr < d, Theorem 2 guarantees that optimizing the initial d×dd \times d rotation matrix is equivalent to optimizing a reduced d×rd \times r matrix on the Stiefel manifold. Notice that in all of our experiments, we do select r<dr < d and work exclusively in this extreme data-efficiency setting. Hence, we get additional speedups from this reduction.

(v) Our experimental result show the benefits obtained by combining our data-selection procedure followed by the optimization reduction discussed in (iv) in terms of both overall efficiency (runtime, disk storage, peak GPU memory all reported in the paper) and downstream performance. These novel tools allow ThinQuant to uniquely scale to a LLaMA-3.1 405B.

We hope that these points make the high-level message clearer. As noted above, we will move the proofs to the Appendix following the reviewer's suggestion and will add more ablations and experimental results in the main paper.

Sensitivity to the calibration budget

We find that the results are not overly sensitive to BB. We provide a table below showing the results between 0.5x and 16x the original budget. WikiText-2 perplexity stays between 7.907.90 and 7.947.94, which is inside the range we observe across seeds at a fixed budget (7.897.89 to 7.957.95). This agrees with Figure 3, where the estimated miss probability on LLaMA-3-8B is nearly flat over this range, so additional activations add little information about the objective. The cost however does depend on BB. The optimization time grows from 23.423.4 s at B=65B = 65 to 146.5146.5 s at B=1040B = 1040, with little gain in quality.

Regarding the choice B=2L+1B = 2L + 1. The calibration set for R1R_1 contains N=LTN = LT activations from two projections in every layer. A budget linear in LL keeps the selected fraction of the calibration set constant across models, and also keeps BB small relative to the hidden dimension (B/dB/d is between 1.5% and 2% for all models we consider), which the reduced formulation relies on. While running the experiments, we found this to be a relatively stable choice and decided to keep the rule simple for this reason.

BudgetWiki-2 ↓C4 ↓0-shot (%) ↑Opt. (s)Runtime (s)
0.5×0.5\times (33)7.94212.89859.9422.8166.9
1×1\times (65, paper)7.91112.83159.7023.4116.0
2×2\times (130)7.89812.89159.6828.0133.8
4×4\times (260)7.93012.86759.9337.9179.2
8×8\times (520)7.92812.92160.0566.0159.5
16×16\times (1040)7.92512.93258.90146.5245.7

Table: ThinQuant activation-budget sweep on LLaMA-3-8B (W4A4KV4)

For a random Gaussian direction gg the projection gTxg^T x has variance ∣∣x∣∣22||x||_2^2. In high dimensions Algorithm~1 therefore favors activations with a large norm. Does the selected set differ in practice from simply taking the top-BB activations by norm?

We thank the reviewer for this interesting question. Your variance calculation is indeed correct. For g∼N(0,Id)g \sim \mathcal N(0,I_d) and any fixed x∈Rdx \in \mathbb R^d, we have

\qquad \operatorname{Var}(g^\top x)=||x||_2^2$$ Thus, an activation with large norm tends to have large projections. However, Algorithm 1 in our paper does not score an activation by the size of its own projection. Instead, it records how often the activation attains the *largest* absolute projection among all activations, and this depends on the geometry of the entire collection. The two rules would only agree when all activation vectors are pairwise orthogonal. We give an example below with a single non-orthogonal pair to provide intuition. Let $e_k \in \mathbb R^d$ be the $k$-th standard basis vector (a vector with all zeros except the $k$th coordinate), and consider $d+1$ activation vectors, each in $d$ dimensions: $$x_1 = 2e_1 = (2,0,0,\dots,0)^\top$$ $$x_2 = \frac{3}{2}e_1 = \left(\frac{3}{2},0,0,\dots,0\right)^\top$$ $$x_{k+1} = e_k = (0,\dots,0,1,0,\dots,0)^\top, \quad k=2,\dots,d$$ With a budget of $B=2$, selection by norm will retain $\{x_1,x_2\}$. But here $x_2=\tfrac34 x_1$, so we will have $\lVert Rx_2 \rVert_\infty = \frac{3}{4}\lVert Rx_1 \rVert_\infty$ for every rotation $R$. The activation $x_2$ can never determine the worst-case objective (1) from the paper, and is thus redundant in terms of selection. Now we will show how Algorithm 1 from the paper gets around this issue. Suppose we generate $g\sim\mathcal N(0,I_d)$, so that the coordinates $g_1,\dots,g_d$ are i.i.d. $\mathcal N(0,1)$. The projections are $g^\top x_1=2g_1$, $g^\top x_2=\tfrac32g_1$, and $g^\top x_{k+1}=g_k$. Letting $p_i$ denote the probability that $x_i$ attains $\max_j |g^\top x_j|$, we have: $$p_2 = \mathbb P\bigl[\tfrac32|g_1|\ge 2|g_1|\bigr] = \mathbb P[g_1=0] = 0$$ $$p_1 = \mathbb P\bigl[2|g_1|>\max_{k\ge2}|g_k|\bigr] \gt \mathbb P\bigl[|g_1|>\max_{k\ge2}|g_k|\bigr] = \frac{1}{d}$$ $$p_3 = \cdots = p_{d+1} = \frac{1-p_1}{d-1}$$ Here, while $x_1$ has the highest probability of selection, Algorithm 1 immediately ignores $x_2$ since it is geometrically redundant given $x_1$. The other vectors, which are orthogonal to $x_1$, each have positive selection probability. Ranking by norm is thus sensitive to redundancy among large activations, whereas Algorithm 1 is not. We see this effect in practice. We ran selection by norm on LLaMA-3-8B and LLaMA-3-70B. In the table below, we show where the selected activations came from in the network. Ranking by norm takes essentially the whole budget from a single layer. It takes all $65$ activations from layer 1 on LLaMA-3-8B, and $159$ of $161$ from the last layer on LLaMA-3-70B, in both cases selecting from the MLP layers only. Since $R_1$ is shared across all layers, the remaining layers are then not represented in the calibration set at all. Our selection algorithm instead spreads the same budget over $19$ of $32$ layers and $48$ of $80$ layers, and across both attention and MLP activations. | Model | Selection | Layers selected from | Largest share from one layer | Attn. / MLP split | |---|---|---:|---:|---:| | LLaMA-3-8B ($B=65$) | Algorithm 1 | 19 / 32 | 30.8% | 50 / 15 | | | $\mathrm{Top}_B(\lVert x \rVert_2)$ | 1 / 32 | 100.0% | 0 / 65 | | LLaMA-3-70B ($B=161$) | Algorithm 1 | 48 / 80 | 10.6% | 107 / 54 | | | $\mathrm{Top}_B(\lVert x \rVert_2)$ | 2 / 80 | 98.8% | 0 / 161 | *Table: Where the selected activations come from.* The following table shows the effect on quality. On LLaMA-3-8B, the largest norms occur in an early layer, which is also the part of the network where Algorithm 1 places most of its budget ($42$ of $65$ activations come from layers 0–2). The norm-based set is redundant but still covers well extreme directions that matter for the whole network, and the loss is small ($8.03$ versus $7.91$ on WikiText-2). On LLaMA-3-70B, the largest norms occur in the last layer, and the norm-based selection gives nearly all of its budget to this layer. The rotation $R_1$ is then fitted primarily to a single layer that is not representative of the earlier layers, whose outliers are never seen during calibration and are left uncontrolled. The result is a perplexity of $393.80$, compared to $5.63$ for Algorithm 1. | Model | Selection | WikiText-2 ↓ | C4 ↓ | |---|---|---:|---:| | LLaMA-3-8B | Algorithm 1 | 7.91 | 12.83 | | | $\mathrm{Top}_B(\lVert x \rVert_2)$ | 8.03 | 13.01 | | LLaMA-3-70B | Algorithm 1 | 5.63 | 10.34 | | | $\mathrm{Top}_B(\lVert x \rVert_2)$ | 393.80 | 619.75 | *Table: Perplexity comparison under W4A4KV4 quantization.*
Reviewer_VwNqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the calibration costs associated with rotation-based quantization. The authors propose a method to select a small subset of activations based on the frequency of extreme values ​​along random directions. They then reduce the full rotation optimization problem to a subspace spanned by these selected activations, solving it efficiently using the Alternating Direction Method of Multipliers (ADMM).

Critical strengths

The geometric selection process reduces the number of activations that must be retained, which in turn enables precise dimensionality reduction within a subspace. These two steps work in tandem to translate data efficiency into lower optimization costs.

The improved calibration efficiency is significant. For Llama-3-70B, the total rotation calibration time is reduced from 111 minutes (as reported for DartQuant) to approximately 12 minutes, while simultaneously improving quantization quality.

Critical weaknesses

Downstream performance is reported only as an average across nine tasks, making it impossible to assess potential degradation on individual tasks.

All experiments are restricted to the Llama family, and the evaluation lacks benchmarks for general knowledge or multi-step reasoning (e.g., MMLU, GSM8K).

The stability under extremely small calibration budgets remains unclear.

Questions for authors
  1. Please provide the per-task results for the nine tasks, and include evaluations such as MMLU and GSM8K on a representative model to determine whether performance retention extends to both general knowledge and multi-step reasoning capabilities?

  2. What's the results for models outside the Llama family, including MoE models?

  3. Could you report the precision and calibration costs for a representative model across a few different activation budgets and random seeds, in order to assess the stability of the selection and rotation-based completion processes?

3 comments

We thank the reviewer for their kind comments and positive assessment of the paper.

Per-task breakdown.
In response to the your request, we provide the per-task results for all nine downstream benchmarks on LLaMA-3-70B below.

MetricDense (BF16)DartQuantThinQuant
WikiText-2 PPL ↓2.867.555.63
C4 PPL ↓7.1715.6910.34
ARC-Easy (%) ↑85.9872.6977.53
ARC-Challenge (%) ↑64.2547.8752.22
HellaSwag (%) ↑84.9967.2580.50
OpenBookQA (%) ↑48.6041.2043.80
PIQA (%) ↑84.5576.7180.03
SocialIQA (%) ↑50.5643.3545.39
WinoGrande (%) ↑80.6670.2473.88
RTE (%) ↑68.5961.3767.51
BoolQ (%) ↑85.3575.4481.53

The per-task results show that ThinQuant consistently outperforms DartQuant across all nine downstream tasks, rather than achieving gains concentrated on a subset of benchmarks. The improvements are particularly pronounced on HellaSwag (+13.25 percentage points), BoolQ (+6.09 points), ARC-Challenge (+4.35 points), and ARC-Easy (+4.84 points). These results suggest that ThinQuant's improved quantization quality translates into broad performance gains across diverse downstream tasks, while requiring substantially less rotation calibration time (approximately 12 minutes versus 111 minutes for DartQuant on LLaMA-3-70B).

New Architectures & Reasoning Tasks

We thank the reviewer for these suggestions. We introduce an additional model, Gemma-4-31B, evaluated on the same benchmarks used in the paper. We then address the questions about model architectures and reasoning capabilities jointly by evaluating Qwen3-8B on general knowledge and mathematical reasoning benchmarks.

Gemma-4-31B

MetricDense (BF16)SpinQuantDartQuantThinQuant
WikiText-2 PPL ↓6.389.3913.289.10
C4 PPL ↓10.1214.1220.1913.96
Nine-task average (%) ↑51.4241.2439.3041.69
Rotation calibration time (min) ↓—171.2018.232.56

ThinQuant achieves the best quantized result on all three quality metrics, while completing rotation calibration in 2.56 minutes, approximately 7× faster than DartQuant and 67× faster than SpinQuant. We apply ThinQuant without changing its selection or activation-budget, providing evidence that its benefits extend beyond the Llama family.

To evaluate both another non-Llama architecture and reasoning capabilities, we additionally evaluate Qwen3-8B on MMLU, GSM8K, MATH500, and AIME 2026.

For calibration, we use reasoning trajectories generated by the full-precision target model, motivated by prior work on reasoning-aware quantization [1] and pruning [2]. These studies highlight the importance of calibration data that reflects the activations encountered during reasoning. We prompt Qwen3-8B with 256 NuminaMath-1.5 training problems, collect its generated reasoning trajectories, and sample 128 sequences of 4,096 tokens, using the same calibration windows across methods. ThinQuant applies its geometric activation selection to these sequences to learn the rotations. Evaluations use W4A4KV4, with symmetric GPTQ weight quantization and asymmetric activation and KV-cache quantization, as in the paper.

MATH500 and AIME results report the mean and sample standard deviation across three generation seeds, keeping the models fixed across tasks.

MetricDense (BF16)SpinQuantDartQuantThinQuant
MMLU, 5-shot (%) ↑75.1765.9065.4365.66
GSM8K (%) ↑95.4594.4793.5694.01
MATH500 (%) ↑97.20 ± 0.2093.47 ± 0.5093.67 ± 0.2392.93 ± 0.42
AIME 2026 (%) ↑63.33 ± 5.7747.78 ± 3.8546.67 ± 3.3351.11 ± 3.85
Rotation calibration time (min) ↓—54.8572.185.14
Calibration disk storage (GiB) ↓—0.065432.1760.194

We find that ThinQuant remains highly competitive on reasoning tasks, while reducing rotation-calibration time by approximately 11× relative to SpinQuant and 14× relative to DartQuant. On AIME 2026, which is the most challenging reasoning benchmark in this evaluation, ThinQuant achieves the highest mean accuracy among quantized methods.

[1] Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models. COLM 2025. https://arxiv.org/abs/2504.04823

[2] Ryan Lucas, Kayhan Behdin, Zhipeng Wang, Qingquan Song, Shao Tang, and Rahul Mazumder. Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction. ICLR 2026. https://arxiv.org/abs/2509.12464

Sensitivity to random seeds and calibration budget
At the default budget B=65B=65 activations, we evaluate five seed configurations, including the original paper run on LLaMA-3-8B. Across configurations, we vary the seed for all of the random parts of the pipeline including calibration sampling, rotation initialization, Gaussians used in Algorithm 1, orthogonal completion, GPTQ calibration, and evaluation.

SeedWikiText-2 PPL ↓C4 PPL ↓Zero-shot (%) ↑Optimization (s)Calibration runtime (s)
0 (original paper)7.91112.83159.7023.42116.00
17.89012.79159.8624.26124.81
27.92712.87459.3624.28126.04
37.94612.81859.9924.16125.10
47.91912.79359.6724.26125.38
Mean ± SD7.919 ± 0.02112.821 ± 0.03459.72 ± 0.2424.08 ± 0.37123.47 ± 4.20

We observe small variation in quality. For example, the sample standard deviations are 0.021 for WikiText-2 perplexity, 0.034 for C4 perplexity, and 0.24 percentage points for zero-shot accuracy. These results support the stability of the overall pipeline, including activation selection and rotation completion.

Sensitivity to the activation budget

We additionally sweep the activation budget BB from approximately 0.5× to 16× the default, holding the original seed configuration and initialization fixed.

BudgetWikiText-2 PPL ↓C4 PPL ↓Zero-shot (%) ↑Optimization (s)Calibration runtime (s)
0.5× (33)7.94212.89859.9422.8166.9
1× (65, paper)7.91112.83159.7023.4116.0
2× (130)7.89812.89159.6828.0133.8
4× (260)7.93012.86759.9337.9179.2
8× (520)7.92812.92160.0566.0159.5
16× (1040)7.92512.93258.90146.5245.7

We find that the perplexity remains stable across this range. Increasing the budget doesn't give a consistent accuracy improvement, while the runtime does increase for the largest budgets.

Dear reviewer VwNq,

Please let us know if our response addresses your questions. We would be glad to provide additional clarifications or run further experiments while the discussion period is still open!

Comments on the paper

1 comment
Area_Chair_i4fvOpenReview ↗

Discussion phase

A request to the authors to respond to the reviewers’ comments and the reviewers to check the responses and see if their concerns have been addressed. Please engage in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.