- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SmoothGPTQ, a framework for optimizing the diagonal smoothing matrix when weights are quantized with GPTQ and activations with RTN. Diagonal smoothing, as in SmoothQuant, balances the errors from weight and activation quantization, but prior work optimizes the smoothing parameters for RTN only, without considering the error compensation of GPTQ. To address this, the authors derive an approximation of the quantization MSE as a function of , which is convex in the log-scale and can thus be minimized efficiently. The authors show that the approximation is accurate at high bit-widths, and report state-of-the-art results for W4A4 INT quantization on several LLMs.
Critical strengths
-
SmoothQuant followed by GPTQ is widely used in practice, while the smoothing scales are designed for RTN. Optimizing for GPTQ is thus directly useful for real quantization pipelines.
-
The effect of GPTQ enters the proxy through the Cholesky diagonal only (Eq. (8)), providing a clear interpretation of how error compensation changes the optimal scales.
Critical weaknesses
-
The comparisons do not support the state-of-the-art claim for W4A4 INT quantization.
- Calibration. All baseline results in Tables 6-10 are copied from the MR-GPTQ paper, while SmoothGPTQ is calibrated on SlimPajama with its own settings (Appendix D.7). SmoothGPTQ and the baselines are thus calibrated with different data and settings, and the gains may partly come from the calibration. In the comparisons run by the authors (Tables 2-4), SmoothGPTQ also determines from 100K activation vectors, whereas SmoothQuant uses only vectors, which may differ.
- Format. In Tables 7-10, SmoothGPTQ is quantized to NVINT4, while most of the compared methods are quantized to NVFP4. Based on these tables, Sec. 5.2 claims that SmoothGPTQ is competitive with or outperforms the NVFP4 methods. This comparison, however, cannot tell whether the gains come from the method or the format, since the two formats differ. For the 70B model (Table 10), this problem is more severe, as all baselines use NVFP4 and SmoothGPTQ is compared only across formats.
- Baselines. Rotation-based methods achieve the state of the art in W4A4. The paper, however, includes QuaRot and SpinQuant only in the NVFP4 rows of Appendix D.7, although their main setting is INT4, so SmoothGPTQ is never compared with them in INT4. The INT4-G32 benchmark (Table 6) includes only RTN, RTN with a Hadamard transform, and GPTQ. OmniQuant, which also optimizes diagonal scales, is compared only at W6A6 (Table 5), not at W4A4. Even within the reported results, SmoothGPTQ does not achieve the best accuracy, and MR-GPTQ outperforms it on Llama-3.1-8B in NVINT4 (76.66 vs. 76.55, Table 9).
The authors should compare SmoothGPTQ with these methods in INT4-G32, using the same calibration data and quantization scope (e.g., whether the KV cache and attention matmuls are quantized).
-
The distortion proxy, the core of SmoothGPTQ, is not validated in the main setting.
- Accuracy at 4 bits. The proxy assumes that clipping in GPTQ is negligible (Sec. 2.1). At 4 bits, however, clipping is likely to occur frequently, since the weight scale has only a small margin (). For the down projection at 4 bits, the measured MSE is indeed only half of the predicted one (Table 1). At 2 and 3 bits, the optimized even gives a higher KL divergence than GPTQ without smoothing (Table 3), suggesting that the proxy becomes unreliable at low bit-widths.
Questions for authors
-
The proxy is validated on Llama-3.1-8B-Instruct only, and the benchmarks are limited to the Llama-3 family. The effectiveness of SmoothGPTQ depends on both the channel correlation and the outlier distribution of activations, which vary across model families. Do both the accuracy of the proxy and the gains hold for other families (e.g., Qwen and Mistral) and MoE models?
-
Minor: Appendix D.7 contains a broken citation ("SlimPajama (?)").
2 comments
Low rate accuracy: Regarding the mediocre performance at bits: Our approximate distortion expression is developed under a high-resolution assumption, and several approximations we make in the derivation are not really justifies when the quantization noise is not much smaller than the signal (weights or activations). For the quantization error is too large, so that our distortion approximation is highly inaccurate, and optimizing with respect to it leads to bad results. In our revision we better emphasize the fact that the approximation assumes high-rate. This was probably not sufficiently emphasized before.
Discrepancy in down projection distortion approximation: The measured/predicted MSE ratio is 0.5075 (Table 1) for has a rather delicate explanation. The activations in the down projection involve the SwiGLU term. Consequently, many of them are very close to . For relatively small rates, even , the conditional distribution of activations falling in the quantization cell around zero is not uniform - it is concentrated around 0. Our high-resolution approximation on the other hand, assumes the quantization error is uniform, and therefore overestimates the distortion. In our revision we added an explanation of this phenomenon to Appendix E.5.
Clipping: We have measured the frequency of clipping in GPTQ and provide the results below. Indeed clipping rarely occurs with
Table F. Percentage of weight entries clipped by GPTQ to with the optimized SmoothGPTQ scaling and margin on Llama-3.1-8B-Instruct, averaged over the 32 layers. Top: the setting of Table 1 (, ). Bottom: the INT4-G32 setting of Table 11 (, ). The largest single-layer value occurs in the Q/K/V projections of the first layer at (5.0% at ); at INT4-G32 it is 0.061% at .
| Setting | Q | K | V | O | GATE | UP | DOWN | |
|---|---|---|---|---|---|---|---|---|
| , | 2 | 0.38 | 0.37 | 0.42 | 0.0084 | 0.063 | 0.060 | <0.001 |
| 3 | 0.28 | 0.27 | 0.29 | 0.0016 | 0.052 | 0.052 | <0.001 | |
| 4 | 0.17 | 0.17 | 0.17 | <0.001 | 0.048 | 0.048 | <0.001 | |
| 6 | 0.037 | 0.037 | 0.037 | <0.001 | 0.036 | 0.036 | 0 | |
| 8 | 0.0062 | 0.0061 | 0.0062 | 0 | 0.014 | 0.014 | 0 | |
| , | 2 | 0.045 | 0.066 | 0.016 | 0.074 | 0.023 | 0.018 | 0.0097 |
| 3 | 0.016 | 0.024 | 0.0048 | 0.026 | 0.0073 | 0.0055 | 0.0033 | |
| 4 | 0.0075 | 0.012 | 0.0012 | 0.0076 | 0.0023 | 0.0016 | <0.001 | |
| 6 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | |
| 8 | <0.001 | <0.001 | 0 | <0.001 | 0 | 0 | 0 |
Validation on more models: We added results also for Gemma-4-12B-it and for Mistral-Small-24B-Instruct-2501 to Appendix c.
We thank the reviewer for their comments.
Calibration: The reviewer is correct that using different calibration dataset in the same experiment was sloppy. We apologize for this embarrassing mistake. In our revision we repeated all SmoothGPTQ experiments with the FineWeb-Edu calibration dataset, and inserted the new results instead of the SlimPajama results. For the reviewer's convenience, we provide in the table below the results with SlimPajama calibration (original submission) and with FineWeb-Edu (revision). For the 1B model changing the calibration set deteriorated the results, but for all other models we have obtained improved results with FineWeb-Edu.
Table E. SmoothGPTQ benchmark accuracy (%) with the SlimPajama calibration of the original submission and the FineWeb-Edu calibration of the revision. Quantization format, optimizer settings and evaluation protocol are otherwise unchanged.
Our SmoothGPTQ is based on minimizing the approximated MSE expression D~(S) over S. The function D~(S) is based on calibration data, but our actual goal is to minimize D(S) which is the expected risk over runtime activations. Thus, in a sense what we are minimizing is an ``empirical risk'' and hopefully the minimizer also achieves a small population risk. We take a large number of calibration samples in order to avoid overfitting.
The situation in SmoothQuant is different for two reasons:
Thus, we agree with the reviewer's statement that SmoothQuant generally requires less activation vectors than SmoothGPTQ. However, we do not view this as a significant shortcoming, as calibration vectors are easy to collect.
Format: We agree with the reviewer that the NVFP4 results we included in Tables 7-10 (now 8-11) were more confusing than helpful, and we therefore removed them from the revision. In the table for Llama3.3-70B (now Table 11) we now also added MR-GPTQ.
Baselines: we have followed the reviewers' suggestion and evaluated SmoothGPTQ on more LLMs and compared against more quantization schemes. The results are reported in Appendix C, and comparison to more competing PTQ schemes is given above in our response "Additional benchmarks experiments requested by the reviewers" to reviewer 5NVJ.
We agree with the reviewer's comment that rotation-based methods generally achieve the state of the art in W4A4. SmoothGPTQ was designed for maximum accuracy with minimum FLOPS overhead. For this reason, SmoothGPTQ only requires diagonal transformations. Some of the competing schemes apply significantly denser linear transformations (with transformation complexity varying between different schemes), which our scheme intentionally avoids. Thus, the comparison to rotation-based schemes is somewhat unfair.
Nevertheless, for the models with more than 1B parameters that we tested SmoothGPTQ performs better than almost all of them, with WUSH being the only exception. WUSH uses denser linear transformations, and still only beats SmoothGPTQ with a small margin.
Note that now SmoothGPTQ is also calibrated on FineWeb (like all other schemes) MR-GPTQ not longer achieves better performance than our SmoothGPTQ on Llama-3.1-8B in NVINT4. The MR-GPTQ method requires more FLOPS than SmoothGPTQ, and in our experiment SmoothGPTQ nevertheless generally performed better.