acceptodds
Under review as a conference paper at ICLR 2027

SmoothGPTQ: Optimizing Diagonal Scaling for GPTQ Weight-Activation Quantization

Abstract

Diagonal smoothing, as in SmoothQuant, is a useful tool for balancing the errors incurred from weights quantization and activations quantization. For weight quantization the error can be further reduced by accounting for the activations second-order statistics via GPTQ. Prior work have considered the optimization of diagonal smoothing parameters for round-to-nearest (RTN) quantization. In this work we develop a framework, termed SmoothGPTQ, for optimizing those parameters for GPTQ-quantized weights and RTN-quantized activations. We derive an approximate expression for the quantization mean-squared error MSE as a function of . This expression is convex in log-scale and is therefore amenable to minimization, and its accuracy is established through experiments. Applying the developed SmoothGPTQ algorithm to various LLMs give state-of-the-art results for W4A4 INT quantization across various benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training Quantization, GPTQ, SmoothQuant

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j15LOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SmoothGPTQ, a framework for optimizing the diagonal smoothing matrix SS when weights are quantized with GPTQ and activations with RTN. Diagonal smoothing, as in SmoothQuant, balances the errors from weight and activation quantization, but prior work optimizes the smoothing parameters for RTN only, without considering the error compensation of GPTQ. To address this, the authors derive an approximation of the quantization MSE as a function of SS, which is convex in the log-scale and can thus be minimized efficiently. The authors show that the approximation is accurate at high bit-widths, and report state-of-the-art results for W4A4 INT quantization on several LLMs.

Critical strengths
  1. SmoothQuant followed by GPTQ is widely used in practice, while the smoothing scales are designed for RTN. Optimizing SS for GPTQ is thus directly useful for real quantization pipelines.

  2. The effect of GPTQ enters the proxy through the Cholesky diagonal only (Eq. (8)), providing a clear interpretation of how error compensation changes the optimal scales.

Critical weaknesses
  1. The comparisons do not support the state-of-the-art claim for W4A4 INT quantization.

    • Calibration. All baseline results in Tables 6-10 are copied from the MR-GPTQ paper, while SmoothGPTQ is calibrated on SlimPajama with its own settings (Appendix D.7). SmoothGPTQ and the baselines are thus calibrated with different data and settings, and the gains may partly come from the calibration. In the comparisons run by the authors (Tables 2-4), SmoothGPTQ also determines SS from 100K activation vectors, whereas SmoothQuant uses only 2n2n vectors, which may differ.
    • Format. In Tables 7-10, SmoothGPTQ is quantized to NVINT4, while most of the compared methods are quantized to NVFP4. Based on these tables, Sec. 5.2 claims that SmoothGPTQ is competitive with or outperforms the NVFP4 methods. This comparison, however, cannot tell whether the gains come from the method or the format, since the two formats differ. For the 70B model (Table 10), this problem is more severe, as all baselines use NVFP4 and SmoothGPTQ is compared only across formats.
    • Baselines. Rotation-based methods achieve the state of the art in W4A4. The paper, however, includes QuaRot and SpinQuant only in the NVFP4 rows of Appendix D.7, although their main setting is INT4, so SmoothGPTQ is never compared with them in INT4. The INT4-G32 benchmark (Table 6) includes only RTN, RTN with a Hadamard transform, and GPTQ. OmniQuant, which also optimizes diagonal scales, is compared only at W6A6 (Table 5), not at W4A4. Even within the reported results, SmoothGPTQ does not achieve the best accuracy, and MR-GPTQ outperforms it on Llama-3.1-8B in NVINT4 (76.66 vs. 76.55, Table 9).

    The authors should compare SmoothGPTQ with these methods in INT4-G32, using the same calibration data and quantization scope (e.g., whether the KV cache and attention matmuls are quantized).

  2. The distortion proxy, the core of SmoothGPTQ, is not validated in the main setting.

    • Accuracy at 4 bits. The proxy assumes that clipping in GPTQ is negligible (Sec. 2.1). At 4 bits, however, clipping is likely to occur frequently, since the weight scale has only a small margin (τ=0.05\tau = 0.05). For the down projection at 4 bits, the measured MSE is indeed only half of the predicted one (Table 1). At 2 and 3 bits, the optimized SS even gives a higher KL divergence than GPTQ without smoothing (Table 3), suggesting that the proxy becomes unreliable at low bit-widths.
Questions for authors
  • The proxy is validated on Llama-3.1-8B-Instruct only, and the benchmarks are limited to the Llama-3 family. The effectiveness of SmoothGPTQ depends on both the channel correlation and the outlier distribution of activations, which vary across model families. Do both the accuracy of the proxy and the gains hold for other families (e.g., Qwen and Mistral) and MoE models?

  • Minor: Appendix D.7 contains a broken citation ("SlimPajama (?)").

2 comments

We thank the reviewer for their comments.

Calibration: The reviewer is correct that using different calibration dataset in the same experiment was sloppy. We apologize for this embarrassing mistake. In our revision we repeated all SmoothGPTQ experiments with the FineWeb-Edu calibration dataset, and inserted the new results instead of the SlimPajama results. For the reviewer's convenience, we provide in the table below the results with SlimPajama calibration (original submission) and with FineWeb-Edu (revision). For the 1B model changing the calibration set deteriorated the results, but for all other models we have obtained improved results with FineWeb-Edu.

Table E. SmoothGPTQ benchmark accuracy (%) with the SlimPajama calibration of the original submission and the FineWeb-Edu calibration of the revision. Quantization format, optimizer settings and evaluation protocol are otherwise unchanged.

ModelFormatCalibrationMMLU-CoTGSM8kHellaSwagWinoGrandeAvg.
Llama-3.2-1BNVINT4SlimPajama36.7336.1658.7459.8347.86
FineWeb33.0732.4558.4661.4046.35
Llama-3.2-3BNVINT4SlimPajama61.6473.2472.0269.3069.05
FineWeb62.4573.4671.8469.2269.24
Llama-3.1-8BINT4-G32SlimPajama67.9079.9178.6776.0975.64
FineWeb68.9379.9179.2275.9376.00
NVINT4SlimPajama69.5081.3579.3376.0176.55
FineWeb69.6082.8779.6975.9377.02
Llama-3.3-70BNVINT4SlimPajama85.4295.0786.0584.2187.69
FineWeb85.5694.6286.1484.5387.71

Our SmoothGPTQ is based on minimizing the approximated MSE expression D~(S)\tilde{D}(S) over SS. The function D~(S)\tilde{D}(S) is based on calibration data, but our actual goal is to minimize D(S)D(S) which is the expected risk over runtime activations. Thus, in a sense what we are minimizing is an ``empirical risk'' and hopefully the minimizer also achieves a small population risk. We take a large number of calibration samples in order to avoid overfitting.

The situation in SmoothQuant is different for two reasons:

  1. It chooses SS as a simple function of WW and the calibration XX, and does not minimize a function of SS. Thus, the risk of overfitting is smaller.
  2. SmoothQuant does not use GPTQ and is therefore not sensitive to joint second-order statistics of different activation channels.

Thus, we agree with the reviewer's statement that SmoothQuant generally requires less activation vectors than SmoothGPTQ. However, we do not view this as a significant shortcoming, as calibration vectors are easy to collect.

Format: We agree with the reviewer that the NVFP4 results we included in Tables 7-10 (now 8-11) were more confusing than helpful, and we therefore removed them from the revision. In the table for Llama3.3-70B (now Table 11) we now also added MR-GPTQ.

Baselines: we have followed the reviewers' suggestion and evaluated SmoothGPTQ on more LLMs and compared against more quantization schemes. The results are reported in Appendix C, and comparison to more competing PTQ schemes is given above in our response "Additional benchmarks experiments requested by the reviewers" to reviewer 5NVJ.

We agree with the reviewer's comment that rotation-based methods generally achieve the state of the art in W4A4. SmoothGPTQ was designed for maximum accuracy with minimum FLOPS overhead. For this reason, SmoothGPTQ only requires diagonal transformations. Some of the competing schemes apply significantly denser linear transformations (with transformation complexity varying between different schemes), which our scheme intentionally avoids. Thus, the comparison to rotation-based schemes is somewhat unfair.

Nevertheless, for the models with more than 1B parameters that we tested SmoothGPTQ performs better than almost all of them, with WUSH being the only exception. WUSH uses denser linear transformations, and still only beats SmoothGPTQ with a small margin.

Note that now SmoothGPTQ is also calibrated on FineWeb (like all other schemes) MR-GPTQ not longer achieves better performance than our SmoothGPTQ on Llama-3.1-8B in NVINT4. The MR-GPTQ method requires more FLOPS than SmoothGPTQ, and in our experiment SmoothGPTQ nevertheless generally performed better.

Low rate accuracy: Regarding the mediocre performance at R=2,3R=2,3 bits: Our approximate distortion expression is developed under a high-resolution assumption, and several approximations we make in the derivation are not really justifies when the quantization noise is not much smaller than the signal (weights or activations). For R=2,3R=2,3 the quantization error is too large, so that our distortion approximation is highly inaccurate, and optimizing SS with respect to it leads to bad results. In our revision we better emphasize the fact that the approximation assumes high-rate. This was probably not sufficiently emphasized before.

Discrepancy in down projection distortion approximation: The measured/predicted MSE ratio is 0.5075 (Table 1) for R=4R=4 has a rather delicate explanation. The activations in the down projection involve the SwiGLU term. Consequently, many of them are very close to 00. For relatively small rates, even R=4R=4, the conditional distribution of activations falling in the quantization cell around zero is not uniform - it is concentrated around 0. Our high-resolution approximation on the other hand, assumes the quantization error is uniform, and therefore overestimates the distortion. In our revision we added an explanation of this phenomenon to Appendix E.5.

Clipping: We have measured the frequency of clipping in GPTQ and provide the results below. Indeed clipping rarely occurs with τ=0.05\tau=0.05

Table F. Percentage of weight entries clipped by GPTQ to [−βR,βR][-\beta_R,\beta_R] with the optimized SmoothGPTQ scaling and margin τ=0.05\tau=0.05 on Llama-3.1-8B-Instruct, averaged over the 32 layers. Top: the setting of Table 1 (g=ng=n, δ=0\delta=0). Bottom: the INT4-G32 setting of Table 11 (g=32g=32, δ=0.05\delta=0.05). The largest single-layer value occurs in the Q/K/V projections of the first layer at g=ng=n (5.0% at R=4R=4); at INT4-G32 it is 0.061% at R=4R=4.

SettingRRQKVOGATEUPDOWN
g=ng=n, δ=0\delta=020.380.370.420.00840.0630.060<0.001
30.280.270.290.00160.0520.052<0.001
40.170.170.17<0.0010.0480.048<0.001
60.0370.0370.037<0.0010.0360.0360
80.00620.00610.006200.0140.0140
g=32g=32, δ=0.05\delta=0.0520.0450.0660.0160.0740.0230.0180.0097
30.0160.0240.00480.0260.00730.00550.0033
40.00750.0120.00120.00760.00230.0016<0.001
6<0.001<0.001<0.001<0.001<0.001<0.001<0.001
8<0.001<0.0010<0.001000

Validation on more models: We added results also for Gemma-4-12B-it and for Mistral-Small-24B-Instruct-2501 to Appendix c.

Reviewer_6a6XOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper optimizes diagonal smoothing for weight quantization via GPTQ and round-to-nearest quantized activations. It derives a high-rate output-distortion proxy via Cholesky pivots of the activation second moment, accounting for GPTQ feedback. The proxy is convex in logarithmic scales and separates across quantization groups. SmoothGPTQ optimizes these scales from calibration data, then applies GPTQ to the transformed weights. The experiments assess prediction accuracy, reconstruction error, full-model KL/perplexity, and downstream W4A4 accuracy.

Critical strengths
  1. The contribution is technically well motivated.
  2. The log-scale convexity argument in Proposition 3.1 is clear and convincing. The grouped formulation and treatment of shared-input projections make the objective applicable to practical quantization pipelines.
  3. The validation connects the analytic proxy to measured distortion and end-to-end quality.
Critical weaknesses

The relationship between the proxy and the target low-bit regime needs better characterization. For example, at 4-bit, the DOWN-projection measured/predicted MSE ratio is 0.5075 (Table 1). At 2- and 3-bit, optimized scaling does not improve full-model KL over GPTQ (Table 3). These results expose limitations, but it remains unclear which approximation errors matter for selecting useful scales at 4-bit.

Questions for authors
  1. With the same optimizer and calibration budget, how does performance change if the Cholesky pivot coefficients in the objective are replaced by marginal activation variances?
  2. At INT4-G32, how accurate is the proxy, and is the discrepancy primarily explained by clipping, residual nonuniformity, or cross terms?
1 comment

We thank the reviewer for their comments.

Regarding the mediocre performance at R=2,3R=2,3 bits: Our approximate distortion expression is developed under a high-resolution assumption, and several approximations we make in the derivation are not really justifies when the quantization noise is not much smaller than the signal (weights or activations). For R=2,3R=2,3 the quantization error is too large, so that our distortion approximation is highly inaccurate, and optimizing SS with respect to it leads to bad results. In our revision we better emphasize the fact that the approximation assumes high-rate. This was probably not sufficiently emphasized before.

There is a rather delicate explanation to why the down-projection measured/predicted MSE ratio is 0.5075 (Table 1) for R=4R=4. The activations in the down projection involve the SwiGLU term. Consequently, many of them are very close to 00. Consequently, the conditional distribution of activations falling in the quantization cell around zero is not uniform - it is concentrated around 0. Our high-resolution approximation on the other hand, assumes the quantization error is uniform, and therefore overestimates the distortion. This effect is more pronounced for small RR, even R=4R=4, where the quantization cells are relatively large. In our revision we added an explanation of this phenomenon to Appendix E.5.

Q1: We have tested this for Llama-3.1-8B-Instruct and report the results in the table below. To our surprise, the difference is found to be negligible (for this model). We nevertheless believe the optimization of SS should be performed with respect to the actual loss (that includes the Cholesky factors, rather than diagonal elements of ΣX\Sigma_X); When diag(U)2\mathrm{diag}(U)^2 has entries sufficiently smaller than diag(ΣX)\mathrm{diag}(\Sigma_X), using the wrong criterion would make a great difference. For some reason, this is not the case for Llama-3.1-8B-Instruct.

Table C. Mean INT4-G32 local distortion across all 32 layers. SQ denotes SmoothQuant; all methods use activation RTN. The last column optimizes SS with a symmetric proxy in which the weight term uses the covariance diagonal (ΣX)ii(\Sigma_X)_{ii} instead of the Cholesky pivots ui2u_i^2, and is otherwise identical to SmoothGPTQ (same GPTQ quantizer, τ\tau, δ\delta, pool, schedule and initialization, with its own calibration-only step-size selection).

ProjectionUnitGPTQ (S=IS=I)SQ+RTN (α=0.5\alpha=0.5)SQ+RTN (α=0.85\alpha=0.85)SQ+GPTQ (α=0.5\alpha=0.5)SQ+GPTQ (α=0.85\alpha=0.85)SmoothGPTQSymmetric (no Cholesky)
Q10−310^{-3}8.47.28.15.15.74.34.4
K10−210^{-2}1.81.82.31.21.61.11.1
V10−310^{-3}3.72.82.22.31.81.81.8
O10−510^{-5}9.612.310.98.67.36.06.0
GATE10−310^{-3}2.12.62.52.22.01.81.8
UP10−310^{-3}1.41.71.51.41.21.21.2
DOWN10−410^{-4}6.68.512.16.79.23.53.5

Q2: The table below measures the fraction of the neglected cross term's energy (energy of eWˉj⊤eXˉke_{\bar W_j}^\top e_{\bar X_k}) from the held-out MSE. In our response to Reviewer j15L we also demonstrate that the effect of clipping is negligible. Thus, the discrepancy mostly follows from neglecting the product of quantization errors in our analysis. As the rate grows, the effect of this term indeed vanishes, consistent with the high-rate assumption in our derivation. For down-projection there is another source of discrepancy that we described above.

Table D. Energy of the neglected term eWˉj⊤eXˉke_{\bar W_j}^\top e_{\bar X_k}, 1ab∑j,k(eWˉj⊤eXˉk)2\frac{1}{ab}\sum_{j,k}(e_{\bar W_j}^\top e_{\bar X_k})^2, as a percentage of the held-out MSE, at the optimized SS, in the setup of Table 1 of the main paper (5,000 held-out vectors, all 32 layers; mean of the per-layer fractions). Top: g=ng=n, δ=0\delta=0. Bottom: INT4-G32 setting with g=32g=32, δ=0.05\delta=0.05.

SettingRRQKVOGATEUPDOWN
g=ng=n, δ=0\delta=0244436360536168
324261820151329
46.67.74.25.83.32.78.7
60.580.650.280.370.320.280.55
80.190.170.0470.0230.270.250.033
g=32g=32, δ=0.05\delta=0.05228282831272724
35.45.94.55.34.03.94.4
41.11.30.901.10.790.760.94
60.0620.0710.0480.0560.0420.0400.054
80.00340.00400.00280.00320.00240.00230.0031
Reviewer_5NVJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

• The paper proposes SmoothGPTQ, which applies GPTQ instead of RTN to the weights after diagonal scaling (migration) in SmoothQuant, while jointly optimizing the migration parameters (S). • The paper theoretically derives an approach to approximating the optimal (S), using a method similar to WaterSIC. • The proposed method achieves better performance than GPTQ on smaller Llama 3-family models.

Critical strengths

The strengths are not clear.

Critical weaknesses

• Limited novelty and incremental contributions. The idea of optimizing the migration scales in SmoothQuant is closely related to OmniQuant, while the proposed methodology bears similarities to WaterSIC. The paper does not sufficiently demonstrate whether applying the WaterSIC-like optimization to the SmoothQuant+GPTQ pipeline provides substantial practical benefits in terms of either computational cost or quantization performance. • Insufficient experimental validation. The main benchmark results compare SmoothGPTQ primarily against GPTQ, and the experiments are limited to the Llama 3 family. Furthermore, the paper does not provide comparisons against the baselines on larger models such as 70B-parameter LLMs. These limitations make it difficult to establish the practical utility and generalizability of the proposed method.

Questions for authors

• Does optimizing the migration parameters with SmoothGPTQ provide a clear advantage over applying GPTQ to weights quantized using OmniQuant? • Does SmoothGPTQ also outperform other baselines, such as AffineQuant, QuaRot, DuQuant, SpinQuant, OSTQuant, and WUSH on benchmark tests? • In particular, does SmoothGPTQ offer meaningful advantages over recent methods such as OSTQuant and WUSH, which also learn transformations to make weight and activation distributions easier to quantize? • Does the proposed method maintain its performance on LLMs beyond the Llama family?

2 comments

We thank the reviewer for their comments.

The reviewer correctly identifies that diagonal smoothing is by no means new, and prior work has considered optimization of the smoothing matrix SS from different angles.

To the best of our knowledge, no prior work have considered the optimization of SS for weights GPTQ and activations RTN quantization. The ``surprise'' in this work is that for high-quantization rate we can actually calculate the optimal SS. In particular, we show that the MSE can be well-approximated using a simple expression, which is convex in log-coordinates and can then be minimized with respect to SS.

The reviewer mentions OmniQuant and WaterSIC. Both differ from our setup significantly:

  • WaterSIC optimizes SS for weight-only GPTQ quantization to the integers, \textbf{followed by entropy coding}. The use of entropy coding drastically changes the optimization criterion: for INT quantization what dictates the MSE distortion is the value of the maximum (smoothed) weight entry. Under entropy coding, large outliers have negligible effect on performance. The WaterSIC choice of SS therefore leads to terrible performance under INT quantization, even if only weights are quantized and activations remain in full precision.

  • OmniQuant optimizes for RTN to INT-like constellations. This is indeed closer to our setup. However, there are several differences: 1)OmniQuant is not optimized for GPTQ; 2)In OmniQuant there is no closed-form convex expression of the smoothing parameters that is minimized. Instead, it learns good smoothing parameters. Since OmniQuant learns the smoothing parameters for all projections in a layer jointly, while SmoothGPTQ optimizes each projection separately, one might expect OmniQuant to come out ahead. It does not: SmoothGPTQ performs significantly better (See enclosed tables). We think this is because SmoothGPTQ’s per-projection loss expression is highly accurate. That accuracy seems to matter more than optimizing jointly.

In terms of efficiency, under SmoothGPTQ inference is essentially as fast as vanilla RTN. This was the main guideline for this scheme: achieve maximum accuracy with essentially no additional FLOPS than the simplest quantization method. Subsection 5.3 in the revised manuscript demonstrates this experimentally.

In terms of experimental results, we have followed the reviewers' suggestion and evaluated SmoothGPTQ on more LLMs and compared against more quantization schemes (including all schemes the reviewer mentioned). The results are inserted in the comment below. See also extended results in Appendix C, including for Gemma and Mistral models.

We would like to point the reviewer's attention to the fact that some of the comparisons are somewhat unfair. SmoothGPTQ was designed for maximum accuracy with minimum FLOPS overhead. For this reason, SmoothGPTQ only requires diagonal transformations. Some of the competing schemes apply significantly denser linear transformations (with transformation complexity varying between different schemes), which our scheme intentionally avoids. Nevertheless, for the models with more than 1B parameters that we tested, SmoothGPTQ performs better than almost all of them, with WUSH being the only exception. Remarkably, the performance of SmoothGPTQ is on par even with WUSH, despite the fact that the latter uses denser linear transformations.

AuthorsOpenReview ↗

Additional benchmarks experiments requested by the reviewers

Table A. NVINT4 (INT4 weights and activations, group 16, E4M3 group scales) on Llama-3.2-1B/3B-Instruct and Llama-3.1-8B-Instruct. All methods are run by us in the same quantization setting (all decoder linear layers; embeddings, LM head, attention and KV cache in BF16), calibrated on the same FineWeb-Edu sequences at each method's default sample count, and evaluated with the same vLLM protocol. Recovery is relative to BF16 in the same protocol.

ModelMethodMMLU-CoTGSM8kHellaSwagWinoGrandeAvg.Recovery%
Llama-3.2-1BBF1646.5545.5661.1161.4053.66–
RTN34.7226.0054.7956.2742.9580.04
GPTQ31.5127.2255.9957.0642.9580.04
MR-GPTQ42.6436.3957.9759.1949.0591.41
QuaRot39.6433.3655.6160.2247.2187.98
SpinQuant34.8031.8457.9159.1245.9285.58
OSTQuant39.5733.5958.0759.1247.5988.69
DuQuant36.2216.2249.0957.6239.7974.15
OmniQuant35.6224.4155.9559.8343.9581.90
OmniQuant+GPTQ39.8326.9157.4059.7545.9785.67
AffineQuant32.6527.9056.2157.2243.5081.07
WUSH (RTN)42.6934.3459.1360.6949.2291.73
WUSH (GPTQ)40.0736.6259.3061.0949.2791.82
SmoothGPTQ33.0732.4558.4661.4046.3586.38
Llama-3.2-3BBF1664.5377.4873.5671.1171.67–
QuaRot58.0966.6469.0367.9665.4391.29
SpinQuant59.6465.9671.0268.3566.2492.42
OSTQuant61.4071.4271.1567.8867.9694.82
DuQuant56.6559.6765.0365.8261.7986.21
OmniQuant60.7766.7970.3465.9865.9792.05
OmniQuant+GPTQ59.0868.0870.6467.4066.3092.51
AffineQuant59.7970.1370.8868.7567.3994.03
WUSH (RTN)61.3673.3171.5869.2268.8796.09
WUSH (GPTQ)61.6273.6271.9270.3269.3796.79
SmoothGPTQ62.4573.4671.8469.2269.2496.61
Llama-3.1-8BBF1672.5585.9080.4077.5879.11–
QuaRot67.5079.4577.7375.7775.1294.96
SpinQuant67.5677.4178.1575.7774.7294.45
OSTQuant69.2179.6178.7275.6975.8195.83
DuQuant61.6368.0873.5274.5169.4387.76
OmniQuant68.6278.4778.6676.4075.5495.49
OmniQuant+GPTQ67.7778.5478.8274.4374.8994.67
AffineQuant68.6379.7678.1376.4075.7395.73
WUSH (RTN)69.5379.6879.2976.0976.1596.26
WUSH (GPTQ)70.1881.7379.0977.1977.0597.40
SmoothGPTQ69.6082.8779.6975.9377.0297.36

Table B. INT4-G32 (INT4 weights and activations, group 32, BF16 group scales) on Llama-3.1-8B-Instruct. Same quantization setting, calibration data and evaluation protocol for all methods, all run by us.

MethodMMLU-CoTGSM8kHellaSwagWinoGrandeAvg.Recovery%
BF1672.5585.9080.4077.5879.11–
RTN66.2275.2177.6676.4873.8993.40
GPTQ67.5475.5178.3175.1474.1393.71
SmoothQuant57.8859.6773.9572.9366.1183.57
SmoothQuant+GPTQ67.6978.5478.7974.0374.7694.50
QuaRot (RTN)61.3169.3774.5874.0369.8288.26
QuaRot (GPTQ)67.7680.6778.2076.6475.8295.84
SpinQuant66.2378.3278.3675.1474.5194.19
OSTQuant68.3879.2378.7875.3775.4495.36
DuQuant64.5674.3775.8173.4872.0691.09
OmniQuant64.3671.8776.4274.1171.6990.62
OmniQuant+GPTQ65.4374.7577.3674.1972.9392.19
AffineQuant65.7072.9377.1974.1172.4891.62
WUSH (RTN)69.0778.7079.2074.4375.3595.25
WUSH (GPTQ)68.5980.5979.1676.8076.2896.42
SmoothGPTQ68.9379.9179.2275.9376.0096.07

Comments on the paper

2 comments

Dear Authors,

Thank you for your revisions. Although the response indicates that the manuscript has been revised, the changes are not highlighted in the revised manuscript, making it difficult to identify and review the modifications.

Could you please highlight the revised portions in a different color in the manuscript to facilitate the review process?

Thank you.

Dear reviewer,

You are absolutely right, we should have done that in the first place. We uploaded a new revision with all changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.