acceptodds
Under review as a conference paper at ICLR 2027

ReSC-Q: Residual-Aware Reliable PTQ with Shrinkage-Controlled Compensation

Abstract

Post-training quantization (PTQ) reduces the memory footprint and inference cost of LLMs under tight deployment constraints. Hessian-based PTQ methods such as GPTQ use estimated curvature information for error compensation, but compensation estimated from calibration data does not always lead to optimal reconstruction loss on unseen inputs. This becomes especially problematic in ultra low-bit quantization. We propose ReSC-Q, which controls uncertain cross-group curvature through shrinkage while explicitly accounting for the reconstruction loss remaining after compensation. ReSC-Q supports practical group-wise quantization by separating the blockwise reconstruction objective into cross-group compensation and within-group quantization, applying shrinkage and residual-aware weighting to each, respectively. Across seven LLMs evaluated at 2-bit, ReSC-Q improves average zero-shot accuracy by 1.09 percentage points over the strongest baseline for each model, reducing the mean accuracy gap to full precision by 23.2%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Quantization, Post Training Quantization, Weight-only Quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q2NBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces ReSC-Q which is a post-training quantization method. It is designed to address sub-optimal error compensation in ultra low-bit quantization. It splits the reconstruction objective into two groups: cross-group compensation and within-group quantization, applying shrinkage to manage curvature uncertainty and residual-aware weighting to handle remaining loss. Evaluation results across seven LLMs at a very low bit quantization, 2-bit quantization, ReSC-Q improves average zero-shot accuracy over the strong baselines, reducing the mean accuracy gap to full precision models by a large ratio (23.2%).

Critical strengths

1: ReSC-Q demonstrates strong effectiveness in extremely low-bit post-training quantization. According to seven evaluated LLMs at 2-bit precision. it improves zero-shot accuracy by 1.09% in average. In addition, it bring benefits to success rate to massive architectures. 2: This paper provides a comprehensive analysis to show that standard hessian-based error compensation suffers from noisy off-diagonal estimates especially, when calibration data is limited. ReSC-Q proposed in this work addresses this bottle neck by introducing a dual-pronged mechanism that apply corss-group shrinkage to suppress unreliable off-diagonal curvature. 3: The proposed residual-aware quantization objective is versatile, it can be seamlessly integrated into various quantization frameworks, ranging from group-wise affine quantization and rotation-based methods to advanced structured-code representations such as E8P lattices and trellis-coded quantizations.

Critical weaknesses

1:The formula used to adjust cross-group values relies on a basic assumption that errors are balanced and average out to zero. Due to this, it can miss hidden biases or differences between the sample data used during setup and the real-world data seen later. Therefore, the method might misjudge how reliable those connections actually are, making the adjustments either too weak or too strong. 2:Similar to other methods, ReSC-Q focuses on making each individual layer match the original model as closely as possible, rather than directly trying to improve final task performance. Hence, getting better layer-matching scores doesn't always guarantee that the model will actually perform better on real-world tasks.

Questions for authors

1:The shrinkage coefficient is based on a simplified noise model that assumes estimation errors have a zero mean. However, this assumption may not reflect differences between the calibration data and real-world deployment data. How much does the estimated coefficient change when the calibration data differs from the actual deployment data? Could using a noise model that adapts to these differences or accounts for systematic errors make the method more robust? 2:ReSC-Q optimizes the reconstruction quality of each layer rather than directly optimizing the performance of the final task. Therefore, better reconstruction does not always lead to better generation results. Could this method be improved by adding a lightweight penalty that considers the target task, or by combining reconstruction quality with task performance in the optimization objective, so that quantization better preserves the final task performance?

5 comments

We thank the reviewer for the questions about the noise assumption and the relationship between reconstruction and final task performance. We clarify the scope of the estimator, report how its coefficient changes with calibration data, and connect the task-aware suggestion to our existing sensitivity-weighting experiment.

W1, Q1-1. Zero-mean noise and systematic distribution differences

The zero-mean noise model describes sampling variation around the calibration-population Hessian. A difference between the calibration and deployment populations introduces an additional bias.

To express both effects, let hh denote one deployment-population matrix entry, β\beta the calibration-to-deployment bias, and ϵ\epsilon the sampling noise with variance vv. Then

\qquad \mathbb{E}[\epsilon]=0, \qquad \mathbb{E}[\epsilon^2]=v,$$ where $\hat h$ is the entry estimated from calibration data. For a fixed shrinkage coefficient $\rho$, the expected squared error, omitting the additional regularization term for clarity, is $$\mathbb{E}[(\rho\hat h-h)^2] = \bigl((\rho-1)h+\rho\beta\bigr)^2+\rho^2v.$$ This expression separates sampling variance from the bias introduced by shrinkage and distribution differences. A bias-aware extension could use this objective, but would require information about $\beta$. Estimating that difference requires representative deployment samples or additional assumptions about the distribution shift. ReSC-Q uses the calibration statistics available during quantization to estimate the sampling uncertainty and select its shrinkage coefficient. The same-population and cross-source experiments in our response to Reviewer 2 also show how the coefficient affects compensation: gains occur with small samples from the same population and also when compensation is evaluated against another source. The zero-mean assumption is used to derive the shrinkage coefficient. The within-group residual upper bound follows from matrix inequalities for the chosen curvature matrix and does not depend on this noise assumption. ### **Q1-2. How much does the coefficient change with calibration data?** We measured $\hat\rho$ (the estimated fraction of cross-group curvature retained for compensation) using the original model's activations while varying the calibration source and sample count. Here, $n$ is the number of calibration sequences. | Calibration source | n=8 | n=32 | n=128 | n=512 | n=1024 | | :-- | --: | --: | --: | --: | --: | | WikiText-2 | 0.342 | 0.486 | 0.665 | 0.800 | 0.847 | | C4 | 0.357 | 0.472 | 0.616 | 0.763 | 0.823 | *Table R5. Mean estimated coefficient across q_proj and down_proj at layers 5, 15, and 25 of Llama-3.1-8B-Instruct. Each sequence $n$ contains 2,048 tokens.* Both sources produce stronger shrinkage at small calibration sizes and retain more cross-group information as the sample count increases. At $n=128$, the mean coefficient is 0.665 for WikiText-2 and 0.616 for C4, a difference of 0.049. Differences within individual modules range from 0.033 to 0.071. The coefficient therefore responds to both calibration size and source. It is computed entirely from calibration statistics, so changing only the evaluation dataset leaves it unchanged. ### **W2, Q2. Layer reconstruction versus task-aware optimization** Our objective is to quantize a pretrained model using calibration data without additional task-specific training. We use layer reconstruction to guide quantization and measure final model performance separately on eight zero-shot tasks. A task-aware objective can be combined with this approach. Appendix C.7 examines a related way of using loss information: replacing the within-group fitting weights with empirical-Fisher sensitivity (importance derived from squared loss gradients). Within each quantization pipeline, the representation and all other components are kept fixed. Table R6 shows the relevant rows of Table 10. | Quantization pipeline | Within-group weighting | PPL | Zero-Shot Avg. | | :-- | :-- | --: | --: | | DASH-Q | Hessian diagonal | 17.17 | 56.69 | | DASH-Q | Empirical Fisher | **15.14** | **58.00** | | ReSC-Q | Residual upper-bound weights | 11.76 | **58.83** | | ReSC-Q | Empirical Fisher | **11.51** | 58.45 | *Table R6. Controlled sensitivity-weighting comparison from Appendix C.7. Llama-3.1-8B-Instruct, nominal 2-bit weights, group size 32, and 128 calibration sequences, each containing 2,048 tokens.* Fisher weighting improves both metrics for DASH-Q. For ReSC-Q, it slightly improves PPL but lowers average zero-shot accuracy. The same sensitivity signal therefore has different effects depending on the surrounding quantization and compensation procedure. This experiment incorporates loss information through the fitting weights. Adding an explicit target-task loss would be a further extension for settings where representative task data and an additional optimization budget are available.
AuthorsOpenReview ↗

Gentle reminder: author–reviewer discussion ending in two days

Dear Reviewer q2NB,

Thank you again for your review and thoughtful questions. With two days remaining in the discussion period, we would like to briefly highlight the explanations and additional evidence provided in our responses and revised manuscript.

Regarding the zero-mean noise assumption, we clarified that it describes sampling variation around the calibration-population Hessian, while systematic calibration-to-deployment differences remain a limitation. Our response reports how the estimated shrinkage coefficient changes with calibration source and sample count. Appendix C.4 additionally examines same-source and cross-source compensation and the associated variance–bias trade-off (Tables 6–7).

Regarding reconstruction versus task performance, our response discusses your task-aware suggestion through a controlled empirical-Fisher weighting comparison (Table 10), with the representation and other pipeline components held fixed. Section 7 also explicitly acknowledges that improvements in layer reconstruction need not translate monotonically to downstream accuracy.

We would appreciate your feedback on whether these explanations and experiments address your concerns. If they do, we would be grateful if you could consider updating your assessment and rating accordingly. If any questions remain, we would be happy to address them within the remaining discussion period.

Thank you again for your time and consideration.

Best regards, The authors

AuthorsOpenReview ↗

Follow-up before the discussion deadline

Dear Reviewer q2NB

With one day remaining in the discussion period, we would appreciate any brief feedback on whether our responses and revised manuscript address your main concerns. If any questions remain, we would be happy to clarify them before the discussion closes.

Thank you again for your time and consideration.

Best regards, The authors

Reviewer_EJqMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

It is a bit difficult to understand the exact scheme, but to my understanding the scheme is a variant of grouped-GPTQ fro weight-only quantization, with the following ingredients: 1)Within each group the quantization is done using round-to-nearest (RTN) 2)Once a group has been quantized, the quantization errors are fed back to the weight groups that were not quantized yet. The feedback coefficients are not calculated using the partial inverse calibration-based hessian, but a regulated version of it. 3)The quantizer for each group is a uniform quantizer with optimized scaled and shift.

The main innovations here are how to choose the filter coefficients in (2) and how to optimize the scales and shifts is (3). The latter seems to use an existing method from DASH-Q and to my understanding is not a major innovation, which leaves us with the optimization in (2).

In regards to that optimization, if I understood correctly, the scheme is essentially standard group-GPDQ (as in block-LDLQ) but with H replaced with H_\rho, where all off-diagonal entries are scaled by \rho, and diagonal entries stay the same. I think this has a very similar effect to damping which is a standard part of the GPTQ algorithm. In particular, if diag(H) is constant, I think the two give the same matrix up to scaling. Of course in practice diag(H) is not fixed, and there is also the issue of how to optimize the damping coefficient, but I still think the "shrinkage" idea is not very innovative in light of the fact that damping is typically used anyways.

Critical strengths

-The paper develops a procedure for processing the Hessian matrix such as to obtain better results when grouped-GPTQ is applied -The experimental results look quite encouraging

Critical weaknesses

-The presentation of the scheme is challenging to read. Modern accounts of GPTQ use the Cholesky decomposition of the Hessian or inverse Hessian as in the Quip# paper and in Chen, Jiale, et al. "The geometry of llm quantization: Gptq as babai's nearest plane algorithm." International Conference on Learning Representations. Vol. 2026. 2026.

Representing the new algorithm under this view will make it more readable.

-The connection to damping is never discussed, although it is crucial to the understanding of what is new in this paper.

-The experimental results are hard to comprehend. Each group is given two FP16 numbers - one for the scale and one for the shift. With the group sizes that are used in the paper, what appear in the table as 2-bits per weight is actually 3-bits per weight, 3-->3.5, 4-->4.25. Thus, I find the claims about, e.g., achieving SOTA in 2-bit quantization misleading.

-The introduction makes some statements that are not completely justified and introduces definitions that seem a bit strange to me. For example: -Why are estimates of "off-diagonal entries" less reliable than diagonal entries? -Where do we have limited calibration data? We can, and usually do, generate as many calibration samples as needed for accurate estimation. The problem is typically different: the dataset we use for calibration is sometimes not a faithful representation of the activation "population" because it doesn't cover all domains. But this is not a issue of having less or more calibration samples. -I don't understand the relation between orthogonal transforms and Hessian-based error compensation. Orthogonal transforms do not change the hessian eigenvalues. Whenever those eigenvalues are not all equal, there will be improvement from hessian-based error compensation (GPTQ). -I find the definition of SNR_{ij} in section 4 very strange. Why treat |E(\hat{H}{ij})| as the signal strength? In general, diagonal entries of H will have larger magnitude than off-diagonal elements because of Cauchy-Schwartz, but what is the operational meaning of this? Why should I care about SNR{ij}?

Questions for authors

-What is conceptually different here from optimized damping? Can you compare some of your results to group-GPTQ with optimized damping coefficient, and explain why ReSC-Q is better? -How can we know the gain is from reducing estimation error of hessian and not from regularizing agains modal mismatch (the calibration set does not represent modes that do exist in the test set) -Does the scheme really give SOTA results in its effective quantization rate (that also accounts for the quantizer's parameters)? -Will

8 comments

We thank the reviewer for the suggestions on the factorized formulation, damping, storage accounting, and calibration diagnostics. We provide the requested controlled damping comparison and new experiments separating finite-sample effects from distribution mismatch.

S0. Clarification for the within-group objective and the shrinkage versus damping

The reviewer’s summary describes the groupwise feedback structure. Two details are important for understanding the contribution: the objective used to fit each group and the Hessian blocks to which shrinkage is applied.

For within-group optimization, Algorithm 1 assigns integer codes by RTN at the current scale ss and offset zz, and alternates this assignment with weighted regression updates of ss and zz to optimize Eq. (14), whose closed-form solution is provided in Appendix A.4. RTN is therefore one step within the optimization; the fitting objective determines the quantization grid on which rounding operates.

Also, the alternating affine solver is adopted from DASH-Q, while the residual-aware objective is derived in this work. DASH-Q uses the Hessian diagonal as its fitting weights. ReSC-Q instead derives djd_j (the fitting weight for coordinate jj) from a conservative upper bound on the post-compensation residual in Eq. (6). These weights include the magnitudes of within-group cross terms, giving the upper-bound relation in Eq. (12) that diagonal-only weighting does not generally provide. Thus, reusing the solver does not mean reusing the objective. DASH-Q also discards cross-group curvature entirely: its perplexity is substantially worse than ReSC-Q’s, its accuracy does not improve with more calibration data (Table 5), and it degrades quickly at larger group sizes (Table 8).

Table 6 isolates this distinction by replacing the residual-aware weights with diagonal-only weights while keeping cross-group shrinkage and blockwise compensation fixed. Table 10 further compares alternative sensitivity weights under the same quantization procedure. These experiments examine the within-group objective separately from the compensation regularizer.

Regarding shrinkage scope with damping, Eq. (9) scales only the entries between quantization groups and preserves each within-group block, including its off-diagonal entries. The shrinkage target is therefore the full block-diagonal matrix. The equivalence to identity damping described in the review holds, up to an overall scaling, when all off-diagonal entries are scaled and all diagonal entries are equal. In our case, equal diagonal entries alone are not sufficient, since the preserved within-group blocks can also contain nonzero off-diagonal entries, so no damping coefficient reproduces H~ρ\widetilde H_\rho. We explain the precise relationship and provide the controlled damping comparison in W2/Q1.

These two choices are connected through Eq. (6): shrinkage regularizes compensation, while the residual-aware objective fits the current group using an upper bound constructed from the preserved within-group curvature. ReSC-Q therefore changes both the feedback coefficients and the criterion used to choose the current group’s quantized representation.

W4-3. Orthogonal transformations and error compensation

Our statement in Sec.2 (“However, even after the transformation, Hessian-based error compensation often remains necessary”) is that rotation may mitigate the quantization difficulty due to the outliers, though compensation still remains useful after rotation; not that rotation makes compensation difficult or changes Hessian eigenvalues. The two operations are presented as complementary.

If the reviewer is referring to the statement in Appendix C.10, it discusses a separate question: conceptually how the rotation works for group-wise affine fitting. The discussion concerns representing a group with a shared scale and offset, and why the benefit of rotation can depend on group size. It does not attribute the observed fitting behavior to cross-group compensation.

W1. Presenting the method in the block LDL/Cholesky view

Our implementation computes groupwise compensation through inverse-Hessian block updates. These updates give the same compensation coefficients as the Cholesky formulation for a fixed Hessian and group order.

Let H(k)H^{(k)} (the positive-definite Hessian used for compensation at step kk) and M(k)M^{(k)} (its inverse) be partitioned over the current group and future coordinates:

\begin{bmatrix} A_k&B_k\\\\ B_k^\top&C_k \end{bmatrix}, \qquad M^{(k)} = \begin{bmatrix} E_k&F_k\\\\ F_k^\top&G_k \end{bmatrix}.$$ Here, $A_k,E_k$ are the current-group blocks, $B_k,F_k$ are the current-to-future blocks, and $C_k,G_k$ are the future-coordinate blocks. For $e_k$ (the current-group quantization error), our implementation solves $E_kY_k=F_k$, where $Y_k$ is the feedback matrix, and computes $$\boldsymbol{\delta}_{\mathcal{R}_k}^{\star}=-e_kY_k, \qquad M^{(k+1)}=G_k-F_k^\top Y_k,$$ where $\boldsymbol{\delta}_{\mathcal{R}_k}^{\star}$ is the future-weight update and $M^{(k+1)}$ is the inverse Hessian for the remaining coordinates. These are the updates in Appendix A.1. To express the same operation in Cholesky form, write $$M^{(k)}=R_k^\top R_k, \qquad R_k= \begin{bmatrix} R_{11}&R_{12}\\\\ 0&R_{22} \end{bmatrix},$$ where $R_k$ is an upper-triangular Cholesky factor with the same block partition. Multiplying the factors gives $E_k=R_{11}^\top R_{11}$ and $F_k=R_{11}^\top R_{12}$. Therefore, $$\boldsymbol{\delta}_{\mathcal{R}_k}^{\star} = -e_kE_k^{-1}F_k = -e_kR_{11}^{-1}R_{12}, \qquad M^{(k+1)} = R_{22}^\top R_{22}.$$ Thus, our inverse-block update produces the same feedback as the triangular factor, and the next inverse corresponds to its remaining trailing block. The residual also has a direct correspondence. With $S_k$ (the Schur complement, or residual curvature after optimal compensation), $$S_k=A_k-B_kC_k^{-1}B_k^\top=E_k^{-1}, \qquad e_kS_ke_k^\top = \left\lVert e_kR_{11}^{-1}\right\rVert _2^2.$$ This is the residual term in Eq. (6), expressed through the Cholesky factor. The identity $S_k=E_k^{-1}$ is also established in Appendix A.1. ### **W2, Q1. Comparison with optimized diagonal damping** Diagonal damping adds the same value to every diagonal entry of the Hessian, which suppresses compensation for low-curvature directions. Cross-group shrinkage instead scales the entries between quantization groups while leaving the within-group blocks unchanged. For a fixed damping coefficient, the attenuation depends solely on the curvature magnitude, while ReSC-Q uses uncertainty estimated from calibration data to determine how strongly cross-group correlations are used for compensation. Let $\widehat{H}$ denote the calibration Hessian and $D_{\mathcal{G}}$ its block-diagonal part (the within-group blocks, with cross-group entries set to zero). With $\rho$ (the fraction of cross-group curvature retained), shrinkage gives $$\widetilde H_\rho =\rho\widehat{H}+(1-\rho)D_{\mathcal{G}}, \qquad \frac{\widetilde H_\rho}{\rho} =\widehat{H}+\frac{1-\rho}{\rho}D_{\mathcal{G}}, \quad 0 < \rho\leq1.$$ An overall positive scaling does not change the compensation update. Thus, shrinkage is equivalent for compensation to adding a multiple of $D_{\mathcal{G}}$, while damping adds a multiple of $I$ (the identity matrix). ReSC-Q estimates $\rho$ from calibration statistics. We compare these choices with the within-group quantizer, fitting weights, and blockwise compensation procedure fixed. Only the Hessian used for compensation changes. Damping adds $\gamma\operatorname{mean}(\operatorname{diag}\widehat{H})I$, where $\gamma$ controls the added value relative to the mean Hessian diagonal. Table R3 reports the five damping coefficients evaluated at every calibration size. | n | Metric | γ=0.01 | γ=0.03 | γ=0.1 | γ=0.3 | γ=1 | **ReSC-Q** | | :-- | :-- | --: | --: | --: | --: | --: | --: | | **8** | PPL | 14.97 | 13.61 | 13.36 | **12.65** | 13.07 | 13.28 | | | Acc. | 52.60 | 53.88 | 56.95 | 57.29 | 57.46 | **58.00** | | **128** | PPL | **11.59** | 11.65 | **11.59** | 12.24 | 12.70 | 11.77 | | | Acc. | 56.24 | 57.23 | 57.69 | 58.09 | 58.66 | **58.72** | | **1024** | PPL | 11.46 | 11.52 | 11.65 | 11.87 | 12.73 | **11.31** | | | Acc. | 57.23 | 57.39 | 58.32 | 58.46 | 58.31 | **58.57** | *Table R3. Controlled damping sweep on Llama-3.1-8B-Instruct. All runs use nominal 2-bit affine quantization, group size 32. Damping columns use $\rho=1$; ReSC-Q uses estimated shrinkage with $\gamma=0.01$.*

ReSC-Q gives the highest accuracy at every calibration size and the lowest PPL at n=1024 as shown in Table R3. The γ\gamma=0.01 column is the direct control, since ReSC-Q uses the same γ\gamma=0.01 (GPTQ’s default): adding shrinkage improves accuracy by 5.4, 2.5 and 1.3 points at n=8, 128 and 1024. A tuned damping coefficient can match or beat ReSC-Q on PPL at small budgets, but the best γ\gamma shifts with n, and no single value is competitive across all three. This is the expected consequence of the distinction above: damping is a fixed conditioning term, whereas ρ^\hat \rho is estimated from the sampling variance of the cross-group entries and adapts to the calibration budget (Table 3).

The two are complementary and ReSC-Q already uses both (γ\gamma=0.01). For a fixed current-group error eke_k and shrinkage coefficient ρ\rho, damping adds a penalty λ∥δRk∥22\lambda\lVert \boldsymbol{\delta}_{\mathcal{R}_k}\rVert _2^2 on the compensation magnitude, where λ=γmean⁡(diag⁡H^)\lambda=\gamma\operatorname{mean}(\operatorname{diag}\widehat{H}). The resulting update is

=\rho\\,e_kB_k\left(C_{k,\rho}+\lambda I\right)^{-1},$$ where $B_k$ is the current-to-future curvature block and $C_{k,\rho}$ is the future-coordinate block after shrinkage. Along an eigenvector of $C_{k,\rho}$ with eigenvalue $c > 0$, damping scales the update component by $c/(c+\lambda)$ relative to shrinkage alone. Shrinkage thus controls how much cross-group correlation is used for compensation, while damping bounds the update magnitude in low-curvature directions. We will add Table R3 and this comparison to the appendix. ### **W3, Q3. Effective bit rate and the scope of the accuracy comparison** Our controlled affine comparisons use the same weight-code precision, group size, and FP16 (16-bit floating-point) scale/offset storage to isolate the effect of the quantization procedure. Including these two parameters per group, the effective rate is $b + 32 / g$, where $b$ is the code precision and $g$ is the group size. Our nominal 2-, 3-, and 4-bit settings therefore use 3.00, 3.50, and 4.25 bits per quantized weight, respectively. We will state these effective rates explicitly alongside the code precision. ReSC-Q is not tied to this affine storage format. Table R1 applies its shrinkage and residual-aware weighting to QuIP# and QTIP while retaining their native representations. Within each pair, the baseline and ReSC-Q combination use the same storage format and approximately 2.0004 bits per quantized weight. These comparisons evaluate the effect of ReSC-Q without increasing the storage budget. Larger groups (Appendix C.8) or lower-precision scales and offsets are also possible ways to reduce affine metadata overhead. These choices change the storage–accuracy trade-off and can be evaluated alongside ReSC-Q. Here, we keep the representation fixed within each controlled comparison and will make the corresponding format and effective bit rate explicit in our performance claims. ### **W4-1, W4-4. Off-diagonal reliability and SNR** The sensitivity of covariance and curvature estimates to sampling noise has been widely studied (Reference Idx 21-24 in our paper). We examine this issue in quantization using entry-wise SNR, which measures the magnitude of each Hessian entry relative to its sampling variation. In Fig. 1(a), off-diagonal entries show lower SNR than diagonal entries across sampled calibration sets. Appendix C.12 extends this comparison across models, layers, and projection modules. Here, the signal is the population value of a Hessian entry, while the noise is the variation introduced by sampling calibration data. Let $h$ (the population value) and $\hat h=h+\epsilon$ (its estimate) follow the zero-mean noise model, where $\epsilon$ is the estimation noise and $v=\mathbb{E}[\epsilon^2]$ is its variance. Since $\mathbb{E}[\epsilon]=0$, we have $\mathbb{E}[\hat h]=h$ and $\mathrm{Std}(\hat h)=\sqrt v$. Thus, the absolute mean measures the signal magnitude, and the standard deviation measures the sampling uncertainty. This ratio has a direct connection to shrinkage. Omitting the additional regularization term for this illustration, the optimal coefficient $\rho^\star$ (the fraction of the estimated entry retained) is $$\rho^\star = \arg\min_{0\leq\rho\leq1} \mathbb{E}[(\rho\hat h-h)^2] = \frac{h^2}{h^2+v} = \frac{\mathrm{SNR}^2}{1+\mathrm{SNR}^2}, \qquad \mathrm{SNR}=\frac{|h|}{\sqrt v}.$$ A lower SNR therefore leads to stronger shrinkage in this single-entry model: less of the estimated correlation is retained when its sampling variation is large relative to its signal. ReSC-Q estimates these quantities from calibration data and averages the entry-wise coefficients, as described in Appendix A.2. Fig. 1(b,c) and the additional compensation experiments measure how using these estimates affects reconstruction loss.
Reviewer_E8rWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose ReSC-Q, a weight-only PTQ method that does cross-group shrinkage and within-group residual-aware weighting. Empirically, ReSC-Q outperforms the best affine PTQ baselines by 1.09 pp in 2-bit average zero-shot accuracy across seven models (Tables 1/2).

Critical strengths
  1. Composability While ReSC-Q is useful on its own, it can be composed with structured quantizers like QuIP# and QTIP as shown by Table 11.
Critical weaknesses
  1. Unclear benefit with many calibration samples Table 5 shows gains with up to 1024 calibration sequences, but GPTQ and DASH-Q are not strong baselines. Table 11 evaluates QuIP# and QTIP only up to 128, and the improvement over QTIP is already negligible at 128. I understand that ReSC-Q is mostly relevant in cases where calibration data is limited (around 128), but the most obvious fix is to just get more calibration sequences. For those don't have the original data used to train the models, they can just use inference samples, with should be abundant.
  2. Limited novelty ReSC-Q "apply a practical group-wise affine quantization procedure previously used in DASH-Q" (i.e., iterative weighted least squares), and QuIP# already proposes BlockLDLQ. The main methodological innovation appears to be the estimated cross-group shrinkage coefficient and the row-sum importance upper bound.
Questions for authors
  1. Where does the limited calibration data occur in practice? Without a good justification, it sounds like the authors are solving a problem that doesn't exist.
  2. How would Table 11 look like for n=1024n = 1024?
  3. [Optional; feel free to skip if you don't have time] An established/standard way to regularize unreliably Hessian is to bump the diagonal by adding identity. In fact, GPTQ already has this dampening. How would that work as a baseline compared to your cross-group shrinkage approach? To be specific, I mean tuned diagonal damping with the within-group quantizer and blockwise compensation held fixed.
6 comments

We thank the reviewer for the comments. We have extended the structured-quantizer evaluation to 1,024 calibration sequences and compared shrinkage with tuned damping under the same quantizer and compensation procedure.

W1-1. Role of GPTQ and DASH-Q in the calibration-size study

Figure 3 and Table 5 examine how the components of ReSC-Q contribute to performance across calibration sizes. GPTQ provides a conventional Hessian-based error-compensation reference, while DASH-Q provides the diagonal-weighted affine fitting procedure on which our default quantizer builds. To isolate the effect of cross-group shrinkage, we compare ReSC-Q with its ρ=1\rho=1 variant (no cross-group shrinkage), keeping the residual-aware quantizer and blockwise compensation unchanged. Table 9 separately examines the effect of within-group weighting while retaining the same cross-group shrinkage and blockwise compensation. Together, these controlled comparisons show how shrinkage and within-group weighting each affect quantization performance and calibration sensitivity. Table 14 complements this component-level analysis by evaluating their integration with QuIP# and QTIP. The new large-calibration results are reported in Table R2 in response to Q2.

W1-2, Q1. When is limited calibration relevant?

Calibration data cannot be expected to fully represent every domain that a model may encounter after deployment. Our goal is therefore to make effective use of the available calibration data while accounting for uncertainty in the Hessian estimates derived from it. ReSC-Q uses the uncertainty observed in these statistics to control how strongly cross-group correlations are used for error compensation. We vary the calibration size as an easy way to examine this uncertainty in a controlled setting. Small calibration sets make sampling variation more pronounced, allowing us to observe its effect on compensation more clearly. However, as Reviewer EJqM also notes, data quantity and distribution coverage are separate issues. Increasing the number of samples from one source does not ensure that they represent future inputs from other sources. In Fig. 1(b,c), corresponding curves directly compare compensation estimated from different calibration sources using the same evaluation data and no-compensation baseline. Table R1 further examines these effects by evaluating compensation when calibration and evaluation samples come from the same source or from different sources.

Evaluation settingCalibration -> referenceCompensationn=8n=8n=32n=32n=128n=128n=512n=512n=1024n=1024
Same populationWikiText-2 -> WikiText-2No shrinkage0.9040.7380.6810.6610.658
ReSC-Q0.7860.7330.6820.6620.655
Same populationC4 -> C4No shrinkage0.9850.8140.7590.7390.736
ReSC-Q0.8460.8010.7600.7360.735
Different sourceWikiText-2 -> C4No shrinkage1.1410.9660.9110.8930.889
ReSC-Q0.8960.8710.8590.8590.861
Different sourceC4 -> WikiText-2No shrinkage1.0540.8830.8260.8060.803
ReSC-Q0.8760.8410.8170.8030.800

Table R1. Reconstruction loss normalized to the no-compensation baseline (lower is better; values below 1 indicate improvement). Results average over three layers and two projection modules. Each sample contains 2,048 tokens.

Table R1 shows that ReSC-Q reduces reconstruction loss by a large amount with small calibration sets for all cases. With larger calibration sets, its loss becomes comparable to that of unshrunk compensation within the same population, while the cross-source gains persist across all tested calibration sizes. More calibration samples can reduce sampling uncertainty, but do not necessarily remove the need to regularize compensation across data sources.

We will revise the motivation and figure descriptions to clarify this broader objective and the distinction of calibration size and distribution coverage problem.

W2. Contribution beyond BlockLDLQ and weighted affine fitting

We thank the reviewer for the careful reading. We agree that ReSC-Q retains BlockLDLQ-style blockwise compensation and DASH-Q’s affine solver, as stated in Sections 5.1 and 5.4. We respectfully disagree that the contribution is limited.

  1. The two new components (cross-group Hessian shrinkage and the row-sum residual bound djd_j) are not small additions; without them, the inherited machinery fails at 2-bit. Hessian compensation with an unshrunk Hessian is what GPTQ and QuIP do, and both collapse on Llama-3.1-8B at 2-bit (Table 1). DASH-Q avoids this collapse by discarding cross-group curvature entirely, but at a cost: its perplexity is substantially worse, its accuracy does not improve with more calibration data (Table 5), and it degrades quickly at larger group sizes (Table 11). ReSC-Q reaches 58.83% / 11.76 PPL (Table 1, 2-bit), and each addition is necessary:
  • Removing only the shrinkage (ReSC-Q$_{\rho=1}$, Figure3, Table 5) consistently achieves lower zero-shot accuracy. Table R1 above shows that ReSC-Q’s shrinkage is better at every cross-source budget while matching the same-population case.
  • Replacing the row-sum weight djd_{j} by DASH-Q’s diagonal hessian, empirical Fisher, or inverse-Hessian weights (Tables 6–7) is worse or inconsistent, and they do not upper-bound the residual (Appendix A.3).
  1. The conceptual contribution is the finding and the decomposition behind these components. Fig. 1(b,c) and Figs. 6–7 show that Hessian compensation can increase reconstruction loss on unseen data, a failure of the compensation mechanism itself that prior work had not isolated. Eq. (6) separates the compensation term from the post-compensation residual; BlockLDLQ uses the residual only for a theoretical bound. ReSC-Q is, to our knowledge, the first to regularize the former for estimation uncertainty and minimize a bound on the latter when fitting the current group, and the two must be designed jointly (Eq. 67–68).
  2. Reusing existing solvers is deliberate: it is what makes the result general. The same weights and shrinkage transfer unchanged to QuIP#’s lattice search and QTIP’s trellis search (Table R2 in our response to Q2 below, extending Table 11 to n=1024). This compatibility with existing methods opens further optimization opportunities, such as rotation-based preprocessing as demonstrated in Appendix C.10, or mixed-precision quantization (using different bit-widths across groups or layers).

Q2. QuIP# and QTIP with 1,024 calibration sequences

We extended Table 11 to 1,024 sequences and include the smaller-budget results to show the trend in Table R2. Here, nn is the number of calibration sequences, each containing 2,048 tokens. PPL (perplexity) is lower-is-better; Acc. is average accuracy across the eight zero-shot tasks.

Methodn=8n=8: PPLAcc.n=128n=128: PPLAcc.n=1024n=1024: PPLAcc.
QuIP#14.2353.5111.5557.9611.4157.39
QuIP# + ReSC-Q12.2757.5511.1959.0911.0158.42
QTIP11.0157.829.8261.819.8561.15
QTIP + ReSC-Q10.4660.639.8561.859.9361.54

Table R2. Llama-3.1-8B-Instruct, 2-bit weights. Each combination retains its baseline's native representation.

For QuIP#, ReSC-Q improves both metrics across all three calibration sizes. At 1,024 sequences, PPL decreases by 0.40 and average zero-shot accuracy increases by 1.03 percentage points. QuIP# + ReSC-Q at 128 sequences also achieves lower PPL and higher accuracy than QuIP# alone at 1,024 sequences. For QTIP, the improvement is larger at eight sequences, while the results remain close at 128 and 1,024. At 1,024 sequences, accuracy increases by 0.39 percentage points while PPL increases by 0.08. Thus, the additional benefit remains clear for QuIP# at the larger calibration budget, whereas it is small for QTIP.

Q3. Comparison with diagonal damping

Diagonal damping adds the same value to every diagonal entry of the Hessian, which basically suppresses compensation for low-curvature directions. Cross-group shrinkage instead scales the entries between quantization groups while leaving the within-group blocks unchanged. For a fixed damping coefficient, the attenuation depends solely on the curvature magnitude, while ReSC-Q uses uncertainty estimated from calibration data to determine how strongly cross-group correlations are used for compensation. Let H^\widehat H denote the calibration Hessian and DGD_{\mathcal G} its block-diagonal part (the within-group blocks, with cross-group entries set to zero). With ρ\rho (the fraction of cross-group curvature retained), shrinkage gives

=\rho\widehat H+(1-\rho)D_{\mathcal G}, \qquad \frac{\widetilde H_\rho}{\rho} =\widehat H+\frac{1-\rho}{\rho}D_{\mathcal G}, \quad 0<\rho\leq1.$$ An overall positive scaling does not change the compensation update. Thus, shrinkage is equivalent for compensation to adding a multiple of $D_{\mathcal G}$, while damping adds a multiple of $I$ (the identity matrix). ReSC-Q estimates $\rho$ from calibration statistics. We compare these choices with the within-group quantizer, fitting weights, and blockwise compensation procedure fixed. Only the Hessian used for compensation changes. Damping adds $\gamma\operatorname{mean}(\operatorname{diag}\widehat H)I$, where $\gamma$ controls the added value relative to the mean Hessian diagonal. Table R3 reports the five damping coefficients evaluated at every calibration size. | $n$ | Metric | $\gamma=0.01$ | $\gamma=0.03$ | $\gamma=0.1$ | $\gamma=0.3$ | $\gamma=1$ | ours | | :-- | :-- | --: | --: | --: | --: | --: | --: | | 8 | PPL | 14.97 | 13.61 | 13.36 | **12.65** | 13.07 | 13.28 | | 8 | Acc. | 52.60 | 53.88 | 56.95 | 57.29 | 57.46 | **58.00** | | 128 | PPL | **11.59** | 11.65 | **11.59** | 12.24 | 12.70 | 11.77 | | 128 | Acc. | 56.24 | 57.23 | 57.69 | 58.09 | 58.66 | **58.72** | | 1024 | PPL | 11.46 | 11.52 | 11.65 | 11.87 | 12.73 | **11.31** | | 1024 | Acc. | 57.23 | 57.39 | 58.32 | 58.46 | 58.31 | **58.57** | *Table R3. Controlled damping sweep on Llama-3.1-8B-Instruct. All runs use nominal 2-bit affine quantization, group size 32. Damping columns use $\rho=1$; ReSC-Q uses estimated shrinkage with $\gamma=0.01$.* ReSC-Q gives the highest accuracy at every calibration size and the lowest PPL at n=1024 as shown in Table R3. The $\gamma$=0.01 column is the direct control, since ReSC-Q uses the same $\gamma$=0.01 (GPTQ’s default): adding shrinkage improves accuracy by 5.4, 2.5 and 1.3 points at n=8, 128 and 1024. A tuned damping coefficient can match or beat ReSC-Q on PPL at small budgets, but the best $\gamma$ shifts with n, and no single value is competitive across all three. This is the expected consequence of the distinction above: damping is a fixed conditioning term, whereas $\hat \rho$ is estimated from the sampling variance of the cross-group entries and adapts to the calibration budget (Table 3). The two are complementary and ReSC-Q already uses both ($\gamma=0.01$). For a fixed current-group error $e_k$ and shrinkage coefficient $\rho$, damping adds a penalty $\lambda\lVert\boldsymbol{\delta}_{\mathcal{R}_k}\rVert_2^2$ on the compensation magnitude, where $\lambda=\gamma\operatorname{mean}(\operatorname{diag}\widehat{H})$. The resulting update is $$\boldsymbol{\delta}_{\mathcal{R}_k}^{\star} =\rho\\,e_k B_k\left(C_{k,\rho}+\lambda I\right)^{-1}.$$ Here, $B_k$ is the current-to-future curvature block and $C_{k,\rho}$ is the future-coordinate block after shrinkage and before damping. Along an eigenvector of $C_{k,\rho}$ with eigenvalue $c>0$, damping scales the update component by $c/(c+\lambda)$ relative to shrinkage alone. Shrinkage thus controls how much cross-group correlation is used for compensation, while damping bounds the update magnitude in low-curvature directions. We will add Table R3 and this comparison to the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

We thank all three reviewers for their careful reading and constructive suggestions. We have revised the manuscript to clarify ReSC-Q’s contributions and report expanded experiments addressing the main concerns. Revised and added text is marked in blue in the uploaded PDF. We summarize the key revisions below. All section, figure, and table references refer to the revised manuscript.

  1. Calibration uncertainty and distribution coverage. We revised the motivation to distinguish sampling uncertainty from distribution coverage. Increasing the number of samples from one source does not ensure that its statistics represent other inputs. To examine how these two factors affect compensation, Appendix C.4 evaluates both same-source and cross-source settings. Table 6 reports compensation losses, while Table 7 examines variance, squared bias, and Hessian-estimation MSE. These experiments show compensation gains even without changing the data source, and show that better compensation need not correspond to lower aggregate Hessian-estimation MSE.

  2. Clarified contribution and the general quantization framework. We revised the abstract and introduction to clarify ReSC-Q as a general quantization framework combining cross-group shrinkage with a conservative residual-aware reconstruction objective. Sections 5.1–5.3 derive the compensation–residual decomposition, cross-group shrinkage, and residual upper bound. The bound holds for any current-group error vector and does not require an affine representation. Section 5.4 presents group-wise affine quantization as our default implementation, using DASH-Q’s alternating solver with the derived fitting weights, and explains application to other quantizers. It also states explicitly which components are adopted from prior work and which are new. Appendix C.11 extends the QuIP# and QTIP comparisons to larger calibration sequences while retaining their native representations and decoding procedures (Table 14).

  3. Controlled comparison with diagonal damping. Section 5.2 explains how damping and cross-group shrinkage modify compensation. Diagonal damping suppresses updates strongly along low-curvature directions, whereas cross-group shrinkage controls the use of inter-group correlations based on uncertainty estimated from calibration statistics. Appendix C.5 reports a controlled sweep of damping strengths, keeping the within-group quantizer, fitting weights, and blockwise compensation procedure fixed at each calibration setting (Table 8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.