- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces ReSC-Q which is a post-training quantization method. It is designed to address sub-optimal error compensation in ultra low-bit quantization. It splits the reconstruction objective into two groups: cross-group compensation and within-group quantization, applying shrinkage to manage curvature uncertainty and residual-aware weighting to handle remaining loss. Evaluation results across seven LLMs at a very low bit quantization, 2-bit quantization, ReSC-Q improves average zero-shot accuracy over the strong baselines, reducing the mean accuracy gap to full precision models by a large ratio (23.2%).
Critical strengths
1: ReSC-Q demonstrates strong effectiveness in extremely low-bit post-training quantization. According to seven evaluated LLMs at 2-bit precision. it improves zero-shot accuracy by 1.09% in average. In addition, it bring benefits to success rate to massive architectures. 2: This paper provides a comprehensive analysis to show that standard hessian-based error compensation suffers from noisy off-diagonal estimates especially, when calibration data is limited. ReSC-Q proposed in this work addresses this bottle neck by introducing a dual-pronged mechanism that apply corss-group shrinkage to suppress unreliable off-diagonal curvature. 3: The proposed residual-aware quantization objective is versatile, it can be seamlessly integrated into various quantization frameworks, ranging from group-wise affine quantization and rotation-based methods to advanced structured-code representations such as E8P lattices and trellis-coded quantizations.
Critical weaknesses
1:The formula used to adjust cross-group values relies on a basic assumption that errors are balanced and average out to zero. Due to this, it can miss hidden biases or differences between the sample data used during setup and the real-world data seen later. Therefore, the method might misjudge how reliable those connections actually are, making the adjustments either too weak or too strong. 2:Similar to other methods, ReSC-Q focuses on making each individual layer match the original model as closely as possible, rather than directly trying to improve final task performance. Hence, getting better layer-matching scores doesn't always guarantee that the model will actually perform better on real-world tasks.
Questions for authors
1:The shrinkage coefficient is based on a simplified noise model that assumes estimation errors have a zero mean. However, this assumption may not reflect differences between the calibration data and real-world deployment data. How much does the estimated coefficient change when the calibration data differs from the actual deployment data? Could using a noise model that adapts to these differences or accounts for systematic errors make the method more robust? 2:ReSC-Q optimizes the reconstruction quality of each layer rather than directly optimizing the performance of the final task. Therefore, better reconstruction does not always lead to better generation results. Could this method be improved by adding a lightweight penalty that considers the target task, or by combining reconstruction quality with task performance in the optimization objective, so that quantization better preserves the final task performance?
5 comments
Gentle reminder: author–reviewer discussion ending in two days
Dear Reviewer q2NB,
Thank you again for your review and thoughtful questions. With two days remaining in the discussion period, we would like to briefly highlight the explanations and additional evidence provided in our responses and revised manuscript.
Regarding the zero-mean noise assumption, we clarified that it describes sampling variation around the calibration-population Hessian, while systematic calibration-to-deployment differences remain a limitation. Our response reports how the estimated shrinkage coefficient changes with calibration source and sample count. Appendix C.4 additionally examines same-source and cross-source compensation and the associated variance–bias trade-off (Tables 6–7).
Regarding reconstruction versus task performance, our response discusses your task-aware suggestion through a controlled empirical-Fisher weighting comparison (Table 10), with the representation and other pipeline components held fixed. Section 7 also explicitly acknowledges that improvements in layer reconstruction need not translate monotonically to downstream accuracy.
We would appreciate your feedback on whether these explanations and experiments address your concerns. If they do, we would be grateful if you could consider updating your assessment and rating accordingly. If any questions remain, we would be happy to address them within the remaining discussion period.
Thank you again for your time and consideration.
Best regards, The authors
Follow-up before the discussion deadline
Dear Reviewer q2NB
With one day remaining in the discussion period, we would appreciate any brief feedback on whether our responses and revised manuscript address your main concerns. If any questions remain, we would be happy to clarify them before the discussion closes.
Thank you again for your time and consideration.
Best regards, The authors
We thank the reviewer for the questions about the noise assumption and the relationship between reconstruction and final task performance. We clarify the scope of the estimator, report how its coefficient changes with calibration data, and connect the task-aware suggestion to our existing sensitivity-weighting experiment.
W1, Q1-1. Zero-mean noise and systematic distribution differences
The zero-mean noise model describes sampling variation around the calibration-population Hessian. A difference between the calibration and deployment populations introduces an additional bias.
To express both effects, let h denote one deployment-population matrix entry, β the calibration-to-deployment bias, and ϵ the sampling noise with variance v. Then
\qquad \mathbb{E}[\epsilon]=0, \qquad \mathbb{E}[\epsilon^2]=v,$$ where $\hat h$ is the entry estimated from calibration data. For a fixed shrinkage coefficient $\rho$, the expected squared error, omitting the additional regularization term for clarity, is $$\mathbb{E}[(\rho\hat h-h)^2] = \bigl((\rho-1)h+\rho\beta\bigr)^2+\rho^2v.$$ This expression separates sampling variance from the bias introduced by shrinkage and distribution differences. A bias-aware extension could use this objective, but would require information about $\beta$. Estimating that difference requires representative deployment samples or additional assumptions about the distribution shift. ReSC-Q uses the calibration statistics available during quantization to estimate the sampling uncertainty and select its shrinkage coefficient. The same-population and cross-source experiments in our response to Reviewer 2 also show how the coefficient affects compensation: gains occur with small samples from the same population and also when compensation is evaluated against another source. The zero-mean assumption is used to derive the shrinkage coefficient. The within-group residual upper bound follows from matrix inequalities for the chosen curvature matrix and does not depend on this noise assumption. ### **Q1-2. How much does the coefficient change with calibration data?** We measured $\hat\rho$ (the estimated fraction of cross-group curvature retained for compensation) using the original model's activations while varying the calibration source and sample count. Here, $n$ is the number of calibration sequences. | Calibration source | n=8 | n=32 | n=128 | n=512 | n=1024 | | :-- | --: | --: | --: | --: | --: | | WikiText-2 | 0.342 | 0.486 | 0.665 | 0.800 | 0.847 | | C4 | 0.357 | 0.472 | 0.616 | 0.763 | 0.823 | *Table R5. Mean estimated coefficient across q_proj and down_proj at layers 5, 15, and 25 of Llama-3.1-8B-Instruct. Each sequence $n$ contains 2,048 tokens.* Both sources produce stronger shrinkage at small calibration sizes and retain more cross-group information as the sample count increases. At $n=128$, the mean coefficient is 0.665 for WikiText-2 and 0.616 for C4, a difference of 0.049. Differences within individual modules range from 0.033 to 0.071. The coefficient therefore responds to both calibration size and source. It is computed entirely from calibration statistics, so changing only the evaluation dataset leaves it unchanged. ### **W2, Q2. Layer reconstruction versus task-aware optimization** Our objective is to quantize a pretrained model using calibration data without additional task-specific training. We use layer reconstruction to guide quantization and measure final model performance separately on eight zero-shot tasks. A task-aware objective can be combined with this approach. Appendix C.7 examines a related way of using loss information: replacing the within-group fitting weights with empirical-Fisher sensitivity (importance derived from squared loss gradients). Within each quantization pipeline, the representation and all other components are kept fixed. Table R6 shows the relevant rows of Table 10. | Quantization pipeline | Within-group weighting | PPL | Zero-Shot Avg. | | :-- | :-- | --: | --: | | DASH-Q | Hessian diagonal | 17.17 | 56.69 | | DASH-Q | Empirical Fisher | **15.14** | **58.00** | | ReSC-Q | Residual upper-bound weights | 11.76 | **58.83** | | ReSC-Q | Empirical Fisher | **11.51** | 58.45 | *Table R6. Controlled sensitivity-weighting comparison from Appendix C.7. Llama-3.1-8B-Instruct, nominal 2-bit weights, group size 32, and 128 calibration sequences, each containing 2,048 tokens.* Fisher weighting improves both metrics for DASH-Q. For ReSC-Q, it slightly improves PPL but lowers average zero-shot accuracy. The same sensitivity signal therefore has different effects depending on the surrounding quantization and compensation procedure. This experiment incorporates loss information through the fitting weights. Adding an explicit target-task loss would be a further extension for settings where representative task data and an additional optimization budget are available.