- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies calibration-spectrum bias in binary post-training quantization of LLMs. It argues that reconstruction objectives preferentially preserve high-variance calibration directions, while some low-variance but important directions are disproportionately suppressed by quantization. The authors introduce a directional survivor rate to characterize this effect and propose DistriQuant, which uses Covariance-Balanced Alignment (CBA) to select salient masks. The method uses standard calibration data and retains the underlying binary representation. Experiments report improvements in perplexity and downstream accuracy across OPT, LLaMA, and Qwen models.
Critical strengths
-
Well-motivated investigation of an important quantization problem.
The paper provides a useful perspective on how reconstruction objectives allocate limited binary capacity across representation directions. The combination of the spectral decomposition and cross-domain diagnostics offers an interpretable explanation for why calibration reconstruction quality may not adequately reflect preservation of capabilities under distribution shift. -
Promising empirical results across multiple architectures and tasks.
The reported improvements extend beyond perplexity to commonsense reasoning, GSM8K, and MMLU. For example, average accuracy across the eight commonsense benchmarks improves over ARB-RC by 7.29, 7.30, and 11.17 percentage points on LLaMA-2-7B, LLaMA-2-13B, and LLaMA-3-8B, respectively. These results suggest that the proposed criterion can provide substantial practical benefits. -
Practical integration and useful initial ablations.
DistriQuant modifies mask selection without requiring domain-specific calibration data or additional inference-time operations. The paper also compares separate covariance normalization against shared FP covariance normalization and examines calibration distributions and projection dimensions. These experiments provide useful initial support for the design.
Critical weaknesses
-
The connection between the theoretical objective and the implemented CBA criterion remains insufficiently established.
Proposition 1 connects survivor-rate preservation to an alignment objective that normalizes both branches using the FP covariance. However, the implemented CBA criterion normalizes each branch using its own covariance, and the equivalence in Eq. (8) holds only under a restrictive covariance-matching condition. Although the authors explicitly acknowledge this surrogate design, a substantial gap remains between the theoretical motivation and the implemented objective. Table 4 further shows that the FP covariance variant performs worse than CBA, raising the question of whether the gains are primarily explained by survivor-rate preservation or by other benefits of covariance normalization.
-
Evaluation at larger model scales is missing.
The experiments cover multiple model families, but the largest evaluated model has only 13B parameters. Evaluation on larger models, such as 30B–70B models, would strengthen the evidence that the proposed allocation criterion remains effective at larger scales and help establish its practical quantization cost in those settings.
Questions for authors
-
Connection between directional preservation and downstream performance. Figure 3 provides useful evidence connecting the preservation of enriched directions to lower OOD activation reconstruction error. Have the authors also examined whether improvements in directional survivor rates correlate with downstream task accuracy across quantization configurations? Such an analysis would further support the proposed interpretation of the empirical gains.
-
The overall presentation requires substantial improvement.
Figures 1 and 2 lack explicit axis titles, and their different grouping criteria are not immediately apparent. The notation also needs more careful explanation and consistency, including the transition from calibration directions to projected-output directions , the switch between and , and the reuse of for both an eigenvector matrix and a Cholesky factor. There is also a citation-formatting issue around lines 40–41: “For example, (Liu et al., 2025) show” should use a narrative citation. A thorough revision of the figures, notation, and exposition would substantially improve readability.
Dear Reviewer DWLy,
We greatly appreciate the time and effort the Reviewer dedicated to considering our paper. Here are our responses to all concerns raised by the Reviewer.
Answer: We agree that Proposition 1 does not directly establish a guarantee for practical CBA, which employs independent whitening. The theoretical analysis serves as motivation for CBA as a practical surrogate, rather than providing a direct theoretical guarantee. We have clarified this distinction and revised the corresponding theoretical claims in the revised manuscript. To further support the practical formulation, we provide additional analysis showing a positive association between CBA loss and directional survivor-rate deviation on MMLU (Appendix B.1, Figure 4).
Answer: We extend the analysis to the final fully quantized models by grouping representation directions according to their cross-domain enrichment. As shown in Appendix B.1, Figure 5, we observe the expected negative correlation between survivor-rate deviation and MMLU accuracy across all three groups: smaller deviation (better directional preservation) corresponds to higher downstream accuracy. The strongest correlation occurs for the highest-enrichment directions (r=−0.630, compared with −0.401 and −0.425 for the middle and lowest groups, respectively). These results provide additional empirical support for the proposed mechanism.
Answer: We provide additional results on OPT-30B below, extending our evaluation beyond the 13B models in the original manuscript.
Table: Perplexity (↓) comparison between ARB-RC and our method on OPT-30B.
Our method improves perplexity across all three benchmarks, providing additional evidence of its effectiveness at the larger scale model.
Answer:
We thank the reviewer for the suggestions. We have revised the manuscript to improve clarity, readability, and consistency throughout the paper. Moreover, we have revised Figures 1–2 to clarify the axis labels and grouping criteria, clarified the distinction between calibration and projected-output directions, standardized the notation and terminologies, and corrected the citation formatting. New results are hightlighted in blue color for better observation.