acceptodds
Under review as a conference paper at ICLR 2027

DISTRIQUANT: BEYOND RECONSTRUCTION FOR ROBUST BINARY LLMS

37 views

Abstract

Post-training quantization (PTQ) for large language models (LLMs) is commonly formulated as a reconstruction problem, where quantized weights are optimized to reproduce full-precision behavior on a small calibration set. We show that, in the 1-bit regime, reconstruction plays a more consequential role: it implicitly determines which representation directions survive quantization. Since reconstruction error places greater emphasis on directions with high variance on the calibration distribution, binary PTQ preferentially preserves calibration-dominant structure while weak directions receive substantially less protection. Crucially, directions that are weak on the calibration distribution are not necessarily unimportant elsewhere. We find that many become relatively more prominent on unseen domains, yet are disproportionately suppressed after binarization. To quantify this effect, we introduce a directional survivor rate that measures how much of each full-precision representation direction remains after quantization. Motivated by this finding, we propose DistriQuant, a spectrum-balanced approach to salient-mask selection for binary PTQ. DistriQuant uses Covariance-Balanced Alignment to remove the variance imbalance among calibration directions before comparing full-precision and quantized representations, preventing dominant calibration directions from overwhelming the allocation of limited binary capacity. The method requires only the standard calibration set, leaves the underlying binary representation unchanged, and introduces no additional inference-time computation. Extensive experiments across different architectures and datasets demonstrate the effectiveness of our proposed method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
binary quantization, post training quantization, large language model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DWLyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies calibration-spectrum bias in binary post-training quantization of LLMs. It argues that reconstruction objectives preferentially preserve high-variance calibration directions, while some low-variance but important directions are disproportionately suppressed by quantization. The authors introduce a directional survivor rate to characterize this effect and propose DistriQuant, which uses Covariance-Balanced Alignment (CBA) to select salient masks. The method uses standard calibration data and retains the underlying binary representation. Experiments report improvements in perplexity and downstream accuracy across OPT, LLaMA, and Qwen models.

Critical strengths
  1. Well-motivated investigation of an important quantization problem.
    The paper provides a useful perspective on how reconstruction objectives allocate limited binary capacity across representation directions. The combination of the spectral decomposition and cross-domain diagnostics offers an interpretable explanation for why calibration reconstruction quality may not adequately reflect preservation of capabilities under distribution shift.

  2. Promising empirical results across multiple architectures and tasks.
    The reported improvements extend beyond perplexity to commonsense reasoning, GSM8K, and MMLU. For example, average accuracy across the eight commonsense benchmarks improves over ARB-RC by 7.29, 7.30, and 11.17 percentage points on LLaMA-2-7B, LLaMA-2-13B, and LLaMA-3-8B, respectively. These results suggest that the proposed criterion can provide substantial practical benefits.

  3. Practical integration and useful initial ablations.
    DistriQuant modifies mask selection without requiring domain-specific calibration data or additional inference-time operations. The paper also compares separate covariance normalization against shared FP covariance normalization and examines calibration distributions and projection dimensions. These experiments provide useful initial support for the design.

Critical weaknesses
  1. The connection between the theoretical objective and the implemented CBA criterion remains insufficiently established.

    Proposition 1 connects survivor-rate preservation to an alignment objective that normalizes both branches using the FP covariance. However, the implemented CBA criterion normalizes each branch using its own covariance, and the equivalence in Eq. (8) holds only under a restrictive covariance-matching condition. Although the authors explicitly acknowledge this surrogate design, a substantial gap remains between the theoretical motivation and the implemented objective. Table 4 further shows that the FP covariance variant performs worse than CBA, raising the question of whether the gains are primarily explained by survivor-rate preservation or by other benefits of covariance normalization.

  2. Evaluation at larger model scales is missing.

    The experiments cover multiple model families, but the largest evaluated model has only 13B parameters. Evaluation on larger models, such as 30B–70B models, would strengthen the evidence that the proposed allocation criterion remains effective at larger scales and help establish its practical quantization cost in those settings.

Questions for authors
  1. Connection between directional preservation and downstream performance. Figure 3 provides useful evidence connecting the preservation of enriched directions to lower OOD activation reconstruction error. Have the authors also examined whether improvements in directional survivor rates correlate with downstream task accuracy across quantization configurations? Such an analysis would further support the proposed interpretation of the empirical gains.

  2. The overall presentation requires substantial improvement.

    Figures 1 and 2 lack explicit axis titles, and their different grouping criteria are not immediately apparent. The notation also needs more careful explanation and consistency, including the transition from calibration directions viv_i to projected-output directions uiu_i, the switch between CDC_D and CDFPC_D^{\mathrm{FP}}, and the reuse of UU for both an eigenvector matrix and a Cholesky factor. There is also a citation-formatting issue around lines 40–41: “For example, (Liu et al., 2025) show” should use a narrative citation. A thorough revision of the figures, notation, and exposition would substantially improve readability.

1 comment

Dear Reviewer DWLy,

We greatly appreciate the time and effort the Reviewer dedicated to considering our paper. Here are our responses to all concerns raised by the Reviewer.

W1: The connection between the theoretical objective and the implemented CBA criterion remains insufficiently established. Proposition 1 uses unified FP covariance normalization, whereas practical CBA employs independent whitening.

Answer: We agree that Proposition 1 does not directly establish a guarantee for practical CBA, which employs independent whitening. The theoretical analysis serves as motivation for CBA as a practical surrogate, rather than providing a direct theoretical guarantee. We have clarified this distinction and revised the corresponding theoretical claims in the revised manuscript. To further support the practical formulation, we provide additional analysis showing a positive association between CBA loss and directional survivor-rate deviation on MMLU (Appendix B.1, Figure 4).

Q1: Connection between directional preservation and downstream performance. Figure 3 provides useful evidence connecting the preservation of enriched directions to lower OOD activation reconstruction error. Have the authors also examined whether improvements in directional survivor rates correlate with downstream task accuracy across quantization configurations? Such an analysis would further support the proposed interpretation of the empirical gains.

Answer: We extend the analysis to the final fully quantized models by grouping representation directions according to their cross-domain enrichment. As shown in Appendix B.1, Figure 5, we observe the expected negative correlation between survivor-rate deviation and MMLU accuracy across all three groups: smaller deviation (better directional preservation) corresponds to higher downstream accuracy. The strongest correlation occurs for the highest-enrichment directions (r=−0.630r=-0.630, compared with −0.401-0.401 and −0.425-0.425 for the middle and lowest groups, respectively). These results provide additional empirical support for the proposed mechanism.

W2: Evaluation at larger model scales is missing. Results on 30B--70B models would strengthen the evidence for scalability and practical quantization costs.

Answer: We provide additional results on OPT-30B below, extending our evaluation beyond the 13B models in the original manuscript.

Table: Perplexity (↓\downarrow) comparison between ARB-RC and our method on OPT-30B.

MethodC4 ↓\downarrowWikiText-2 ↓\downarrowPTB ↓\downarrow
ARB-RC13.3611.1116.91
Ours12.7510.5415.82

Our method improves perplexity across all three benchmarks, providing additional evidence of its effectiveness at the larger scale model.

Q2: Figures 1 and 2 lack explicit axis titles, and their different grouping criteria are not immediately apparent. The notation also needs more careful explanation and consistency, including the transition from calibration directions to projected-output directions , the switch between and , and the reuse of for both an eigenvector matrix and a Cholesky factor. There is also a citation-formatting issue around lines 40–41: “For example, (Liu et al., 2025) show” should use a narrative citation. A thorough revision of the figures, notation, and exposition would substantially improve readability.

Answer:

We thank the reviewer for the suggestions. We have revised the manuscript to improve clarity, readability, and consistency throughout the paper. Moreover, we have revised Figures 1–2 to clarify the axis labels and grouping criteria, clarified the distinction between calibration and projected-output directions, standardized the notation and terminologies, and corrected the citation formatting. New results are hightlighted in blue color for better observation.

Reviewer_FudjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows that reconstruction-based binary PTQ preferentially preserves high-variance directions from the calibration distribution, while suppressing weaker directions that may become important under domain shifts. It introduces the directional survivor rate to measure this effect and proposes DistriQuant, which uses Covariance-Balanced Alignment for spectrum-balanced mask selection. DistriQuant preserves the original binary representation without additional inference-time computation and improves perplexity and downstream accuracy across OPT, LLaMA, and Qwen models.

Critical strengths
  1. The paper provides a clear diagnosis of the spectral bias induced by reconstruction-based binary PTQ: high-variance directions on the calibration distribution receive greater protection, whereas weaker directions that may become relatively important under distribution shift are more likely to be suppressed.
  2. Covariance-Balanced Alignment (CBA) translates this observation into a practical criterion for salient-mask selection while preserving the original binary representation and introducing no additional inference-time computation.
  3. The experimental evaluation is comprehensive, covering OPT, LLaMA-2/3, and multiple generations of Qwen. The paper reports perplexity, commonsense reasoning, GSM8K, and MMLU results. Its analyses of cross-domain-enriched directions, covariance-normalization variants, and calibration distributions also provide useful support for the proposed motivation.
Critical weaknesses
  1. The theoretical analysis does not fully align with the practically adopted CBA objective: Proposition 1 controls the representation error after unified normalization using FP covariance, whereas the practical CBA performs whitening using the respective covariances of FP and quantized representations. Independent whitening eliminates the scale and spectral differences of the two representations separately. Therefore, a small CBA loss does not necessarily imply that the direction survival rate in Equation (6) is close to 1. For example, when quantization only scales a certain FP principal direction, the two representations may become identical after independent whitening, while the survival rates of different directions still deviate significantly from 1. The paper refers to CBA as a practical surrogate, yet sufficient conditions or error bounds are missing to clarify when this surrogate can achieve the claimed direction preservation.

  2. Experiments have not fully isolated the core mechanism of "preserving cross-domain-enriched directions": DistriQuant also yields substantial improvements on the calibration domain C4. Hence, these gains may stem from more effective general mask selection or output alignment, rather than primarily from the preservation of cross-domain-enriched directions. Figure 3(b) provides supporting evidence, but Figure 3(a) is based on a controlled model that retains some full-precision columns, which still differs from the final fully binarized model. Current results can demonstrate a correlation between the two, yet are insufficient to definitively establish the link between the proposed mechanism and the final performance gains.

  3. Several budget-matched controls for identifying the effective components of CBA are absent: The appendix compares the reconstruction baseline, unified FP covariance normalization and full CBA, which is valuable. However, simpler controls such as output alignment without whitening are still missing. With fixed candidate masks and search budgets, these controls would help identify the source of performance gains.

  4. Randomness, hyperparameter sensitivity and calibration-domain generalization have not been thoroughly validated: CBA employs random projection matrices and relies on projection dimension and whitening regularization terms, yet the paper mainly analyzes only the projection dimension. Main experiments appear to be based on a single C4 calibration sampling; results with WikiText2 calibration are reported only for OPT-1.3B. Given the large magnitude of some performance improvements, especially the perplexity changes of LLaMA-2 on PTB, fluctuations from random projections, calibration samples and evaluations still need to be reported to confirm result stability.

Questions for authors
  1. When whitening the two representations using FP and Q respectively, what additional conditions must be satisfied to minimize Equation (9) so as to control the directional survival rate defined in Equation (6)? If rigorous guarantees cannot be provided at present, can the authors explicitly weaken the relevant theoretical statements and present the relationship between CBA loss and survivor-rate deviation?
  2. Can the survivor rates for different cross-domain enrichment groups be reported directly on the final fully binarized model, with an analysis of their correlation with the performance on NuminaMath, MBPP, GSM8K or MMLU? This would help verify whether the relationship observed on partially quantized models in Figure 3(a) extends to the final model.
  3. With candidate masks, average bit-width, search iterations and other quantization settings kept identical, what are the respective gains brought by full CBA compared with the following methods: non-whitened output alignment, diagonal-only covariance normalization, correlation scoring, and unified FP covariance normalization?
  4. Can experiments with different calibration distributions be supplemented on at least one LLaMA and one Qwen model, with the mean and standard deviation across multiple calibration/projection seeds reported for several representative settings? There is no need to repeat all experiments, but both language modeling metrics and downstream task metrics should be covered.
2 comments

Dear Reviewer Fudj,

We greatly appreciate the time and effort the Reviewer dedicated to considering our paper. Here are our responses to all concerns raised by the Reviewer.

Q1 and W1: When whitening the two representations using FP and Q respectively, what additional conditions must be satisfied to minimize Equation (9) so as to control the directional survival rate defined in Equation (6)? If rigorous guarantees cannot be provided at present, can the authors explicitly weaken the relevant theoretical statements and present the relationship between CBA loss and survivor-rate deviation?

Answer: We agree that Proposition 1 does not directly establish a theoretical guarantee for practical CBA. Our intention is to use the theoretical analysis to motivate CBA as a practical surrogate for directional preservation. We have clarified this distinction and revised the theoretical claims appropriately in the revised manuscript. Furthermore, we provide additional analysis showing a positive association between CBA loss and directional survivor-rate deviation, providing empirical support for the practical surrogate (Appendix B.1, Figure 4).

Q2 and W2: Can the survivor rates for different cross-domain enrichment groups be reported directly on the final fully binarized model, with an analysis of their correlation with the performance on NuminaMath, MBPP, GSM8K or MMLU? This would help verify whether the relationship observed on partially quantized models in Figure 3(a) extends to the final model.

Answer: We extend the analysis to the final fully quantized models by grouping representation directions according to their cross-domain enrichment. As shown in Appendix B.1, Figure 5, we observe the expected negative correlation between survivor-rate deviation and MMLU accuracy across all three groups: smaller deviation (better directional preservation) corresponds to higher downstream accuracy. The strongest correlation occurs for the highest-enrichment directions (r=−0.630r=-0.630, compared with −0.401-0.401 and −0.425-0.425 for the middle and lowest groups, respectively). These results provide additional empirical support for the proposed mechanism.

Q3 and W3: With candidate masks, average bit-width, search iterations and other quantization settings kept identical, what are the respective gains brought by full CBA compared with the following methods: non-whitened output alignment, diagonal-only covariance normalization, correlation scoring, and unified FP covariance normalization?

Answer: We provide additional budget-matched ablations comparing CBA with non-whitened output alignment, diagonal-only covariance normalization, correlation scoring (CKA), and unified FP covariance normalization.

The results in Table 1 show that CBA achieves the highest average accuracy, outperforming all evaluated alternatives. Although individual alternatives perform better on certain tasks, CBA achieves the strongest overall performance.

Table 1: Comparison of mask-selection criteria under matched quantization settings.

Mask-selection criterionLAMBADAPIQAOBQAWinoGrandeARC-EARC-CHellaSwagCOPAAvg.
Correlation scoring (CKA)53.8566.8718.0056.8351.3024.8337.6467.0047.04
Non-whitened alignment47.2571.2721.0061.8862.6329.6144.2276.0051.73
Diagonal-only covariance52.7867.6321.4065.2747.8123.1244.2280.0050.28
Unified FP covariance normalization63.8970.6220.4060.9359.0927.9943.7965.0051.46
Ours (CBA)66.7670.7821.4060.8564.9430.6344.4377.0054.60

Q4 and W4: Can experiments with different calibration distributions be supplemented on at least one LLaMA and one Qwen model, with the mean and standard deviation across multiple calibration/projection seeds reported for several representative settings? There is no need to repeat all experiments, but both language modeling metrics and downstream task metrics should be covered.

Answer: Following Reviewer's suggestion, we extend the calibration-domain experiments to LLaMA-3-8B and Qwen3-8B, evaluating both C4 and WikiText2 calibration across three seeds. We provide additional results reporting the mean and standard deviation across these runs, covering language-modelling perplexity and downstream QA accuracy.

Table 2: Robustness to calibration distributions on LLaMA-3-8B and Qwen3-8B.

ModelCalibrationMethodC4 ↓WikiText2 ↓PTB ↓AvgQA ↑
LLaMA-3-8BC4ARB-RC35.65 ± 0.4127.42 ± 0.6545.23 ± 2.8446.71 ± 0.77
Ours21.77 ± 0.5416.37 ± 0.7025.46 ± 0.8856.85 ± 0.21
WikiText2ARB-RC46.71 ± 4.0819.26 ± 0.5048.18 ± 2.2642.85 ± 1.15
Ours27.52 ± 0.0812.75 ± 0.1026.66 ± 0.8954.22 ± 0.82
Qwen3-8BC4ARB-RC34.08 ± 1.9427.89 ± 2.6345.07 ± 0.7254.94 ± 1.95
Ours24.73 ± 0.3623.33 ± 0.6230.77 ± 0.6658.08 ± 0.49
WikiText2ARB-RC40.02 ± 0.6918.81 ± 0.3644.05 ± 0.8752.95 ± 0.23
Ours31.10 ± 0.8615.79 ± 0.1934.52 ± 0.8055.70 ± 0.69
Reviewer_U9G8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes how the reconstruction objective in 1-bit PTQ weights high-variance directions on the calibration distribution more heavily. Consequently, directions that are weak on calibration data may be more readily lost after quantization even when they matter on unseen domains. The authors introduce a directional survivor rate and propose DistriQuant, which uses covariance-balanced alignment (CBA) and hierarchical salient-mask selection to preserve representation directions more evenly. Experiments cover OPT, LLaMA, and Qwen, reporting improved perplexity and downstream performance over several 1-bit baselines.

Critical strengths
  1. The paper analyzes how individual representation directions survive cross-domain quantization, rather than relying only on aggregate reconstruction error, giving a useful diagnosis of calibration bias.
  2. The survivor rate makes the suppression of weak directions measurable, and CBA translates this diagnosis into salient-mask selection with a coherent motivation.
  3. Experiments across several model families and language-modeling/downstream tasks show consistent gains in the 1-bit regime, and the paper describes its experimental protocol in useful detail.
Critical weaknesses
  1. CBA aligns separately normalized full-precision and quantized representations. This is a useful surrogate, but its quantitative relationship to per-direction survival may weaken when their covariance spectra or principal directions differ substantially. The paper should directly validate surrogate-to-survivor-rate agreement across more settings.
  2. Evaluation focuses on particular calibration/test domains and a 1-bit implementation. Robustness to mask fraction, calibration-token count, calibration-domain choice, and unseen domains is unclear. Statistical significance and runtime costs relative to baselines also need clearer reporting.
Questions for authors
  1. Can you report the correlation between CBA loss and actual survivor rates, including domains with substantial covariance-spectrum shifts?
  2. How do mask fraction and calibration-token count affect performance? Can you report variance across seeds and comparisons under matched budgets?
2 comments

Dear Reviewer U9G8,

We greatly appreciate the time and effort the Reviewer dedicated to considering our paper. Here are our responses to all concerns raised by the Reviewer.

W1 and Q1: CBA aligns separately normalized full-precision and quantized representations. This is a useful surrogate, but its quantitative relationship to per-direction survival may weaken when their covariance spectra or principal directions differ substantially. The paper should directly validate surrogate-to-survivor-rate agreement across more settings. Can you report the correlation between CBA loss and actual survivor rates, including domains with substantial covariance-spectrum shifts?

Answer: We provide additional analysis examining the relationship between CBA loss and directional survivor-rate deviation on the final quantized models. As shown in Appendix B.1, Figure 4, we observe a positive association between CBA loss and survivor-rate deviation on MMLU, providing additional empirical support for CBA as a practical surrogate for directional preservation.

W2 and Q2: Evaluation focuses on particular calibration/test domains and a 1-bit implementation. Robustness to mask fraction, calibration-token count, calibration-domain choice, and unseen domains is unclear. Statistical significance and runtime costs relative to baselines also need clearer reporting. How do mask fraction and calibration-token count affect performance? Can you report variance across seeds and comparisons under matched budgets?

Answer: Following the reviewer's suggestion, we extend the calibration-domain experiments to LLaMA-3-8B and Qwen3-8B, using C4 and WikiText2 calibration with multiple random seeds under matched budgets. We provide additional results reporting the mean and standard deviation across multiple seeds.

Table: Robustness to calibration distributions on LLaMA-3-8B and Qwen3-8B. We report perplexity (↓\downarrow) on C4, WikiText2, and PTB, and average zero-shot QA accuracy (↑\uparrow). Results are reported as mean ±\pm standard deviation over three seeds.

ModelCalibrationMethodC4 ↓WikiText2 ↓PTB ↓AvgQA ↑
LLaMA-3-8BC4ARB-RC35.65 ± 0.4127.42 ± 0.6545.23 ± 2.8446.71 ± 0.77
Ours21.77 ± 0.5416.37 ± 0.7025.46 ± 0.8856.85 ± 0.21
WikiText2ARB-RC46.71 ± 4.0819.26 ± 0.5048.18 ± 2.2642.85 ± 1.15
Ours27.52 ± 0.0812.75 ± 0.1026.66 ± 0.8954.22 ± 0.82
Qwen3-8BC4ARB-RC34.08 ± 1.9427.89 ± 2.6345.07 ± 0.7254.94 ± 1.95
Ours24.73 ± 0.3623.33 ± 0.6230.77 ± 0.6658.08 ± 0.49
WikiText2ARB-RC40.02 ± 0.6918.81 ± 0.3644.05 ± 0.8752.95 ± 0.23
Ours31.10 ± 0.8615.79 ± 0.1934.52 ± 0.8055.70 ± 0.69

Regarding the runtime cost, we have provided both the quantization time and inference throughput of our method relative to the baseline in the Appendix A and Table 5 of the revised paper, which is modest.

We further examine the sensitivity of model's performance to mask fraction and calibration sample count. As shown in Table 1, the combination of our method and ARB-RC improves over ARB-RC even at a lower average bit-width, but there is little variance in performance when the mask fraction varies. On the other hand, Table 2 shows consistent improvements over ARB-RC across different calibration sizes, each sample consistes of 2048 tokens.

Table 1: Sensitivity to the fraction of 2-bit weights on OPT-1.3B.

MethodBitsC4↓WikiText2 ↓
ARB-RC1.1127.7026.40
ARB-RC + Ours1.0423.5521.72
1.0723.1022.02
1.1522.3421.06

Table 2: Sensitivity to calibration sample count on OPT-1.3B

SamplesTokensMethodC4↓WikiText2 ↓PTB ↓
3265,536ARB-RC28.7529.4146.40
ARB-RC + Ours24.0823.5035.30
64131,072ARB-RC27.7526.5442.48
ARB-RC + Ours23.4221.9834.43
128262,144ARB-RC27.7026.4043.03
ARB-RC + Ours23.1022.0233.73

Thank you for the rebuttal. The added robustness and sensitivity results are helpful. The CBA–survivor-rate check is still limited (mainly one MMLU setting, not broader spectrum-shift cases), but my main concerns are sufficiently addressed, and I maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.