- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ODES, a mixed-precision PTQ method. Assuming equal quantization errors across projections at the same bit width, ODES uses off-diagonal Hessian information to guide bit allocation. It employs Hutchinson–Welford estimation and compressed probe-centered estimation to reduce the memory overhead of computing this information.
Critical strengths
None
Critical weaknesses
The Method section ends with computing off-diagonal information but lacks how to use this information to guide bit allocation. The actual bit-allocation procedure is instead in the Experiments section.
ODES (k_hqq) and AMQ both use the C4 evaluation set as the optimization proxy, but ODES performs worse C4 PPL than uniform bit allocation in some settings.
ODES relies on the unreasonable assumption of "equal quantization errors across projections at the same bit width". Based on this assumption, ODES simplifies the entire bit-allocation procedure, but only reduces runtime by up to 50% compared with AMQ, without consistenly achieving better results. This simplification is undesirable for mixed-precision PTQ, where improving performance should be the primary priority.
Questions for authors
Why does ODES (k_hqq) achieve worse C4 PPL than uniform bit allocation in some settings, despite using the same C4 evaluation set as AMQ for optimization?
5 comments
Response to Reviewer 57T8 (2/2)
W3 (equal quantization error at the same bit width). The assumption is neither unreasonable nor an oversight, for three reasons.
First, using a common perturbation scale to isolate curvature sensitivity has clear precedent. Our analysis builds on the standard independent, zero-mean additive quantization-noise model (Sripad & Snyder, 1977; Widrow & Kollár, 2008; L127–129), with a common baseline variance introduced to derive the sufficient condition (L143–148). HAWQ-V2 (NeurIPS 2020, Assumption 1 and Lemma 1) compares layer sensitivity under equal perturbation norms, with the same coefficient along every Hessian eigenvector, and SAM (ICLR 2021, Appendix A.1) uses isotropic Gaussian parameter perturbations with a common variance across coordinates in its PAC-Bayesian analysis. Ranking sensitivity without a realized error term also has precedent, as HAWQ (ICCV 2019) and Q-BERT (AAAI 2020) rank layers by the Hessian spectrum alone. In ODES, the common-variance baseline isolates the contribution of off-diagonal curvature, and the score S_m (Eq. 4) requires no realized quantization error. With projection-specific variances, Appendix A.1 extends to a variance-weighted bound that retains the factor 16 under the same one-bit scaling.
Second, the assumption is not only well precedented but also a deliberate design that takes into account how modern PTQ works. For calibration-based quantizers such as GPTQ and LeanQuant the per-projection perturbation is known only after quantization, because it depends on the calibration data and on the bit-widths of neighboring projections (L97–102). Exact candidate-specific errors require quantizing the corresponding configurations. Approximations can avoid this step but introduce assumptions about those errors. ODES is computed once from the pretrained weights and reused for every quantizer and every K (L297–299).
Third, the cross-quantizer results provide empirical support for this design (Table 4). The same ODES ranking, with K selected using HQQ, improves average accuracy across the evaluated models over uniform under RTN, HQQ and GPTQ, which use different quantization procedures, and the accuracy optimum did not exceed the theory-guided boundary K_th on any of the eight models (L320–321). The gap between K_th and K_HQQ that the idealization implies is stated in Limitations (L465–467).
In the revision the Introduction emphasizes that the common-variance baseline is uninformative by design because it targets calibration-based PTQ, where the realized error is unknown before quantization, so that no projection is favored by an assumed error scale and one ranking works generally across RTN, HQQ, GPTQ and LeanQuant (Table 4, revised L55–57).
W4 (only 50% faster than AMQ, not consistently better). Our claim is comparable quality at lower allocation cost.
First, the 50% figure is relative to EvoPress. Relative to AMQ the reduction is 36% (L399–402), with per-model reductions of 25–56% against AMQ and 26–78% against EvoPress (L401–402). We have revised the Conclusion to state that the allocation time is 36% lower than AMQ and 50% lower than EvoPress (revised L538).
Second, we did not claim consistently better results (Abstract L32–33, Conclusion L483–484). We claim comparable quality on the following two metrics (Table 2).
Avg. PPL: ODES is better than AMQ and EvoPress on Llama-2-7B and 13B. On both models, AMQ improves C4 perplexity but worsens PTB perplexity relative to uniform. EvoPress worsens both, whereas ODES improves PTB perplexity (L425–428). Against uniform, AMQ is better on six of the eight models and ODES on five. Where ODES is worse, the gap is at most 0.24 PPL. Where AMQ is worse (Llama-2-13B and 7B), it is 7.5 and 20.5 PPL.
ACC: ODES (K_HQQ) is the only configuration that improves accuracy over uniform on all eight models, whereas AMQ falls below uniform on Qwen3-8B and EvoPress on five of the eight. On Qwen3-8B ODES exceeds AMQ by 3.04 pp.
Third, we agree that performance is one of the primary objectives of mixed-precision PTQ, and allocation cost is another. The contribution of this work is a lower allocation cost together with an attempt to expose a structure that search-based methods treat as a black box (L50–51), while remaining comparable in quality to those methods. The structure is the off-diagonal interaction of the task-loss Hessian at the pretrained weights, which indicates which projections should receive precision before quantization (L484–485). The added experiments support this. The 3-seed SD motivates our 1 pp tolerance (revised Table C.4), MeanDiag under the same HQQ search shows the gains do not depend on the search alone (revised Table 2), random rankings show the ranking matters (revised Figure 5), and 10 seeds show the K = 2 gains on two Llama models exceed seed variation (revised Table 4).
We would appreciate it if the reviewer would reconsider the score in light of these points.
We still have the following concerns:
-
Bit allocation is an important part of the method. Sensitivity estimation is intended to guide bit allocation. Even a simple bit-allocation rule is an essential component of the algorithm, as sensitivity values alone don't constitute a complete method.
-
Applying the same sensitivity ranking to RTN, HQQ, GPTQ, and LeanQuant isn't necessarily an advantage. We want to select the best-performing quantization algorithm and optimize bit allocation based on its quantization errors. For PTQ combined with bit allocation, a quantizer-agnostic ranking can be a disadvantage because it ignores differences in quantization errors across projections and algorithms.
-
The assumption of "equal quantization errors across projections at the same bit width" is a simplification adopted by prior work. Such simplifications can reduce quantization performance to lower the computational cost of bit allocation. However, even a 50% reduction in bit-allocation cost offers limited practical value in the overall PTQ + bit-allocation pipeline. Achieving only comparable quantization performance at this cost reduction provides little benefit.
Response to Reviewer 57T8 (1/2)
We thank the reviewer for the time spent on our paper. Most of the points raised are already described in the paper, but we believe they were not emphasized enough. For the points that are included, we indicate below where each appears, citing the line and table numbers of the pre-revision PDF (L162 = line 162), and we have revised the paper with a clearer emphasis on them. Changes in the revised PDF are marked in red, and its line numbers are written as revised L.
W1 (allocation procedure not in the Method section). As the reviewer notes, the procedure is in the paper. The pairing rule is Eq. 5 in Sec. 3.2 (L162–169), and the full procedure, promoting the top-K and demoting the bottom-K projections from a uniform b0-bit baseline and selecting K by a one-dimensional HQQ search, opens Sec. 4.2 (L289–297). Its placement followed the two reasons below. Since the reviewer finds that this structure hurts readability, we have added a final subsection to Sec. 3 (Sec. 3.4, "ODES-Guided Bit Allocation") stating the full procedure, covering the pairing rule, the top-K/bottom-K reallocation, the one-dimensional search over K and the transfer to the target quantizer (revised L211–222).
First, we wanted to keep the CPC experiments separate from the allocation procedure. Sec. 4 opens with the CPC experiments (Sec. 4.1, L243–262), which validate the estimator rather than the allocation. Had the procedure been stated at the end of Sec. 3, these experiments would have stood between the procedure and its results, and we were concerned that this would interrupt the reader's flow. We had therefore kept Sec. 3 to the sensitivity itself, which the title names as the contribution of this paper, including its derivation from the off-diagonal interaction term, the sufficient condition and the CPC estimator.
Second, once the sensitivity is computed, the allocation is a simple rule that promotes the top-K and demotes the bottom-K projections and chooses K by a one-dimensional search. We therefore separated this simple rule from the sensitivity, which carries the complexity of the method, and placed it beside the experiments that show this simple rule is sufficient.
W2 and Question (C4 perplexity vs. uniform). Using a corpus dataset during bit allocation does not guarantee a lower perplexity than uniform on that corpus, for three reasons.
First, the evaluation set is disjoint from the data used for profiling, calibration and K selection (L214–215), and K is selected under HQQ and transferred to GPTQ without re-search (L294–297). Nothing in our pipeline fits the C4 evaluation windows, and the same holds for AMQ under our protocol.
Second, the same phenomenon is reported across the literature. The following are reported cases around 3 bits in which mixed-precision methods yield higher perplexity than uniform-bit baselines on the same corpus used for bit allocation. AMQ (EMNLP 2025, Tables 3 and 16), on Llama-2-13B and Llama-3.1-70B at 3.25 bits, relative to uniform AWQ on WikiText-2; MxMoE (ICML 2025, Table 1), on Qwen2-MoE at 3.25 bits, relative to uniform GPTQ⋆ on WikiText-2; the IntProg and 10% 2-bit/10% 4-bit allocation baselines reported in CMPQ (TMLR 2026, Table 1), on OPT-2.7B and OPT-6.7B at an average of 3 bits, relative to uniform 3-bit allocation on C4; the LLM-MQ baseline reported in CMPQ (Table 2), on OPT-2.7B, OPT-6.7B, Llama-2-7B and Llama-2-13B at 3 bits, relative to uniform AWQ on C4; and CMPQ itself (Table 12), on OPT-30B at 3 bits, relative to uniform AWQ on C4. EvoPress (ICML 2025) uses FineWeb-Edu as its calibration corpus for compression and allocation, yet its Table 3 reports that the DP allocation baseline yields higher C4 perplexity than uniform GPTQ at 3 bits on Mistral-7B-v0.3, Llama-3-8B and Phi-3-Medium-14B. Using a corpus for allocation therefore does not by itself make the perplexity on that corpus lower than uniform.
Third, our paper already discusses differences across evaluation corpora (L425–431). In our experiments, AMQ's C4 improvements did not always transfer to other corpora. On Llama-2-7B and 13B, AMQ and EvoPress raise PTB above uniform while ODES lowers it (Table 2). We therefore concluded that "these results support evaluating projection sensitivity using a criterion that is less directly tied to a finite calibration set or a particular search objective" (L431). ODES ranks projections by the strength of their perturbation interactions with the rest of the network (Eq. 4, L154–159) and reallocates precision to reduce the variance of the off-diagonal loss contribution under the stated perturbation model (Eq. 2–3, L137–151). The ranking therefore reflects how the model processes inputs through these interactions (L484–485), providing a criterion less directly tied to candidate perplexity on a single corpus. The revision states this explicitly in the Introduction and in Sec. 4.3 (revised L52–53, L490–494).