acceptodds
Under review as a conference paper at ICLR 2027

Off-Diagonal Hessian Sensitivity for Post-Training Quantization of Large Language Models

Abstract

Mixed-precision post-training quantization (MP-PTQ) is a promising approach for improving resource efficiency under constrained computational and memory budgets through effective bit allocation. MP-PTQ methods based on independently optimized layer-wise or module-wise surrogate objectives are computationally efficient, but primarily rely on local quantization errors and therefore do not explicitly account for cross-layer interactions under the task loss. In contrast, search-based methods treat mixed-precision allocation as a black-box optimization problem, leading to costly exploration of a large search space. To address these limitations, we propose Off-Diagonal Energy Sensitivity (ODES), which captures interactions within and across projections through the off-diagonal structure of the task-loss Hessian. Starting from a second-order expansion of the task loss, we separate its off-diagonal interaction term. Then, we derive a sufficient condition under which ODES-guided precision reallocation reduces the second moment of the off-diagonal task-loss contribution. To estimate ODES at LLM scale without materializing the Hessian, we develop a compressed probe-centered (CPC) estimator. CPC reduced estimator state memory by approximately \(126\times\) while producing ODES estimates that closely matched those obtained with the direct Welford estimator. Using the estimated ODES ranking, we reduce mixed-precision bit allocation to a one-dimensional search over the number of reallocated projection pairs and evaluate these candidates using HQQ as a lightweight quantization proxy. This reduced the total allocation time by 36–50% relative to search-based methods while maintaining comparable performance in downstream accuracy and perplexity. ODES-guided allocations also improved over uniform allocation with RTN, HQQ, and GPTQ, indicating that the ODES ranking transfers across different quantizers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, large language models, mixed-precision quantization, off-diagonal Hessian, Hessian sensitivity profiling, matrix-free Hessian estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_57T8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ODES, a mixed-precision PTQ method. Assuming equal quantization errors across projections at the same bit width, ODES uses off-diagonal Hessian information to guide bit allocation. It employs Hutchinson–Welford estimation and compressed probe-centered estimation to reduce the memory overhead of computing this information.

Critical strengths

None

Critical weaknesses

The Method section ends with computing off-diagonal information but lacks how to use this information to guide bit allocation. The actual bit-allocation procedure is instead in the Experiments section.

ODES (k_hqq) and AMQ both use the C4 evaluation set as the optimization proxy, but ODES performs worse C4 PPL than uniform bit allocation in some settings.

ODES relies on the unreasonable assumption of "equal quantization errors across projections at the same bit width". Based on this assumption, ODES simplifies the entire bit-allocation procedure, but only reduces runtime by up to 50% compared with AMQ, without consistenly achieving better results. This simplification is undesirable for mixed-precision PTQ, where improving performance should be the primary priority.

Questions for authors

Why does ODES (k_hqq) achieve worse C4 PPL than uniform bit allocation in some settings, despite using the same C4 evaluation set as AMQ for optimization?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 57T8 (1/2)

We thank the reviewer for the time spent on our paper. Most of the points raised are already described in the paper, but we believe they were not emphasized enough. For the points that are included, we indicate below where each appears, citing the line and table numbers of the pre-revision PDF (L162 = line 162), and we have revised the paper with a clearer emphasis on them. Changes in the revised PDF are marked in red, and its line numbers are written as revised L.

W1 (allocation procedure not in the Method section). As the reviewer notes, the procedure is in the paper. The pairing rule is Eq. 5 in Sec. 3.2 (L162–169), and the full procedure, promoting the top-K and demoting the bottom-K projections from a uniform b0-bit baseline and selecting K by a one-dimensional HQQ search, opens Sec. 4.2 (L289–297). Its placement followed the two reasons below. Since the reviewer finds that this structure hurts readability, we have added a final subsection to Sec. 3 (Sec. 3.4, "ODES-Guided Bit Allocation") stating the full procedure, covering the pairing rule, the top-K/bottom-K reallocation, the one-dimensional search over K and the transfer to the target quantizer (revised L211–222).

First, we wanted to keep the CPC experiments separate from the allocation procedure. Sec. 4 opens with the CPC experiments (Sec. 4.1, L243–262), which validate the estimator rather than the allocation. Had the procedure been stated at the end of Sec. 3, these experiments would have stood between the procedure and its results, and we were concerned that this would interrupt the reader's flow. We had therefore kept Sec. 3 to the sensitivity itself, which the title names as the contribution of this paper, including its derivation from the off-diagonal interaction term, the sufficient condition and the CPC estimator.

Second, once the sensitivity is computed, the allocation is a simple rule that promotes the top-K and demotes the bottom-K projections and chooses K by a one-dimensional search. We therefore separated this simple rule from the sensitivity, which carries the complexity of the method, and placed it beside the experiments that show this simple rule is sufficient.

W2 and Question (C4 perplexity vs. uniform). Using a corpus dataset during bit allocation does not guarantee a lower perplexity than uniform on that corpus, for three reasons.

First, the evaluation set is disjoint from the data used for profiling, calibration and K selection (L214–215), and K is selected under HQQ and transferred to GPTQ without re-search (L294–297). Nothing in our pipeline fits the C4 evaluation windows, and the same holds for AMQ under our protocol.

Second, the same phenomenon is reported across the literature. The following are reported cases around 3 bits in which mixed-precision methods yield higher perplexity than uniform-bit baselines on the same corpus used for bit allocation. AMQ (EMNLP 2025, Tables 3 and 16), on Llama-2-13B and Llama-3.1-70B at 3.25 bits, relative to uniform AWQ on WikiText-2; MxMoE (ICML 2025, Table 1), on Qwen2-MoE at 3.25 bits, relative to uniform GPTQ⋆ on WikiText-2; the IntProg and 10% 2-bit/10% 4-bit allocation baselines reported in CMPQ (TMLR 2026, Table 1), on OPT-2.7B and OPT-6.7B at an average of 3 bits, relative to uniform 3-bit allocation on C4; the LLM-MQ baseline reported in CMPQ (Table 2), on OPT-2.7B, OPT-6.7B, Llama-2-7B and Llama-2-13B at 3 bits, relative to uniform AWQ on C4; and CMPQ itself (Table 12), on OPT-30B at 3 bits, relative to uniform AWQ on C4. EvoPress (ICML 2025) uses FineWeb-Edu as its calibration corpus for compression and allocation, yet its Table 3 reports that the DP allocation baseline yields higher C4 perplexity than uniform GPTQ at 3 bits on Mistral-7B-v0.3, Llama-3-8B and Phi-3-Medium-14B. Using a corpus for allocation therefore does not by itself make the perplexity on that corpus lower than uniform.

Third, our paper already discusses differences across evaluation corpora (L425–431). In our experiments, AMQ's C4 improvements did not always transfer to other corpora. On Llama-2-7B and 13B, AMQ and EvoPress raise PTB above uniform while ODES lowers it (Table 2). We therefore concluded that "these results support evaluating projection sensitivity using a criterion that is less directly tied to a finite calibration set or a particular search objective" (L431). ODES ranks projections by the strength of their perturbation interactions with the rest of the network (Eq. 4, L154–159) and reallocates precision to reduce the variance of the off-diagonal loss contribution under the stated perturbation model (Eq. 2–3, L137–151). The ranking therefore reflects how the model processes inputs through these interactions (L484–485), providing a criterion less directly tied to candidate perplexity on a single corpus. The revision states this explicitly in the Introduction and in Sec. 4.3 (revised L52–53, L490–494).

AuthorsOpenReview ↗

Response to Reviewer 57T8 (2/2)

W3 (equal quantization error at the same bit width). The assumption is neither unreasonable nor an oversight, for three reasons.

First, using a common perturbation scale to isolate curvature sensitivity has clear precedent. Our analysis builds on the standard independent, zero-mean additive quantization-noise model (Sripad & Snyder, 1977; Widrow & Kollár, 2008; L127–129), with a common baseline variance introduced to derive the sufficient condition (L143–148). HAWQ-V2 (NeurIPS 2020, Assumption 1 and Lemma 1) compares layer sensitivity under equal perturbation norms, with the same coefficient along every Hessian eigenvector, and SAM (ICLR 2021, Appendix A.1) uses isotropic Gaussian parameter perturbations with a common variance across coordinates in its PAC-Bayesian analysis. Ranking sensitivity without a realized error term also has precedent, as HAWQ (ICCV 2019) and Q-BERT (AAAI 2020) rank layers by the Hessian spectrum alone. In ODES, the common-variance baseline isolates the contribution of off-diagonal curvature, and the score S_m (Eq. 4) requires no realized quantization error. With projection-specific variances, Appendix A.1 extends to a variance-weighted bound that retains the factor 16 under the same one-bit scaling.

Second, the assumption is not only well precedented but also a deliberate design that takes into account how modern PTQ works. For calibration-based quantizers such as GPTQ and LeanQuant the per-projection perturbation is known only after quantization, because it depends on the calibration data and on the bit-widths of neighboring projections (L97–102). Exact candidate-specific errors require quantizing the corresponding configurations. Approximations can avoid this step but introduce assumptions about those errors. ODES is computed once from the pretrained weights and reused for every quantizer and every K (L297–299).

Third, the cross-quantizer results provide empirical support for this design (Table 4). The same ODES ranking, with K selected using HQQ, improves average accuracy across the evaluated models over uniform under RTN, HQQ and GPTQ, which use different quantization procedures, and the accuracy optimum did not exceed the theory-guided boundary K_th on any of the eight models (L320–321). The gap between K_th and K_HQQ that the idealization implies is stated in Limitations (L465–467).

In the revision the Introduction emphasizes that the common-variance baseline is uninformative by design because it targets calibration-based PTQ, where the realized error is unknown before quantization, so that no projection is favored by an assumed error scale and one ranking works generally across RTN, HQQ, GPTQ and LeanQuant (Table 4, revised L55–57).

W4 (only 50% faster than AMQ, not consistently better). Our claim is comparable quality at lower allocation cost.

First, the 50% figure is relative to EvoPress. Relative to AMQ the reduction is 36% (L399–402), with per-model reductions of 25–56% against AMQ and 26–78% against EvoPress (L401–402). We have revised the Conclusion to state that the allocation time is 36% lower than AMQ and 50% lower than EvoPress (revised L538).

Second, we did not claim consistently better results (Abstract L32–33, Conclusion L483–484). We claim comparable quality on the following two metrics (Table 2).

Avg. PPL: ODES is better than AMQ and EvoPress on Llama-2-7B and 13B. On both models, AMQ improves C4 perplexity but worsens PTB perplexity relative to uniform. EvoPress worsens both, whereas ODES improves PTB perplexity (L425–428). Against uniform, AMQ is better on six of the eight models and ODES on five. Where ODES is worse, the gap is at most 0.24 PPL. Where AMQ is worse (Llama-2-13B and 7B), it is 7.5 and 20.5 PPL.

ACC: ODES (K_HQQ) is the only configuration that improves accuracy over uniform on all eight models, whereas AMQ falls below uniform on Qwen3-8B and EvoPress on five of the eight. On Qwen3-8B ODES exceeds AMQ by 3.04 pp.

Third, we agree that performance is one of the primary objectives of mixed-precision PTQ, and allocation cost is another. The contribution of this work is a lower allocation cost together with an attempt to expose a structure that search-based methods treat as a black box (L50–51), while remaining comparable in quality to those methods. The structure is the off-diagonal interaction of the task-loss Hessian at the pretrained weights, which indicates which projections should receive precision before quantization (L484–485). The added experiments support this. The 3-seed SD motivates our 1 pp tolerance (revised Table C.4), MeanDiag under the same HQQ search shows the gains do not depend on the search alone (revised Table 2), random rankings show the ranking matters (revised Figure 5), and 10 seeds show the K = 2 gains on two Llama models exceed seed variation (revised Table 4).

We would appreciate it if the reviewer would reconsider the score in light of these points.

We still have the following concerns:

  • Bit allocation is an important part of the method. Sensitivity estimation is intended to guide bit allocation. Even a simple bit-allocation rule is an essential component of the algorithm, as sensitivity values alone don't constitute a complete method.

  • Applying the same sensitivity ranking to RTN, HQQ, GPTQ, and LeanQuant isn't necessarily an advantage. We want to select the best-performing quantization algorithm and optimize bit allocation based on its quantization errors. For PTQ combined with bit allocation, a quantizer-agnostic ranking can be a disadvantage because it ignores differences in quantization errors across projections and algorithms.

  • The assumption of "equal quantization errors across projections at the same bit width" is a simplification adopted by prior work. Such simplifications can reduce quantization performance to lower the computational cost of bit allocation. However, even a 50% reduction in bit-allocation cost offers limited practical value in the overall PTQ + bit-allocation pipeline. Achieving only comparable quantization performance at this cost reduction provides little benefit.

Reviewer_gLKtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an off-diagonal task-loss Hessian sensitivity for mixed-precision LLM quantisation and proposes CPC, a compressed estimator aiming to have memory savings while approximating direct estimation. The empirical results establish that the resulting ranking is useful and transfers across several quantisers, while reducing allocation cost relative to search-based alternatives. However, the theoretical result controls only the second moment of an idealised off-diagonal interaction term and the experiments do not fully isolate the benefit of off-diagonal information from simpler sensitivity measures.

Critical strengths
  • Novel interaction-aware sensitivity measure: ODES uses projection-level off-diagonal task-loss Hessian energy to capture interactions within and across projections, addressing information missed by local reconstruction losses and purely diagonal curvature measures.

  • Memory-efficient CPC estimator: it approximates well the direct coordinate-wise Hutchinson–Welford estimates while reducing estimator-state memory by approximately 126×.

  • Connection between theory and method design: the paper derives a sufficient condition under which transferring precision from a low-ODES projection to a high-ODES projection reduces the second moment of the off-diagonal loss interaction. It is limited to an idealised noise model, but it provides a good motivation for the ranking and bit-reallocation strategy.

  • Practical and transferable mixed-precision allocation: ODES reduces the combinatorial bit-allocation problem to a one-dimensional search over paired promotions and demotions. The resulting allocations improve over uniform precision with RTN, HQQ, and GPTQ and reduce allocation time by 36–50% relative to the evaluated search-based methods while maintaining comparable performan

Critical weaknesses
  • ODES reduces the second moment of the off-diagonal loss interaction, not the expected total task loss, perplexity, or downstream error. The connection between reduced interaction variance and improved model quality is indirect.

  • The theoretical analysis assumes independent, zero-mean quantisation errors with simplified variance scaling across bit widths. Real quantisation errors can be biased, correlated, group-structured, and dependent on clipping, scaling, and the quantisation algorithm.

  • The experiments do not convincingly show that off-diagonal Hessian energy is more informative than carefully matched alternatives such as diagonal curvature, Hessian trace, gradient sensitivity, or direct quantisation-error sensitivity.

  • Final allocation still requires empirical search. ODES determines the projection ranking, but the number of promoted and demoted projection pairs is selected through an HQQ-based search. The theoretical condition does not reliably determine the final allocation, and using HQQ for both selection and evaluation may favour the reported HQQ results.

Questions for authors
  1. Does lower predicted off-diagonal interaction variance correlate with better loss, perplexity, and accuracy?

  2. Does ODES outperform matched diagonal, gradient, norm, quantisation-error, and random sensitivity rankings?

  3. Does the theoretical condition remain predictive when practical quantisation errors violate its assumptions?

  4. How much of the improvement comes from the ODES ranking rather than the 20-candidate HQQ search over KK?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gLKt

We thank the reviewer for the careful reading. We have run the requested matched baseline, and each weakness is answered with its paired question. Line and table numbers (L162 = line 162) refer to the pre-revision PDF, and numbers in the revised PDF are written as revised L, revised Table and revised Figure. Changes in the revised PDF are marked in red.

W1 and Q1 (second moment vs. loss, perplexity and accuracy). The condition certifies only the off-diagonal second moment, but in our experiments its reduction was accompanied by higher accuracy. First, within the range where the condition predicts a reduction, window-NLL variance decreased with K on three of four models and the accuracy optimum lay inside that range on all eight models (L320–323, Figure 3). Second, the ranking with the lower off-diagonal fluctuation also had the higher accuracy. Under a controlled perturbation that isolates this fluctuation, ODES was 7.02% and 17.06% below MeanDiag on the two models measured (L384–388), and our answers to W3 and Q2 and to W4 and Q4 below show that the same ranking gave clearly higher accuracy. Third, J_off bounds the interaction-induced loss shift of a deployed model (L137–140) and is the largest term of the second-order loss fluctuation (Table A.1).

W2 and Q3 (idealized error assumptions). Our paper already states the gap between this idealized model and real quantization errors (L465–467). The condition nevertheless remained predictive in practice, in both accuracy and perplexity. Across RTN, HQQ, GPTQ and LeanQuant, whose errors differ in bias, correlation and clipping, the accuracy optimum never exceeded the boundary K_th given by the condition on any of the eight models (L320–321). With the same ODES ranking, average accuracy rose over uniform from 53.55 to 53.73 under RTN, from 52.74 to 57.95 under HQQ and from 57.16 to 59.05 under GPTQ, and stayed level under LeanQuant (63.57 and 63.53), while model-averaged perplexity was lower than uniform under all four quantizers (Table 4). This also holds when Llama-2-7B is excluded because of its large seed variation in perplexity (Table C.2, revised Table 4). Under GPTQ, which was not used to select K, accuracy was above uniform on all eight models and PTB perplexity was lower on all eight (Table 2).

W3 and Q2 (matched simpler sensitivities). We had considered the diagonal comparison shown by Figures 3 and C.1, and in response to the question we have now run MeanDiag through exactly the pipeline used for ODES (HQQ proxy over K ∈ {1, …, 20}, then GPTQ). Here too the ODES ranking proved meaningful. ODES (K_HQQ) was above uniform and above MeanDiag in accuracy on all eight models, whereas MeanDiag (K_HQQ) was above uniform on only three (revised Table 2, revised L401–411). MeanDiag ranks projections by the mean diagonal Hessian entry, the per-parameter Hessian trace of HAWQ-V2 (L306–308). It lowers WikiText-2 and C4 perplexity on five models but PTB on only three, with large PTB degradations on three Llama models, whereas ODES lowers PTB on all eight (revised Table C.3). We also replaced the ODES ranking with five random rankings per model on Llama-3.2-1B, Llama-3.2-3B, Qwen3-1.7B and Qwen3-4B and evaluated every K from 1 to 10 under GPTQ. The mean accuracy of random rankings fell below uniform as K increased and stayed below ODES (K_HQQ) at every K on all four models (revised Figure 5, revised L401–407).

W4 and Q4 (ranking vs. the HQQ search over K). Along the path defined by the ranking, accuracy first rose and then fell as K increased, peaking at an intermediate K on seven of the eight models (Figure 3 and revised Figure C.1), and the HQQ search over K located K near this peak. First, the ranking reduces on the order of 3^M paired ±1-bit configurations to 20 candidates on one path, and the search selects one point on it (L289–297). The search scores candidates by HQQ perplexity and never sees accuracy, yet the selected K coincided with the accuracy optimum on two models and lay within one pair of it on five of eight (Figure 3 and revised Figure C.1). Second, the same search with MeanDiag produces a much smaller gain. With the identical 20-candidate HQQ search, the eight-model average accuracy under GPTQ rose by 0.32 pp over uniform for MeanDiag and by 1.89 pp for ODES (revised Table 2). Third, the gains also transfer to GPTQ, which was not used for selection. Table 2 is under GPTQ, where accuracy improved on all eight models (L455–457). In an additional experiment over ten GPTQ seeds, the gain of ODES (K_HQQ) over uniform was +1.50 pp [+1.08, +1.93] on Llama-2-7B and +2.54 pp [+1.99, +3.09] on Llama-3.2-1B (seed-paired 95% intervals, revised Table 4), with all ten seeds improving.

Reviewer_gLKtOpenReview ↗

Rebuttal comment

The rebuttal addresses my main concerns. The matched comparison with MeanDiag under the same HQQ-selection and GPTQ-evaluation pipeline, together with the random-ranking control, demonstrates that the ODES ranking contributes meaningfully beyond the search over KK. The controlled perturbation experiment and cross-quantizer results also provide good empirical support for the proposed off-diagonal criterion. The theoretical result remains limited to an idealised second-order interaction term and does not guarantee improved task loss or accuracy, so KthK_{\mathrm{th}}​ should be interpreted as a conservative boundary rather than a predictor of the optimal allocation. I'll maintain my weak-accept rating with increased confidence.

AuthorsOpenReview ↗

Dear Reviewer gLKt

Thank you for your constructive feedback, which has helped us improve our paper. We are glad that our responses and additional experiments have addressed your main concerns.

If you have any further questions or suggestions, we would be happy to continue the discussion.

Reviewer_Mn83OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ODES, a methods that scores each projection by its off-diagonal task-loss Hessian energy, estimated with a memory-efficient Hutchinson-style estimator (CPC), and uses the ranking for a one-dimensional mixed-precision search around a 3-bit base with HQQ as proxy. Evaluated on eight Llama/Qwen models and four quantizers against AMQ and EvoPress.

Critical strengths
  • CPC matches the direct estimator at a fraction of the memory; the ranking is stable across profiling windows and transfers across quantizers.
  • Comprehensive evaluation experiments.
Critical weaknesses
  • Parity with search-based methods is not supported: ODES loses to AMQ on most models in both accuracy and perplexity, and the reported wins come mainly against EvoPress, which falls below uniform allocation on five of eight models and looks misconfigured. On half the models the selected K is 1 or 2, i.e. essentially uniform.
  • The theory-guided boundary is often worse than uniform, and the diagonal-Hessian baseline is never run through the same pipeline, so the value of the off-diagonal criterion itself is not isolated.
Questions for authors
  1. Why does EvoPress underperform uniform on five models?
  2. Report variance over calibration seeds for the K∈{1,2} cases.
  3. Run MeanDiag / HAWQ-V2 trace through the identical K-search with the HQQ proxy.
  4. How does the method extend beyond ±1 bit around one base width?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Mn83 (1/2)

We thank the reviewer for the concrete requests. Line and table numbers (L162 = line 162) refer to the pre-revision PDF, and numbers in the revised PDF are written as revised L and revised Table. Changes in the revised PDF are marked in red.

Parity with AMQ. In accuracy ODES does not lose to AMQ on most models. Using a 1 pp tolerance informed by the observed scale of calibration-seed variation (SD 0.04–0.98 pp over three GPTQ seeds in our existing runs), the accuracy comparison gives four on-par results and two wins for each method, and ODES (K_HQQ) is the only allocation in Table 2 whose accuracy is above uniform on all eight models. The revision reports these SDs and confidence intervals (revised L465–468, L1406–1409). In average perplexity AMQ is lower on six models, but its margin is concentrated in the three smallest models and its search corpus, and ODES exceeds uniform by at most 0.24. On the Llama-2 models, AMQ improves C4 but raises PTB well above uniform, whereas ODES lowers PTB (L425–431). This is what we mean by comparable quality at lower allocation cost (Abstract L32–33, Conclusion L483–484), at about 36% less total allocation time than AMQ and 50% less than EvoPress (Table 3, L399–402).

Selected K of 1 or 2 on half the models (Q2). Since the reviewer suspects the gain at K = 1 or 2 is seed variation, we ran ten GPTQ calibration seeds for uniform and the selected K on the four models with seed-paired 95% confidence intervals. The existing-run SDs and ten-seed intervals are in revised Tables C.4 and 4. On Llama-2-7B (K = 2) the gain is +1.50 pp [+1.08, +1.93] and on Llama-3.2-1B (K = 2) +2.54 pp [+1.99, +3.09], with all ten seeds improving. On Qwen3-4B and Qwen3-8B (K = 1) the gain is +0.30 pp [−0.20, +0.79] and +0.12 pp [−0.07, +0.31], within seed variation of uniform. So on two of the four models the gain is significant. For the two Qwen models our paper already describes this. With their small K_th the effect of reallocation is confined to a much narrower range (Figure 3, L323, L350–352), so they are less favorable to precision reallocation, and K_th signals it before quantization.

EvoPress below uniform on five models (Q1). EvoPress was run with its official search code and published hyperparameters. The fitness is the KL divergence to the original model, bit candidates {2, 3, 4, 5, 6}, 150 generations × 128 offspring, and the original survivor and token schedule. The only change is the calibration data. AMQ, EvoPress and ODES received the same C4 calibration budget (Appendix C.1), with evaluation windows disjoint from all profiling, calibration and search samples (L214–215). Run on Llama-2-7B with its original calibration data, EvoPress reproduces Table 7 of its paper. Over three GPTQ seeds and the three best configurations per search, our WikiText-2, C4 and FineWeb-Edu perplexities are 5.77 ± 0.07, 7.89 ± 0.10 and 6.75 ± 0.10, against 5.70, 7.87 and 6.76 in that table. EvoPress fits its search to the calibration corpus, with 8M fresh tokens in its paper against about 65K in our common budget, so under that budget the search may overfit this limited sample. The revision discusses this calibration-data dependence in Section 4.3 (revised L490–492).

Diagonal-Hessian baseline through the identical pipeline (Q3). We had considered the comparison sufficiently shown by Figures 3 and C.1, and in response to the question we ran MeanDiag through the identical pipeline (HQQ proxy over K ∈ {1, …, 20}, then GPTQ). In accuracy MeanDiag (K_HQQ) is above uniform on three of eight models, while ODES (K_HQQ) is above uniform and MeanDiag on all eight. MeanDiag lowers WikiText-2 and C4 perplexity on five models but PTB on only three, with large PTB losses on three Llama models, while ODES lowers PTB on all eight. See revised Tables 2 and C.3 (revised L401–411, L1352–1354).

Theory-guided boundary often worse than uniform. K_th is the boundary at which the pairwise sufficient condition first fails. The sufficient condition (Eq. 3) certifies a reduction of the off-diagonal fluctuation J_off under the idealized perturbation model. The diagonal mean, higher-order terms and the difference between the idealized model and real quantizer errors (L465–467) are not covered by the condition, so the realized performance at K_th can be below uniform. ODES accounts for this gap between the idealized model and the real quantizer by selecting the actual operating point K_HQQ on the path defined by the ranking with the HQQ proxy (L289–297). The observed accuracy optimum did not exceed K_th on any of the eight models (L320–321).

AuthorsOpenReview ↗

Response to Reviewer Mn83 (2/2)

Beyond ±1 bit around one base width (Q4). This is a direction for future work, and our paper states the current restriction in Limitations (L467–468). With paired one-bit promotion and demotion a single K describes the search space, but with more bit-width choices it no longer does. What we wanted to show in this paper is that a ranking built from an idealized perturbation model, combined with the simplest ±1-bit rule around one base width, already yields a measurable efficiency gain. The idealized model is the less exact part of the current form, and it is also the part that can be improved. We believe it can be replaced by one fitted to the realized errors of a given quantizer, which should bring the constants in the bound, and hence the predicted gain from moving a projection by one or more bits, closer to the realized values. The trade-off between bit budget and loss could then be approximated numerically before quantization, and moves of more than one bit and other base widths would follow from that calculation rather than from a larger search.

AuthorsOpenReview ↗

Gentle Reminder for Reviewer Mn83

Dear Reviewer Mn83,

Thank you again for taking the time to review our paper. Following your feedback, we have provided additional experiments and clarifications, particularly regarding the baseline comparisons, calibration-seed variability, and performance evaluation.

We would appreciate hearing your thoughts on these updates and whether any concerns remain unresolved. We would be glad to provide further clarification and continue the discussion if needed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

We thank the reviewers for their feedback. We have uploaded a revised manuscript with changes marked in red and provided detailed responses under each review. All section, table and figure references in this comment refer to the revised manuscript.

The main additions are as follows.

  1. Ranking ablations. In Table 2, MeanDiag and ODES are evaluated with GPTQ after the same HQQ selection pipeline. Their mean accuracy gains over uniform across eight models are 0.32 pp and 1.89 pp, respectively. Figure 5 adds five random rankings per model on four models, with every K from 1 to 10 evaluated using GPTQ. The corresponding perplexity results are provided in Appendix C.6.

  2. Calibration-seed variability. Table 4 reports additional experiments over ten GPTQ calibration seeds for uniform and ODES on the four models whose selected K is 1 or 2. We report seed-paired accuracy differences, 95% confidence intervals and the number of seeds showing an improvement. Appendix C.5 separately reports standard deviations from the existing three-seed runs.

  3. Method presentation and design rationale. Section 3.4 now explicitly describes paired precision reallocation, the one-dimensional HQQ search over K and the application of the selected allocation to the target quantizer. The Introduction explains the rationale for using a common perturbation variance, and the Conclusion specifies the allocation-time reductions relative to AMQ and EvoPress separately.

We would appreciate your consideration of these additional results together with our responses to the individual reviews.

AuthorsOpenReview ↗

Summary of the Discussion

We thank the reviewers for the discussion. For each reviewer, we summarize the initial review and the comments that followed. L denotes pre-revision line numbers unless marked revised.

Reviewer 57T8. The initial review listed no strengths ("57T8: Critical Strengths: None") and raised four weaknesses. First, "57T8: The Method section ... lacks how to use this information to guide bit allocation." Second, "57T8: ... ODES performs worse C4 PPL than uniform ..." Third, "57T8: ODES relies on the unreasonable assumption ..." Fourth, "57T8: ... only reduces runtime by up to 50% compared with AMQ, without consistenly achieving better results." (spelling as in the original)

Our rebuttal and revision addressed each of these. First, the complete allocation procedure is now stated in the Method section (revised Sec. 3.4). 57T8 nevertheless raised this point again after the revision ("57T8: Bit allocation ... complete method."). Second, on the C4 PPL, using a corpus during allocation does not guarantee a lower PPL than uniform on that corpus, as reported in previously published papers (our response to W2) and discussed in our paper (L425–431). This concern was not raised again. Third, a common perturbation scale at the same bit width has precedent in previously published papers (our response to W3), and it is a deliberate design for calibration-based PTQ, where the realized error is known only after quantization (L97–102, revised L55–57). Later, 57T8 calls this "unreasonable" assumption a simplification adopted by prior work. Fourth, our paper claims comparable overall quality at substantially lower allocation cost, and the reduction is 36% relative to AMQ, not 50%, and 50% relative to EvoPress (revised L538). 57T8 described the performance as comparable but argued that the 36–50% reduction is a small benefit. After the rebuttal, 57T8 also raised a new point, that applying one ranking to several PTQ methods is not necessarily an advantage. As we answered the reviewer, our paper presents this (Abstract, Sec. 4.4, Conclusion) not as an advantage in itself but for what it implies. The ranking works when the quantization outcome is uncertain because it depends on the calibration data and the PTQ method.

Taken together, the comments of 57T8 appear to acknowledge the following four points.

The ODES-guided allocation achieves comparable quantization performance ("57T8: Achieving only comparable quantization performance at this cost reduction provides little benefit.").

The reduced allocation cost is a gain, although the reviewer regards it as small ("57T8: even a 50% reduction in bit-allocation cost offers limited practical value in the overall PTQ + bit-allocation pipeline").

ODES builds on prior work ("57T8: The assumption of 'equal quantization errors across projections at the same bit width' is a simplification adopted by prior work.").

The bit allocation guided by ODES is simple ("57T8: Sensitivity estimation is intended to guide bit allocation. Even a simple bit-allocation rule is an essential component of the algorithm ...").

Points 1–4 show that a simple allocation (4) guided by ODES, built on the assumptions of prior work (3), reaches comparable performance (1) at a lower allocation cost (2), which shows the importance and significance of ODES, the sensitivity we propose.

What remains is whether comparable performance at a lower allocation cost is a sufficient contribution, and we consider this a value judgment of the reviewer. 57T8 sees 36% (AMQ) and 50% (EvoPress) time savings as little benefit ("57T8: Achieving only ... little benefit.").

Reviewer gLKt. The review recognized the novel interaction-aware sensitivity measure and the practical and transferable allocation that reduces allocation time by 36–50% relative to search-based methods while maintaining comparable performance. We answered the weaknesses with additional experiments, including MeanDiag through the identical pipeline, five random rankings per model and ten GPTQ calibration seeds. gLKt explicitly stated that the rebuttal addresses the reviewer's main concerns and that the confidence would be raised.

Reviewer Mn83. Mn83 did not reply to our response. Two concerns of Mn83 (the diagonal-Hessian baseline through the same pipeline and the theory-guided boundary) correspond to two main concerns of gLKt, which gLKt stated were addressed. Our response to Mn83 directly addresses parity with AMQ. We reported a reproduction check on Llama-2-7B using EvoPress's original calibration data and discussed the limited common calibration budget as a possible explanation for the observed gap. On the gains at small K, ten GPTQ seeds showed significant gains on two of four models (revised Table 4). We therefore believe that most concerns of Mn83 have also been resolved.

For these reasons, we believe that most of the main concerns raised by the reviewers have been resolved. What remains is the value judgment of 57T8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.