acceptodds
Under review as a conference paper at ICLR 2027

Two-pass Trajectory Calibration for Compensation-based Post-training Quantization

Abstract

In post-training quantization (PTQ), learned per-row codebooks are typically fitted once on the weights where an error compensation engine such as GPTQ or LDLQ starts from. The engine, however, does not round those weights: after each column is rounded, the remaining columns are updated to compensate for its rounding error. By the time a later column is rounded, its values have therefore already been shifted by the compensation from every column rounded before it. We call this sequence of shifted values the layer's trajectory. We propose 2-Pass Trajectory Calibration (2PTC), a drop-in correction for the mismatch between the static weights a codebook is fit on and the shifted trajectory the engine actually rounds: run the engine once to record the trajectory, refit the codebook on it, and rerun the same engine on the same starting weights with the refitted grid. 2PTC stores a single codebook of the same memory size and leaves inference unchanged; it only costs one additional offline pass. Across six open LLMs and two compensation engines, on top of QEP inter-layer correction, per-channel 2PTC lowers perplexity at every bit-width with GPTQ and at 2 and 3 bits with LDLQ. The gains are largest at 2 bits: WikiText-2 perplexity decreases in 36 of 36 paired runs, by 15.5% on average with GPTQ and 5.7% with LDLQ, and average zero-shot accuracy improves by up to 9.4 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, LLM compression, model compression, non-uniform quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SrxeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a mismatch in compensation-based PTQ, where codebooks are fitted to initial weights rather than the compensated weights actually being quantized. It proposes 2-Pass Trajectory Calibration (2PTC), which refits codebooks using the first-pass quantization trajectory, supported by a layer-wise reconstruction error analysis. Experiments show some improvements, although the benefits are sensitive to the initial codebook design.

Critical strengths
  1. The paper identifies a meaningful mismatch between codebook fitting and the actual rounding trajectory in compensation-based PTQ. The proposed two-pass approach is easy to integrate into existing quantization pipelines.
  2. The paper provides an extensive empirical evaluation on six LLMs, two quantization engines, and three bit-widths across three random seeds. The fitting-target ablation convincingly isolates the benefits of trajectory-based codebook fitting from those of QEP correction. The authors also transparently report negative results with group-wise codebooks, uniform grids, and LeanQuant-style configurations, strengthening the credibility of the experimental findings.
Critical weaknesses
  1. I believe there is a critical contradictory here. The second-pass codebook is fitted to a trajectory generated using the first-pass codebook, but changing the codebook also changes the rounding decisions and compensation trajectory. Consequently, improving the frozen-trajectory objective does not guarantee improving the actual reconstruction loss. This mismatch weakens the theoretical motivation, and I believe there will be a better solution than the proposed 2PTC.
  2. The improvement largely comes from imperfections in the original codebook. If the original codebook is more advanced, 2PTC can even have a negative effect (as shown by the comparison with LeanQuant), which raises concerns about the method's robustness and general applicability. The proposed 2PTC provides a simple and practical alternative.
Questions for authors
  1. Can the authors reconsider this inconsistency and try to develop a more optimal solution using techniques such as a lookahead optimization objective or compensation-aware codebook generation?
  2. Can the authors provide a theoretical analysis of what benefits or drawbacks 2PTC could bring if the original codebook is theoretically optimal?
  3. Can the authors add experiments comparing 2PTC with more advanced codebook-based PTQ methods, including QTIP, GPTVQ, and AQLM?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Srxe

W1 – Frozen-trajectory objective vs. the actual loss. We agree that updating the codebook also alters the rounding decisions and, consequently, the quantization trajectory. While improving the frozen-trajectory objective does not guarantee a reduction in the actual loss a priori, Eq. 7 bounds this discrepancy in terms of trajectory displacement, and we now directly quantify its empirical behavior through a new set of experiments.

For every layer, we computed the exact objective minimized by GPTQ/LDLQ:

L(G)=∥(W∗−W^(G))X∥F2=tr((W∗−W^(G)) H (W∗−W^(G))⊤).\mathcal{L}(\mathcal{G}) = \Vert (W^* - \widehat{W}(\mathcal{G}))X \Vert_F^2 = \mathrm{tr}\big((W^* - \widehat{W}(\mathcal{G}))\,H\,(W^* - \widehat{W}(\mathcal{G}))^\top\big).

We evaluated it for both the first-pass codebook (G1\mathcal{G}_1) and the codebook refitted by 2PTC on the first-pass trajectory (G2\mathcal{G}_2). The experiment covers 108 full quantization runs: 6 models ×\times 3 seeds ×\times 3 bit-widths ×\times 2 engines (GPTQ and QuIP), for a total of 31,248 layers. G2\mathcal{G}_2 achieves strictly lower reconstruction error than G1\mathcal{G}_1 in 99.4% of these layers. The proportion is consistently high in every setting, including 4 bits (99.2% with GPTQ and 99.98% with QuIP). In the few layers where G1\mathcal{G}_1 is better, the margin is small. Although the trajectory does change between passes, its effect on the loss is minor, and refitting the codebook on the first-pass trajectory almost always brings the quantized matrix closer to the full-precision one.

Here we attach a table with the differences between how much G2\mathcal{G}_2 reduces loss with respect to G1\mathcal{G}_1:

EngineBitsMedian per layerMean per layerReduction in total model error (median per run)
GPTQ21.0%2.9%1.7%
GPTQ30.5%1.9%1.2%
GPTQ40.8%2.4%1.8%
QuIP20.8%3.2%1.0%
QuIP30.3%0.5%0.3%
QuIP40.5%0.7%0.4%

To remove any remaining risk of degradation, we now add a per-layer acceptance test to 2PTC that keeps the first-pass codebook whenever L(G1)<L(G2)\mathcal{L}(\mathcal{G}_1)<\mathcal{L}(\mathcal{G}_2). Both losses are by-products of the two passes, so the test has negligible cost. It acts as a one-step lookahead on the true objective and guarantees that no layer ends up worse than with a single pass.

Finally, we also studied whether further iterations improve the performance of the proposed approach. In the three-pass experiment of Appendix D.3, the codebook is refitted again on the second-pass trajectory and a third quantization pass is run, using the same protocol and configurations as 2PTC. The third pass improves over 2PTC in only 4 of 18 cases at 2 bits and is 1.8% worse on average. This indicates that the main correction happens with the first refit, and that two passes are sufficient.

W2 – Dependence on the initial codebook. We agree that the benefit of 2PTC depends on the initial codebook. With the LeanQuant codebook as published, the second pass does not help: it degrades results at 2 bits and is neutral at 3–4 bits. Specifically, this occurs because LeanQuant's codebook is optimized for a different criterion: its kk-means step weights columns by (1/Ukk)4=dk2(1/U_{kk})^4 = d_k^2, whereas the compensation objective weights columns by dkd_k. Refitting with dk2d_k^2 on the trajectory therefore optimizes a mismatched objective.

This is visible per layer, and the acceptance test handles it. With the LeanQuant codebook, the second pass increases the layer loss. In exactly these cases, our acceptance test (explained above) retains the first-pass codebook.

Nevertheless, our weighting is the stronger baseline. Our Hessian-diagonal weighting already outperforms LeanQuant with a single pass at 3–4 bits (17/17 model–seed pairs). Also, at 2 bits, 2PTC on our simple initialization outperforms one-pass LeanQuant in 18/18 pairs (median −14.5%-14.5\% perplexity). We thus argue that the gain is not only a correction of a weak initial codebook. It reflects the open-loop vs. closed-loop mismatch, which is largest at low bit-widths.

AuthorsOpenReview ↗

Continuation of the reponse

Q1 – Lookahead or compensation-aware codebook. Our per-layer acceptance test already functions as a one-step lookahead on the true post-compensation loss. Designing a fully compensation-aware codebook generator is an exciting direction. We will explicitly discuss both approaches as promising directions for future work in the revision.

Q2 – Theoretically optimal initial codebook. Here it is helpful to distinguish between two notions of optimality:

  • Open-loop optimal: Optimal for the unperturbed full-precision weights WW.
  • Closed-loop optimal: Optimal for the actual compensated trajectory that the quantizer rounds.

Any codebook fitted prior to quantization is at best open-loop optimal, as it cannot anticipate the perturbations introduced by error feedback. 2PTC bridges this gap by moving the codebook toward the closed-loop optimum on the observed trajectory (whose distribution shifts very little between passes, as confirmed by our frozen-trajectory accuracy above). Conversely, if an initial codebook were already closed-loop optimal, the refit would leave it essentially unchanged to first order, and the acceptance test guarantees that no layer can degrade.

Q3 – Comparison with other methods. QTIP and AQLM belong to distinct quantization paradigms. Neither uses the GPTQ-style sequential error feedback that 2PTC targets. We will clearly position both works in the revised Related Work section.

GPTVQ is the closest baseline to our approach among the three methods mentioned. We integrated its official implementation into our pipeline. Both methods therefore share the same calibration data, models, evaluation protocol and metrics, and we compare them in terms of effective bits per weight.

On five models, 1D GPTVQ uses 0.08–0.17 more bits per weight than our per-channel codebooks (e.g., 2.19 vs. 2.01 at 2 bits) and requires roughly 12×12\times more quantization time (∼4 h\sim 4\text{ h} vs. ∼20 min\sim 20\text{ min} on 7B-scale models). In terms of perplexity, 2PTC with QEP outperforms 1D GPTVQ at 2 bits (12/15 model–seed pairs) and is comparable at 3–4 bits. The same holds for its 2D variant. We emphasize that GPTVQ's performance comes at a higher memory and compute cost.

BitsMethodBits/weightPPL WikiText-2 (mean)GPTVQ better than 2PTC + QEPGPTVQ better than 2PTC w/o QEPQuant. time (7B-scale)
21D GPTVQ2.1920.573/158/15∼4.1 h\sim 4.1\text{ h}
22PTC + QEP2.0116.90----∼20 min\sim 20\text{ min}
31D GPTVQ3.166.956/1412/14∼4.3 h\sim 4.3\text{ h}
32PTC + QEP3.036.73----∼19 min\sim 19\text{ min}
41D GPTVQ4.146.216/119/11∼4.4 h\sim 4.4\text{ h}
42PTC + QEP4.065.94----∼20 min\sim 20\text{ min}

Importantly, the two approaches are orthogonal. GPTVQ initializes each codebook when its column group is reached, but within each group, the codebook is still fitted before compensating for the remaining columns. Applying 2PTC on top of GPTVQ's codebooks is therefore a natural extension.

Dear Reviewer Srxe,

Thank you again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

Reviewer_AJv4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes running quantization with error redistribution to unquantized values (like GPTQ, LDLQ etc) in a loop with VQ methods to better fit the vector code book to the redistributed weight matrix.

Critical strengths
  • The insight that sequential weight updating shifts the weight distribution causing a misalignment to the code-book of code-book-based quantizers and the data they represent is sound.
Critical weaknesses
  • The proposed method does not solve the problem of misaligned codebook and weight-distribution shift (it just empirically mitigates it in some cases) and an underlying theory is missing.

  • Perplexity results in Table 1 show that the improvements are usually small and sometimes negative (!) and Figure 3 shows that refitting the codebook sometimes decreases ppl. This further demonstrates the lack of a consistent underpinning theory.

  • Accuracy results are highly unreliable in quantization evaluation (see 'Accuracy is not all you need' paper).

Questions for authors

Can the authors explain 1) why the iterative process sometimes yields worse results and 2) how this can be prevented?

5 comments
AuthorsOpenReview ↗

Response to Reviewer AJv4

W1, W2, Q1, Q2 – Does the second codebook really fit better? To directly verify whether the second-pass codebook G2\mathcal{G}_2 improves layer-level reconstruction (Eq. 5), we added a diagnostic tracking the exact compensation objective minimized by GPTQ/LDLQ, across every layer for both the first-pass (G1\mathcal{G}_1) and second-pass (G2\mathcal{G}_2) codebooks:

L(G)=∥(W∗−W^(G))X∥F2=tr((W∗−W^(G)) H (W∗−W^(G))⊤).\mathcal{L}(\mathcal{G}) = \Vert (W^* - \widehat{W}(\mathcal{G}))X \Vert_F^2 = \mathrm{tr}\big((W^* - \widehat{W}(\mathcal{G}))\,H\,(W^* - \widehat{W}(\mathcal{G}))^\top\big).

Across 108 new experiments (6 models ×\times 3 seeds ×\times 3 bit-widths ×\times 2 modes [QuIP, GPTQ]), totalling 31,248 layers, G2\mathcal{G}_2 achieves strictly lower reconstruction error than G1\mathcal{G}_1 in 99.4% of all layers. This consistency holds even at 4 bits, where G2\mathcal{G}_2 outperforms G1\mathcal{G}_1 in 99.2% of layers with GPTQ and 99.98% with QuIP.

Furthermore, our frozen-trajectory approximation proves highly accurate in practice: it predicts the actual second-pass loss within 0.2–0.5% of the first-pass loss and correctly identifies the better codebook in over 97.5% of layers.

Regarding the occasional cases in Table 1 where 2PTC appears marginally worse, these occur exclusively at 4 bits, where the differences (≤0.04%\le 0.04\% perplexity) fall well within standard GPU run-to-run non-determinism, which we measured at 0.1–0.15% at 4 bits and 0.2–0.4% at 3 bits for identical reruns. Thus, these differences reflect statistical ties rather than genuine degradations.

Here we attach a table with the differences between how much G2\mathcal{G}_2 reduces loss with respect to G1\mathcal{G}_1:

EngineBitsMedian per layerMean per layerReduction in total model error (median per run)
GPTQ21.0%2.9%1.7%
GPTQ30.5%1.9%1.2%
GPTQ40.8%2.4%1.8%
QuIP20.8%3.2%1.0%
QuIP30.3%0.5%0.3%
QuIP40.5%0.7%0.4%

Nevertheless, to guarantee that no layer ever degrades, we now incorporate a per-layer acceptance criterion at negligible cost. Since L(G1)\mathcal{L}(\mathcal{G}_1) and L(G2)\mathcal{L}(\mathcal{G}_2) are already computed as natural by-products of the two passes, we simply retain G1\mathcal{G}_1 whenever L(G1)<L(G2)\mathcal{L}(\mathcal{G}_1)<\mathcal{L}(\mathcal{G}_2). With our initialization, this safeguard keeps G2\mathcal{G}_2 in 99.4% of layers, confirming its consistent advantage while providing a strict layer-wise guarantee.

In addition to that, the gain scales with bit-width, as our analysis predicts. The trajectory widens in proportion to quantization distortion, which is ∼12×\sim 12\times larger at 2 bits than at 4 bits.

  • At 2 bits, the improvement is large and systematic, as shown in the table below across a wide range of metrics.
  • At 3–4 bits it is smaller but consistent with GPTQ. With QuIP at 3–4 bits it is a tie.

Underlying theory. We now provide an exact characterization, which we will include in the revised draft. When the assignment of weights to centroids is unchanged between the two passes, the second-pass quantized weights differ by Δ=W^2−W^1=PS(Z1−W^1)\Delta=\widehat{W}_2-\widehat{W}_1=P_S(Z_1-\widehat{W}_1) where PSP_S is the weighted cell-averaging projector. In other words, each centroid shifts by the Hessian-weighted mean rounding error accumulated by its cell along the first-pass trajectory. The true loss then satisfies

L(G2)−L(G1)=−2 ⟨E1X,ΔX⟩F+∥ΔX∥F2,\mathcal{L}(\mathcal{G}_2)-\mathcal{L}(\mathcal{G}_1)=-2\,\langle E_1X,\Delta X\rangle_F+\Vert\Delta X\Vert_F^2 ,

where E1=W∗−W^1E_1=W^{*}-\widehat{W}_1, so that E1XE_1X is the first-pass output error.

The second pass improves the layer if and only if ⟨E1X,ΔX⟩F>12∥ΔX∥F2\langle E_1X,\Delta X\rangle_F>\frac{1}{2}\Vert\Delta X\Vert_F^2. This answers both questions: (1) a layer worsens precisely when the displacement term ∥ΔX∥F2\Vert\Delta X\Vert_F^2 outweighs the alignment between the correction and the first-pass output error, and (2) the per-layer acceptance test described above prevents this. The bound in Eq. 7 follows by applying Cauchy–Schwarz to this identity.

AuthorsOpenReview ↗

Continuation of the reponse

W3 – Evaluation metrics. We expanded all new runs to report full-vocabulary KL divergence relative to the FP16 model, top-1 token agreement, per-question prediction flips, WikiText-2 and C4 perplexity, and zero-shot accuracy. All metrics move consistently in the same direction as perplexity (mean across 6 models and 3 seeds; lower is better except for top-1 agreement and accuracy).

Looking at paired comparisons across all 18 model–seed pairs:

  • With GPTQ: 2PTC lowers KL divergence in 17/18 (2-bit), 18/18 (3-bit), and 18/18 (4-bit) cases, and reduces C4 perplexity in 18/18, 18/18, and 16/18 cases, respectively.
  • With QuIP: The 2-bit gains are equally consistent (18/18 across metrics), while 3- and 4-bit regimes behave as statistical ties.

We will include the complete set of tables and layer-wise diagnostics in the revised manuscript.

Results table — means across 6 models and 3 seeds:

EngineBitsMethodPPL WikiPPL C4KLTop-1 (%)Accuracy (%)Flips (%)
GPTQ21 pass17.0415.681.07459.464.220.0
GPTQ22PTC15.0513.830.88062.967.017.1
GPTQ31 pass6.2578.2170.15284.976.56.22
GPTQ32PTC6.2028.1690.14385.576.75.81
GPTQ41 pass5.6277.5750.05291.478.03.66
GPTQ42PTC5.6057.5570.04991.778.13.52
QuIP21 pass10.3811.920.61769.269.414.2
QuIP22PTC9.8011.460.56470.670.013.5
QuIP31 pass5.9717.9650.10887.377.04.91
QuIP32PTC5.9677.9540.10787.477.14.77
QuIP41 pass5.5057.4610.03193.378.22.52
QuIP42PTC5.5057.4600.03193.378.12.51

Dear Reviewer AJv4,

Thank you again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

Reviewer_CLLxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes 2-Pass Trajectory Calibration (2PTC) for Post-training quantization of llms, which recalculate the codebook of compensated weights and run a second quantization process. Experiments over different models and bit-widths show improvement over fixed codebooks without adding inference overhead.

Critical strengths
  1. The motivation is clear and easy to follow
  2. The experiments are expensive, covering different models and frameworks.
Critical weaknesses
  1. The advantage over dynamic group-wise calibration remains unclear. Per-group quantization can partially address the mismatch problem by calculating the quantization scales/codebook on already compensated weights.
  2. In Appendix D.4, results show that the proposed method is consistently harmful for group-wise quantization. As per-group quantization formats such as NVFP4 and MXFP4 become increasingly prevalent, the general applicability and impact of this method may be limited.
  3. The method requires running the quantization process a second time, which is time-consuming, especially for large models
Questions for authors

please see the weakness above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer CLLx

W1 – Advantage over dynamic group-wise calibrations. We agree that dynamic per-group calibration can partially absorb the trajectory mismatch. When per-group scales or codebooks are computed dynamically as each group is reached, they are fitted directly on already-compensated weights and therefore partially absorb the trajectory mismatch, which also explains the behavior observed in Appendix D.4. However, this dynamic adaptation comes with memory overhead. Standard group-wise quantization (g=128g = 128) requires storing 0.14–0.25 extra bits per weight. In contrast, the per-channel codebooks store a much smaller number of extra bits per weight, from 0.016 bits (at 2-bit) to 0.063 bits (at 4-bit) for channel size =4096= 4096.

To better understand the advantage of 2PTC over group-wise calibration, we evaluate across 6 models and both GPTQ and QuIP:

  • At 2 bits: Per-channel 2PTC achieves lower perplexity than uniform g128 in all cases (−11%-11\% to −68%-68\%).
  • At 3 bits: 2PTC outperforms uniform g128 in 10/12 cases, using fewer bits per weight.
  • At 4 bits: g128 yields a 0.5–2.5% perplexity advantage, but at the cost of 0.07–0.22 additional bits per weight.

WikiText-2 perplexity, average 3 seeds (bits per weight in brackets):

EngineModel2-bit: 2PTC (2.01)2-bit: g128 (2.25)3-bit: 2PTC (3.03)3-bit: g128 (3.25)4-bit: 2PTC (4.05)4-bit: g128 (4.25)
GPTQLlama-2-7B10.6722.186.176.195.695.64
GPTQLlama-2-13B8.7310.915.455.375.094.96
GPTQLlama-2-70B5.816.963.823.843.493.42
GPTQLlama-3-8B35.36101.508.179.156.756.77
GPTQMistral-7B13.7315.295.975.805.495.35
GPTQSheared-2.7B14.1920.247.687.687.147.05
QuIPLlama-2-7B9.5315.326.056.095.635.59
QuIPLlama-2-13B7.5610.065.305.335.004.97
QuIPLlama-2-70B5.196.463.713.803.433.41
QuIPLlama-3-8B16.0350.057.507.796.536.48
QuIPMistral-7B8.7111.655.705.745.375.35
QuIPSheared-2.7B11.7517.207.517.577.077.04

W2 – Group-wise formats (NVFP4/MXFP4). We agree with the reviewer that 2PTC is not designed for group-wise quantization, and that its benefit at 4 bits, the currently most common setting in industry, is modest. While 4-bit quantization is the current industry standard, the aim of this line of research is to push bit-widths as low as possible while retaining model performance. Our goal, however, is not to claim universal applicability, but to identify a concrete, well-defined scope where the method is useful.

That scope is the setting where memory is the main constraint, i.e., when the goal is to reduce the total size of the model as much as possible. This is achieved with per-channel quantization (which, as discussed in W1, minimizes metadata overhead) at very low bit-widths. It is precisely there that 2PTC gives its largest gains. At 2 bits, per-channel 2PTC reduces perplexity by 13.4% with GPTQ and 5.2% with QuIP with respect to a single pass (average over 6 models and 3 seeds). In addition, it outperforms group-wise quantization with g=128g = 128 while using fewer bits per weight.

In summary, 2PTC is especially suited to memory-constrained deployment, where models must be as small as possible. We will state this scope explicitly in the revised version of the paper.

W3 – Computational cost of the second pass. While 2PTC introduces a second quantisation step, it is significantly cheaper than running the quantization pipeline twice. Specifically, the second pass reuses both the Hessian matrices and the calibration activations computed in the first pass, eliminating any additional forward passes through the model; it only repeats the lightweight per-layer GPTQ update and kk-means fitting.

End-to-end runtime on the same compute node (excluding model loading and evaluation; mean over 2/3/4 bits and both GPTQ and QuIP, with QEP):

  • Llama-2-7B: 1.72×1.72\times single-pass time (10.5→18.010.5 \to 18.0 min)
  • Llama-2-13B: 1.63×1.63\times single-pass time (16.8→27.416.8 \to 27.4 min)
  • Llama-2-70B: 1.55×1.55\times single-pass time (75→11975 \to 119 min)

Notably, the relative overhead decreases as model size grows. Furthermore, this is strictly a one-off offline cost paid prior to deployment, introducing zero latency or memory overhead at inference time.

Dear Reviewer CLLx,

Thank you again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.