acceptodds
Under review as a conference paper at ICLR 2027

Beyond 4 Bits: Data-Centric Post-Training Quantization and Recovery

Abstract

Post-training quantization (PTQ) aims at compressing the weights of a large trained model using a limited compute and data budget. Down to 4 bits, state-of-the-art PTQ methods are near-lossless; below this limit, the tested scalar PTQ configurations collapse to near-random accuracy and model quality becomes acutely sensitive to choices such as the quantization representation and the calibration data. At such low bitwidths, recovery fine-tuning after quantization becomes crucial, and its effectiveness depends heavily on these same choices. Yet there is little systematic evidence on how the quantization method, the quality and task alignment of calibration data, and the design of the recovery procedure interact to determine the best final 2-bit model. We present a systematic study of 2-bit quantization and recovery fine-tuning across multiple model families, scales, and data distributions, and distill our findings into a practical, architecture-dependent set of best practices for efficiently obtaining an accurate 2-bit model. This recipe, which combines a vector codebook representation, a small task-aligned calibration mixture, and a short codebook-only distillation stage, restores %–% average accuracy across seven question-answering tasks at 2 bits (a %–% bf16 ceiling), versus the % random accuracy floor of scalar PTQ (more so on MoE than dense models), at roughly half the compute budget of the strongest Quantization Aware Training (QAT) baseline. Our contribution is empirical: a systematic, architecture-aware map of the 2-bit design space, together with released code, Hessian mixtures, and checkpoints that make the findings directly actionable and reproducible.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation or frontier models, including LLMs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_viTJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper empirically compares various existing 2-bit quantization methods and provides a recipe for how to utilize these methods to quantize existing LLMs to get good performance. Some of the methods that the paper looked at are as follows: SpinQuant, QuaRot, VPTQ, YAQA, AQLM, GSQ, and BitDistiller. The general recommendation of the paper is to use vector representations instead of scalar representations, YAQA quantization for dense models, and VPTQ for MOEs.

Critical strengths

None.

Critical weaknesses
  1. The paper reads more like a survey of existing methodology. It is unclear why the same conclusion cannot be reached by reading the original methods.
  2. The empirical suggestions provided in the paper do not offer a proper theoretical explanation of why the recipe works. The results are also not surprising.
Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Response to Reviewer viTJ

Thank you for reviewing our paper.

W1 (reads like a survey; conclusions available from the original papers). Our work goes well beyond a survey by providing a unified empirical evaluation of 2-bit quantization methods under unified protocol, practically relevant evaluation methods and a practical actionable recipe for researchers. Each method paper evaluates one quantizer under its own protocol, bit accounting and models, so their results cannot be compared fairly. Our findings come from experiments on each part of the quantization pipeline, including representation, calibration, recovery, and evaluation, under a unified protocol. We summarize our key contributions below. They are now supported by additional experiments that other reviewers recommended.

  • Calibration and recovery data interact. The best recovery data changes with the calibration data. We also confirm this on a held-out model (Figure 2; Tables 25-27).
  • Vector PTQ beats all the trained scalar baselines we study on generation tasks, including the newly evaluated EfficientQAT given the same data, at a fraction of the training compute (Tables 4, 5, 11). On Qwen3-MoE, its generation lead over EfficientQAT is within noise.
  • Zero-shot QA averages hide most of the 2-bit damage, which shows up in generation (Sec. 4.4).

We believe that our results set a new standard for how 2-bit PTQ pipelines should be built and evaluated, which no single method paper can provide. Our comparisons show that bit rates must count codebooks and scales, especially on MoEs, that QA and generation must be reported separately, and that calibration and recovery data must be chosen jointly. With thorough experiments in a unified framework, we present a unified recipe for choosing the appropriate representation, calibration and recovery data and evaluation protocol for practitioners and researchers.

W2 (no theoretical explanation; results not surprising). To explain what we observe in practice, we turn to rate-distortion theory for a theoretical justification (Section 3.2). At 2 bits, a scalar grid has a fixed gap to the rate-distortion bound that a vector codebook can partly close. Measured errors on rotated real weights follow this ordering (Appendix B.1, Table 1). Equations 2-3 explain the calibration result, because data enter only through a mixture of per-domain Hessians. Held-out-domain perplexity shows the calibration-recovery coupling directly (Appendix C.2). Furthermore, we show that tuning calibration and recovery data separately, the usual practice, picks the wrong recovery data, which we predicted before running the held-out model (Table 27). We also show that evaluation on generation tasks is needed to capture the quality degradation at 2 bits. Based on our thorough and extensive experiments, practitioners now have an actionable recipe for choosing calibration and fine-tuning data and for evaluating 2-bit quantization methods.

Thank you again for your review. We hope we have addressed your concerns. If you have any further questions or comments, we would be happy to address them and engage in further discussion.

Dear Reviewer viTJ,

A gentle reminder that the discussion closes on November 18. Our response explains what the unified protocol adds beyond the original method papers and adds a rate-distortion explanation of the 2-bit collapse (Sec. 3.2). We would be happy to answer any further questions you may have. If your concerns are resolved, we would appreciate it if you would consider updating your assessment.

Best regards,

The authors

Reviewer_x4KGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what it actually takes to get a usable 2-bit weight-only LLM. Its framing claim is that three choices are normally hidden in implementation detail: the quantization representation, the calibration data, and the recovery fine-tuning procedure are irrelevant at 4 bits but decisive and coupled at 2 bits, so studying any one alone misleads. The contribution is a sweep rather than a new algorithm. The authors fix an evaluation protocol (a seven-task QA average, and a generation average over GSM8K and HumanEval) and cross it with scalar PTQ (SpinQuant, QuaRot at group 128), vector PTQ (VPTQ, YAQA, AQLM), two trained scalar baselines (GSQ, BitDistiller), four calibration domains plus convex mixtures obtained by reweighting precomputed Hessians, and a recovery stage that distills from a full-precision teacher while updating only the continuous codebook and freezing the discrete assignments. The main results are: scalar PTQ collapses to roughly 29% QA and zero generation at 2 bits on all three primary models, while vector codebooks give 63-68% QA; vector PTQ beats both trained scalar baselines by 5 to 15 QA points at two to three orders of magnitude less compute; calibration composition moves generation far more than QA; codebook-only distillation adds about two QA points on Phi-4-mini with large generation gains, at roughly half the compute of BitDistiller; and QA retention is a uniform 90% across model families while generation retention spans 51% to 101%, with the MoE at the top of that range. Supporting validation includes a bit-matched VPTQ control at 2.25 bits per weight, a QTIP ablation that isolates the two-sided Fisher Hessian, a router-precision ablation, two auxiliary models, and a decoding analysis on MATH-500 showing a repetition penalty of 1.3 restores 2-bit accuracy to the full-precision level. The output is a practical decision guide.

Critical strengths

The cost accounting, and the controls that separate representation from bit budget. Most low-bit papers compare checkpoints and leave the reader to assume the comparison is fair. This work audits algorithmic FLOPs per method from the actual run configurations, clearly shows where the accounting is uncertain (BitDistiller and GSQ differ by 2.7%, which the authors call smaller than the uncertainty of their own estimate), separates three distinct notions of memory, and then runs the two controls readers would demand. Raising VPTQ from 2.00 to 2.25 bits per weight closes about three quarters of its gap to YAQA but leaves a 2.6-point residual; and swapping YAQA's two-sided Fisher Hessian for QTIP's single input-side Hessian, holding the codebook, rotation and per-domain Hessians fixed, isolates the rounding objective. These are what make the "representation, not bitwidth" claim credible.

Splitting QA from generation, and diagnosing the decoding failure rather than assuming capability loss. The finding that QA retention sits near 90% across every family while generation retention spans 51% to 101% is the paper's most transferable result: it means reporting a zero-shot QA average systematically understates 2-bit damage. The follow-through is better still. Instead of attributing the MATH-500 drop to lost capability, the authors measure the hesitation-marker rate and the four-gram loop score, show both roughly double at 2 bits, and demonstrate that a repetition penalty of 1.3 recovers full-precision accuracy while halving the loop score. The qualitative example of a 2-bit chain consuming its entire token budget without ever emitting an answer makes the point concrete. That reframes a meaningful part of the reported regression as a decoding artifact, which is both practically actionable and a caution to anyone benchmarking low-bit reasoning models.

Critical weaknesses

The central claim is made without a competitive scalar 2-bit baseline. The paper states that the tested scalar PTQ methods collapse to the random-accuracy floor in every family, then generalises to "every evaluated scalar method collapses.". However EfficientQAT, which is cited in the related work but never run, reports a 2-bit Llama-2-70B at 69.48 against 72.41 in full precision (arXiv:2407.11062), and ParetoQ places 2-bit scalar networks on the size-accuracy frontier (arXiv:2502.02631). The metrics differ from this paper's QA average, so these are not drop-in numbers, but they sit far enough from 29% that the cliff looks like a property of the configurations chosen, not of scalar representation as such. Describing BitDistiller as "the strongest QAT baseline" likewise means the strongest of two.

The framing hypothesis is never tested by experimental design. The introduction argues the three choices are strongly coupled and that studying any one in isolation gives a misleading picture. Yet the experiments are strictly one-factor-at-a-time throughout, in the recovery protocol, in every ablation block and in the aggregate tables. No representation-by-calibration-by-recovery cell is reported and no interaction is quantified anywhere. The coupling premise is the paper's reason for being a joint study rather than three sweeps, and it is asserted rather than shown.

The cross-method rankings are protocol-dependent, and there is no noise floor. The headline table mixes three selection rules: best-over-recipe for VPTQ and YAQA, one fixed 50/50 recipe for AQLM, and an enlarged mixed corpus for the trained scalars. The appendix then concedes that finer VPTQ mixtures on the MoE beat anything in the headline table and, if pooled, would lift VPTQ's best MoE generation above YAQA but excludes them. Every "plus X points" in the comparison section, and every row of the decision guide, is therefore grid-dependent. There is also a single seed and no error bars or significance tests anywhere, yet the headline inference compares ranges: generation spans 13 to 27 points across recipes while QA spans 4 to 8. The generation average rests on two tasks, one with 164 problems, and the only variance disclosed is a 7.5-point gap between two runs of one MATH-500 configuration. Recovery also loses QA on Llama, 65.7 down to 64.8 and 64.6, at roughly 160 times the base quantization compute.

Questions for authors

Can you add one competitive 2-bit scalar baseline at a comparable budget? EfficientQAT at w2g64, or ParetoQ's 2-bit setting. If the vector quantization still wins by a wide margin, the representation claim gets much stronger. What evidence supports the coupling premise? The best and worst calibration recipe crossed with scalar versus vector, or with and without recovery would show whether the effects are additive or genuinely interacting. What is the run-to-run spread of your QA average, GSM8K and HumanEval? Since the calibration and recovery conclusions argue from range comparisons, two or three seeds on one model would tell how much of the generation span is signal. Why does codebook-only recovery reduce the QA average on Llama-3.1-8B while helping the other two models? Is this an artifact of the shared learning rate or data scale, or a property of that base checkpoint? The recipe is currently stated unconditionally. The cross-paradigm appendix lists the Qwen3-MoE BitDistiller checkpoint at 53.8 QA average, while the setup section and the main comparison table give 55.9. Which is the reported checkpoint? Is the GSQ collapse to the random floor on both MoE models a method limitation or an implementation failure? The appendix diagnosed exactly such a false collapse for OLMoE, so what invariance check did the GSQ MoE exports pass?

2 comments
AuthorsOpenReview ↗

Response to Reviewer x4KG

Thank you for the detailed review of our paper, and for recognizing the cost accounting and the experimental protocol that separates representation from bit budget, as well as the split of QA from generation, which you called the paper's most transferable result. Corrections C1-C14 are listed in the supplementary PDF. We use W for weaknesses and Q for questions and address each of your points below.

W1/Q1 (competitive scalar baseline). We added EfficientQAT (w2g128 at YAQA's bit rate, and w2g64) with our data on all four models (Tables 4, 22). It avoids the collapse but trails YAQA on generation on all models (within noise on Qwen3-MoE) and on QA on three of four (Sec. 4.3, Table 5). The collapse claim now covers only the studied scalar PTQ pipelines (Sec. 4.2). BitDistiller is now "a full-model QAT baseline" rather than "the strongest QAT baseline". Published EfficientQAT and ParetoQ results use their own data, budgets and metrics. ParetoQ's full-model QAT budget is far beyond those compared here, so we study EfficientQAT under our protocol (Appendix B.7). Our work focuses on obtaining a 2-bit model at a limited compute budget, and the evaluated methods now reflect this constraint.

W2/Q2 (coupling not tested). We now test the coupling directly (General response (2)). The best recovery data changes with the calibration data on three models, one of which is held out (Figure 2, Section 4.5; Tables 25-30). Representation and calibration also interact. With EfficientQAT, the best recovery data does not change (Appendix B.7).

W3 (selection rules). Table 4 (submitted Table 3) now selects VPTQ and YAQA from the same five calibration domains. The submitted table had selected YAQA from more recipes, including Hessian mixtures. Under the shared rule, YAQA's QA lead over VPTQ is 5.7 and 1.8 AvgQA on the dense models and 1.2 on Qwen3-MoE (Table 4). AQLM is now removed because it used a single recipe and exceeded the budget on the MoEs (correction C3). The finer MoE VPTQ mixtures were over budget because they stored one codebook per expert (correction C1). In budget, the best of them reaches 67.2 AvgQA but only 55.2 AvgGen, well below YAQA's generation (Table 41).

W3/Q3 (noise floor, significance). General response (4) describes the new evidence. We report seed standard deviations (Figure 2, Tables 25-27) and independently quantized bases (Table 29). We also run an item-level paired bootstrap of the performance gaps between methods, which covers the small HumanEval set (Table 5; Sec. 4.3). Seed spread is far below the calibration span (Table 6).

W3 (MATH-500). The 7.5-point gap compared two 40-problem runs. Table 47 now uses all 262 level-4/5 problems, the in-budget checkpoint and paired intervals (correction C5, the rerun MATH-500 decoding study).

W3/Q4 (Llama QA loss; cost). Thank you for raising this point. We found a RoPE configuration error in our Llama recovery pipeline due to a transformers version mismatch (correction C10). With the corrected configuration, short and long recovery raise Llama QA and generation (Tables 7, 26). The 160x figure compares long recovery with base quantization alone. Quantization plus long recovery costs about half as much as BitDistiller (Tables 10-11), and short recovery uses about 100x fewer tokens (Table 35).

Q5 (which Qwen3-MoE BitDistiller checkpoint is reported?). We apologize for the confusion. The reported checkpoint scores 55.9. The 53.8 was a stale value from an older checkpoint, and Table 14 now shows 55.9 (correction C7).

Q6 (GSQ on MoEs). Thank you for this question. Following the paper's procedure, we initially applied GSQ with rotation and norm fusion on the submitted exports. Papers such as QuaRot and SpinQuant do not study rotations on MoEs, and we found that our rotation procedure broke OLMoE. The rotated, unquantized model was not invariant to the rotation (Table 23). The Qwen run used a different model variant. Hence, we withdrew the older GSQ collapse results on the MoEs (correction C11) and reran GSQ without rotation on the paper's checkpoints. It does not collapse and stays below YAQA (Tables 4, 5; Appendix B.8).

Additional updates. Two highlighted results changed. The bit-matched VPTQ experiment (a separate experiment on Phi-4-mini, not the main-table VPTQ) previously stored one codebook per linear layer at about 3.6 effective bits. It now shares one codebook across layers at exactly YAQA's 2.26 bits, and YAQA leads consistently (Table 17; correction C2). The repetition penalty also helps BF16, so a 2-bit gap remains (Table 47; correction C5, the MATH-500 study rerun at each studied precision).

Thank you again for your review, which helped us improve the clarity and completeness of our experimental evaluation. We hope we have addressed your concerns. If you have any further questions or comments, we would be happy to address them and engage in further discussion.

Dear Reviewer x4KG,

A gentle reminder that the discussion phase closes on November 18. Our response adds EfficientQAT with matched data (Tables 4, 22), a direct test of the calibration-recovery coupling (Figure 2), and significance tests (Table 5). We are glad to answer any further questions you may have and engage in further discussions with you. If your concerns are resolved, we would appreciate it if you would consider updating your assessment.

Best regards,

The authors

Reviewer_b1vzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper empirically studies approximately 2-bit weight quantization of language models, examining how the quantization representation, calibration data, and subsequent recovery fine-tuning affect performance. It compares scalar and vector quantizers, including trained scalar baselines, across three primary models spanning dense and mixture-of-experts architectures, with two auxiliary models. The main findings are that vector representations preserve substantial QA accuracy where the tested scalar PTQ configurations collapse, calibration composition affects generation more strongly than aggregate QA, and distillation that updates only the codebook can recover generative performance, with benefits depending on the model and task. The contribution is an empirical characterization of these tradeoffs and a practical recipe combining vector quantization, task-aligned calibration, and codebook recovery. Validation includes seven QA benchmarks, mathematical reasoning, code generation, and instruction following, together with calibration-mixture sweeps, controls for index rate, rounding curvature, and router precision, recovery ablations, and estimated training-compute comparisons.

Critical strengths
  1. The calibration sweeps demonstrate that data composition can substantially affect generation at a fixed quantization format, while aggregate QA changes much less. For example, Table 31 reports that changing the web/reasoning mixture from 50/50 to 70/30 raises HumanEval from 31.7 to 53.0, with AvgQA moving from 67.0 to 68.4. These comparisons identify a consequential experimental variable and show why QA alone can give an incomplete picture of a compressed model's performance. This finding is the strongest reason for my positive assessment.

  2. The recovery experiments show that tuning only the codebook, while freezing the discrete assignments and preserving the storage format, can recover substantial generative performance. For example, both reported Phi web-recovery experiments raise HumanEval from 8.5 to 19.5. Together with the comparisons across models and recovery settings, these results help practitioners assess when restricted fine-tuning is worth pursuing and which capabilities it may improve. This adds practical value to the empirical study.

Critical weaknesses
  1. The relationship between the central recovery experiments is insufficiently documented. Tables 5 and 15 report identical unrecovered baseline scores, but their reasoning-recovery rows give HumanEval scores of 36.6 and 10.4, respectively. Appendix A.8 refers to different reference runs, yet the starting checkpoints, recovery data, and training budgets remain difficult to identify. This matters because the magnitude and task dependence of recovery gains are central to the paper's recommendations. A precise account of these runs would substantially improve confidence in the conclusions.

  2. The recommendations about data selection extend beyond the controlled comparisons. The mixture sweeps establish substantial sensitivity to calibration composition, but the claims that quality matters more than quantity and that calibration and recovery should be chosen jointly receive weaker experimental support. The limited specification of the corpora and filtering procedures also makes the proposed recipe difficult to reproduce or apply elsewhere. This limits the practical reach of the paper's main contribution. More complete documentation and narrower claims could address much of this concern.

Questions for authors
  1. Clarification of the recovery comparisons. Could you reconcile the Phi results in Tables 5 and 15? Both report the same unrecovered baseline scores, but reasoning recovery gives HumanEval scores of 36.6 and 10.4, respectively. Appendix A.8 refers to different bases when explaining the web-reference discrepancy. Could you identify the starting checkpoint for each table and specify what differs in the recovery data, training budget, checkpoint selection, or evaluation? This would help establish the magnitude of the recovery benefit and whether the reversal reflects an interaction with calibration.

  2. Clarification of the data-selection evidence. Could you identify the calibration and recovery datasets, filtering procedures, and token budgets used in the principal comparisons? Which existing comparisons support the stronger claims that quality matters more than quantity and that calibration and recovery should be chosen jointly? I am asking which conclusions the reported experiments establish. Additional matched-budget or crossed experiments could be left to future work if the claims are narrowed accordingly.

  3. Clarification of Figure 1(a). Could you identify the task set, quantizers, and calibration recipes used in this panel and reconcile them with the caption and Appendix A.21? The plotted BF16 value of 72.4 matches the five-task average reconstructed from Table 15, whereas the caption describes seven-task results consistent with panel (b) and Table 11. The scalar legend names KronQ and rotation plus GPTQ, while the caption names SpinQuant. The YAQA point is labeled "web" and scores 65.6, whereas Appendix A.21 identifies the checkpoint as rw64, whose five-task average from Table 15 is 64.22. Is panel (a) intended to show a separate experiment?

2 comments
AuthorsOpenReview ↗

Response to Reviewer b1vz

Thank you for the review and for your positive assessment. We appreciate that you found the calibration sweeps, which show that data composition substantially changes generation at a fixed format, and the codebook-only recovery results useful for practitioners.

W1/Q1 (Tables 7 and 34; submitted Tables 5 and 15). They recover different bases. Table 34 recovers one fixed base (math+web 0.6/0.4, "rw64") with each recovery source. Each Table 7 row recovers the base calibrated on its own recipe, although the submitted Table 5 had the rw64 base for all rows (correction C8, which restores each row's own base). The Table 7 reasoning row thus starts from a base with much higher HumanEval. Table 35 and Appendix D.2 ('Relation between Tables 7 and 34') give the recovery data, the budgets (about 100x apart), the checkpoint selection (final checkpoint in both) and the evaluation. The interaction you suspected holds strongly in practice and is now tested directly (General response (2); Figure 2).

W2/Q2 (data documentation and claims). The data card (Appendix D.2) and Table 35 give the sources, pool sizes, filtering and token budgets. Table 31 reports the benchmark overlap. We revised the three claims as follows. (i) Calibration moves generation more than QA, as Tables 6 and 24 show. (ii) Calibration and recovery should be chosen jointly. The crossed design now tests this claim (Figure 2, Tables 25-30), and we scope it to one domain pair at a short budget (Sec. 4.5). (iii) Our fixed-budget sweeps do not establish that quality matters more than quantity, so we removed this claim (Supplementary Table S2).

Q3 (Figure 1(a)). The submitted figure points to an earlier version of the experiment, which used a five-task average, KronQ and rotation+GPTQ, and another YAQA checkpoint. The revised panel plots exactly the caption's data. It uses the seven-task AvgQA, SpinQuant with web calibration, and the rw64 YAQA checkpoint. The conclusion remains unchanged (correction C9, which regenerates Figure 1(a) from the data its caption describes).

Thank you again for your careful review and positive assessment. Your questions on the recovery runs and data documentation led us to test the calibration-recovery interaction directly, which is now a central result of the paper. We hope we have addressed your concerns. If you have any further questions or suggestions, we would be happy to address them and engage in further discussion.

Dear Reviewer b1vz,

A gentle reminder that the discussion closes on November 18. Our response pairs each recovered model with its own base (Table 7), adds a data card (Appendix D.2), and redraws Figure 1(a) to address your primary questions. We are happy to answer any further questions you may have and engage in discussions before the end of the rebuttal period.

Best regards,

The authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3Bs5 (part 1 of 2)

Thank you for the review, and for finding the central result, that the 2-bit collapse is mainly a representation failure, clear and practically useful, and the separation of QA from generation a valuable caution against judging low-bit models on QA averages alone.

W1 (limited novelty). Our primary contribution is an actionable calibration, recovery and evaluation protocol for 2-bit quantization methods. We believe its primary value lies in conclusions that the field cannot draw from the existing method papers. Each quantizer paper reports results with its own data, bit accounting, models and metrics, so the literature cannot tell which stage of the PTQ pipeline actually determines 2-bit quality. Our work answers this question under a single protocol. We believe the paper makes three contributions that are new and impactful for the field.

First, we identify a crucial interaction in the PTQ pipeline. Calibration data and recovery data are usually tuned separately, but the best recovery data changes with the calibration data. The interaction is -8.5, -5.0 and -9.4 AvgGen on Phi-4-mini, Llama-3.1-8B and Qwen3-8B, and all 95% intervals exclude zero. We stated three predictions before running the held-out Qwen3-8B, and all three held (Figure 2, Table 27). Tuning each stage in isolation therefore picks the wrong recovery data. This affects any pipeline that combines post-training quantization with fine-tuning.

Second, we establish the evaluation protocols that a fair 2-bit comparison requires. (i) Bits must be accounted for properly on MoEs. Vector quantizers store codebooks in addition to indices, and a separate codebook per expert multiplies this storage by the number of experts. A nominally 2-bit VPTQ checkpoint with per-expert codebooks therefore stores 7.23 bits per weight on Qwen3-MoE. We count indices, codebooks and scales for all methods, and with codebooks shared across experts the MoE runs stay at 2.09-2.14 bits, so all methods are compared at the same budget (Tables 9, 41). (ii) QA and generation accuracy must be reported separately for quantized models. QA retention stays at 89-94% of BF16 in each model family, while generation retention spans 51-100% (Sec. 4.4). Calibration moves QA by only 3.9-7.4 points but generation by 13.5-26.7 points (Table 6). A QA average alone therefore hides most of the damage and most of the differences between pipelines. (iii) Calibration and recovery data must be selected jointly. Because the best recovery data depends on the calibration data, recovery data should be chosen on the base calibrated with the intended recipe, and both choices should be judged on generation rather than on QA, which is nearly flat across recipes (Figure 2, Tables 6, 25).

Third, we give a compute-aware recipe that practitioners can follow directly. Vector PTQ with task-aligned calibration and short codebook-only recovery outperforms all trained scalar baselines we study on generation (within noise against EfficientQAT on Qwen3-MoE), at a fraction of their training compute (Tables 4, 5, 11). The resulting 2-bit weights cut weight memory by 3.2-4.2x (Table 13).

We believe this is impactful for the field because 2-bit quantization is moving from research into deployment, while choices of quantizer, data and evaluation are still made from results that cannot be fairly compared. The most common way to report 2-bit results, a QA average, does not fully capture the damage that 2-bit quantization causes and can change which method or data appears best. Our protocol and findings give the community a common reference for fair comparison and a clear protocol for making pipeline decisions. General response (1)-(12) lists the experiments that test these findings.

Our responses to W2-W4 and Q1-Q2 continue in part 2 of this response.

AuthorsOpenReview ↗

Response to Reviewer 3Bs5 (part 2 of 2)

W2 (unequal bit rates). We now report effective bits counting indices, codebooks and scales for each method (Sec. 4.1, Table 9). We found that VPTQ and AQLM used per-expert codebooks for the MoEs, so these entries exceeded the budget. MoE VPTQ now uses in-budget expert-shared codebooks, and AQLM is removed (Table 41; corrections C1 and C3, which cover the over-budget per-expert codebooks). The bit-matched VPTQ experiment on Phi-4-mini, which is separate from the main-table VPTQ, previously stored one 2^18-entry codebook per linear layer at about 3.6 effective bits. It now shares one codebook across layers and matches YAQA's effective bits exactly, and YAQA is the recommended 2-bit vector quantization recipe across our experiments (Table 17; correction C2).

W3 (seeds, latency, memory). General response (4) covers seeds and significance. We now measure latency and memory for different 2-bit formats on one H100 GPU (Appendix A.5, Table 13). The 2-bit weights cut weight memory several-fold, and decode speed depends on the kernels available and the batch size.

W4 (recovery gain only +0.7). The 61.8 base is the math+web checkpoint of the ablations, not the web-calibrated base. In the submitted version, Table 5 listed it for all rows, although each row recovered the base calibrated on its own recipe (correction C8). Table 7 (submitted Table 5) now shows each recovered model next to its own base. We agree that recovery adds little QA, and the gain is within noise on the web base. Its value is primarily on generation, where all gains are significant (Table 33), at a much lower cost (Table 7, Figure 2). Table 32 shows that the generation gains also hold under the chat template.

Q1 (Llama recovery). Thank you for your question. On closer inspection, we discovered that the drop came from a RoPE configuration error in our Llama recovery pipeline due to a transformers version mismatch (correction C10). With the corrected configuration, recovery improves Llama QA and generation in all cells (Table 26) and in the long runs (Table 7).

Q2 (IFEval 39.2). The caption value was leftover text from an earlier version and is removed (correction C6). Table 7 now lists IFEval before and after recovery, and Table 45 pairs the base with its own recovered model. Completion-style IFEval does not improve with codebook-only recovery, but IFEval under the chat template does (Table 32).

Thank you again for your review. Your questions on bit accounting, uncertainty and Llama recovery led to important corrections and made our comparisons fairer and more complete. We hope we have addressed your concerns. If you have any further questions or comments, we would be happy to address them and continue the discussion.

AuthorsOpenReview ↗

General response to all reviewers (part 1 of 2)

We thank all reviewers for their careful and constructive reviews, which helped us improve the paper. We have incorporated the detailed feedback from the reviewers in the updated version of the paper. In the revised paper, all additions and changes are marked in blue. In the response below, we summarize the primary changes in the paper.

We are grateful that the reviewers recognized several contributions of the paper. Reviewer x4KG noted that the cost accounting and the experiments that separate representation from bit budget make the "representation, not bitwidth" claim credible. Reviewer 3Bs5 found it clear and practically useful that the 2-bit collapse is mainly a representation failure and that vector PTQ beats full-model QAT. Reviewers x4KG and 3Bs5 valued splitting QA from generation, calling it the paper's most transferable result and a valuable caution against judging low-bit models on QA averages alone. Reviewer b1vz called the calibration sweeps, which show that data composition substantially changes generation at a fixed quantization format, "the strongest reason for my positive assessment". Reviewer b1vz also noted that codebook-only recovery preserves the bit-width while recovering generative performance, which helps practitioners decide when fine-tuning on a small budget is worth pursuing.

Additional Experiments and Evaluations

(1) Matched-data trained strong scalar baseline (Reviewer x4KG). We study EfficientQAT with its public code and hyperparameters on our calibration and recovery data for all four models (Table 4, Sec. 4.3; Appendix B.7, Table 22). The grid covers w2g128, which is comparable to YAQA's bit rate, and w2g64 on the dense models. It uses web data, math data and the combined web, math, instruction and reasoning mixture that the GSQ and BitDistiller baselines use, together with three end-to-end training settings (none, web and math). EfficientQAT avoids the scalar collapse and beats BitDistiller on all models. It trails YAQA on generation and on QA on the dense models and OLMoE. On Qwen3-MoE, its best-QA checkpoint leads YAQA by 3.0 AvgQA but trails YAQA on generation, within noise (Table 5).

(2) Calibration and recovery coupling (Reviewers x4KG, b1vz). We study a crossed calibration x recovery design with three recovery seeds on Phi-4-mini and Llama-3.1-8B and repeat it on a held-out model, Qwen3-8B (Figure 2, Sec. 4.5; Appendix C.2, Tables 25-27). The best recovery data changes with the calibration data on all three models. The negative interactions are -8.5, -5.0 and -9.4 AvgGen on Phi-4-mini, Llama-3.1-8B and Qwen3-8B, and all 95% confidence intervals exclude zero. Appendix C.3 tests robustness to held-out tasks, re-quantized bases and disjoint recovery samples (Tables 28-30). Appendix B.7 shows that representation and calibration also interact. Switching from web to math calibration helps generation more with YAQA than with EfficientQAT, by 15.2 [11.8, 18.6] AvgGen on Phi-4-mini and 7.7 [4.6, 10.9] on Llama-3.1-8B.

(3) Effective bits (Reviewer 3Bs5). We now count indices, codebooks and scales for each method. MoE VPTQ now uses in-budget expert-shared codebooks instead of the per-expert codebooks used earlier for VPTQ (Sec. 4.1, Table 9; Appendix E.3, Table 41).

(4) Noise floor and significance (Reviewers x4KG, 3Bs5). We report seed standard deviations, re-quantized bases and a paired item-level bootstrap of the performance gaps, calibration effects and recovery gains (Sec. 4.3 'Noise floor and significance'; Tables 5, 25, 29, 33). Following Reviewer x4KG's suggestion, we also evaluate MATH-500 on all 262 level-4/5 problems (Appendix F.6, Table 47).

(5) Mechanism and theory (Reviewer viTJ). Sec. 3.2 adds a rate-distortion argument, which we measure on real weights (Appendix B.1, Table 1). Appendix C.2 reports held-out-domain perplexity of the crossed cells.

(6) Documentation (Reviewers b1vz, 3Bs5). Table 7 shows each recovered model next to its own base, and Table 35 lists the base, data and budget of the recovery runs. Appendix D.2 adds a data card. Appendix C.3 adds a benchmark-overlap check and a chat-template evaluation (Tables 31-32).

(7) Deployment cost (Reviewer 3Bs5). We measure the weight memory and decode throughput of 2-bit formats (Appendix A.5, Table 13).

The list of additional experiments continues in part 2 of the general response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.