acceptodds
Under review as a conference paper at ICLR 2027

Multi-Bitwidth Quantization for LLMs Using Additive Codebooks

Abstract

As large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints, the ability to adaptively manage the performance-efficiency trade-off without retraining is critical. We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework enabling inference-time precision control over LLM weights from a single quantized model. As theoretical motivation, we prove that Gaussian sources are successively refinable under a weighted mean squared error distortion motivated by LLM loss functions: successive refinement across distortion levels incurs no rate penalty relative to encoding at each level separately. Drop-by-Drop approximates this hierarchical structure in practice by incorporating Matryoshka-style supervision into additive codebook training, inducing an ordering in which codebook prefixes yield accurate partial reconstructions at each precision level. Furthermore, a block-Hadamard rotation brings the weight distribution closer to our Gaussian source assumption. The result is a single model that serves multiple bitwidths by dropping codebooks, reducing storage and quantization cost relative to the static per-bitwidth models. Across Qwen, LLaMA, Gemma, and Mistral, Drop-by-Drop achieves lower perplexity than state-of-the-art multi-bitwidth methods with competitive zero-shot accuracy, while our specialized kernels further reduce decoding latency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi bit width quantization, adaptive inference, successive refinement, large language model compression, post-training quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1UVmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Drop-by-Drop, a multi-bitwidth post-training quantization framework that enables inference-time precision control from a single quantized model. The method incorporates Matryoshka-style supervision into additive codebook training, allowing lower-precision models to be obtained by dropping codebooks. The authors provide a theoretical motivation through successive refinability of Gaussian sources under weighted mean squared error. Across Qwen, LLaMA, Gemma, and Mistral, the method achieves lower perplexity than the evaluated multi-bitwidth baselines in most settings, while maintaining competitive zero-shot accuracy.

Critical strengths

The paper addresses a practical problem: supporting multiple precision levels without retraining or storing separate models.

The method is clear and shows substantial improvements over directly dropping codebooks from AQLM.

Experiments cover several model families and sizes, with generally consistent perplexity improvements.

The storage comparisons and specialized inference kernels strengthen the practical value of the work.

Critical weaknesses

1 The method builds on AQLM by introducing Matryoshka-style supervision over additive codebook prefixes. While effective, the methodological advance appears incremental, and the technical challenges addressed by this extension could be explained more clearly.

2 The theoretical result provides a useful motivation under Gaussian assumptions, but it does not establish a performance guarantee for the practical algorithm. The paper could clarify what the result explains about the observed performance.

3 At lower bitwidths, a noticeable gap remains compared with independently quantized models in some settings. Lower perplexity also does not consistently translate into better zero-shot accuracy.

4 Could the authors include an independently quantized AQLM baseline with the same Hadamard preprocessing to better separate the effects of rotation and prefix supervision?

Questions for authors

No further questions for the authors.

4 comments
AuthorsOpenReview ↗

1/4 Response by Authors

Thank you very much for your time and feedback. We appreciate your support.

Comment 1: The method builds on AQLM by introducing Matryoshka-style supervision over additive codebook prefixes. While effective, the methodological advance appears incremental, and the technical challenges addressed by this extension could be explained more clearly.

Re:
Thank you for this suggestion. We agree that the technical motivation deserves clearer explanation. Our key contribution is a successive-refinement-inspired multi-bitwidth additive quantization framework that integrates Gaussian-motivated Hadamard rotation with Matryoshka-supervised codebook refinement. This enables additive codebook prefixes to serve as progressively refined reconstructions at different precision levels.

We have a theory-motivated design. We prove that Gaussian sources are successively refinable under WMSE, establishing that nested representations can theoretically achieve optimal rate-distortion trade-offs at multiple precision levels. However, practical LLM weights deviate from Gaussianity. Motivated by this theorem, we introduce block-Hadamard rotation to multi-bitwidth quantization to bring weight distributions closer to the Gaussian assumption, as empirically supported by the reduced excess kurtosis in Appendix C. We then use Matryoshka supervision to realize nested precision through additive codebook prefixes.

Technical challenges of Matryoshka supervision: Unlike MatQuant and MatGPTQ, which exploit inherent integer bit-slicing, AQLM jointly optimizes additive codebooks without guaranteeing useful intermediate prefixes. Two challenges arise:

  1. Learning a codebook hierarchy: Simply dropping codebooks from AQLM causes severe degradation. Our prefix-level WMSE supervision explicitly induces an ordering in which earlier codebooks provide coarse reconstructions and later ones progressively refine them.
  2. Balancing multiple precisions: Since all precision levels share codebooks, optimizing one prefix can compromise others. We address this through selective loss weighting (35W), which outperforms uniform supervision (Appendix G).

These design choices are supported empirically. On Qwen2.5-3B at 3 bits, AQLM (Drop) yields a C4 perplexity of 283.97, compared with 14.94 for DbyD (Uniform), 14.32 for DbyD (35W), and 13.96 with Hadamard rotation.

We will revise Sections 4-5 to make the connection between successive refinement theory, Gaussian-motivated rotation, and Matryoshka-supervised additive quantization more explicit.

AuthorsOpenReview ↗

2/4 Response by Authors

Comment 2: The theoretical result provides a useful motivation under Gaussian assumptions, but it does not establish a performance guarantee for the practical algorithm. The paper could clarify what the result explains about the observed performance.

Re:
We agree that Theorem 1 is an asymptotic result for Gaussian sources, not a performance guarantee for the practical algorithm. We will revise Section 5 to clarify what the result explains about the observed performance:

  1. How can DbyD, a single model, approach the performance of separately quantized models? Theorem 1 shows that, under WMSE, a nested code can match separate codes at every rate, so nesting has no inherent cost. Consistent with this, at 4 and 5 bits on all models of 7B and above, DbyD (Had.+35W) is at most 0.09 C4 perplexity above individually trained AQLM (Ind.) and within one standard error in zero-shot accuracy.

  2. Why does supervising prefixes work? The code constructed in the proof is built in stages: v^i(k)=v^i(k−1)+zi(k)\widehat{v}_i^{(k)} = \widehat{v}_i^{(k-1)} + z_i^{(k)}, where v^i(k)\widehat{v}_i^{(k)} is the reconstruction of component ii after stage kk and zi(k)z_i^{(k)} is the refinement added at that stage. The key property is that every intermediate reconstruction v^i(k)\widehat{v}_i^{(k)} is a good code at its own rate. Our Matryoshka-style loss LDbyDL_{\text{DbyD}} enforces this property in additive codebooks: each term L(k)L^{(k)} is the distortion of the sum of the first kk codebooks. AQLM trains only the full sum, so the property does not hold and dropping codebooks fails (C4 perplexity 283.97 vs. 12.94 on Qwen2.5-3B at 3 bits); with LDbyDL_{\text{DbyD}}, the same setting reaches 14.32.

  3. Why does rotation help? Theorem 1 holds for Gaussian sources, while LLM weights are heavy-tailed. The block-Hadamard rotation brings the weights closer to Gaussian (average excess kurtosis drops from 2.48 to 0.50 on Qwen2.5-7B), and therefore closer to the setting of Theorem 1. Empirically, it narrows the gap to individually trained models on nearly all models and bitwidths.

AuthorsOpenReview ↗

3/4 Response by Authors

Comment 3: At lower bitwidths, a noticeable gap remains compared with independently quantized models in some settings. Lower perplexity also does not consistently translate into better zero-shot accuracy.

Re:

1. The gap to independently quantized models is concentrated at 3 bits and on small models, and it shrinks steadily with model size:

3-bit C4 (↓\downarrow)Q-0.5BQ-3BQ-7BQ-14BQ-32B
AQLM (Ind.)21.8712.9411.8510.019.46
DbyD (Had.+35W)27.6713.9612.2710.389.49
Gap5.801.020.420.370.03

Firstly, this gap is the cost of serving all three bitwidths from one quantization run and one checkpoint, where AQLM (Ind.) needs three of each. Secondly, it is a tradeoff the user controls: serving two bitwidths instead of three lowers 3-bit C4 perplexity on Qwen2.5-3B from 14.32 to 13.91 (App. G.3). Third, even with this gap, DbyD remains competitive with state-of-the-art multi-bitwidth methods at 3 bits. We already list lower bitwidths as a limitation in Section 7, and, as you point out, we will state that the gap is currently largest for small models at 3 bits.

2. Regarding the perplexity vs. zero-shot accuracy: The two metrics measure different things. Perplexity averages log-likelihood over every token, so it registers small, uniform losses. Zero-shot accuracy only asks whether the correct answer outranks a few alternatives, so a model can lose likelihood without flipping these rankings. This mismatch is known for compressed LLMs: multiple-choice accuracy can stay nearly unchanged while model behaviour differs [1], and it degrades more mildly than tasks where the model must generate the answer [2]. With a standard error of about 1 point, the five-task average cannot separate methods that are close: at 3 bits, DbyD's accuracy differences to the best baseline are 1.0 to 1.4 standard errors, so none is significant.

We have added additional benchmarks (Question 3 of Reviewer 3ZuX). On GSM8K at 3 bits, DbyD leads AnyPrecLLM, the strongest baseline, by 3.3 to 3.9 points on Mistral-7B, Qwen2.5-7B-Instruct, and Llama-3-8B, in line with the perplexity ordering.

3-bit GSM8K (%, ↑\uparrow)DbyD (3.02 bpw)AnyPrecLLM (3.04 bpw)Difference
Mistral-7B25.80 ± 0.7922.49 ± 1.46+3.31
Qwen2.5-7B-Instruct82.26 ± 1.1478.37 ± 0.97+3.89
Llama-3-8B22.54 ± 1.1018.93 ± 0.50+3.61

We will add this discussion and the GSM8K results to the revised paper.

[1] Dutta, A., Krishnan, S., Kwatra, N. and Ramjee, R., 2024. Accuracy is not all you need. Advances in Neural Information Processing Systems, 37, pp.124347-124390.
[2] Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z. and Yang, Y., 2024, May. Compressing llms: The truth is rarely pure and never simple. In International Conference on Learning Representations (Vol. 2024, pp. 36973-36993).

AuthorsOpenReview ↗

4/4 Response by Authors

Comment 4: Could the authors include an independently quantized AQLM baseline with the same Hadamard preprocessing to better separate the effects of rotation and prefix supervision?

Re:
Yes, we appreciate this suggestion. We added AQLM (Ind.)+Had., and also AQLM (Drop)+Had., to separate the effects of rotation and prefix supervision. The tables below report both on Qwen2.5-0.5B and Qwen2.5-3B.

C4 (↓)
ModelFP16AQLM (Ind.)AQLM (Ind.)+Had.AQLM (Drop)AQLM (Drop)+Had.DbyD (35W)DbyD (Had.+35W)
Q-0.5B5×817.4117.8717.7917.8717.7918.0017.87
4×818.7318.5620.9120.2021.4819.99
3×821.8721.2558.7634.3937.5827.67
         
Q-3B5×811.1411.3511.3011.3511.3011.3911.32
4×811.7211.5822.3217.1712.2411.93
3×812.9412.55283.9711,748.2514.3213.96

 

W2 (↓)
ModelFP16AQLM (Ind.)AQLM (Ind.)+Had.AQLM (Drop)AQLM (Drop)+Had.DbyD (35W)DbyD (Had.+35W)
Q-0.5B5×813.0613.3013.3513.3013.3513.3613.36
4×813.7613.8416.2115.9616.6515.49
3×815.4315.5951.7129.6528.4022.05
         
Q-3B5×87.457.547.557.547.557.577.57
4×87.717.7430.709.568.228.05
3×88.258.34365.0016,949.839.919.73

The two effects separate as follows:

  1. Rotation alone barely changes independent AQLM. AQLM (Ind.)+Had. is within about 0.6 perplexity of AQLM (Ind.) everywhere, with a small gain on C4 and essentially no change on WikiText-2.
  2. Rotation alone does not make prefixes reliable. AQLM (Drop)+Had. improves over AQLM (Drop) on Qwen2.5-0.5B but still collapses on Qwen2.5-3B at 3×8 (C4 perplexity 11,748.25).
  3. Prefix supervision is what makes dropping work. AQLM (Drop)+Had. and DbyD (Had.+35W) share the rotation and differ only in the prefix loss. With it, 3×8 C4 perplexity goes from 34.39 to 27.67 on Qwen2.5-0.5B and from 11,748.25 to 13.96 on Qwen2.5-3B.
  4. Rotation and prefix supervision are complementary. Adding the rotation to DbyD (35W) improves it at every bitwidth on both models.

This pattern is consistent with our theoretical motivation. AQLM learns its codebooks, so at a fixed bitwidth it adapts to the weights whether or not they are Gaussian, which is why the rotation makes little difference for AQLM (Ind.). Successive refinement is a different matter: Theorem 1 shows that a nested code can match separate codes at every rate for Gaussian sources. The rotation brings the weights closer to that setting, but the theorem only says that such a code exists. AQLM optimizes the full five-codebook sum and never asks the first three or four codebooks to be good on their own, which is why AQLM (Drop)+Had. still fails. The prefix loss is what trains the codebooks toward the nested code.

We will add this comparison to Section 6 of the revised paper, with the full ablation in a new subsection of Appendix G that isolates the effect of the rotation from that of the prefix loss.

Reviewer_3ZuXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Drop-by-Drop (DbyD) extends the any-precision quantization paradigm to vector quantization (VQ) using additive codebooks. It establishes a baseline for any-precision VQ and provides a theoretical analysis based on successive refinement. Evaluations show lower perplexity and kernel latency than existing any-precision scalar quantization (SQ) baselines.

Critical strengths
  1. It generalizes the any-precision paradigm from SQ methods, such as Any-Precision LLM, MatQuant, and AnyBCQ, to VQ and exploits the advantages of VQ. It can also be directly compared against AQLM, its single-precision VQ counterpart.

  2. It provides a theoretical foundation for successive refinement under weighted MSE, which also motivates the use of Hadamard rotation.

Critical weaknesses

I currently lean towards rejection, mainly because (1) the baseline should incorporate rotation, and (2) comparisons on SOTA models and generation benchmarks are needed, given the weaker zero-shot results.

  1. Missing rotated baseline. The comparison between DbyD (Had.+35W) and AQLM (Had.+Ind.) is missing. Since rotation can be applied to both DbyD and AQLM, this comparison is necessary to isolate the gain of the proposed method.

  2. Models and benchmarks. The comparison should include recent state-of-the-art (SOTA) models, such as Gemma 4 (Gemma Team, 2026) and Qwen3 (Qwen Team, 2025). It should also include generation and reasoning benchmarks, such as GSM8K (Cobbe et al., 2021) / MATH (Hendrycks et al., 2021), BBH (Suzgun et al., 2022), and IFEval (Zhou et al., 2023). More comparisons are especially needed given the weaker zero-shot results in Table 24.

Questions for authors
  1. Why is AQLM (Ind.) considered the ideal performance bound? Please include a comparison with AQLM (Had.+Ind.).
  2. Could you report results on more recent models, such as Qwen3 or Gemma 4?
  3. Could you report results on generation and reasoning benchmarks, such as GSM8K, BBH, and IFEval?

Additional Questions:

  1. How could DbyD be extended to trellis-coded quantizers, such as QTIP (Tseng et al., 2024)?
  2. Could DbyD support automatic bit-width selection for each layer or module?
11 comments
AuthorsOpenReview ↗

1/6 Response by Authors

Thank you very much for your time and feedback. We followed your recomendations.

Question 1: Why is AQLM (Ind.) considered the ideal performance bound? Please include a comparison with AQLM (Had.+Ind.).

Re:
AQLM (Ind.) trains a separate model per bitwidth, so each is optimized for its own budget, whereas a droppable model must serve every bitwidth with one shared set of codebooks, which is a more constrained problem. Initially, we used plain AQLM (Ind.) as the reference, because the Hadamard rotation is part of our contribution and is not in the original AQLM. However, we agree that AQLM (Ind.)+Had. is the appropriate empirical performance bound for DbyD, and we have added it below:

C4 (↓)
ModelFP16AQLM (Ind.)AQLM (Ind.)+Had.AQLM (Drop)DbyD (Had.+35W)
Q-0.5B5x817.4117.8717.7917.8717.87
4x818.7318.5620.9119.99
3x821.8721.2558.7627.67
       
Q-3B5x811.1411.3511.3011.3511.32
4x811.7211.5822.3211.93
3x812.9412.55283.9713.96
       
Q-7B5x810.4010.6110.5110.6110.51
4x810.9410.6811.7910.93
3x811.8511.2815.1012.27

 

W2 (↓)
ModelFP16AQLM (Ind.)AQLM (Ind.)+Had.AQLM (Drop)DbyD (Had.+35W)
Q-0.5B5x813.0613.3013.3513.3013.36
4x813.7613.8416.2115.49
3x815.4315.5951.7122.05
       
Q-3B5x87.457.547.557.547.57
4x87.717.7430.708.05
3x88.258.34365.009.73
       
Q-7B5x86.866.936.926.936.93
4x87.087.037.847.28
3x87.477.3910.338.40

 

AQLM (Ind.)+Had. on single run; others use mean of 3 runs.

Rotation changes the AQLM (Ind.) reference only slightly: AQLM (Ind.)+Had. is within about 0.6 perplexity of AQLM (Ind.) everywhere, with a small gain on C4 and essentially no change on WikiText-2. This pattern is consistent with our theoretical motivation. AQLM learns its codebooks, so at a fixed bitwidth it adapts to the weights whether or not they are Gaussian, which is why the rotation makes little difference for independent training. Meanwhile, Theorem 1 shows that one nested code can be optimal at every rate for Gaussian sources, and the rotation brings the weights closer to that setting. Plain AQLM optimizes only the full five-codebook sum and never asks the first three or four codebooks to be good on their own, which is why AQLM (Drop) collapses; the DbyD loss is what trains the codebooks to find the nested code.

AuthorsOpenReview ↗

2/6 Response by Authors

Question 2: Could you report results on more recent models, such as Qwen3 or Gemma 4?

Re:
We added Qwen3-0.6B and Qwen3-4B, and the trend matches the models in the submission. The columns AQLM (Drop) → AQLM (Drop)+Had. → DbyD (Had.+35W) show the two components step by step. Adding the rotation alone already reduces the loss from dropping codebooks in most settings, and adding prefix supervision on top reduces it further in every setting. The independently trained AQLM (Ind.)+Had. models, which need a separate training run and separate storage per bitwidth, remain the reference.

Qwen3-0.6B
DatasetBitsFP16AQLM (Ind.)+Had.AQLM (Drop)AQLM (Drop)+Had.DbyD (Had.+35W)
WikiText-2 (↓)519.1719.5719.5919.5719.80
420.5528.5229.0223.24
323.59477.75208.8545.64
C4 (↓)523.8324.5524.7524.5524.51
425.8534.3832.4527.94
330.87577.53154.3450.52

 

Qwen3-4B
DatasetBitsFP16AQLM (Ind.)+Had.AQLM (Drop)AQLM (Drop)+Had.DbyD (Had.+35W)
WikiText-2 (↓)512.4312.6712.6412.6712.80
412.9816.8215.1814.96
314.3470.0934.9420.59
C4 (↓)515.4615.9515.8915.9515.94
416.5320.2018.3417.66
319.3876.1438.7522.49
AuthorsOpenReview ↗

3/6 Response by Authors

Question 3: Could you report results on generation and reasoning benchmarks, such as GSM8K, BBH, and IFEval?

Re:
Thank you for this suggestion. We added all three benchmarks: GSM8K [1] and BBH [2] for reasoning, and IFEval [3] for instruction following (on Instruct models only).

  • No baseline is consistently better than DbyD. Across all three benchmarks, baselines are significantly worse than DbyD in 31 comparisons and significantly better in only 3. All three are on BBH against AnyPrecLLM, which uses slightly more storage. On GSM8K and IFEval, no baseline is significantly better than DbyD in any cell.
  • DbyD's advantage is clearest on GSM8K at 3 bits, where reasoning degrades most. DbyD has the highest GSM8K accuracy on 5 of 6 models, including a significant +3.9 points over AnyPrecLLM on Qwen2.5-7B-Instruct.

For each model, every method answers identical prompts built by lm-eval-harness 0.4.9.1 and is decoded and scored by the same script (greedy decoding, FP16, batch size 8). To show which differences are meaningful, we use a paired McNemar test [4,5] on per-question correctness: ▼/▲ = significantly worse/better than DbyD at the same bit-width (p < 0.05, uncorrected).

GSM8K (%, ↑)
BitsMethodbpwQ-0.5B-InstG-2BQ-3BM-7BQ-7B-InstL-8B
16FP161637.91±1.4917.89±0.3374.98±0.4536.95±0.7584.74±0.1651.00±1.36
         
5DbyD (Had.+35W)5.0234.55±1.3215.59±0.1872.20±0.2334.80±0.3384.53±0.3548.14±0.40
5AnyBCQ5.0422.54±1.10▼8.62±0.16▼67.42±0.42▼n/rn/rn/r
5AnyPrecLLM5.1734.65±0.5015.39±0.9972.55±0.7635.08±0.4285.37±0.8248.50±1.31
5MatGPTQ5.2528.56±1.52▼15.26±0.4470.00±1.3034.50±0.53*42.76±1.21▼
         
4DbyD (Had.+35W)4.0219.43±0.3912.64±0.8267.75±0.6431.79±0.9883.37±0.4241.02±0.30
4AnyBCQ4.0310.77±0.55▼5.33±0.12▼61.38±0.46▼n/rn/rn/r
4AnyPrecLLM4.0920.92±1.2513.34±0.4768.74±0.4232.98±0.8484.13±0.3242.00±1.12
4MatGPTQ4.2516.10±0.43▼10.77±0.5964.87±1.0733.08±1.38*36.21±0.24▼
         
3DbyD (Had.+35W)3.023.16±0.235.59±1.0345.94±0.5525.80±0.7982.26±1.1422.54±1.10
3AnyBCQ3.032.05±0.552.00±0.32▼31.26±1.30▼n/rn/rn/r
3AnyPrecLLM3.042.58±0.475.86±0.2742.51±0.0922.49±1.4678.37±0.97▼18.93±0.50
3MatGPTQ3.250.30±0.26▼2.58±0.13▼1.77±0.27▼18.63±0.16▼*13.72±0.38▼

GSM8K: 5-shot, all 1,319 test problems, flexible-extract scoring. Mean ± std over three few-shot seeds (1234, 42, 7). Instruct models use their chat template with multi-turn few-shot and up to 1,024 new tokens; base models use the plain prompt and up to 256 new tokens.

 

BBH (%, ↑)
BitsMethodbpwQ-3BM-7B
16FP161655.3758.35
     
5DbyD (Had.+35W)5.0254.2557.40
5AnyBCQ5.0447.90 ▼n/r
5AnyPrecLLM5.1754.2658.33 ▲
5MatGPTQ5.2553.03 ▼56.03 ▼
     
4DbyD (Had.+35W)4.0250.2154.68
4AnyBCQ4.0346.12 ▼n/r
4AnyPrecLLM4.0950.0256.11 ▲
4MatGPTQ4.2549.5254.74
     
3DbyD (Had.+35W)3.0237.2848.00
3AnyBCQ3.0332.85 ▼n/r
3AnyPrecLLM3.0440.33 ▲45.60 ▼
3MatGPTQ3.25**41.48 ▼

BBH: 3-shot chain-of-thought, all 27 tasks (6,511 questions), exact match. Greedy decoding, one run.

Reviewer_k7sDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multi-bitwidth post-training quantization of LLM weights: one quantized checkpoint whose precision is chosen when the model is loaded. It builds on AQLM, which represents each group of 8 weights as a sum of codewords from MM learned 8-bit codebooks. The proposed method, Drop-by-Drop (DbyD), changes AQLM’s codebook-update step: the layer-wise output-weighted MSE is evaluated on every prefix of the first kk codebooks and combined as ∑kαkL(k)\sum_k \alpha_k L^{(k)}. Dropping trailing codebooks at inference then yields 3-, 4- and 5-bit models from a single 5×8 checkpoint. An optional block-wise Walsh–Hadamard rotation with random signs is applied before quantization. A lookup-table-based batch-1 GEMV kernel (codeNx8) is provided for N×8N\times 8 codebook configurations.

As theoretical motivation, Theorem 1 (Sec. 4; App. D–E) shows that an i.i.d. Gaussian weight matrix is successively refinable under the weighted distortion E∣(W−W^)X∣F2\mathbb{E}|(W-\hat W)X|_F^2 for a fixed XX. The proof uses the change of variables Y=WXY=WX, an eigendecomposition and nested reverse water-filling.

Experiments cover Qwen2.5 (0.5B–32B), Gemma-2B, Mistral-7B-v0.3 and Llama-3-8B. Metrics are C4/WikiText-2 perplexity and average accuracy over five zero-shot tasks, mostly over three seeds. DbyD is compared with independently quantized AQLM models, naive codebook dropping, and three multi-bitwidth methods (Any-Precision LLM, AnyBCQ, MatGPTQ). The paper also reports storage, quantization cost and per-layer kernel latency.

Critical strengths

Prefix supervision makes AQLM codebook dropping usable, across a broad model sweep. Tables 7–9 (App. H) show that naively dropping AQLM codebooks breaks the model. At 3×8, C4 perplexity is 7.8×1047.8\times10^4 for Llama-3-8B, 283.97 for Qwen2.5-3B and 104.10 for Mistral-7B. With prefix-supervised codebooks the same checkpoint stays usable: 12.97 / 14.32 / 9.14 for DbyD (35W). With the rotation, DbyD comes within 8% of independently quantized AQLM at 3×8 on all six models of 3B parameters or more. The sweep covers eight models from 0.5B to 32B, mostly with three seeds. This is solid evidence that supervising codebook prefixes is a workable route to a single multi-precision vector-quantized checkpoint. It needs one quantization run and one 5-bit checkpoint instead of three (Fig. 14).

A practical kernel for 3–5 codebooks. App. I.3 notes that AQLM ships kernels only for 1×16 and 2×8. The proposed codeNx8 kernel computes the input–codebook dot products once and replaces each code with a table lookup. Measured against each baseline’s own unmodified kernel (AnyBCQ, Any-Precision LLM, MatGPTQ), it is 1.02–1.46× faster on every layer shape in Tables 25–26, including the online Hadamard transform. This matters for the deployment claim: a multi-bitwidth vector-quantized format is only useful if every prefix can be decoded efficiently.

Critical weaknesses

The headline comparison with multi-bitwidth methods (Table 1; App. I.1–I.2) is not yet reliable.

Evaluation protocol unspecified; no FP16 references. Sec. 6 and App. H–I state neither the evaluation sequence length nor the C4 subset or token count. No table includes an FP16 row. The numbers suggest the methods may not have been evaluated identically. For Llama-3-8B, DbyD (Had.+35W) at 5 bits reports WikiText-2 5.96 and C4 8.63 (AQLM (Ind.) 5×8: 5.89 / 8.71). Both are below FP16 values commonly reported for this model (e.g., 6.1 / 9.2 in Huang et al., 2024). All three baselines at 5 bits lie above them (6.25–7.22 / 9.68–10.24). A 5-bit model scoring better than FP16 is unexpected under an identical protocol. A different evaluation setting for DbyD than for the baselines (for example a longer context window) would explain it. The perplexity advantage does not appear in zero-shot accuracy (Table 24). At 3 bits, DbyD (Had.+35W) has the highest average accuracy on 0 of 8 models. It is below Any-Precision LLM on 6 of 8, despite much lower perplexity: Mistral-7B 70.20 vs. 72.14 with C4 8.32 vs. 9.39, and Qwen2.5-7B 68.66 vs. 70.13 with C4 12.27 vs. 15.52. Averaged over the eight models, DbyD is 0.40 / 0.52 / 0.06 points below Any-Precision LLM at 3 / 4 / 5 bits. Each difference on its own is within noise, but the direction is consistent. This conflicts with the large perplexity margins in Table 1. Several baseline results look degenerate or out of range. AnyBCQ on Mistral-7B gives C4 17.48 / 14.28 / 10.49 at 5 / 4 / 3 bits, i.e., it gets better as bits are removed. AnyBCQ on Gemma-2B reaches WikiText-2 40.80 at 5 bits. MatGPTQ gives 134.12 at 3 bits on Qwen2.5-3B, and 23.25 at 4 bits on Gemma-2B where Any-Precision LLM gives 14.40. AnyBCQ and MatGPTQ both fail on Qwen2.5-7B. Table 1 runs AnyBCQ at 5 bits, outside its native 2–4-bit range (App. I.1), and MatGPTQ has no 5-bit kernel (App. I.3).

Why it matters. “Lower perplexity than state-of-the-art multi-bitwidth methods” is the main empirical claim (Abstract, Sec. 6, Sec. 7), and the points above leave it unestablished. What would resolve it: state the exact protocol for every method and add FP16 rows. Evaluate all checkpoints with one script. Check the anomalous baseline runs, for example against the baselines’ published numbers where models overlap.

  1. The contribution of the Matryoshka loss is not separated from the Hadamard rotation, and the theory explains neither.

The main variant combines prefix supervision with a block-Hadamard rotation, but the AQLM baselines are never rotated (Sec. 6, “Method Definitions”). The data suggest that rotation alone substantially improves AQLM. DbyD (Had.+35W) matches or beats AQLM (Ind.) at 5×8 on C4 for all 8 models (Table 7), e.g., Mistral 7.54 vs. 7.93 and Gemma 11.79 vs. 12.50. This holds even though half of DbyD’s loss weight is on the 3-codebook prefix. So unrotated AQLM (Ind.) is not the “ideal performance bound” claimed in Sec. 6. The claim that DbyD “closely approaches individually trained models” (Sec. 7) is therefore measured against a weaker reference. Without rotation, DbyD (35W) at 3×8 is 11% to 72% worse than AQLM (Ind.) in C4 perplexity on 5 of 8 models (Llama-3-8B +21%, Qwen2.5-0.5B +72%; Table 7).

Two further gaps:

An obvious baseline is missing: quantize at 3×8 with AQLM, then fit codebooks 4–5 on the residual with the first three frozen. This gives the 3-bit model exactly and shows the cost at 4 and 5 bits. The choice of 35W is attributed to App. G.2, but G.2 reports no numbers (Fig. 13 is a schematic). Under 35W the 4×8 prefix that is evaluated receives no supervision (α4=0\alpha_4=0).

The theory does not compensate for this. Theorem 1 is correct as stated: I checked the change of variables, the nested water-filling and the Markov chain. However, it follows directly from a known case. For fixed XX the distortion is tr((W−W^)XX⊤(W−W^)⊤)\mathrm{tr}\big((W-\hat W)XX^\top(W-\hat W)^\top\big). Rotating by the eigenvectors of XX⊤XX^\top and rescaling turns an i.i.d. Gaussian WW into independent Gaussian components under sum-MSE. That is exactly the setting of the paper’s own Theorem 3 (App. D.4).

More importantly, the theorem describes asymptotically long random codes whose per-stage rate is allocated by reverse water-filling over the eigendirections of XX⊤XX^\top. DbyD instead adds a fixed 1 bit/weight per stage with 8-dimensional codebooks. Its codes are chosen against L(M)L^{(M)} only (App. F, Alg. 2, line 19), and block fine-tuning targets k=Mk=M only. So the theorem does not predict whether a codebook prefix can be nested, or how well. It also does not motivate the rotation: under its i.i.d. Gaussian assumption an orthogonal rotation leaves the source unchanged. The paper’s contribution therefore rests on the empirical effect of the Matryoshka loss, and that effect is the part not isolated.

What would resolve it: report AQLM (Ind.)+Had, AQLM (Drop)+Had and the greedy stacking baseline on at least 2–3 models. Report the G.2 numbers and the data used to select α\alpha. Present Theorem 1 as motivation rather than as a foundation of the method.

Questions for authors
  1. For every method in Tables 1 and 20–23: what evaluation sequence length, C4 subset or number of tokens, and evaluation script were used? Please add FP16 perplexities for all eight models under that pipeline. If all methods were evaluated with one script and the FP16 rows show DbyD’s 5-bit perplexities at or above FP16, I would consider the first point of Weakness 1 resolved.
  2. How do you explain the gap between perplexity and accuracy in Table 24 at 3 bits (e.g., Mistral-7B, Qwen2.5-7B, Llama-3-8B)? Were all zero-shot runs done with the same lm-eval-harness version and the same metric (acc vs. acc_norm)?
  3. Can you report AQLM (Ind.)+Had, AQLM (Drop)+Had and the greedy baseline (freeze the first three codebooks, fit codebooks 4–5 on the residual) on a subset of models, e.g., Llama-3-8B and Qwen2.5-3B/7B? If DbyD (Had.+35W) keeps a clear advantage over AQLM (Drop)+Had and the greedy baseline, and stays close to AQLM (Ind.)+Had, I would consider Weakness 2 resolved.
  4. On which data were the α\alpha coefficients (35W) chosen: the 32-sample calibration validation split, or WikiText-2/C4 test perplexity? Please give the App. G.2 results, including 4×8 performance for weightings with α4>0\alpha_4>0 (e.g., 345W, Uni.).
  5. For the anomalous baseline entries (AnyBCQ on Mistral-7B worse at 5 bits than at 3 bits; AnyBCQ on Gemma-2B with WikiText-2 ≥ 40; MatGPTQ on Qwen2.5-3B at 134.12 for 3 bits), which configurations were used (group size, parent bitwidth, calibration data and size)? Do the same pipelines reproduce the baselines’ published numbers where models overlap?
  6. Which concrete design choice in DbyD does Theorem 1 inform, such as per-stage rate allocation, the choice of α\alpha, or the need for rotation? Relatedly, why does App. E.2 model block length as NN i.i.d. weight matrices rather than the doutd_\text{out} i.i.d. rows of a single layer, which share the same XX⊤XX^\top?
8 comments
AuthorsOpenReview ↗

1/7 Response by Authors

Thank you very much for your time and feedback. We followed your recommendations and address your concerns below.

Question 1: For every method in Tables 1 and 20–23: what evaluation sequence length, C4 subset or number of tokens, and evaluation script were used? Please add FP16 perplexities for all eight models under that pipeline. If all methods were evaluated with one script and the FP16 rows show DbyD’s 5-bit perplexities at or above FP16, I would consider the first point of Weakness 1 resolved.

Re: All methods are now scored with one script on identical token streams, and FP16 rows are included. Under this pipeline, DbyD's 5-bit perplexity is above FP16 on all eight models and both datasets.

W2 ↓\downarrowC4 ↓\downarrow
BitsbpwMethodQ-0.5BG-2BQ-3BQ-7BM-7BL-8BQ-14BQ-32BQ-0.5BG-  2BQ- 3BQ- 7BM-7BL-8BQ-14BQ-32B
1616FP1613.068.347.456.864.955.845.254.9017.4111.5711.1410.407.488.438.958.73
55.02DbyD (Had.+35W)13.368.507.576.934.995.965.384.9917.8711.8011.3210.517.548.639.068.79

 

In the original submission, the baseline numbers in Tables 1 and 20-23 came from each method's own published protocol. These differed in context length (1024, 2048 or 4096 tokens), in the C4 subset, and in how storage overhead was counted. All methods now use the following:

ItemSetting
ScriptOne evaluation script for all methods and FP16
Context4096 tokens, non-overlapping windows
C4256 random 4096-token windows from c4-validation.00000-of-00008, seed 0 (1,048,576 tokens)
WikiText-2Test split (299,078 tokens)

For each model, every method is scored on the identical token tensor, and both evaluation sets are disjoint from every method's calibration data. DbyD is the mean of 3 runs; baselines are a single run of each trained checkpoint. "bpw" (bits per weight) is the mean over models and includes all overheads (scales, codebooks, lookup tables, sign vectors). For the baseline comparison we re ran three models (Qwen2.5-0.5B-Instruct, Gemma-2B, Qwen2.5-3B) under this protocol:

W2 (↓\downarrow)C4 (↓\downarrow)
BitsbpwMethodQ-0.5BG-2BQ-3BQ-0.5BG-2BQ-3B
1616FP1613.068.347.4517.4111.5711.14
         
55.02DbyD (Had.+35W)13.368.507.5717.8711.8011.32
55.04AnyBCQ15.269.488.1919.4512.9411.97
55.17AnyPrecLLM13.499.077.5417.8212.5411.26
55.25MatGPTQ14.549.147.7218.9012.4811.51
         
44.02DbyD (Had.+35W)15.499.698.0519.9913.2411.93
44.03AnyBCQ17.4010.178.6621.3513.6912.53
44.09AnyPrecLLM14.779.827.9119.1313.3611.67
44.25MatGPTQ16.8310.318.2721.8014.2912.09
         
33.02DbyD (Had.+35W)22.0513.199.7327.6717.0813.96
33.03AnyBCQ29.8515.7611.0932.5419.2515.20
33.04AnyPrecLLM23.1512.919.5727.1916.7113.58
33.25MatGPTQ66.9621.85128.8563.9327.1795.29

 

For easier reading, the table below averages perplexity over these three models at each bitwidth.

MethodW2 5bW2 4bW2 3bW2 Avg (↓\downarrow)C4 5bC4 4bC4 3bC4 Avg (↓\downarrow)
FP16–––9.62–––13.37
DbyD (Had.+35W)9.8111.0814.9911.9613.6615.0519.5716.10
AnyPrecLLM10.0310.8315.2112.0313.8714.7219.1615.92
AnyBCQ10.9812.0818.9013.9814.7915.8622.3317.66
MatGPTQ10.4711.8072.5531.6114.3016.0662.1330.83

(continued in the next comment)

AuthorsOpenReview ↗

2/7 Response by Authors

Disk size (GB) of the single checkpoint serving 3/4/5 bits, including the unquantized embeddings and output layer:

Disk (GB, ↓\downarrow)
MethodQ-0.5BG-2BQ-3BM-7BQ-7BL-8BQ-14BQ-32B
DbyD0.502.292.364.916.276.4711.3922.64
AnyBCQ0.512.322.404.956.316.5311.4522.80
AnyPrecLLM0.542.412.485.066.436.6411.6323.15
MatGPTQ0.673.103.467.63*9.2116.5634.83

* MatGPTQ did not converge on Q-7B.

 

DbyD has the smallest storage: at every precision it has the lowest bits per weight of any method (3.02 / 4.02 / 5.02), and its single checkpoint serving all three precisions is the smallest on all eight models. In quality, DbyD has lower perplexity than AnyBCQ and MatGPTQ, while using fewer bits per weight than both. Against AnyPrecLLM, the strongest baseline, it is on par while using fewer bits, so we will state DbyD as "comparable" to AnyPrecLLM regarding perplexity.

We hope this resolves the first point of Weakness 1: all methods are evaluated with one script, and the FP16 rows show DbyD's 5-bit perplexities above FP16 on all eight models and both datasets. We will revise the abstract and Section 6 accordingly, and add the protocol and these results to the Appendix I.

AuthorsOpenReview ↗

3/7 Response by Authors

Question 2: How do you explain the gap between perplexity and accuracy in Table 24 at 3 bits (e.g., Mistral-7B, Qwen2.5-7B, Llama-3-8B)? Were all zero-shot runs done with the same lm-eval-harness version and the same metric (acc vs. acc_norm)?

Re: Every method was scored on the same tasks with the same metrics, the harness version has no measurable effect, the 3-bit accuracy gaps are not statistically significant, and we also evaluated on a generative tasks.

1. Same tasks and metrics for every method. All zero-shot results use the same five tasks (ARC-e, ARC-c, HellaSwag, PIQA, Winogrande), full evaluation splits, 0-shot, FP16. The metric is acc_norm where the task defines it (ARC-e, ARC-c, HellaSwag, PIQA) and acc for Winogrande, averaged over the five tasks. No method is scored differently.

2. Harness version has no measurable effect. Each method was evaluated in its official environment, and these pin different lm-eval-harness versions (0.4.0-0.4.12). To rule this out as a source of differences, we ran the same FP16 model through every method's harness. The five-task averages agree to within 0.09 points, an order of magnitude below the standard error of about 1 point:

FP16 avg. acc (%)DbyD harnessAnyBCQ harnessAnyPrecLLM harnessMatGPTQ harness
Qwen2.5-0.5B-Instruct54.3754.3454.3754.39
Qwen2.5-3B68.2268.2668.1968.19
Qwen2.5-7B-Instruct73.4873.5073.4273.51

3. The 3-bit accuracy gaps are not statistically significant. DbyD trails the best baseline by 1.4-1.9 points on the three models you mention, which is 1.0-1.4 standard errors of the difference, so none is significant at the 5% level (two-sided z-test, |z| < 1.96).

3-bit avg. acc (%, ↑\uparrow)DbyD (Had.+35W)Best baselineDifferenceSE of difference
Mistral-7B70.20 ± 1.0272.14 ± 0.98 (AnyPrecLLM)-1.941.41
Qwen2.5-7B-Instruct68.66 ± 1.0870.13 ± 1.00 (AnyPrecLLM)-1.471.47
Llama-3-8B67.81 ± 1.0369.25 ± 1.00 (AnyBCQ)-1.441.44

4. Why perplexity and accuracy trends can differ. Perplexity averages log-likelihood over every token of held-out text, so it registers small, uniform losses in prediction quality. Zero-shot accuracy only asks whether the correct answer outranks a few alternatives, so a model can lose likelihood across the board without flipping these rankings. This mismatch is known for compressed LLMs: multiple-choice accuracy can stay nearly unchanged while model behaviour differs [1], and it degrades more mildly than tasks where the model must generate the answer [2]. We therefore also evaluated GSM8K, a generative benchmark (Question 3 of Reviewer 3ZuX). At 3 bits and matched budget, DbyD leads AnyPrecLLM by 3.3-3.9 points on the same three models, roughly 2-3 standard errors of the difference:

3-bit GSM8K (%, ↑\uparrow)DbyD (3.02 bpw)AnyPrecLLM (3.04 bpw)Difference
Mistral-7B25.80 ± 0.7922.49 ± 1.46+3.31
Qwen2.5-7B-Instruct82.26 ± 1.1478.37 ± 0.97+3.89
Llama-3-8B22.54 ± 1.1018.93 ± 0.50+3.61

DbyD also outperforms MatGPTQ and AnyBCQ on every model reported.

We will add the evaluation protocol details to the "Experimental Setup & Hyperparameters" paragraph and the GSM8K results to a new Appendix J.

[1] Dutta, A., Krishnan, S., Kwatra, N. and Ramjee, R., 2024. Accuracy is not all you need. Advances in Neural Information Processing Systems, 37, pp.124347-124390.
[2] Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z. and Yang, Y., 2024, May. Compressing llms: The truth is rarely pure and never simple. In International Conference on Learning Representations (Vol. 2024, pp. 36973-36993).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors' Response Summary

Thank you to all reviewers for their time and thorough feedback. We appreciate your recognition of both the practical and the theoretical contributions of Drop-by-Drop:

  • "This is solid evidence that supervising codebook prefixes is a workable route to a single multi-precision vector-quantized checkpoint." (k7sD)
  • "It provides a theoretical foundation for successive refinement under weighted MSE, which also motivates the use of Hadamard rotation." (3ZuX)
  • "The storage comparisons and specialized inference kernels strengthen the practical value of the work." (1UVm)

The reviews helped us clarify and strengthen the paper. Since the original submission, we have:

  • added generation and reasoning benchmarks (GSM8K, BBH, IFEval) with significance tests,
  • added rotated AQLM baselines that isolate the effect of the rotation from that of prefix supervision,
  • added results on the newer Qwen3-0.6B and Qwen3-4B models,
  • implemented per-layer bit-width selection (mixed precision) from a single checkpoint,
  • re-evaluated all methods under one unified protocol with FP16 references and full storage accounting,
  • showed an ablation of the loss weights α\alpha on held-out data,
  • clarified the scope of Theorem 1 and what it explains about the observed performance.

Detailed answers are below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.