- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Drop-by-Drop, a multi-bitwidth post-training quantization framework that enables inference-time precision control from a single quantized model. The method incorporates Matryoshka-style supervision into additive codebook training, allowing lower-precision models to be obtained by dropping codebooks. The authors provide a theoretical motivation through successive refinability of Gaussian sources under weighted mean squared error. Across Qwen, LLaMA, Gemma, and Mistral, the method achieves lower perplexity than the evaluated multi-bitwidth baselines in most settings, while maintaining competitive zero-shot accuracy.
Critical strengths
The paper addresses a practical problem: supporting multiple precision levels without retraining or storing separate models.
The method is clear and shows substantial improvements over directly dropping codebooks from AQLM.
Experiments cover several model families and sizes, with generally consistent perplexity improvements.
The storage comparisons and specialized inference kernels strengthen the practical value of the work.
Critical weaknesses
1 The method builds on AQLM by introducing Matryoshka-style supervision over additive codebook prefixes. While effective, the methodological advance appears incremental, and the technical challenges addressed by this extension could be explained more clearly.
2 The theoretical result provides a useful motivation under Gaussian assumptions, but it does not establish a performance guarantee for the practical algorithm. The paper could clarify what the result explains about the observed performance.
3 At lower bitwidths, a noticeable gap remains compared with independently quantized models in some settings. Lower perplexity also does not consistently translate into better zero-shot accuracy.
4 Could the authors include an independently quantized AQLM baseline with the same Hadamard preprocessing to better separate the effects of rotation and prefix supervision?
Questions for authors
No further questions for the authors.
4 comments
2/4 Response by Authors
Comment 2: The theoretical result provides a useful motivation under Gaussian assumptions, but it does not establish a performance guarantee for the practical algorithm. The paper could clarify what the result explains about the observed performance.
Re:
We agree that Theorem 1 is an asymptotic result for Gaussian sources, not a performance guarantee for the practical algorithm. We will revise Section 5 to clarify what the result explains about the observed performance:
-
How can DbyD, a single model, approach the performance of separately quantized models? Theorem 1 shows that, under WMSE, a nested code can match separate codes at every rate, so nesting has no inherent cost. Consistent with this, at 4 and 5 bits on all models of 7B and above, DbyD (Had.+35W) is at most 0.09 C4 perplexity above individually trained AQLM (Ind.) and within one standard error in zero-shot accuracy.
-
Why does supervising prefixes work? The code constructed in the proof is built in stages: , where is the reconstruction of component after stage and is the refinement added at that stage. The key property is that every intermediate reconstruction is a good code at its own rate. Our Matryoshka-style loss enforces this property in additive codebooks: each term is the distortion of the sum of the first codebooks. AQLM trains only the full sum, so the property does not hold and dropping codebooks fails (C4 perplexity 283.97 vs. 12.94 on Qwen2.5-3B at 3 bits); with , the same setting reaches 14.32.
-
Why does rotation help? Theorem 1 holds for Gaussian sources, while LLM weights are heavy-tailed. The block-Hadamard rotation brings the weights closer to Gaussian (average excess kurtosis drops from 2.48 to 0.50 on Qwen2.5-7B), and therefore closer to the setting of Theorem 1. Empirically, it narrows the gap to individually trained models on nearly all models and bitwidths.
3/4 Response by Authors
Comment 3: At lower bitwidths, a noticeable gap remains compared with independently quantized models in some settings. Lower perplexity also does not consistently translate into better zero-shot accuracy.
Re:
1. The gap to independently quantized models is concentrated at 3 bits and on small models, and it shrinks steadily with model size:
| 3-bit C4 () | Q-0.5B | Q-3B | Q-7B | Q-14B | Q-32B |
|---|---|---|---|---|---|
| AQLM (Ind.) | 21.87 | 12.94 | 11.85 | 10.01 | 9.46 |
| DbyD (Had.+35W) | 27.67 | 13.96 | 12.27 | 10.38 | 9.49 |
| Gap | 5.80 | 1.02 | 0.42 | 0.37 | 0.03 |
Firstly, this gap is the cost of serving all three bitwidths from one quantization run and one checkpoint, where AQLM (Ind.) needs three of each. Secondly, it is a tradeoff the user controls: serving two bitwidths instead of three lowers 3-bit C4 perplexity on Qwen2.5-3B from 14.32 to 13.91 (App. G.3). Third, even with this gap, DbyD remains competitive with state-of-the-art multi-bitwidth methods at 3 bits. We already list lower bitwidths as a limitation in Section 7, and, as you point out, we will state that the gap is currently largest for small models at 3 bits.
2. Regarding the perplexity vs. zero-shot accuracy: The two metrics measure different things. Perplexity averages log-likelihood over every token, so it registers small, uniform losses. Zero-shot accuracy only asks whether the correct answer outranks a few alternatives, so a model can lose likelihood without flipping these rankings. This mismatch is known for compressed LLMs: multiple-choice accuracy can stay nearly unchanged while model behaviour differs [1], and it degrades more mildly than tasks where the model must generate the answer [2]. With a standard error of about 1 point, the five-task average cannot separate methods that are close: at 3 bits, DbyD's accuracy differences to the best baseline are 1.0 to 1.4 standard errors, so none is significant.
We have added additional benchmarks (Question 3 of Reviewer 3ZuX). On GSM8K at 3 bits, DbyD leads AnyPrecLLM, the strongest baseline, by 3.3 to 3.9 points on Mistral-7B, Qwen2.5-7B-Instruct, and Llama-3-8B, in line with the perplexity ordering.
| 3-bit GSM8K (%, ) | DbyD (3.02 bpw) | AnyPrecLLM (3.04 bpw) | Difference |
|---|---|---|---|
| Mistral-7B | 25.80 ± 0.79 | 22.49 ± 1.46 | +3.31 |
| Qwen2.5-7B-Instruct | 82.26 ± 1.14 | 78.37 ± 0.97 | +3.89 |
| Llama-3-8B | 22.54 ± 1.10 | 18.93 ± 0.50 | +3.61 |
We will add this discussion and the GSM8K results to the revised paper.
[1] Dutta, A., Krishnan, S., Kwatra, N. and Ramjee, R., 2024. Accuracy is not all you need. Advances in Neural Information Processing Systems, 37, pp.124347-124390.
[2] Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z. and Yang, Y., 2024, May. Compressing llms: The truth is rarely pure and never simple. In International Conference on Learning Representations (Vol. 2024, pp. 36973-36993).
4/4 Response by Authors
Comment 4: Could the authors include an independently quantized AQLM baseline with the same Hadamard preprocessing to better separate the effects of rotation and prefix supervision?
Re:
Yes, we appreciate this suggestion. We added AQLM (Ind.)+Had., and also AQLM (Drop)+Had., to separate the effects of rotation and prefix supervision. The tables below report both on Qwen2.5-0.5B and Qwen2.5-3B.
| C4 (↓) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Model | FP16 | AQLM (Ind.) | AQLM (Ind.)+Had. | AQLM (Drop) | AQLM (Drop)+Had. | DbyD (35W) | DbyD (Had.+35W) | |
| Q-0.5B | 5×8 | 17.41 | 17.87 | 17.79 | 17.87 | 17.79 | 18.00 | 17.87 |
| 4×8 | 18.73 | 18.56 | 20.91 | 20.20 | 21.48 | 19.99 | ||
| 3×8 | 21.87 | 21.25 | 58.76 | 34.39 | 37.58 | 27.67 | ||
| Q-3B | 5×8 | 11.14 | 11.35 | 11.30 | 11.35 | 11.30 | 11.39 | 11.32 |
| 4×8 | 11.72 | 11.58 | 22.32 | 17.17 | 12.24 | 11.93 | ||
| 3×8 | 12.94 | 12.55 | 283.97 | 11,748.25 | 14.32 | 13.96 |
| W2 (↓) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Model | FP16 | AQLM (Ind.) | AQLM (Ind.)+Had. | AQLM (Drop) | AQLM (Drop)+Had. | DbyD (35W) | DbyD (Had.+35W) | |
| Q-0.5B | 5×8 | 13.06 | 13.30 | 13.35 | 13.30 | 13.35 | 13.36 | 13.36 |
| 4×8 | 13.76 | 13.84 | 16.21 | 15.96 | 16.65 | 15.49 | ||
| 3×8 | 15.43 | 15.59 | 51.71 | 29.65 | 28.40 | 22.05 | ||
| Q-3B | 5×8 | 7.45 | 7.54 | 7.55 | 7.54 | 7.55 | 7.57 | 7.57 |
| 4×8 | 7.71 | 7.74 | 30.70 | 9.56 | 8.22 | 8.05 | ||
| 3×8 | 8.25 | 8.34 | 365.00 | 16,949.83 | 9.91 | 9.73 |
The two effects separate as follows:
- Rotation alone barely changes independent AQLM. AQLM (Ind.)+Had. is within about 0.6 perplexity of AQLM (Ind.) everywhere, with a small gain on C4 and essentially no change on WikiText-2.
- Rotation alone does not make prefixes reliable. AQLM (Drop)+Had. improves over AQLM (Drop) on Qwen2.5-0.5B but still collapses on Qwen2.5-3B at 3×8 (C4 perplexity 11,748.25).
- Prefix supervision is what makes dropping work. AQLM (Drop)+Had. and DbyD (Had.+35W) share the rotation and differ only in the prefix loss. With it, 3×8 C4 perplexity goes from 34.39 to 27.67 on Qwen2.5-0.5B and from 11,748.25 to 13.96 on Qwen2.5-3B.
- Rotation and prefix supervision are complementary. Adding the rotation to DbyD (35W) improves it at every bitwidth on both models.
This pattern is consistent with our theoretical motivation. AQLM learns its codebooks, so at a fixed bitwidth it adapts to the weights whether or not they are Gaussian, which is why the rotation makes little difference for AQLM (Ind.). Successive refinement is a different matter: Theorem 1 shows that a nested code can match separate codes at every rate for Gaussian sources. The rotation brings the weights closer to that setting, but the theorem only says that such a code exists. AQLM optimizes the full five-codebook sum and never asks the first three or four codebooks to be good on their own, which is why AQLM (Drop)+Had. still fails. The prefix loss is what trains the codebooks toward the nested code.
We will add this comparison to Section 6 of the revised paper, with the full ablation in a new subsection of Appendix G that isolates the effect of the rotation from that of the prefix loss.
1/4 Response by Authors
Thank you very much for your time and feedback. We appreciate your support.
Re:
Thank you for this suggestion. We agree that the technical motivation deserves clearer explanation. Our key contribution is a successive-refinement-inspired multi-bitwidth additive quantization framework that integrates Gaussian-motivated Hadamard rotation with Matryoshka-supervised codebook refinement. This enables additive codebook prefixes to serve as progressively refined reconstructions at different precision levels.
We have a theory-motivated design. We prove that Gaussian sources are successively refinable under WMSE, establishing that nested representations can theoretically achieve optimal rate-distortion trade-offs at multiple precision levels. However, practical LLM weights deviate from Gaussianity. Motivated by this theorem, we introduce block-Hadamard rotation to multi-bitwidth quantization to bring weight distributions closer to the Gaussian assumption, as empirically supported by the reduced excess kurtosis in Appendix C. We then use Matryoshka supervision to realize nested precision through additive codebook prefixes.
Technical challenges of Matryoshka supervision: Unlike MatQuant and MatGPTQ, which exploit inherent integer bit-slicing, AQLM jointly optimizes additive codebooks without guaranteeing useful intermediate prefixes. Two challenges arise:
These design choices are supported empirically. On Qwen2.5-3B at 3 bits, AQLM (Drop) yields a C4 perplexity of 283.97, compared with 14.94 for DbyD (Uniform), 14.32 for DbyD (35W), and 13.96 with Hadamard rotation.
We will revise Sections 4-5 to make the connection between successive refinement theory, Gaussian-motivated rotation, and Matryoshka-supervised additive quantization more explicit.