- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DepthQuant, a mixed-precision weight-only post-training quantization framework for large language models. It first ranks candidate precision assignments using quantization perturbations propagated over a short network segment. It then redistributes bit budgets across layers using isolated model-output KL divergences and dynamic programming over the resulting allocation paths, while preserving the total parameter-bit budget. Finally, it refines per-row quantization ranges with the bit allocation fixed. Experiments cover eight models from 4B to 32B at average precisions of 2, 2.5, and 3 bits.
Critical strengths
-
Well-motivated study of an important quantization problem.
The paper is motivated by an important practical limitation: local reconstruction error may not reliably reflect the downstream impact of quantization. Accounting for error propagation in mixed-precision allocation remains an important challenge, particularly at extremely low precision. The paper's focus on this problem is meaningful and practically relevant.
-
Promising empirical results.
The paper reports improvements over the evaluated baselines across multiple model families and precision settings, including substantial gains in several configurations at an average precision of 2 bits. Both perplexity and downstream task accuracy are evaluated.
Critical weaknesses
-
Insufficient justification of short-depth response scoring. The limitations of local reconstruction criteria have already been discussed in previous works (see reference). However, the paper does not establish why short-depth responses offer a better accuracy–cost trade-off than sensitivity estimates based on the final model loss. However, Table 17 shows no consistent advantage of over , which could indicates short-depth response scoring is not a reliable proxy. Although Tables 19, 21, and 25 provide evidence supporting that short-depth response scoring is consistent with the final outputs, they do not isolate the benefit of observing an additional layer or compare against global-loss scoring under controlled conditions.
-
Limited validation of the restricted allocation space and surrogate objective.
Full-Depth Allocation optimizes over frozen within-layer allocation paths and a narrow per-layer budget window. Dynamic programming is exact within this restricted domain, but potentially beneficial allocations may be excluded. Table 16 only examines a narrow range of caps on two models at 3 bits. Moreover, the objective sums isolated per-layer KL divergences rather than directly optimizing joint-model divergence. The diagnostics in Tables 22 and 23 are useful but limited. Broader cap ablations and stronger validation under joint, particularly 2-bit, quantization would strengthen the method.
-
Incremental contribution of range refinement.
The range search in Section 3.4 is closely related to established clipping and range-optimization techniques, including AWQ's clipping search and OmniQuant's learnable weight clipping. Existing ablations demonstrate its usefulness but not its advantage over these alternatives. Since this component contributes substantially to some reported gains, comparisons using the same bit allocation and quantization backend are needed to better isolate the contribution of the proposed scoring and allocation mechanisms.
-
Incomplete strong-baseline comparisons at 2 bits.
SignRoundV2 is evaluated at 2.5/3 bits but omitted from the average-2-bit comparison. Its use of uniform precision at this budget does not sufficiently justify exclusion, since uniform RTN and GPTQ are included. Adding SignRoundV2's W2A16 configuration would make the extreme-low-bit evaluation more convincing. SFMP would also be a relevant additional mixed-precision baseline, with budget accounting and evaluation protocols carefully aligned.
-
Limited model scale and downstream evaluation coverage.
Most of the models evaluated in the experiment are small-scale, and the largest is the only 32B. Besides, the evaluation lacks knowledge-intensive benchmarks such as MMLU. I would suggest the authors provide at least two more models at 32B scale or above.
[1]MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design
[2]AutoMixQ: Automatic Mixed-precision Quantization for Deploying Bit-Efficient LLMs
[3]GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
[4]SqueezeLLM: Dense-and-Sparse Quantization
Questions for authors
See weakness
9 comments
Response to Reviewer gAmd (2)
W2. Limited validation of the restricted allocation space and surrogate objective.
Thank you for these suggestions. We added experiments on joint 2-bit quantization and wider budget windows with different candidate-state resolutions.
On Qwen3-4B at a requested average of 2 bits, we compared allocations before and after Full-Depth Allocation, keeping the total parameter-bit budget and reconstruction procedure fixed. All layers are quantized simultaneously when measuring the following results:
| Allocation | Joint KL on pricing text ↓ | Joint KL on held-out text ↓ | Held-out NLL ↓ |
|---|---|---|---|
| Short-Depth Response Scoring | 2.04330 | 1.89148 | 3.86442 |
| + Full-Depth Allocation | 1.93952 | 1.78339 | 3.75129 |
Full-Depth Allocation reduces joint KL by 5.08% on the pricing text and 5.71% on held-out text, while also lowering held-out NLL. This supports the usefulness of isolated-layer pricing for selecting allocations that improve the jointly quantized model in this setting.
We also evaluated wider per-layer budget windows on the same model at the same total budget:
| Cap c | States per layer | Step (bit) | Pricing time (h) | WikiText-2 PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|---|---|---|
| 0.1 (reported) | 5 | 0.05 | 2.75 | 21.72 | 37.60 | 51.18 |
| 0.5 | 5 | 0.25 | 2.74 | 21.40 | 38.74 | 50.81 |
| 0.5 | 11 | 0.10 | 5.36 | 21.40 | 38.26 | 51.81 |
| 0.5 | 21 | 0.05 | 9.79 | 21.09 | 38.26 | 51.37 |
Widening the window and refining the step size do not consistently improve performance, while increasing the number of states from 5 to 21 raises pricing time from 2.74 to 9.79 hours. Broader and denser candidate coverage therefore does not guarantee better final allocations, supporting the reported window as a practical accuracy–cost trade-off. We will include these additional ablations and joint-quantization measurements in the revised manuscript.
W3. Incremental contribution of range refinement.
Thank you for this comment. To distinguish the contributions of allocation and reconstruction, we conducted two complementary comparisons: different allocation maps under the same reconstruction procedure, and different reconstruction procedures under a fixed allocation map.
We first compare the frozen allocation maps produced by DepthQuant, SliM, AMQ, and SignRoundV2 on Qwen3-8B at a nominal average of 3 bits. Each method retains its allocation and bit budget from the corresponding main-table configuration. Within each block below, all maps use the same reconstruction procedure and evaluation protocol.
GPTQ
| Allocation | Wiki PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|
| SliM | 11.17 | 17.88 | 68.12 |
| AMQ | 11.11 | 17.89 | 67.98 |
| SignRoundV2 | 14.11 | 21.97 | 67.30 |
| DepthQuant | 10.79 | 17.23 | 69.99 |
GPTQ Per-Row Range Refinement
| Allocation | Wiki PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|
| SliM | 11.37 | 18.28 | 70.63 |
| AMQ | 10.78 | 17.54 | 69.92 |
| SignRoundV2 | 12.08 | 19.60 | 69.05 |
| DepthQuant | 10.64 | 17.09 | 71.95 |
AWQ
| Allocation | Wiki PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|
| SliM | 11.09 | 17.62 | 70.69 |
| AMQ | 11.43 | 17.82 | 70.39 |
| SignRoundV2 | 13.37 | 21.00 | 69.67 |
| DepthQuant | 11.16 | 17.55 | 71.48 |
DepthQuant achieves the highest six-task accuracy among the tested allocations under all three reconstruction settings, exceeding the strongest competing map by 1.87 points under GPTQ, 1.32 points under GPTQ with refinement, and 0.79 points under AWQ. Its allocation advantage therefore persists both without our range refinement and under an alternative reconstruction procedure.
We then compare reconstruction procedures using the same fixed DepthQuant allocation on two models at a requested average of 3 bits. The OmniQuant backend uses learnable weight clipping with the allocation fixed.
| Model | Reconstruction | Wiki PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|---|
| Qwen3-8B | AWQ | 11.16 | 17.55 | 71.48 |
| Qwen3-8B | OmniQuant | 11.48 | 18.02 | 69.58 |
| Qwen3-8B | GPTQ + Per-Row Range Refinement | 10.64 | 17.09 | 71.95 |
| Qwen2.5-7B | AWQ | 7.93 | 13.43 | 71.89 |
| Qwen2.5-7B | OmniQuant | 7.67 | 13.39 | 72.61 |
| Qwen2.5-7B | GPTQ + Per-Row Range Refinement | 7.45 | 13.10 | 72.98 |
For the same allocation, GPTQ with Per-Row Range Refinement achieves lower perplexity on both datasets and higher six-task accuracy than AWQ and OmniQuant on both models. Together with the common-backend comparisons, these results support both the allocation and reconstruction choices.
Response to Reviewer gAmd (3)
W4. Incomplete strong-baseline comparisons at 2 bits.
Thank you for pointing out these missing comparisons. We added SignRoundV2’s uniform W2A16 configuration under the same perplexity and six-task evaluation protocol.
| Model | Method, average 2 bits | Wiki PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|---|
| Qwen3-8B | SignRoundV2 W2A16 | 24.88 | 27.54 | 61.45 |
| Qwen3-8B | DepthQuant | 18.67 | 40.39 | 57.12 |
| Qwen2.5-7B | SignRoundV2 W2A16 | 14.04 | 19.97 | 63.38 |
| Qwen2.5-7B | DepthQuant | 12.69 | 31.83 | 57.75 |
SignRoundV2 achieves lower C4 perplexity and higher six-task accuracy on both models, while DepthQuant achieves lower WikiText-2 perplexity. The gap may reflect two factors: SignRoundV2 optimizes rounding and quantization ranges through block reconstruction, whereas DepthQuant focuses on bit allocation with GPTQ and Per-Row Range Refinement; DepthQuant’s 1-bit groups also introduce more aggressive compression at this budget. DepthQuant’s allocation advantage is supported by the main-table mixed-precision results and the common-backend comparisons in W3. Learned reconstruction could further complement our allocation.
We also added SFMP at an average target-weight precision of 3 bits on Qwen3-4B and Qwen2.5-7B, using its native reconstruction procedure and the shared evaluation protocol.
| Model | Method | Wiki PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|---|
| Qwen3-4B | SFMP | 16.46 | 23.03 | 67.79 |
| Qwen3-4B | DepthQuant | 14.50 | 21.54 | 67.37 |
| Qwen2.5-7B | SFMP | 7.79 | 13.02 | 72.08 |
| Qwen2.5-7B | DepthQuant | 7.45 | 13.10 | 72.98 |
Budget labels count target-weight bits and exclude quantization metadata, consistent with the main tables. SFMP’s recorded storage cost is approximately 3.25 bits per weight when metadata is included. DepthQuant’s realized target-weight averages are 3.0008 and 3.0016 bits, respectively.
We will incorporate both baselines into the corresponding evaluations and revise the associated comparisons and claims to reflect these results.
W5. Limited model scale and downstream evaluation coverage.
Thank you for these suggestions. We expanded the evaluation with Llama-2-70B, Qwen2.5-32B, and zero-shot MMLU.
Larger-model evaluation. We evaluated Llama-2-70B and Qwen2.5-32B at an average of 2 bits under the same perplexity and six-task evaluation protocol:
| Model, average 2 bits | Method | Wiki PPL ↓ | C4 PPL ↓ | ZS (%) ↑ |
|---|---|---|---|---|
| Llama-2-70B | RTN | 27.04 | 47.63 | 48.77 |
| Llama-2-70B | GPTQ | 6.88 | 11.77 | 56.30 |
| Llama-2-70B | SliM | 5.83 | 10.75 | 66.03 |
| Llama-2-70B | DepthQuant | 5.16 | 8.63 | 69.68 |
| Qwen2.5-32B | RTN | 1,145.82 | 1,594.42 | 39.95 |
| Qwen2.5-32B | GPTQ | 13.87 | 32.23 | 51.18 |
| Qwen2.5-32B | SliM | 9.14 | 18.43 | 65.72 |
| Qwen2.5-32B | DepthQuant | 8.11 | 15.52 | 70.61 |
DepthQuant improves six-task accuracy over the strongest evaluated baseline, SliM, by 3.65 percentage points on Llama-2-70B and 4.89 points on Qwen2.5-32B, while lowering perplexity on both datasets.
Knowledge-intensive evaluation. We evaluated zero-shot MMLU across all 57 subjects and 14,042 test questions, aggregating accuracy over questions. Each quantized method uses its native reconstruction procedure, with precision settings following the corresponding configurations in our 3-bit comparisons.
| Method | Qwen3-4B MMLU (%) ↑ | Qwen2.5-7B MMLU (%) ↑ |
|---|---|---|
| FP reference, 16 bits | 68.35 | 71.88 |
| RTN | 52.02 | 65.15 |
| GPTQ | 56.62 | 65.24 |
| SliM | 60.38 | 65.13 |
| AMQ | 60.15 | 67.48 |
| SignRoundV2 | 63.86 | 67.74 |
| SFMP | 63.43 | 67.37 |
| DepthQuant | 63.43 | 68.42 |
Among the evaluated quantized methods, DepthQuant ranks first on Qwen2.5-7B and joint second on Qwen3-4B. We will incorporate these additional evaluations into the revised manuscript.
We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.
Response to Reviewer gAmd (1)
Thank you for this question. We select d=2 as a practical accuracy–cost trade-off. It is the shortest horizon that includes the remainder of the source layer and one complete succeeding Transformer block. Table 17 shows that increasing the horizon does not consistently improve downstream performance, motivating a closer examination of the ranking benefit and computational cost of extending the observation depth.
To isolate the effect of observation depth, we keep the scoring procedure unchanged and vary only the observation point: the source layer (d=1), the succeeding Transformer block (d=2), or the last Transformer block (full-depth VJP). We also evaluate final-model NLL-gradient scoring, using the mean absolute first-order change in next-token loss induced by each candidate perturbation.
On Qwen3-8B, we evaluated the same quantization candidates in layers 8 and 24. Ranking accuracy measures the fraction of equal-size group pairs for which the predicted ordering of bit-upgrade benefits agrees with the measured reduction in held-out model-output KL.
Compared with d=1, d=2 improves ranking accuracy in three of the four audited conditions. Extending the horizon to the last Transformer block provides higher ranking accuracy at layer 24 but lower accuracy at layer 8, showing that a longer response horizon does not consistently improve ranking qality. The d=2 score also achieves higher ranking accuracy than NLL-gradient scoring in all four comparisons.
We also measured scoring time and peak allocated GPU memory under the same experimental setup:
Both full-depth VJP and NLL-gradient scoring incur higher recorded time and peak memory in these runs. These results support d=2 as a practical balance between capturing cross-layer propagation and limiting scoring cost. We will include these controlled comparisons and clarify this rationale in the revised manuscript.
We also have expanded the Related Work section to discuss the suggested papers and clarify the distinction: