acceptodds
Under review as a conference paper at ICLR 2027

DepthQuant: Depth Error Propagation for Mixed-Precision Bit Allocation

Abstract

Mixed-precision quantization assigns a bit width to each unit of a large language model under a fixed bit budget, but most current methods decide the allocation from single-layer information and fall well short of full precision. This may be partly because single-layer methods miss information from deeper layers: similar quantization errors can produce different responses at the model output. To address this, we propose DepthQuant, a mixed-precision quantization method that uses depth error propagation as the signal that drives the bit allocation. By measuring the response each precision change induces one layer deeper and how much it changes the model output, DepthQuant spends the bit budget where more precision reduces that measured change the most. Built on this principle, DepthQuant introduces a cohesive multi-scale design: short-depth response scoring ranks precision changes by the response they induce one layer deeper, full-depth allocation then adjusts the resulting allocation by the error's effect on the model output, and a final step sets each row's range from information in its own layer. Experiments show that DepthQuant improves on the compared baselines in most evaluated settings and in every reported average-2-bit setting, reducing perplexity by up to 40.6% at 2.5-bit average precision relative to the best-performing baseline and raising zero-shot accuracy by up to 11.6% at 2-bit average precision, where these methods degrade the most.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixed-precision quantization, low-bit quantization, post-training quantization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gAmdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DepthQuant, a mixed-precision weight-only post-training quantization framework for large language models. It first ranks candidate precision assignments using quantization perturbations propagated over a short network segment. It then redistributes bit budgets across layers using isolated model-output KL divergences and dynamic programming over the resulting allocation paths, while preserving the total parameter-bit budget. Finally, it refines per-row quantization ranges with the bit allocation fixed. Experiments cover eight models from 4B to 32B at average precisions of 2, 2.5, and 3 bits.

Critical strengths
  1. Well-motivated study of an important quantization problem.

    The paper is motivated by an important practical limitation: local reconstruction error may not reliably reflect the downstream impact of quantization. Accounting for error propagation in mixed-precision allocation remains an important challenge, particularly at extremely low precision. The paper's focus on this problem is meaningful and practically relevant.

  2. Promising empirical results.

    The paper reports improvements over the evaluated baselines across multiple model families and precision settings, including substantial gains in several configurations at an average precision of 2 bits. Both perplexity and downstream task accuracy are evaluated.

Critical weaknesses
  1. Insufficient justification of short-depth response scoring. The limitations of local reconstruction criteria have already been discussed in previous works (see reference). However, the paper does not establish why short-depth responses offer a better accuracy–cost trade-off than sensitivity estimates based on the final model loss. However, Table 17 shows no consistent advantage of d=2d=2 over d=1d=1, which could indicates short-depth response scoring is not a reliable proxy. Although Tables 19, 21, and 25 provide evidence supporting that short-depth response scoring is consistent with the final outputs, they do not isolate the benefit of observing an additional layer or compare against global-loss scoring under controlled conditions.

  2. Limited validation of the restricted allocation space and surrogate objective.

    Full-Depth Allocation optimizes over frozen within-layer allocation paths and a narrow per-layer budget window. Dynamic programming is exact within this restricted domain, but potentially beneficial allocations may be excluded. Table 16 only examines a narrow range of caps on two models at 3 bits. Moreover, the objective sums isolated per-layer KL divergences rather than directly optimizing joint-model divergence. The diagnostics in Tables 22 and 23 are useful but limited. Broader cap ablations and stronger validation under joint, particularly 2-bit, quantization would strengthen the method.

  3. Incremental contribution of range refinement.

    The range search in Section 3.4 is closely related to established clipping and range-optimization techniques, including AWQ's clipping search and OmniQuant's learnable weight clipping. Existing ablations demonstrate its usefulness but not its advantage over these alternatives. Since this component contributes substantially to some reported gains, comparisons using the same bit allocation and quantization backend are needed to better isolate the contribution of the proposed scoring and allocation mechanisms.

  4. Incomplete strong-baseline comparisons at 2 bits.

    SignRoundV2 is evaluated at 2.5/3 bits but omitted from the average-2-bit comparison. Its use of uniform precision at this budget does not sufficiently justify exclusion, since uniform RTN and GPTQ are included. Adding SignRoundV2's W2A16 configuration would make the extreme-low-bit evaluation more convincing. SFMP would also be a relevant additional mixed-precision baseline, with budget accounting and evaluation protocols carefully aligned.

  5. Limited model scale and downstream evaluation coverage.

    Most of the models evaluated in the experiment are small-scale, and the largest is the only 32B. Besides, the evaluation lacks knowledge-intensive benchmarks such as MMLU. I would suggest the authors provide at least two more models at 32B scale or above.

[1]MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design

[2]AutoMixQ: Automatic Mixed-precision Quantization for Deploying Bit-Efficient LLMs

[3]GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance

[4]SqueezeLLM: Dense-and-Sparse Quantization

Questions for authors

See weakness

9 comments
AuthorsOpenReview ↗

Response to Reviewer gAmd (1)

W1. Insufficient justification of short-depth response scoring.

Thank you for this question. We select d=2 as a practical accuracy–cost trade-off. It is the shortest horizon that includes the remainder of the source layer and one complete succeeding Transformer block. Table 17 shows that increasing the horizon does not consistently improve downstream performance, motivating a closer examination of the ranking benefit and computational cost of extending the observation depth.

To isolate the effect of observation depth, we keep the scoring procedure unchanged and vary only the observation point: the source layer (d=1), the succeeding Transformer block (d=2), or the last Transformer block (full-depth VJP). We also evaluate final-model NLL-gradient scoring, using the mean absolute first-order change in next-token loss induced by each candidate perturbation.

On Qwen3-8B, we evaluated the same quantization candidates in layers 8 and 24. Ranking accuracy measures the fraction of equal-size group pairs for which the predicted ordering of bit-upgrade benefits agrees with the measured reduction in held-out model-output KL.

Layer indexUpgraded = 1d = 2Full-depth VJPFinal-model NLL gradient
82 → 385.93%89.93%49.78%75.11%
83 → 477.48%78.52%57.19%70.52%
242 → 389.48%88.74%91.26%84.15%
243 → 489.93%90.67%91.85%76.44%

Compared with d=1, d=2 improves ranking accuracy in three of the four audited conditions. Extending the horizon to the last Transformer block provides higher ranking accuracy at layer 24 but lower accuracy at layer 8, showing that a longer response horizon does not consistently improve ranking qality. The d=2 score also achieves higher ranking accuracy than NLL-gradient scoring in all four comparisons.

We also measured scoring time and peak allocated GPU memory under the same experimental setup:

ScorerLayer 8 wall time (s)Layer 24 wall time (s)Peak allocated GPU memory (GiB)
d = 1174.07(×0.9\times 0.9)256.94(×1.1\times 1.1)2.21(×0.7\times 0.7)
d = 2(ours)192.96(×1.0\times 1.0)241.11(×1.0\times 1.0)3.23(×1.0\times 1.0)
Full-depth VJP5,771.25(×29.9\times 29.9)3,306.29(×13.7\times 13.7)16.69(×5.2\times 5.2)
Final-model NLL gradient1,432.22 (×7.4\times 7.4)728.13 (×3.0\times 3.0)16.69 (×5.2\times 5.2)

Both full-depth VJP and NLL-gradient scoring incur higher recorded time and peak memory in these runs. These results support d=2 as a practical balance between capturing cross-layer propagation and limiting scoring cost. We will include these controlled comparisons and clarify this rationale in the revised manuscript.

We also have expanded the Related Work section to discuss the suggested papers and clarify the distinction:

Prior work incorporates information beyond local reconstruction in several ways. MixQuant evaluates distortion under quantized upstream contexts. MixLLM, SqueezeLLM, and GuidedQuant use loss-derived sensitivity to guide precision allocation or quantizer optimization. AMQ, AutoMixQ, and MixFrag guide allocation through predicted model quality, global losses, or output-distribution changes, respectively. DepthQuant couples short-depth response scoring with full-depth output calibration, capturing error amplification beyond the perturbed module with short backward passes while retaining global guidance for cross-layer allocation.

AuthorsOpenReview ↗

Response to Reviewer gAmd (2)

W2. Limited validation of the restricted allocation space and surrogate objective.

Thank you for these suggestions. We added experiments on joint 2-bit quantization and wider budget windows with different candidate-state resolutions.

On Qwen3-4B at a requested average of 2 bits, we compared allocations before and after Full-Depth Allocation, keeping the total parameter-bit budget and reconstruction procedure fixed. All layers are quantized simultaneously when measuring the following results:

AllocationJoint KL on pricing text ↓Joint KL on held-out text ↓Held-out NLL ↓
Short-Depth Response Scoring2.043301.891483.86442
+ Full-Depth Allocation1.939521.783393.75129

Full-Depth Allocation reduces joint KL by 5.08% on the pricing text and 5.71% on held-out text, while also lowering held-out NLL. This supports the usefulness of isolated-layer pricing for selecting allocations that improve the jointly quantized model in this setting.

We also evaluated wider per-layer budget windows on the same model at the same total budget:

Cap cStates per layerStep (bit)Pricing time (h)WikiText-2 PPL ↓C4 PPL ↓ZS (%) ↑
0.1 (reported)50.052.7521.7237.6051.18
0.550.252.7421.4038.7450.81
0.5110.105.3621.4038.2651.81
0.5210.059.7921.0938.2651.37

Widening the window and refining the step size do not consistently improve performance, while increasing the number of states from 5 to 21 raises pricing time from 2.74 to 9.79 hours. Broader and denser candidate coverage therefore does not guarantee better final allocations, supporting the reported window as a practical accuracy–cost trade-off. We will include these additional ablations and joint-quantization measurements in the revised manuscript.

W3. Incremental contribution of range refinement.

Thank you for this comment. To distinguish the contributions of allocation and reconstruction, we conducted two complementary comparisons: different allocation maps under the same reconstruction procedure, and different reconstruction procedures under a fixed allocation map.

We first compare the frozen allocation maps produced by DepthQuant, SliM, AMQ, and SignRoundV2 on Qwen3-8B at a nominal average of 3 bits. Each method retains its allocation and bit budget from the corresponding main-table configuration. Within each block below, all maps use the same reconstruction procedure and evaluation protocol.

GPTQ

AllocationWiki PPL ↓C4 PPL ↓ZS (%) ↑
SliM11.1717.8868.12
AMQ11.1117.8967.98
SignRoundV214.1121.9767.30
DepthQuant10.7917.2369.99

GPTQ ++ Per-Row Range Refinement

AllocationWiki PPL ↓C4 PPL ↓ZS (%) ↑
SliM11.3718.2870.63
AMQ10.7817.5469.92
SignRoundV212.0819.6069.05
DepthQuant10.6417.0971.95

AWQ

AllocationWiki PPL ↓C4 PPL ↓ZS (%) ↑
SliM11.0917.6270.69
AMQ11.4317.8270.39
SignRoundV213.3721.0069.67
DepthQuant11.1617.5571.48

DepthQuant achieves the highest six-task accuracy among the tested allocations under all three reconstruction settings, exceeding the strongest competing map by 1.87 points under GPTQ, 1.32 points under GPTQ with refinement, and 0.79 points under AWQ. Its allocation advantage therefore persists both without our range refinement and under an alternative reconstruction procedure.

We then compare reconstruction procedures using the same fixed DepthQuant allocation on two models at a requested average of 3 bits. The OmniQuant backend uses learnable weight clipping with the allocation fixed.

ModelReconstructionWiki PPL ↓C4 PPL ↓ZS (%) ↑
Qwen3-8BAWQ11.1617.5571.48
Qwen3-8BOmniQuant11.4818.0269.58
Qwen3-8BGPTQ + Per-Row Range Refinement10.6417.0971.95
Qwen2.5-7BAWQ7.9313.4371.89
Qwen2.5-7BOmniQuant7.6713.3972.61
Qwen2.5-7BGPTQ + Per-Row Range Refinement7.4513.1072.98

For the same allocation, GPTQ with Per-Row Range Refinement achieves lower perplexity on both datasets and higher six-task accuracy than AWQ and OmniQuant on both models. Together with the common-backend comparisons, these results support both the allocation and reconstruction choices.

AuthorsOpenReview ↗

Response to Reviewer gAmd (3)

W4. Incomplete strong-baseline comparisons at 2 bits.

Thank you for pointing out these missing comparisons. We added SignRoundV2’s uniform W2A16 configuration under the same perplexity and six-task evaluation protocol.

ModelMethod, average 2 bitsWiki PPL ↓C4 PPL ↓ZS (%) ↑
Qwen3-8BSignRoundV2 W2A1624.8827.5461.45
Qwen3-8BDepthQuant18.6740.3957.12
Qwen2.5-7BSignRoundV2 W2A1614.0419.9763.38
Qwen2.5-7BDepthQuant12.6931.8357.75

SignRoundV2 achieves lower C4 perplexity and higher six-task accuracy on both models, while DepthQuant achieves lower WikiText-2 perplexity. The gap may reflect two factors: SignRoundV2 optimizes rounding and quantization ranges through block reconstruction, whereas DepthQuant focuses on bit allocation with GPTQ and Per-Row Range Refinement; DepthQuant’s 1-bit groups also introduce more aggressive compression at this budget. DepthQuant’s allocation advantage is supported by the main-table mixed-precision results and the common-backend comparisons in W3. Learned reconstruction could further complement our allocation.

We also added SFMP at an average target-weight precision of 3 bits on Qwen3-4B and Qwen2.5-7B, using its native reconstruction procedure and the shared evaluation protocol.

ModelMethodWiki PPL ↓C4 PPL ↓ZS (%) ↑
Qwen3-4BSFMP16.4623.0367.79
Qwen3-4BDepthQuant14.5021.5467.37
Qwen2.5-7BSFMP7.7913.0272.08
Qwen2.5-7BDepthQuant7.4513.1072.98

Budget labels count target-weight bits and exclude quantization metadata, consistent with the main tables. SFMP’s recorded storage cost is approximately 3.25 bits per weight when metadata is included. DepthQuant’s realized target-weight averages are 3.0008 and 3.0016 bits, respectively.

We will incorporate both baselines into the corresponding evaluations and revise the associated comparisons and claims to reflect these results.


W5. Limited model scale and downstream evaluation coverage.

Thank you for these suggestions. We expanded the evaluation with Llama-2-70B, Qwen2.5-32B, and zero-shot MMLU.

Larger-model evaluation. We evaluated Llama-2-70B and Qwen2.5-32B at an average of 2 bits under the same perplexity and six-task evaluation protocol:

Model, average 2 bitsMethodWiki PPL ↓C4 PPL ↓ZS (%) ↑
Llama-2-70BRTN27.0447.6348.77
Llama-2-70BGPTQ6.8811.7756.30
Llama-2-70BSliM5.8310.7566.03
Llama-2-70BDepthQuant5.168.6369.68
Qwen2.5-32BRTN1,145.821,594.4239.95
Qwen2.5-32BGPTQ13.8732.2351.18
Qwen2.5-32BSliM9.1418.4365.72
Qwen2.5-32BDepthQuant8.1115.5270.61

DepthQuant improves six-task accuracy over the strongest evaluated baseline, SliM, by 3.65 percentage points on Llama-2-70B and 4.89 points on Qwen2.5-32B, while lowering perplexity on both datasets.

Knowledge-intensive evaluation. We evaluated zero-shot MMLU across all 57 subjects and 14,042 test questions, aggregating accuracy over questions. Each quantized method uses its native reconstruction procedure, with precision settings following the corresponding configurations in our 3-bit comparisons.

MethodQwen3-4B MMLU (%) ↑Qwen2.5-7B MMLU (%) ↑
FP reference, 16 bits68.3571.88
RTN52.0265.15
GPTQ56.6265.24
SliM60.3865.13
AMQ60.1567.48
SignRoundV263.8667.74
SFMP63.4367.37
DepthQuant63.4368.42

Among the evaluated quantized methods, DepthQuant ranks first on Qwen2.5-7B and joint second on Qwen3-4B. We will incorporate these additional evaluations into the revised manuscript.


We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.

Reviewer_tiEqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper makes an interesting observation that quantization in a given layer has a significantly different impact on downstream error. It then presents DepthQuant, a mixed-precision post-training quantization (PTQ) technique that employs three steps. First, it uses Short Depth Scoring to capture the impact on the immediately succeeding layer. Second, a Full-Depth Allocation is used to further adjust quantization based on the impact on the output. Finally, there is a per-row range assignment.

Critical strengths
  • The paper presents an interesting motivating experiment on the impact of quantization on downstream error
  • Significant gains compared to prior work in the low-bit regime
Critical weaknesses
  • The experiment results are presented on single runs
  • Most of the gains come from per-row range refinement in Table 6
Questions for authors

Thanks for submitting your paper to ICLR ‘27. The paper motivates the problem well by showing significant differences in downstream error impact. The three-stage design is interesting. However, two key issues stand out to me.

The experimental results are presented on single runs. The results do not include any standard deviations or error bars. Given that quantization has a varying impact downstream, it is essential to evaluate each experiment with multiple runs. It would help to include standard deviations across a few probe seeds, at least for a subset of settings.

In Table 6, most of the gains come from per-row range refinement. This runs counter to the paper's original argument that end-to-end impact is important. The refinement minimizes local weight-reconstruction error. How would you reconcile these differences? Also, would the baselines benefit in a similar manner from the same per-row range search?

7 comments
AuthorsOpenReview ↗

Response to Reviewer tiEq (1)

W1. The experiment results are presented on single runs.

Q1. The experimental results are presented on single runs. The results do not include any standard deviations or error bars. Given that quantization has a varying impact downstream, it is essential to evaluate each experiment with multiple runs. It would help to include standard deviations across a few probe seeds, at least for a subset of settings.

We thank the reviewer for asking this question. We repeated the complete native pipeline of SliM, SignRoundV2 and DepthQuant on Qwen3-4B at a requested average of 3 bits under three independent random seeds. Each method keeps its own configuration. The evaluation data, the evaluation protocol and the hardware are fixed.

For reference, the main-table results for this model and budget are:

MethodWikiText-2 PPL ↓C4 PPL ↓ZS (%) ↑
SliM15.5222.4264.95
SignRoundV219.4326.8265.52
DepthQuant14.5021.5467.37

Seed 1

MethodWikiText-2 PPL ↓C4 PPL ↓ZS (%) ↑Δ ZS vs. main table (pp)
SliM15.1922.4565.97+1.02
SignRoundV219.6826.4364.36−1.16
DepthQuant14.7321.6167.84+0.47

Seed 2

MethodWikiText-2 PPL ↓C4 PPL ↓ZS (%) ↑Δ ZS vs. main table (pp)
SliM15.3622.4265.99+1.04
SignRoundV219.5326.3365.78+0.26
DepthQuant14.5121.7267.64+0.27

Seed 3

MethodWikiText-2 PPL ↓C4 PPL ↓ZS (%) ↑Δ ZS vs. main table (pp)
SliM14.9622.3666.38+1.43
SignRoundV219.6926.9265.50−0.02
DepthQuant14.5821.7867.67+0.30

Across the three seeds

MethodWikiText-2 PPL ↓C4 PPL ↓ZS (%) ↑Δ ZS vs. main table (pp)
SliM15.17 ± 0.2022.41 ± 0.0466.11 ± 0.23+1.16
SignRoundV219.63 ± 0.0926.56 ± 0.3265.22 ± 0.75−0.30
DepthQuant14.61 ± 0.1121.70 ± 0.0967.72 ± 0.11+0.35

Δ is the difference in ZS from the main table for the same model and budget. Across the three runs the complete method varies by 0.11 in WikiText-2 perplexity, 0.09 in C4 and 0.11 points in ZS (sample standard deviations), and it keeps the same rank under every seed: DepthQuant has the lowest perplexity on both datasets and the highest ZS in all three runs. This spread is small relative to the margins we report, so the comparison in the main table does not rest on the choice of seed.

AuthorsOpenReview ↗

Response to Reviewer tiEq (2)

W2. Most of the gains come from per-row range refinement in Table 6

Q2. In Table 6, most of the gains come from per-row range refinement. This runs counter to the paper's original argument that end-to-end impact is important. The refinement minimizes local weight-reconstruction error. How would you reconcile these differences? Also, would the baselines benefit in a similar manner from the same per-row range search?

Thank you for this observation. Table 6 starts from Short-Depth Response Scoring, which already uses propagated responses to determine the allocation. Its incremental gains therefore do not isolate the contribution of propagation-aware scoring itself. Allocation determines which precision each group receives, while range refinement calibrates the quantization parameters for that fixed allocation.

To examine these contributions, we reconstructed the allocation maps produced by DepthQuant, SliM, AMQ, and SignRoundV2 on Qwen3-8B at a nominal average of 3 bits using the same GPTQ backend, with and without the same range refinement. We retain each method’s allocation and bit budget from the corresponding main-table configuration. All tests use group size 128, the same 128 calibration sequences of 2,048 tokens, and the same evaluation protocol. This control replaces each method’s native quantizer while preserving its allocation.

BackendAllocationWikiText-2 PPL ↓C4 PPL ↓ZS (%) ↑
GPTQSliM11.1717.8868.12
GPTQAMQ11.1117.8967.98
GPTQSignRoundV214.1121.9767.30
GPTQDepthQuant10.7917.2369.99
GPTQ + range refinementSliM11.3718.2870.63
GPTQ + range refinementAMQ10.7817.5469.92
GPTQ + range refinementSignRoundV212.0819.6069.05
GPTQ + range refinementDepthQuant10.6417.0971.95

Applying the same range refinement improves six-task accuracy for all three competing allocations, showing that its benefit is not specific to DepthQuant. DepthQuant retains the highest accuracy among the tested allocations, exceeding the strongest competitor by 1.87 percentage points without refinement and 1.32 points with refinement, under the shared GPTQ backend. We will include this controlled comparison and clarify the interpretation of Table 6 in the revised manuscript.


We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.

AuthorsOpenReview ↗

Thank You for Your Review. We Welcome Further Discussion

Dear Reviewer tiEq,

Thank you again for the time and care you have devoted to reviewing our work. Your suggestions have helped us strengthen the experimental evidence, and we have provided point-by-point responses with additional experiments and corresponding revisions to the manuscript.

As the discussion deadline approaches, we would be grateful if you could take a look at our replies when your schedule permits. We hope the additions address your concerns and would welcome any further feedback. If anything remains unclear or unresolved, we would appreciate the opportunity to clarify it before the discussion closes.

Best regards,

Authors of Submission 2496

Reviewer_TTzJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DepthQuant, a mixed-precision post-training quantization (MPQ) method for large language models (LLMs). Traditional MPQ methods typically rely on single-layer local reconstruction errors or activation statistics, failing to capture how errors propagate and evolve across network depth. To address this, DepthQuant introduces a multi-scale framework consisting of: (1) Short-Depth Response Scoring to rank layer-wise precision changes using randomized Hutchinson-type estimators and vector-Jacobian products (VJPs); (2) Full-Depth Allocation utilizing a dynamic programming approach to redistribute bits across layers based on model-output KL divergence under a fixed parameter budget; and (3) Per-Row Range Refinement to optimize local weight ranges. Experiments across multiple model families demonstrate strong perplexity.

Critical strengths
  1. Moving beyond naive single-layer metrics, the incorporation of cross-layer error propagation and output-level calibration provides a more solid theoretical foundation for bit allocation.
  2. The pipeline effectively bridges local weight adaptation, short-range directional response, and global output-distribution matching.
Critical weaknesses
  1. The reliance on randomized Rademacher probes, Jacobian-vector products across horizons, and full-depth state pricing introduces a heavy computational burden. As reported by the authors, a single one-time probe pass for an 8B model takes roughly 10.4 hours, making the offline calibration pipeline extremely resource-intensive.
  2. Because DepthQuant is strictly restricted to weight-only quantization (keeping activations in floating-point), and employs mixed-precision bit widths across fine-grained weight groups, the paper does not claim or evaluate inference-time latency or memory bandwidth gains.
  3. Since activations remain in high-precision float (e.g., BF16/FP16) while weights are stored in diverse low-bit mixed precisions, the runtime engine must continuously dequantize weights back to activation formats (or execute specialized mixed-precision GEMM kernels). The paper fails to discuss whether such non-uniform bit allocations introduce decoding/dequantization bottlenecks during online serving, offsetting the theoretical parameter-bit savings.
Questions for authors
  1. Given that your framework relies on irregular, fine-grained mixed-precision allocation across layers and groups, how do you handle the runtime dequantization overhead? Since activations are kept in full precision (BF16/FP16), the compute units must dynamically handle mixed-bit weight dequantization. Without specialized custom CUDA kernels, irregular bit-widths typically result in severe memory divergence and worse latency than uniform quantization. Do you have benchmark results regarding actual inference speedup, or is this work purely theoretical/accuracy-oriented?
  2. The pipeline requires extensive offline profiling (e.g., ~10.4 hours for an 8B model). While acceptable for post-training compression of static foundation models, please elaborate on how scalable this approach is as model scales increase further (e.g., 70B+ or MoE models).
  3. The choice to target weight-only quantization avoids activation quantization errors, but modern high-performance LLM serving systems heavily rely on activation quantization (e.g., W8A8 or W4A4) to alleviate memory bandwidth walls. Could the depth error propagation principle be extended to activation quantization as well? A brief discussion on this limitation would strengthen the conclusion.
5 comments
AuthorsOpenReview ↗

Response to Reviewer TTzJ (1)

W1. The reliance on randomized Rademacher probes, Jacobian-vector products across horizons, and full-depth state pricing introduces a heavy computational burden. As reported by the authors, a single one-time probe pass for an 8B model takes roughly 10.4 hours, making the offline calibration pipeline extremely resource-intensive.

Q2. The pipeline requires extensive offline profiling (e.g., ~10.4 hours for an 8B model). While acceptable for post-training compression of static foundation models, please elaborate on how scalable this approach is as model scales increase further (e.g., 70B+ or MoE models).

Thank you for this question. To directly assess scalability, we evaluated the complete DepthQuant pipeline on Llama-2-70B, measuring both offline runtime and quantization accuracy. For clarity, the 10.4-hour figure in Table 27 covers the complete scoring pass over 128 calibration sequences for an 8B model.

For Llama-2-70B, scoring and final reconstruction each use one RTX A6000 GPU, while candidate evaluation uses four. The stage-wise wall-clock times are:

Requested average precisionShort-Depth Response ScoringFull-Depth Allocation: candidate evaluationFinal reconstruction with Per-Row Range Refinement
2 bits131.87 h37.78 h1.50 h
2.5 bits129.54 h39.49 h1.64 h

Scoring and candidate evaluation dominate the offline cost; the subsequent dynamic-programming solve is inexpensive. Shared probes over a fixed short horizon constrain the scoring cost. The resulting allocation and quantization parameters remain fixed during inference.

Accuracy on Llama-2-70B is reported below:

Requested average precisionMethodWikiText-2 PPL ↓C4 PPL ↓Six-task ZS (%) ↑
2 bitsRTN27.0447.6348.77
2 bitsGPTQ6.8811.7756.30
2 bitsSliM5.8310.7566.03
2 bitsDepthQuant5.168.6369.68(+3.65)
2.5 bitsSignRoundV24.757.0574.89
2.5 bitsDepthQuant4.086.6875.91(+1.02)

DepthQuant improves six-task accuracy over the strongest evaluated baseline by 3.65 percentage points at 2 bits and 1.02 percentage points at 2.5 bits, with lower perplexity on both datasets. These results demonstrate applicability at 70B scale, with an offline cost–accuracy tradeoff. We will include these measurements and hardware details in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer TTzJ (2)

W2. ... and employs mixed-precision bit widths across fine-grained weight groups, the paper does not claim or evaluate inference-time latency or memory bandwidth gains.

W3. Since activations remain in high-precision float (e.g., BF16/FP16) while weights are stored in diverse low-bit mixed precisions, the runtime engine must continuously dequantize weights back to activation formats (or execute specialized mixed-precision GEMM kernels). The paper fails to discuss whether such non-uniform bit allocations introduce decoding/dequantization bottlenecks during online serving, offsetting the theoretical parameter-bit savings.

Q1. Given that your framework relies on irregular, fine-grained mixed-precision allocation across layers and groups, how do you handle the runtime dequantization overhead? Since activations are kept in full precision (BF16/FP16), the compute units must dynamically handle mixed-bit weight dequantization. Without specialized custom CUDA kernels, irregular bit-widths typically result in severe memory divergence and worse latency than uniform quantization. Do you have benchmark results regarding actual inference speedup, or is this work purely theoretical/accuracy-oriented?

Thank you for raising these deployment concerns. We implemented packed inference for models quantized with the complete DepthQuant pipeline and benchmarked it against uniform 3-bit GPTQ under the same backend.

DepthQuant assigns one bit width to each group of 128 consecutive input columns, shared by all output rows. Our custom CUDA kernel adapts SliM’s mixed-precision design, using group-aligned 64-column tiles and precomputed packed offsets. The bit width is therefore uniform within each tile. During decoding, dequantization is fused with the matrix–vector product: the kernel directly consumes packed codes, scales, and zero points without materializing a full-precision weight matrix in GPU memory. Allocation and Per-Row Range Refinement are performed offline.

We evaluated Qwen2.5-7B on one RTX 3090, with BF16 activations, batch size 1, KV caching, and the same PyTorch/custom-CUDA packed backend. GPTQ uses uniform 3-bit weights; DepthQuant uses an average of 3.0016 bits. Generation throughput is measured over ten fixed prompts, generating 512 tokens per prompt.

MethodWM (GiB)RM (GiB)Token/s ↑
GPTQ4.5285.37412.06
DepthQuant4.5295.37813.71

WM denotes stored model size, and RM denotes peak allocated GPU memory during generation.

Prefill and decoding are also timed separately at two input lengths. Each value is the median of five runs after warm-up.

Input tokensMethodPrefill latency (ms) ↓Decode latency (ms/token) ↓
128GPTQ306.0479.74
128DepthQuant291.5070.31
512GPTQ1,053.7681.17
512DepthQuant1,037.4271.70

Under the same packed backend, DepthQuant achieves higher generation throughput and lower decoding latency than uniform 3-bit GPTQ, with nearly identical storage and peak memory. Thus, we observe no additional decoding slowdown in this setting. We will include these implementation details and measurements in the revision.

W2. Because DepthQuant is strictly restricted to weight-only quantization (keeping activations in floating-point)...

Q3. The choice to target weight-only quantization avoids activation quantization errors, but modern high-performance LLM serving systems heavily rely on activation quantization (e.g., W8A8 or W4A4) to alleviate memory bandwidth walls. Could the depth error propagation principle be extended to activation quantization as well? A brief discussion on this limitation would strengthen the conclusion.

Thank you for this suggestion. Our current evaluation focuses on weight-only quantization, while the depth-propagation principle could also be extended to activation quantization. For a linear module, the joint output perturbation is

δy=QW(W)QA(x)−Wx=ΔWx+WΔx+ΔWΔx,\delta_{y} = Q_W(W)Q_A(x) - Wx = \Delta W x + W \Delta x + \Delta W \Delta x,

where ΔW=QW(W)−W\Delta W = Q_W(W)- W and Δx=QA(x)−x\Delta x = Q_A(x) - x.

This captures both error sources and their interaction, providing a starting point for extending the propagation-based scoring to joint quantization. We will clarify the current weight-only scope and discuss this extension in the revised manuscript.


We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.

AuthorsOpenReview ↗

Kind Reminder: We Would Appreciate Your Feedback on Our Responses

Dear Reviewer TTzJ,

Thank you again for your time and constructive feedback. We have provided point-by-point responses and additional experimental results, and hope these address your concerns.

As the discussion deadline approaches, we would appreciate it if you could review our replies when your schedule permits. Please let us know if anything remains unclear or requires further clarification; we would be happy to respond.

Best regards,

Authors of Submission 2496

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Summary of Reviews

We thank all three reviewers for their time and constructive feedback. We are grateful that the reviewers recognized the following strengths of DepthQuant:

  1. "Well-motivated study of an important quantization problem." (gAmd)
  2. "Moving beyond naive single-layer metrics, the incorporation of cross-layer error propagation and output-level calibration provides a more solid theoretical foundation for bit allocation." (TTzJ)
  3. "The pipeline effectively bridges local weight adaptation, short-range directional response, and global output-distribution matching." (TTzJ)
  4. "Significant gains compared to prior work in the low-bit regime" (tiEq), and "Promising empirical results." (gAmd)

Following the reviewers' suggestions, we have supplemented the paper with the following experiments and discussions:

  1. Benchmarked packed inference against uniform GPTQ, reporting throughput, prefill and decoding latency, model storage, and peak GPU memory under a shared CUDA backend. We also explain the group-aligned layout and fused dequantization used for mixed-bit execution.
  2. Extended evaluation to Llama-2-70B and Qwen2.5-32B, with perplexity and six-task accuracy comparisons, and reported the offline computational costs of the main stages at 70B scale.
  3. Repeated the complete DepthQuant, SliM, and SignRoundV2 pipelines with three random seeds on Qwen3-4B at nominal 3 bits, reporting individual runs and mean ± sample standard deviation. DepthQuant retains the best perplexity and six-task accuracy among these methods in every run.
  4. Compared local, short-depth, full-depth response, and final-model NLL-gradient scoring, reporting ranking accuracy on identical candidates and computational-cost measurements to examine the accuracy–cost trade-off.
  5. Validated Full-Depth Allocation under joint 2-bit quantization and expanded the budget-window ablation to wider ranges and denser candidate grids, reporting joint divergence, downstream performance, and pricing cost.
  6. Separated allocation and reconstruction through controlled comparisons: frozen maps from four methods under shared GPTQ, GPTQ with Per-Row Range Refinement, and AWQ backends; and fixed-DepthQuant-map comparisons with AWQ and OmniQuant.
  7. Added native SignRoundV2 W2A16, SFMP, and zero-shot MMLU comparisons, expanded the discussion of the suggested related work, and outlined how propagation-based scoring could extend to joint weight–activation quantization.

You can find our individual responses below your review comments. If you have any more concerns or questions, we are entirely open to continuing the discussion with you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.