acceptodds
Under review as a conference paper at ICLR 2027

RampQ: Quantizing What Recovery Cannot Fix for Mixed-Precision LLM Quantization

Abstract

Low-bit quantization reduces the memory footprint of large language models but often degrades model quality at aggressive precision. Mixed-precision quantization alleviates this problem by assigning different bit-widths based on module sensitivity or quantization error, while low-rank recovery adds a low-rank branch to recover the quantization error. Pre-recovery quantization error, however, does not necessarily reflect the error that remains after recovery. We show that quantization-error recoverability varies substantially across modules and bit-widths, such that recovery can reorder where higher precision is most beneficial. This motivates allocating precision according to the error that survives recovery. We propose RampQ, a Recovery-Aware Mixed-Precision Quantization framework that performs allocation in the recovered representation space. RampQ constructs bit-specific representations, evaluates each precision choice using module sensitivity and post-recovery residual error, and selects representations under their joint memory cost. A model-level refinement further searches complete-model assignments using selective end-to-end perplexity feedback. Our evaluation results show that RampQ improves the quality memory trade off over mixed precision and low rank recovery baselines. Compared with the best baseline under comparable memory settings, RampQ reduces perplexity on average by 8.1% on WikiText-2 and 5.7% on C4, and improves average zero-shot accuracy by 1.67 percentage points.

Primary area:
optimization
Keywords:
Large Language Model Compression, Recovery-Aware Quantization, Mixed-Precision Bit Allocation, Low-Rank Correction, Post-Training Quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1ftQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RampQ, which allocates module bit-widths based on the residual error after low-rank correction and further refines the allocation through full-model perplexity search, demonstrating a favorable trade-off between model quality and storage across seven LLMs.

Critical strengths
  • The paper shows that low-rank correction can change the precision needs of different modules, providing a clear and useful motivation for mixed-precision quantization.
  • The method jointly considers the error and storage costs of the quantized backbone and correction branch, aligning the allocation with the final model representation.
  • The experiments cover multiple model families, bit budgets, and downstream tasks, with extensive ablations and cost analyses.
Critical weaknesses
  • The allocation score needs further justification: The score measures each module’s error relative to its own 4-bit version, but it remains unclear whether this ratio reliably identifies which module would benefit the overall model most from a higher bit-width.
  • Error propagation is not sufficiently evaluated: The local score summarizes module sensitivity and error magnitude, but its reliability under nonlinear error propagation and interactions among jointly quantized modules remains unclear.
  • Limited scope: The experiments only consider weight-only quantization, leaving the method’s effectiveness under joint weight and activation quantization, and its advantages over SpinQuant and QuaRot, unexplored.
  • Incomplete inference evaluation: The paper lacks an inference performance comparison with AWQ and speedup results across different batch sizes, making its practical deployment benefits difficult to assess.
Questions for authors
  1. Why is each module’s error relative to its own 4-bit version used to guide bit allocation? For example, suppose two modules have the same sensitivity and require the same additional storage to upgrade to 4 bits: A’s error decreases from 100 to 50, while B’s decreases from 20 to 2. The score would favor upgrading B because its relative improvement is larger, although A removes more error in absolute terms. How do the authors verify which choice benefits the overall model more?
  2. How well do the local scores match the actual model loss when multiple modules are quantized together? Although full-model PPL search accounts for cross-module effects, how much does it correct inaccuracies in the local scores?
  3. Can the method be extended to joint weight and activation quantization, where activation quantization errors may also affect low-rank correction, and compared with SpinQuant and QuaRot at the same activation precision and comparable total storage budgets?
  4. Could the authors add an inference performance comparison with AWQ and specify whether it uses vLLM, SGLang, or another engine, while reporting throughput and relative speedups across different batch sizes under the same hardware and input/output lengths?
Reviewer_5DEcOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RampQ addresses post-training weight quantization of LLMs in the Wq + AB deployment form (quantized weights plus low-rank correction A·B). The central claim is that precision allocation should be guided by post-recovery residual rather than pre-recovery quantization error, because low-rank recovery reshuffles which modules are actually bottlenecked. The method has two levels: (1) a module-level ranking criterion that scores each module's importance using the residual after applying its low-rank correction, used for fast bit-width assignment; (2) a model-level search that uses a lightweight surrogate (proxy predictor) to explore inter-module configurations under a unified memory budget that accounts for both the quantized backbone and the low-rank branches. Experiments cover seven models at 2.5–3.5 bit, compared against the strongest available baseline per setting (including AMLQ, which jointly optimizes mixed precision and low-rank rank).

Critical strengths

Recovery-aware allocation is a well-motivated angle for the Wq + AB setting. When the deployment form already includes a low-rank correction branch, using the post-recovery residual — rather than the raw quantization error — to decide precision is the more physically correct signal. This is the paper's non-trivial insight and, if validated quantitatively, would be a useful contribution to the mixed-precision + low-rank quantization literature.

Unified memory budget and strongest-baseline comparison. Counting both the quantized backbone and the low-rank branches toward the memory budget is the honest accounting — some prior work reports bit-width only for Wq and treats AB as free. The comparison protocol (strongest baseline at each bit-width setting) is stricter than reporting a single cherry-picked baseline.

Critical weaknesses

Incremental over AMLQ with mischaracterized relationship. AMLQ already jointly optimizes mixed-precision bit-width and low-rank rank under a memory budget — it is not a simple "quantize-then-recover" pipeline as the introduction implies. RampQ's actual delta is: (1) the allocation criterion switches from pre-recovery error to post-recovery residual, and (2) a model-level search with a surrogate is added. The introduction frames the prior work as "first find the most harmful, then recover" (a serial process), which does not match AMLQ's own description of joint optimization. This mischaracterization inflates the perceived novelty. If the introduction accurately described AMLQ as already joint, RampQ's contribution would read as a refinement of the allocation criterion plus added search — a narrower but more honest claim.

Sub-4-bit cost-benefit not justified: 3.5-bit loses ~6 points while 4-bit GPTQ/AWQ/MXFP4 are near-lossless. The paper operates at 2.5–3.5 bit, but at 3.5 bit — the highest tested setting — RampQ still suffers roughly 6-point degradation on GSM8K relative to FP16. Meanwhile, standard INT4 methods (GPTQ, AWQ) and MXFP4 are well-established as near-lossless versus FP16 on these same benchmarks. This raises a fundamental question about the practical value proposition: saving 0.5 bit at the cost of ~6 accuracy points is a deeply unfavorable trade for the vast majority of deployment scenarios. The paper does not provide a head-to-head comparison with 4-bit GPTQ/AWQ/MXFP4 to let the reader judge this trade-off, nor does it articulate a concrete use case where the extra 0.5-bit savings (and the associated mixed-precision + low-rank operator complexity) are worth the accuracy hit. The sub-4-bit regime only makes sense under extreme memory constraints (e.g., fitting a 70B model into a single GPU), but the paper does not frame its contribution this way or demonstrate that its target models cannot fit at 4-bit. Furthermore, the paper evaluates Qwen3 reasoning models — models designed for extended chain-of-thought and agent/tool-use scenarios — yet restricts evaluation to perplexity and zero-shot classification, entirely missing the agent-related tasks where these models are actually deployed and where quantization damage may compound over long generation horizons.

Questions for authors

Q1: Please provide the 2×2 ablation: {pre-recovery criterion, post-recovery criterion} × {no model-level search, with model-level search}, under identical memory budgets. Did baselines (AMLQ, OmniQ, etc.) receive the same search budget as RampQ? This is the critical experiment for attributing gains to recovery awareness rather than to search.

Q2: At 3.5 bit, RampQ loses ~6 points on MMLU/GSM8K vs. FP16, while 4-bit GPTQ/AWQ/MXFP4 are near-lossless on the same benchmarks. Please provide a direct head-to-head comparison between RampQ at 3.5 bit and GPTQ/AWQ/MXFP4 at 4.0 bit on the same models and tasks. What is the concrete deployment scenario where the 0.5-bit savings justify the accuracy loss — and can the target models not fit at 4-bit?

Q3: The paper evaluates Qwen3 reasoning models, but only on perplexity and zero-shot tasks. Have you measured accuracy on agent-related tasks (tool use, multi-step reasoning, function calling) where these models are actually deployed? Quantization damage may compound over long generation horizons — does RampQ's recovery help mitigate this, or does the gap to FP16 widen on agentic benchmarks?

Q4: What is the quantitative evidence that recovery reshuffles module importance? Please report the rank correlation (Spearman or Kendall τ) between pre-recovery and post-recovery importance scores, broken down by module type (attention vs. MLP, Q/K/V/O vs. gate/up/down) and depth. How often does the top-k set actually change?

Q5: AMLQ simultaneously adapts both bit-width and rank. Does RampQ also adapt rank per module, or is rank fixed? If fixed, the comparison to AMLQ (more degrees of freedom) is not apples-to-apples. If adaptive, how is rank allocated? Also, what are the search cost and inference latency of RampQ vs. baselines?

6 comments

I thank the authors for the additional data and clarifications, and for the substantial work behind them. W1 is accepted and withdrawn; W2 and Q4 are largely answered; the independent gain of the recovery-aware criterion is now supported by the 2×2 ablation in Q1.

One core concern, however, remains unaddressed: the value proposition of sub-4-bit quantization is not established. The newly added 4-bit table in fact reinforces this concern — [email protected] (MMLU 55.41 / GSM8K 22.23 / 6.29 GB) already outperforms AWQ, GPTQ, MXFP4, and even FP16. 4-bit is the best operating point of the proposed method. So what problem does sub-4-bit actually solve?

(1) The cost–benefit trade-off is not justified. Relative to its own 4.0-bit variant, [email protected] saves 11.4% storage at the cost of 2.59 points on GSM8K; relative to AWQ-4.0 it saves 12.2% at the cost of 2.19 points. Please specify the concrete deployment scenario: for which model / GPU-memory configuration does 4-bit fail to fit while 3.5-bit fits exactly? Without such a scenario, sub-4-bit is a compression point with no clearly identifiable use case.

(2) Agent capability degrades fastest at lower bit-widths. MMLU drops only a few points at 2.5–3.0 bit, whereas BFCL Multi-turn drops 7.5 / 13 points — below AWQ-4.0 — and it is precisely long-chain reasoning and tool use that constitute the deployment value of Qwen3-class models. Please provide a capability-wise breakdown of degradation (knowledge /single-step reasoning / long-chain reasoning / tool use), and state explicitly whether sub-4-bit has any viable regime in agentic settings.

(3) The supporting evidence does not yet meet the bar. (a) Q4 shows 70% re-ranking in shallow layers versus 20% in deep layers; please report the bit-width actually allocated per depth segment, and add a negative control replacing recovery with a residual-independent random low-rank perturbation (same rank, same storage) — if accuracy improves equally, the gain stems from parameter count rather than from the criterion. (b) The claim that 4-bit exceeds FP16 requires ≥3 seeds with mean ± SD and paired tests on MMLU/GSM8K; the PPL std < 0.003 in D.2.4 covers only the search process and does not substitute for downstream variance. (c) AMLQ is calibrated on WikiText-2 while RampQ uses SlimPajama — a cross-calibration comparison; please add the two-way control and tabulate calibration set / sample count / sequence length / overlap verification for every method, reporting calibration-subset variance separately from training-seed variance. (d) Please add AIME26/HMMT26, LiveCodeBench v5+v6, and ScreenSpot-Pro/MathVista/V*Bench/MMMU, aggregate by capability axis, and pre-declare a degradation threshold.

If the authors can demonstrate a clear deployment value and applicability boundary for sub-4-bit quantization — or reposition the contribution from "a better method" to "a complementary option under extreme memory constraints" — I would consider raising my rating.

Dear Reviewer 5DEc,

Thank you for taking the time to review our rebuttal. To address remaining concerns about the practical value and limitations of sub-4-bit quantization, we conducted additional analyses and evaluations.

(1)

Consider serving Qwen3-32B on a single 24 GiB RTX 4090 with a required context length of 28,672 tokens. At this length, the estimated memory for weights and the FP16 KV cache exceeds 24 GiB for both 4.0-bit configurations. RampQ at 3.5 bits leaves an estimated 1.34 GiB for other runtime allocations.

The estimate assumes batch size 1, 28,672 total prompt and generated tokens, an FP16 KV cache, and no CPU offloading. For RampQ, we scale the measured storage of the LLaMA-2-13B projection weights and their correction factors by the ratio of projection parameter counts, then add the FP16 embeddings and output head. For AWQ, we use the tensor size from the official Qwen3-32B-AWQ checkpoint index, which includes both the embedding and output tensors. The Qwen3-32B configuration gives an FP16 KV cache size of 7.00 GiB at the target context length.

Qwen3-32B configurationEstimated weight memoryFP16 KV cacheSubtotalHeadroom within 24 GiB
RampQ 3.5≈15.66 GiB7.00 GiB≈22.66 GiB≈+1.34 GiB
RampQ 4.0≈17.31 GiB7.00 GiB≈24.31 GiB≈ −0.31 GiB
AWQ 4.0≈18.01 GiB7.00 GiB≈25.01 GiB≈−1.01 GiB

On LLaMA-2-13B, RampQ at 3.5 bits reduces weight storage by 11.4% relative to RampQ at 4.0 bits and by 12.2% relative to AWQ at 4.0 bits, with corresponding GSM8K decreases of 2.59 and 2.19 percentage points. The Qwen3-32B estimate illustrates when weight savings of this kind can make room for the required context.

SqueezeLLM reports that peak memory for 3.02-bit LLaMA-30B on an A6000 increases from 12.5 GB to 16.1 GB as generation length increases from 128 to 1,024 tokens [1]. At 70B, the weights alone can exceed the memory budget. We estimate 28.96 GiB for the LLaMA-2-70B weights at 3.5 bits. QuIP# reports successful generation at 2 bits and an out-of-memory error at 4 bits on a single RTX 4090 [2].

The same constraint applies to agents that retain messages, tool calls, and tool outputs across model invocations [3]. In the Qwen3-32B example, the 3.5-bit configuration can accommodate the target context if other runtime allocations fit within 1.34 GiB. We will measure peak GPU memory during prefill and decoding and check whether each configuration completes the same workload without offloading.

(2)

We report Qwen3-14B accuracy across weight bit budgets.

CapabilityBenchmarkFP16AWQ 4.0RampQ 4.0RampQ 3.5RampQ 3.0RampQ 2.5
KnowledgeMMLU78.8877.9878.5177.5476.9074.30
Scientific reasoningGPQA Diamond47.9845.9647.4747.4743.4339.90
Tool use in one turnBFCL Multiple92.592.595.095.095.091.0
Tool use across turnsBFCL Multi-turn Base33.532.532.530.526.020.5

At 3.0 and 2.5 bits, the MMLU losses relative to FP16 are 1.98 and 4.58 percentage points, respectively. On GPQA Diamond, which we use as an indirect measure of extended reasoning, the corresponding losses are 4.55 and 8.08 points. On BFCL Multiple, accuracy changes by +2.5 and −1.5 points. On BFCL Multi-turn Base, it drops by 7.5 and 13.0 points.

RampQ at 3.5 bits is an option for agents that require the memory headroom illustrated in (1) and can tolerate the observed 2.0 percentage point loss on BFCL Multi-turn Base relative to AWQ at 4.0 bits. The gap grows to 6.5 and 12.0 points at 3.0 and 2.5 bits, respectively, limiting their suitability for workloads that require reliable tool use across turns.

(3a)

We report the actual bit allocations for LLaMA-3.2-3B using the four depth segments in Q4. Both criteria target a 2.5-bit budget. Pre and Post denote the pre-recovery and post-recovery criteria, respectively. The 1–4 bit columns show the number and share of modules assigned each backbone bit width.

LayersCriterion1 bit2 bits3 bits4 bitsEffective bits
L0–6Pre6 (12.24%)021 (42.86%)22 (44.90%)2.9464
L0–6Post021 (42.86%)20 (40.82%)8 (16.33%)2.8394
L7–13Pre5 (10.20%)024 (48.98%)20 (40.82%)2.9860
L7–13Post035 (71.43%)7 (14.29%)7 (14.29%)2.3906
L14–20Pre20 (40.82%)015 (30.61%)14 (28.57%)2.1135
L14–20Post035 (71.43%)12 (24.49%)2 (4.08%)2.4483
L21–27Pre22 (44.90%)012 (24.49%)15 (30.61%)1.9539
L21–27Post041 (83.67%)8 (16.33%)02.3212
AllPre53 (27.04%)072 (36.73%)71 (36.22%)2.499953
AllPost0132 (67.35%)47 (23.98%)17 (8.67%)2.499884

The post-recovery criterion lowers effective precision in the first two segments and raises it in the last two. In L21–27, it assigns 2 bits to 41 of 49 modules and raises effective precision from 1.954 to 2.321 bits.

For the negative control, we fixed the post-recovery allocation and WqW_q, then replaced the fitted ABAB with random low-rank factors generated independently of the quantization residual. The fitted and random branches have the same per-module rank, factor precision, and storage. We also evaluated WqW_q alone by disabling the correction in the same checkpoint. Before allocation, we alternately update WqW_q and ABAB to minimize the reconstruction error of Wq+ABW_q+AB. Disabling the fitted branch leaves the error that ABAB was fitted to recover uncompensated.

CorrectionPPL ↓NLL per token ↓PIQA length-normalized accuracy ↑
Disabled (WqW_q only)655,437.4513.39351.85%
Random ABAB573,985.67 ± 126,042.7513.243 ± 0.23752.63 ± 0.38%
Fitted ABAB11.262.42275.08%

For the random branch, we report the mean and sample standard deviation over three seeds. The fitted branch reaches 75.08% on PIQA, 22.45 percentage points above the random branch at the same rank and storage. This control tests the effect of fitting the correction to the residual at a fixed allocation. The Q1 ablation tests the effect of the allocation criterion with the same set of recovered representations for both criteria.

(3b)

Appendix D.2.4 reports perplexity variation during search. Given the remaining rebuttal time, we evaluated three existing RampQ runs per bit width on LLaMA-3.2-3B using the same five-shot MMLU and GSM8K protocol as in the paper.

RampQBenchmarkSeed 1 (%)Seed 2 (%)Seed 3 (%)Mean (%)Sample SD (pp)
3.5 bitsMMLU54.11854.07754.10654.1000.021
3.5 bitsGSM8K20.14820.08120.13120.1200.035
4.0 bitsMMLU54.90854.89654.92654.9100.015
4.0 bitsGSM8K21.89721.89321.94021.9100.026
MethodBitsMMLU (%)GSM8K (%)
FP1616.056.4426.00
GPTQ4.054.4620.62
AWQ4.054.5120.24
MXFP44.051.9518.57
RampQ (three-seed mean)3.554.1020.12
RampQ (three-seed mean)4.054.9121.91

The sample standard deviation is at most 0.035 percentage points. On LLaMA-3.2-3B, RampQ at 4.0 bits achieves the highest accuracy among the quantized methods on both benchmarks. Its mean scores are 1.53 points below FP16 on MMLU and 4.09 points below FP16 on GSM8K. In the camera-ready version, we will add a three-seed evaluation of 4.0-bit RampQ on LLaMA-2-13B, the model used in the original comparison, and report the mean and sample standard deviation for both benchmarks.

Reviewer_u9AkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RampQ (Recovery-Aware Mixed-Precision Quantization), a novel framework for low-bit LLM quantization that addresses the misalignment between traditional mixed-precision allocation and low-rank recovery. The authors demonstrate that quantization-error recoverability varies heterogeneously across modules and bit-widths, meaning pre-recovery errors do not reflect true post-recovery residuals. To resolve this, RampQ performs precision allocation directly in the recovered representation space using post-recovery errors and module sensitivity, followed by a surrogate-guided refinement to improve end-to-end performance.

Critical strengths
  1. Identifies and tackles the "recovery-induced precision reprioritization" problem, effectively bridging the gap between low-rank correction and mixed-precision quantization.
  2. Backed by activation-weighted error bounds and spectral analysis, showing that recovery-aware priorities align much better with actual NLL impacts than raw quantization errors.
Critical weaknesses
  1. Despite search acceleration, the offline pipeline remains exceptionally resource-intensive (taking around 40 hours for an 8B model), posing scalability challenges for larger models.
  2. Computing the required post-recovery residual error mandates fully constructing, optimizing, and evaluating low-rank branches across all candidate bit-widths for every module before allocation can even begin.
  3. Generates a highly heterogeneous bit-width configuration combined with dynamic low-rank correction branches across layers, introducing significant memory divergence and runtime serving overheads that hinder practical hardware acceleration.
Questions for authors
  1. Given the heavy burden of fully optimizing Wq+ABW_q + AB representations for multiple candidate bit-widths per module prior to allocation, could this strategy be approximated or streamlined? How do you plan to scale this to 70B+ or MoE models?
  2. Because the allocation yields irregular bit-widths and parallel low-rank branches, standard engines (like vLLM) cannot execute this efficiently. Do you plan to release customized, fused CUDA kernels to achieve real-world latency speedups?
  3. Since the greedy initialization (RampQ-G) already performs competitively, is the expensive surrogate-guided refinement strictly necessary, or can a tuned greedy allocation serve as a lightweight alternative?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.