- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents RampQ, which allocates module bit-widths based on the residual error after low-rank correction and further refines the allocation through full-model perplexity search, demonstrating a favorable trade-off between model quality and storage across seven LLMs.
Critical strengths
- The paper shows that low-rank correction can change the precision needs of different modules, providing a clear and useful motivation for mixed-precision quantization.
- The method jointly considers the error and storage costs of the quantized backbone and correction branch, aligning the allocation with the final model representation.
- The experiments cover multiple model families, bit budgets, and downstream tasks, with extensive ablations and cost analyses.
Critical weaknesses
- The allocation score needs further justification: The score measures each module’s error relative to its own 4-bit version, but it remains unclear whether this ratio reliably identifies which module would benefit the overall model most from a higher bit-width.
- Error propagation is not sufficiently evaluated: The local score summarizes module sensitivity and error magnitude, but its reliability under nonlinear error propagation and interactions among jointly quantized modules remains unclear.
- Limited scope: The experiments only consider weight-only quantization, leaving the method’s effectiveness under joint weight and activation quantization, and its advantages over SpinQuant and QuaRot, unexplored.
- Incomplete inference evaluation: The paper lacks an inference performance comparison with AWQ and speedup results across different batch sizes, making its practical deployment benefits difficult to assess.
Questions for authors
- Why is each module’s error relative to its own 4-bit version used to guide bit allocation? For example, suppose two modules have the same sensitivity and require the same additional storage to upgrade to 4 bits: A’s error decreases from 100 to 50, while B’s decreases from 20 to 2. The score would favor upgrading B because its relative improvement is larger, although A removes more error in absolute terms. How do the authors verify which choice benefits the overall model more?
- How well do the local scores match the actual model loss when multiple modules are quantized together? Although full-model PPL search accounts for cross-module effects, how much does it correct inaccuracies in the local scores?
- Can the method be extended to joint weight and activation quantization, where activation quantization errors may also affect low-rank correction, and compared with SpinQuant and QuaRot at the same activation precision and comparable total storage budgets?
- Could the authors add an inference performance comparison with AWQ and specify whether it uses vLLM, SGLang, or another engine, while reporting throughput and relative speedups across different batch sizes under the same hardware and input/output lengths?
I thank the authors for the additional data and clarifications, and for the substantial work behind them. W1 is accepted and withdrawn; W2 and Q4 are largely answered; the independent gain of the recovery-aware criterion is now supported by the 2×2 ablation in Q1.
One core concern, however, remains unaddressed: the value proposition of sub-4-bit quantization is not established. The newly added 4-bit table in fact reinforces this concern — [email protected] (MMLU 55.41 / GSM8K 22.23 / 6.29 GB) already outperforms AWQ, GPTQ, MXFP4, and even FP16. 4-bit is the best operating point of the proposed method. So what problem does sub-4-bit actually solve?
(1) The cost–benefit trade-off is not justified. Relative to its own 4.0-bit variant, [email protected] saves 11.4% storage at the cost of 2.59 points on GSM8K; relative to AWQ-4.0 it saves 12.2% at the cost of 2.19 points. Please specify the concrete deployment scenario: for which model / GPU-memory configuration does 4-bit fail to fit while 3.5-bit fits exactly? Without such a scenario, sub-4-bit is a compression point with no clearly identifiable use case.
(2) Agent capability degrades fastest at lower bit-widths. MMLU drops only a few points at 2.5–3.0 bit, whereas BFCL Multi-turn drops 7.5 / 13 points — below AWQ-4.0 — and it is precisely long-chain reasoning and tool use that constitute the deployment value of Qwen3-class models. Please provide a capability-wise breakdown of degradation (knowledge /single-step reasoning / long-chain reasoning / tool use), and state explicitly whether sub-4-bit has any viable regime in agentic settings.
(3) The supporting evidence does not yet meet the bar. (a) Q4 shows 70% re-ranking in shallow layers versus 20% in deep layers; please report the bit-width actually allocated per depth segment, and add a negative control replacing recovery with a residual-independent random low-rank perturbation (same rank, same storage) — if accuracy improves equally, the gain stems from parameter count rather than from the criterion. (b) The claim that 4-bit exceeds FP16 requires ≥3 seeds with mean ± SD and paired tests on MMLU/GSM8K; the PPL std < 0.003 in D.2.4 covers only the search process and does not substitute for downstream variance. (c) AMLQ is calibrated on WikiText-2 while RampQ uses SlimPajama — a cross-calibration comparison; please add the two-way control and tabulate calibration set / sample count / sequence length / overlap verification for every method, reporting calibration-subset variance separately from training-seed variance. (d) Please add AIME26/HMMT26, LiveCodeBench v5+v6, and ScreenSpot-Pro/MathVista/V*Bench/MMMU, aggregate by capability axis, and pre-declare a degradation threshold.
If the authors can demonstrate a clear deployment value and applicability boundary for sub-4-bit quantization — or reposition the contribution from "a better method" to "a complementary option under extreme memory constraints" — I would consider raising my rating.