- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to recover the residual R = W − Ŵ that training-free LLM compression discards. The central claim is that, for both activation-aware SVD truncation and INT4 quantization (AWQ), the discarded residual is dense and spectrally flat — the quantization residual is statistically close to a Marchenko–Pastur noise bulk, and the truncation residual is a gapless spectral continuum — so a low-rank (LoRA-style) corrector is "structurally mismatched" because it can capture only a vanishing fraction of the residual. The authors propose RC-VQ, a full-rank vector-quantized corrector fitted to R by weight-space k-means, with no gradients and no recovery data, and which composes with SVD factors, INT4 weights, and their stacked combination.
Critical strengths
-
Clear problem framing. Isolating the discarded residual of a training-free compressor as a standalone correction target is natural, training-free, and orthogonal to existing factorization/quantization pipelines (composable with either base).
-
Reproducibility materials are complete. Public models (except Qwen3.5), calibration set, seed ranges, environment (Appendix H), and code + sweep configs are provided.
Critical weaknesses
W1. No trained LoRA baseline — the central thesis is over-generalized (most serious). The flagship claim is that "the standard recovery tool, a low-rank adapter, is structurally mismatched." Yet every low-rank comparison uses training-free truncated SVD of the residual (i.e., LoRC), never the actual standard corrector practitioners deploy: trained LoRA / QLoRA. The spectral-flatness argument (no isolated spikes ⇒ low rank captures little energy) only applies to a corrector that fits the residual E itself. A trained low-rank adapter optimizes loss/perplexity, not a direct fit to E, and can recover functionally important directions even when the residual spectrum is flat. The authors never train a LoRA to disconfirm this, so the conclusion is extended to an entire method class it was not tested against.
W2. No equal-total-bit comparison — the deployment question is unanswered. All comparisons are at equal corrector storage (3 bpw side-channel); the paper never compares against higher-precision bases at equal total bits. For the stated on-device motivation the real question is whether INT4 + 3 bpw VQ (≈7 bpw total) beats INT6 / INT8 AWQ. INT8 AWQ is 8 bpw and nearly lossless, while needing no throughput penalty. The paper does not show RC-VQ is more bit-efficient than simply raising base precision — it only wins the narrower "residual correction" sub-problem versus LoRC.
W3. Severe throughput cost undermines the on-device motivation. The unfused path drops decode throughput from 74.5 to 24.4 tok/s (≈3× slowdown, Appendix G); the LUT-fused kernel is deferred to future work. For a method sold on memory-constrained edge deployment, the only runnable path is 3× slower while the memory win is modest (31% decode-memory reduction on LLaMA-2 7B, and MLP-only compression caps the achievable ratio). In throughput-sensitive settings, 3× slower usually dominates a 31% memory saving. This practical cost should be prominent in the main text with a throughput estimate for the fused kernel, otherwise the "fits on-device" narrative does not hold.
W4. MLP-only scope and modest ratios limit significance. Only MLP projections are compressed (~2/3 of params) and everything collapses near 50% size, so the operating range is 60–90% (models shrink only 10–40%). The 3–13× degradation reduction occurs only in this gentle regime; for the aggressive 2–4 bit full-model compression that edge deployment actually needs, applicability is narrow and significance is at best medium.
Questions for authors
-
At equal total bits, how does INT4 + RC-VQ compare to INT6 / INT8 AWQ in perplexity, memory, and throughput?
-
What is the expected throughput (tok/s) and memory of the LUT-fused kernel, at least at an order-of-magnitude level, to assess edge feasibility?
I thank the authors for their detailed responses and efforts to address the concerns raised in my review. I appreciate that several claims have been corrected and additional experiments provided.
However, my main concerns remain largely unaddressed, especially regarding comparisons with relevant and more recent compression methods. Moreover, the newly added results in Appendix Table 7 show that RC-VQ performs at best comparably to standard AWQ at matched total storage, with AWQ even outperforming it in several settings. Given that AWQ is no longer the state of the art in quantization, I find it difficult to assess the actual benefits of the proposed method without stronger baselines.
The deployment aspect further weakens the practical significance of the work, as the current implementation introduces substantial throughput degradation without demonstrating improvements in runtime memory consumption. Therefore, even the practical benefits of the proposed approach remain questionable.
Overall, while I acknowledge the improvements made during the rebuttal, I am still not convinced that the methodological novelty and experimental evidence are sufficient for acceptance at ICLR. Therefore, I prefer to maintain my original rating.