acceptodds
Under review as a conference paper at ICLR 2027

RC-VQ: Recovering What Calibration Discards with Vector-Quantized Residual Correction

Abstract

Calibration data decides what training-free LLM compression keeps. In activationaware factorization and post-training quantization alike, a small calibration set defines an importance measure over the weights, and whatever it does not certify is discarded as a residual. We argue that failing the calibration proxy does not make weights unimportant, and that the discarded residual should be recovered. The standard recovery tool, a low-rank adapter, is structurally mismatched to this task. On LLaMA-2 7B the residuals are dense and spectrally flat. Quantization residuals are statistically indistinguishable from a Marchenko–Pastur noise bulk, and truncation residuals form a gapless spectral continuum without isolated spikes, so a rank-r corrector recovers a vanishing fraction of either. Across four model families, the energy a low-rank corrector recovers grows only linearly in its rank. We propose RC-VQ, a full-rank vector-quantized corrector fitted to the residual without gradients or recovery data, which composes with SVD factors, INT4 weights, and their stacked combination. On structural compression of LLaMA-2 7B, LLaMA-3 8B, and Qwen3 8B, RC-VQ reduces perplexity degradation by 3–13× against the strongest reported structured baseline at matched checkpoint size. On the INT4-AWQ quantization gap of Qwen3.5 4B, it recovers 96% of the distance to FP16 at three bits per parameter, a budget at which an identically sized low-rank corrector recovers under 40% and a scalar residual branch needs 37% more bits to match. Sweeping codebook objectives across structural, quantization, and stacked residuals, we show that each pure objective has a failure regime, while the hierarchical direction-then-refinement objective, PolarVQ, never collapses, making it the robust default when residual geometry is unknown.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model compression, vector quantization, large language models, low-rank factorization, residual correction, on-device AI

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yfsUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to recover the residual R = W − Ŵ that training-free LLM compression discards. The central claim is that, for both activation-aware SVD truncation and INT4 quantization (AWQ), the discarded residual is dense and spectrally flat — the quantization residual is statistically close to a Marchenko–Pastur noise bulk, and the truncation residual is a gapless spectral continuum — so a low-rank (LoRA-style) corrector is "structurally mismatched" because it can capture only a vanishing fraction of the residual. The authors propose RC-VQ, a full-rank vector-quantized corrector fitted to R by weight-space k-means, with no gradients and no recovery data, and which composes with SVD factors, INT4 weights, and their stacked combination.

Critical strengths
  • Clear problem framing. Isolating the discarded residual of a training-free compressor as a standalone correction target is natural, training-free, and orthogonal to existing factorization/quantization pipelines (composable with either base).

  • Reproducibility materials are complete. Public models (except Qwen3.5), calibration set, seed ranges, environment (Appendix H), and code + sweep configs are provided.

Critical weaknesses

W1. No trained LoRA baseline — the central thesis is over-generalized (most serious). The flagship claim is that "the standard recovery tool, a low-rank adapter, is structurally mismatched." Yet every low-rank comparison uses training-free truncated SVD of the residual (i.e., LoRC), never the actual standard corrector practitioners deploy: trained LoRA / QLoRA. The spectral-flatness argument (no isolated spikes ⇒ low rank captures little energy) only applies to a corrector that fits the residual E itself. A trained low-rank adapter optimizes loss/perplexity, not a direct fit to E, and can recover functionally important directions even when the residual spectrum is flat. The authors never train a LoRA to disconfirm this, so the conclusion is extended to an entire method class it was not tested against.

W2. No equal-total-bit comparison — the deployment question is unanswered. All comparisons are at equal corrector storage (3 bpw side-channel); the paper never compares against higher-precision bases at equal total bits. For the stated on-device motivation the real question is whether INT4 + 3 bpw VQ (≈7 bpw total) beats INT6 / INT8 AWQ. INT8 AWQ is 8 bpw and nearly lossless, while needing no throughput penalty. The paper does not show RC-VQ is more bit-efficient than simply raising base precision — it only wins the narrower "residual correction" sub-problem versus LoRC.

W3. Severe throughput cost undermines the on-device motivation. The unfused path drops decode throughput from 74.5 to 24.4 tok/s (≈3× slowdown, Appendix G); the LUT-fused kernel is deferred to future work. For a method sold on memory-constrained edge deployment, the only runnable path is 3× slower while the memory win is modest (31% decode-memory reduction on LLaMA-2 7B, and MLP-only compression caps the achievable ratio). In throughput-sensitive settings, 3× slower usually dominates a 31% memory saving. This practical cost should be prominent in the main text with a throughput estimate for the fused kernel, otherwise the "fits on-device" narrative does not hold.

W4. MLP-only scope and modest ratios limit significance. Only MLP projections are compressed (~2/3 of params) and everything collapses near 50% size, so the operating range is 60–90% (models shrink only 10–40%). The 3–13× degradation reduction occurs only in this gentle regime; for the aggressive 2–4 bit full-model compression that edge deployment actually needs, applicability is narrow and significance is at best medium.

Questions for authors
  1. At equal total bits, how does INT4 + RC-VQ compare to INT6 / INT8 AWQ in perplexity, memory, and throughput?

  2. What is the expected throughput (tok/s) and memory of the LUT-fused kernel, at least at an order-of-magnitude level, to assess edge feasibility?

Reviewer_P3cUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work studies the residual left after different types of LLM compression, including activation-aware SVD, AWQ, and their combination. The authors argue that these residuals have relatively flat singular-value spectra, so low-rank correction is not a good choice. Instead, they propose to approximate the residual with vector quantization and add the resulting correction to the compressed weights. They also propose PolarVQ, which combines directional clustering with subsequent MSE refinement. The paper reports perplexity improvements over rank-only correction and some other residual-correction baselines.

Critical strengths

I cannot identify clear strengths that are sufficiently supported in the current version of the paper.

Critical weaknesses
  1. In general, the same claims are repeated from section to section, especially regarding spectral flatness, the limitations of low-rank correction, and the information discarded by calibration. These claims are often much stronger than what is actually demonstrated. Writing needs to be significantly improved in several places and the current version of this paper gives the impression of a draft than a final polished manuscript.

  2. The methodological novelty seems rather limited. As I understand it, the main idea is to analyze the compression residual and then use vector quantization to approximate it. The proposed PolarVQ is a combination of directional and MSE clustering. I cannot see a sufficiently convincing explanation of why this particular combination should be considered a significant methodological contribution beyond existing residual correction and vector quantization methods.

  3. While related works seems sufficient, comparisons with several relevant baselines in the main experimental section are missing. For example, CALDERA is already mentioned but not evaluated, despite being directly relevant to the proposed composite compression setting. Moreover, several recent methods for SVD-based compression are missing, including SVD-LLM V2, LEMS and Swift-SVD. I would also expect comparisons with established pruning approaches such as Wanda and SparseGPT, or structured compression methods such as SliceGPT, especially given the rather broad claims made in the paper. For the quantization experiments, the absence of direct higher-bit quantization baselines is particularly concerning, since the proposed INT4+VQ configuration requires approximately 7 bits per compressed weight, while the comparisons are mostly against other correction methods rather than alternative ways to spend the same memory budget.

  4. In lines 237-238, the spectral flatness argument is not well justified. SFM grows from 0.722 for the original weights to 0.851 for the residual at 20% compression. This supports the statement that the residual spectrum becomes flatter, but does not justify treating it as almost flat or featureless. Additionally, for exact truncated SVD the full residual spectrum contains zero singular values, so its SFM would be zero if calculated over all singular values. The authors should specify whether they exclude zero singular values or apply some other convention, as well as clarify the precise matrix and spectrum used for this analysis. Lines 249-251 contain a much more serious issue. A good fit to the Marchenko-Pastur distribution does not imply that the residual entries are i.i.d noise. Moreover, the argument based on the Baik–Ben Arous–Péché transition appears incorrect. The absence of isolated spectral spikes does not mean that a rank-r corrector can only recover the energy above the bulk edge. Even a completely flat nonzero spectrum admits a rank-r approximation that recovers an r/d fraction of its energy, where d is the number of nonzero singular values. Therefore, the theoretical conclusion about the inability of low-rank correction to recover the residual is not established.

  5. Given that the authors repeatedly argue that compression discards important information because of limitations of the calibration data, I would expect experiments with different calibration datasets, distributions, and ideally calibration sizes that support whether the reported residual behavior and the benefits of RC-VQ are actually a general problem rather than an observation for the particular calibration setup used in this paper. Unfortunately, this motivation argument is currently not sufficiently supported.

  6. Section 5.2 is very hard to read because important numerical comparisons are given as plain text instead of tables. Table 1 itself misses important baselines. In particular, it would be useful to report whether spending the same memory on a better base compression method gives comparable or better results than adding the VQ branch.

  7. The quantization experiments need a more meaningful total-storage comparison. For example, INT4-AWQ with an approximately 3-bit residual branch is already around 7 bits per parameter for the targeted weights, before other quantization metadata. Why not compare against direct higher-bit quantization at a similar final budget? The claim of recovering 96% of the AWQ gap also sounds more impressive than the absolute improvement from 9.735 to 9.591 WikiText-2 perplexity, compared with 9.585 for FP16.

  8. The claims regarding vision model behavior are not sufficiently supported. There is a summary table and a figure for VisualWebBench, but the raw per-task results are deferred to supplementary material, and only one selected seed per model is used for that evaluation. Given the emphasis placed on recovering capabilities missed by text calibration, I would expect these experiments to be reported more completely.

  9. Several appendix sections are empty or filled with placeholders, including the sections intended to contain quantization Pareto numbers, cross-domain results, iso-size baseline details, and rate-distortion analysis. Some related data appear elsewhere, but this is still not an acceptable level of completeness for a conference submission and makes a number of statements difficult to verify.

  10. In Section 5.7 the authors honestly report the inference cost and this is a very positive sign. However, the actual result is concerning: the hybrid implementation drops from 74.5 to 24.4 tokens/s while reducing the reported decode memory from 33.7 to 23.1 GiB. This is a significant slowdown. The expected improvement from a future fused kernel is speculative until it is implemented and measured. Since the paper is motivated by practical model compression, I consider this a significant limitation.

Questions for authors

Please refer to the weaknesses section

1 comment

I thank the authors for their detailed responses and efforts to address the concerns raised in my review. I appreciate that several claims have been corrected and additional experiments provided.

However, my main concerns remain largely unaddressed, especially regarding comparisons with relevant and more recent compression methods. Moreover, the newly added results in Appendix Table 7 show that RC-VQ performs at best comparably to standard AWQ at matched total storage, with AWQ even outperforming it in several settings. Given that AWQ is no longer the state of the art in quantization, I find it difficult to assess the actual benefits of the proposed method without stronger baselines.

The deployment aspect further weakens the practical significance of the work, as the current implementation introduces substantial throughput degradation without demonstrating improvements in runtime memory consumption. Therefore, even the practical benefits of the proposed approach remain questionable.

Overall, while I acknowledge the improvements made during the rebuttal, I am still not convinced that the methodological novelty and experimental evidence are sufficient for acceptance at ICLR. Therefore, I prefer to maintain my original rating.

Reviewer_TVrrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RC-VQ, a training-free vector-quantized residual correction framework with first direction selection then MSE clustering, and validates its ability to recover compressed LLM performance across multiple models against low-rank correction baselines at comparable storage budgets.

Critical strengths
  1. RC-VQ provides a simple residual-correction framework that can complement low-rank factorization and quantization without gradients or additional recovery data.
  2. The experiments cover several model families and compression settings, and demonstrate promising recovery at matched budgets.
Critical weaknesses
  1. I have doubt about using bulk spike energy to bound optimal low-rank residual reconstruction, the relevant quantity seems to be the cumulative singular-value energy.

  2. A relatively flat weight-space residual does not imply that the activation-weighted residual has a similarly flat spectrum; the latter is more directly related to functional reconstruction error. The paper should analyze this distinction and compare against activation-aware low-rank correction before concluding that low-rank methods are structurally unsuitable.

  3. The quantization experiments mainly focus on a 4-bit AWQ base, with relatively small initial perplexity degradation in the highlighted recovery results. Evaluations at 3 bits and even 2 bits are more important and needed, because larger quantization errors may exhibit different spectral properties and change the relative effectiveness of VQ and low-rank correction.

  4. Residual geometry can depend on the quantization algorithm, model, and calibration distribution, so AWQ alone provides limited support for the paper’s broad conclusions. Additional comparisons should include other quantizers, as well as recent approaches that explicitly shape quantization error to low rank structures.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.