acceptodds
Under review as a conference paper at ICLR 2027

RECAP: Refine Quantized LLM Checkpoints, Don't Replace Them

Abstract

When a deployed quantized large language model requires higher accuracy, transmitting a complete replacement checkpoint repeats information already available at the client. We formulate successive model refinement as encoding an incremental correction with the cached checkpoint as shared side information. The central challenge is to recover model quality with an incremental correction substantially smaller than a replacement checkpoint. We introduce RECAP, which combines two correction representations with different communication costs. An activation-subspace code transmits coefficients in a basis reconstructed at both endpoints from the cached model and public calibration data, with water-filling allocating bits by activation-weighted coefficient energy; a complement code transmits low-rank factors to correct the remaining residual. Our analysis establishes the shared basis's minimax optimality, derives optimal coefficients and approximate rate allocation, and quantifies the complement's benefit. We ask whether a correction can deliver at least the quality of the next-higher-bit replacement checkpoint while transmitting fewer bytes. From four 1.58-bit checkpoints of 7B models, RECAP reaches the perplexity of same-family 2-bit replacements with 2–19% of their bytes; at a fixed budget, it also achieves lower perplexity than conventional reconstruction baselines at both 7B and 13B scale. These findings show that even a checkpoint with poor standalone performance can remain valuable side information for communication-efficient model refinement. Code is available https://anonymous.4open.science/r/RECAP-7F3A/README.mdhere.

Primary area:
foundation or frontier models, including LLMs
Keywords:
successive model refinement, post-training quantization, LLMs, edge LLMs, low rank decomposition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nv7FOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to incrementally refine a quantized LLM already cached at the client, so as to achieve a higher quality target with less transmission than replacing the model. RECAP combines two correction representations with different communication costs: one transmits coefficients under a shared activation-eigenvector basis, which both endpoints reconstruct from a public calibration context; the other transmits low-rank factors to correct the residual error remaining after decoding. The paper proves the minimax optimality of the shared basis when the residual is unobserved, provides an approximate continuous rate-allocation rule, and derives a joint distortion lower bound for the unquantized problem under a common metric.

Critical strengths
  1. The paper formulates a meaningful incremental communication problem and provides extensive experimental support. RECAP distinguishes algebraic rank from full transmission cost and exploits a locally reconstructable activation basis to reduce the transmitted payload.

  2. The theoretical scope and ablation studies are fairly rigorous. The paper clearly separates exact subspace identities, approximate low-bit rate allocation, and unquantized optimality under a common metric. Component comparisons, cross-corpus evaluation on C4, and endpoint-agreement tests help explain both the benefits and limitations of the design, strengthening the credibility of the reported gains.

Critical weaknesses
  1. The source of RECAP’s full-system advantage is not fully isolated. Appendix C.2–C.3 indicates that RECAP and the baselines use different factor precisions, scaling, and fitting procedures, so these comparisons do not fully separate the benefit of the shared basis from that of the factor coding and fitting choices. In Table 13, complement-only actually achieves lower perplexity in two GPTQ-2 settings.

  2. Table 12 shows that for Qwen-2.5-7B-Instruct GPTQ-2, RECAP perplexity varies from 21.04 to 23.12 depending on the numerical environment, indicating that the exact reconstruction configuration matters greatly. Please clarify how sensitive the reported gains are to calibration and eigensolver settings choices.

Questions for authors
  1. Appendix D.6 states that allocating one quarter of the budget to the shared subspace was selected after component comparisons. How was this ratio selected?

  2. Section 4.1 allocates bits according to SjS_j, the product of the activation eigenvalue and the correction coefficient energy, whereas Table 7 shows that allocation based only on the activation eigenvalues, λj\lambda_j, achieves even better results in some settings. Could the authors explain this observation?

  3. GlowQ (An et al., 2026) also corrects quantization error using shared factors. Please discuss how RECAP differs from GlowQ. Given the differences in sharing mechanisms and optimization objectives, a comparison under settings suitable for both methods could help clarify their relative quality–communication trade-offs.

  4. Figure 1 needs clearer presentation. Please redraw it with consistent typography, cleaner alignment, and clearly labeled components and connections.

Reference: An, S., Suh, I., & Kim, Y. (2026, April). GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs. In International Conference on Learning Representations (Vol. 2026, pp. 124445-124486).

2 comments
AuthorsOpenReview ↗

Response to Reviewer nv7F (1/2): Weaknesses 1–2 and Question 1

We thank the reviewer for recognizing the incremental communication problem, the distinction between algebraic rank and transmission cost, and the rigor of the theoretical scope and ablations. Table and appendix numbers below refer to the revised PDF, where new material is in blue.


Weakness 1: isolating the source of the advantage

A control that differs from RECAP only in the shared basis isolates it: with exactly RECAP's 4-bit factor code, per-row scaling and rounding-aware refits given to QERA, the shared basis still lowers perplexity by 17–71% on the 1.58-bit bases and by 1–6% on the naive 2-bit bases, and on GPTQ-2, where it pays little, the validation-selected split moves RECAP to its complement code or near it (Tables 3, 20 and 21).

The control is QERA's closed-form initialization with RECAP's factor code, at RECAP's payload (the "complement code alone" of Tables 3 and 20; Table 20 was Table 13 in the submission). At about 350 MB (WikiText-2 perplexity):

BaseQERA, 8-bit factors (published setting)QERA with RECAP's factor codeRECAP
Llama SINQ-1.58837.6199.296.5
Llama HQQ-1.583,008.11,217.4350.8
Qwen SINQ-1.58928.5110.574.2
Qwen HQQ-1.582,569.3190.7158.8
Llama SINQ-210.869.539.30
Llama HQQ-211.8910.069.48
Llama GPTQ-224.8118.0718.50
Qwen SINQ-217.0013.8213.63
Qwen HQQ-217.4613.9113.47
Qwen GPTQ-230.5119.7521.03

Matching the factor procedure improves QERA substantially (837.6 to 199.2 on Llama SINQ-1.58), and the shared basis then lowers it to 96.5.

Why GPTQ is the exception. GPTQ's error feedback moves the activation-weighted residual out of the leading activation directions, which hold only 25–26% of it, against 64–65% on SINQ (Appendix D.8; D.6 in the submission). The complement code alone is RECAP's d = 0 case, so the revision selects the split on validation data with d = 0 among the candidates: RECAP falls back to it on Qwen GPTQ-2 and selects d = 64 on Llama GPTQ-2, within 0.07 of it (Table 21).

Why 8 bits for QERA and 4 bits for RECAP. The better precision depends on the rounding: under QERA's own one-shot rounding, 4-bit factors at twice the rank are worse than its published 8 bits on every naive 2-bit base (Llama SINQ-2: 20.76 against 10.86), while with RECAP's rounding-aware refits the same 4-bit payload reaches 9.53.

Weakness 2: sensitivity to the numerical environment

Because the client rebuilds the shared basis, the protocol now fixes the numerics that determine it (an exact double-precision eigendecomposition and a float32 calibration pass at both endpoints): every audited client lands within 0.004 perplexity of the server, where the gap reached 2.1 before, and even before the fix RECAP kept its lead over the best baseline in every audited setting.

We thank the reviewer for raising this: it led us to trace the variation in Table 17 (Table 12 in the submission) to two causes in our reference implementation, and we have now fixed both.

The causes. The widest inputs used a randomized eigensolver whose draws differ between devices, and the CPU calibration pass ran at a different arithmetic precision from the GPU pass.

The fix. The reviewer's 2.1 gap, 23.12 against 21.04 on Qwen GPTQ-2, closes under the protocol (Appendix C.4; third block of Table 17): a CPU client now rebuilds the GPU server's correction exactly, so what remains is evaluation noise.

The reported gains. Even before the fix, RECAP's worst audited case was 23.12 against the best baseline's 25.21 on Qwen GPTQ-2. As a safeguard, the server sends a fingerprint of each basis, and a client that cannot reproduce it receives the basis at 4 bits instead, at a cost of at most 0.24 perplexity.

Question 1: how the quarter split was chosen

The quarter rule came from comparing a small and a larger subspace on the component grid, using the test perplexities the paper reports, and because the best split depends on the base, the server now selects it on held-out validation data, which matches the best test configuration in three of the four Table 20 settings and comes within 0.07 perplexity in the fourth (Table 21).

The choice. We compared a small fixed subspace (d = 64) with a larger one (d = 256) at about 350 MB on the twelve 2- and 3-bit 7B settings; the larger one gave lower perplexity in ten, losing only on the two GPTQ-2 settings. A fixed d = 256 does not fit the smallest payloads, so we made the rule proportional: the subspace code takes a quarter of the payload. It is one rule for all bases, and these comparisons used the test perplexities the paper reports.

The revision. Because the best split depends on the base, and to keep the choice off the test data, the split is now selected at the server on held-out validation data, with the complement code alone (d = 0) among the candidates (Appendix C.2; Table 21).

AuthorsOpenReview ↗

Response to Reviewer nv7F (2/2): Questions 2–4

Question 2: S_j against λ_j alone

The two signals tie, and the reason is measurable: the eigenvalues vary far more across columns than the coefficient energies, so they dominate the product S_j = λ_j‖A*_j‖² and both rules allocate almost the same rates (Appendix D.4).

The tie. Across the four settings of Table 12 (Table 7 in the submission), S_j is better in three (by at most 0.15) and worse in one (by 0.08).

The reason. We measured both factors on two layers of each Table 12 setting. Across the columns, the eigenvalues vary far more than the coefficient energies, so λ_j alone explains about 99% of the variation in log S_j, and the two rules assign the same bit rate to 98% of the columns on SINQ and 86–87% on GPTQ. Table 12 shows the same from the other side: allocating by coefficient energy alone is worse than equal column rates in every setting, while λ_j alone beats them.

Why we keep S_j. It is the quantity the allocation analysis yields, and it needs no extra statistic. What matters more than the choice between the two is the rate: water-filling's gain over equal column rates grows as the rate falls, from at most 0.6 perplexity at four bits to as much as 9.1 at one bit, and it is what lets the subspace code trade coefficient precision for basis dimension: with equal rates, one bit at d = 2048 is worse than four bits at d = 512 on both SINQ bases (Table 11).

Question 3: GlowQ

GlowQ fits its shared factor to the quantization errors and stores it for serving speed, whereas RECAP's shared basis is rebuilt by the client and never sent; compared in RECAP's setting, quality per transmitted byte on a frozen 2-bit base, RECAP is 13–29% lower in perplexity on all six settings (Table 20; Appendix D.10).

We thank the reviewer for the pointer. We now cite GlowQ (An et al., 2026) and compare with it (Section 2; Table 20; Appendix D.10).

How the two differ. Both methods use the fact that projections reading the same input (q, k and v; gate and up) can share an input-side factor, and RECAP groups them the same way. GlowQ fits its shared factor to the group's quantization errors, so the factor is part of the stored correction, and its purpose is serving efficiency: computing the shared projection once per group lowers latency. RECAP's shared basis depends only on the deployed checkpoint and public calibration text, so the client rebuilds it and it never crosses the link. What is transmitted is the residual's coefficients in that basis, coded at one to two bits, plus a low-rank complement. GlowQ also corrects a model when it is quantized, whereas RECAP refines a checkpoint that is already deployed, under a byte budget.

Comparison. GlowQ reports perplexity, accuracy and serving latency at 4 bits, so we compared the two in RECAP's setting: quality per transmitted byte on a frozen 2-bit base. We re-implemented GlowQ's closed form (one shared factor per group, sent once, and one factor per module) with 8-bit factors, as for the other baselines, at RECAP's payload of about 350 MB (WikiText-2 perplexity):

BaseQERAGlowQRECAP
Llama SINQ-210.8610.669.30
Llama HQQ-211.8911.699.48
Llama GPTQ-224.8124.5418.50
Qwen SINQ-217.0017.2313.63
Qwen HQQ-217.4617.9313.47
Qwen GPTQ-230.5129.7621.03

Sharing the factor helps slightly over per-module QERA on four of the six settings. The two ideas also compose: GlowQ's cached group projection is a serving optimization that RECAP's factorized layout could adopt.

Question 4: Figure 1

We have redrawn Figure 1 as a vector figure in one consistent typeface, on a single aligned grid.

Each component is labeled by where it lives (server, link, client), and the connections show what the server computes, what crosses the link, and what the client rebuilds from its cached checkpoint and the public calibration text.


We hope these responses resolve the reviewer's concerns. The revision adds the shared-basis control, the specified numerical protocol, the validation-selected split, the GlowQ comparison and the redrawn Figure 1 discussed above. If the concerns are resolved, we would be grateful if the reviewer would consider raising the score, and we are happy to answer any further questions.

Reviewer_VhDnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of how to efficiently upgrade deployed quantized models at the client. Transmitting a complete higher-bit checkpoint repeats information already available at the client. The paper proposes RECAP. The method consists of two codes: a shared activation-subspace code can be reconstructed with a public calibration dataset ,only the coefficient matrix needs to be transmitted, and bits allocated via water-filling according to activation-weighted energy; a complement-factor code corrects the remaining residual outside the subspace via a low-rank factorization. The paper establishes the minimax optimality of the shared basis and the joint distortion floor of the two codes. Experiments show that starting from 1.58-bit bases, RECAP achieves perplexity parity with 2-bit replacements using only 2–19% of their bytes.

Critical strengths
  1. The overall solution is innovative. It uses client-side computing to generate the shared activation-subspace code, and combines it with coefficient matrix and the complement code to reconstruct the checkpoint with higher precision, which only has 2%-19% of the transmission bytes.
  2. The theoretical foundation is solid, including the minimax optimality of the shared basis and the joint distortion floor of the two codes
  3. This paper accounts for every transmitted bit (including quantized values and codebooks), providing realistic communication cost estimates for deployment.
Critical weaknesses
  1. The paper quantifies the reduction in transmitted bytes, but does not clearly state the additional storage and computational overhead with empirical measurements, especially the one-time cost incurred during the pre-calibration and reconstruction stages. It is therefore unclear whether RECAP is more efficient than directly downloading a relacement checkpoint.
  2. The experimental models evaluated in the paper do not include recent MoE models, and the scheme has not been evaluated under repeated update-and-reconstruction cycles.
Questions for authors
  1. Could you clarify the overhead (including the computing cost and extra storage) required for calibration?
  2. At the boundary between reconstructing with RECAP and direct 2bit replacement, is there a trade-off between accuracy and transmitted byte? How should one choose between RECAP refinement and direct 2-bit replacement when balancing model accuracy and communication cost?
  3. Is the transmitted payload in Fig.2 measured or theoretical?
4 comments
AuthorsOpenReview ↗

Response to Reviewer VhDn (1/2): Weaknesses 1–2

We thank the reviewer for recognizing the shared activation-subspace code rebuilt on the client, the theoretical foundation, and our accounting of every transmitted bit. Table and appendix numbers below refer to the revised PDF, where new material is in blue.


Weakness 1: storage and compute overhead

RECAP sends only what the client's checkpoint lacks, at a one-time preparation cost, and it is more efficient than downloading the replacement: on Llama-2-7B, its 348 MB update, about a fifth of the 1.82 GB 2-bit replacement it passes, is received and applied in 4.7 minutes at 10 Mbit/s against the replacement's 24.3, after a one-time preparation of 11 minutes that the client completes before any update arrives (Tables 6 and 19).

Compute. The client's one-time cost is preparation, done before any update arrives: it builds the shared basis from its deployed checkpoint and public calibration text, once per checkpoint, so no update waits for it and every later update reuses it (11 minutes on an RTX A6000 for Llama-2-7B; Question 1 gives the breakdown). When an update arrives, the client only receives the payload and reconstructs the refined weights in 6 s. For Llama-2-7B, whose 348 MB update passes the quality of the 1.82 GB 2-bit replacement:

Link2-bit replacement (1.82 GB)RECAP update (348 MB), transfer plus reconstruction
10 Mbit/s24.3 min4.7 min
50 Mbit/s4.9 min1.0 min
100 Mbit/s2.4 min0.6 min

Storage. At the payload where RECAP matches the 2-bit replacement, the base plus the stored correction takes about the same space as the replacement: between 13% less and 6% more across the four 7B bases, measured on disk. The 1.58-bit base is smaller than a 2-bit checkpoint, and the correction fits into the difference. Larger payloads buy more quality for more storage, and the server can cap the payload to a client's budget.

Serving. Keeping the correction factorized holds about 1.8 times the payload in memory and raises the decode step's linear-projection time to 1.56 times the base's, below the 1.72 of the two-factor baselines. In exchange for a download about a fifth of the replacement's, the client thus pays a one-time preparation, about the replacement's storage at matched quality and a slower decode step. The revision reports these costs in Section 5 (Table 6) and Appendix D.7 (Tables 18 and 19).

Weakness 2: MoE models and repeated updates

Nothing in RECAP is specific to dense models or to a single update, so it applies to MoE models and to repeated updates without change, and we now measure both: on a 60-expert MoE model its update gives perplexity a third below QERA's at the same bytes and the highest accuracy, and four successive updates reach each quality level with at most 10% more bytes than a single update (Tables 5 and 22).

MoE. Each routed expert is a set of ordinary weight matrices, and its shared basis comes from the calibration tokens the router sends to it, which both endpoints reproduce from the same checkpoint and public text. On Qwen1.5-MoE-A2.7B-Chat (14.3B parameters, 60 routed experts), refining the 2-bit SINQ checkpoint with a 540 MB update:

Qwen1.5-MoE-A2.7B-Chat, SINQ-2 basePayloadWikiText-2 perplexitySeven-task accuracy (%)
Base, unrefined0 (already deployed)14,814.539.3
LQER540 MB35.3348.3
QERA540 MB27.8347.6
RECAP540 MB18.2149.6
FP16(reference)8.0765.2

RECAP keeps the ordering it has on dense models: at the same bytes, its perplexity is about a third below QERA's, and its accuracy is the highest of the three. This is one MoE model and one base, and we add it to the appendix (D.10, Table 22).

Repeated updates. Each update sends only what is new; the client adds it to its correction, reuses the basis it built once, and reconstructs in about 6 s. Four successive updates on Llama-2-7B SINQ-1.58 (WikiText-2 perplexity; Table 5 and Appendix D.9):

UpdateCumulative bytesSuccessive RECAPOne-shot RECAP at about the same bytes
187 MB1,3601,393 (87 MB)
2182 MB405422 (175 MB)
3364 MB98.496.5 (348 MB)
4719 MB27.924.8 (696 MB)

The sequence follows the one-shot trajectory: the first update is itself a single update, and from the second on the sequence needs only 2–10% more bytes than a single update for the same quality, and it passes the 2-bit replacement at the third update. Storage grows with the bytes received, and nothing is sent or stored twice.

AuthorsOpenReview ↗

Response to Reviewer VhDn (2/2): Questions 1–3

Question 1: calibration overhead

Calibration only rebuilds the shared basis on the client, so all of it is one-time preparation, done once per cached checkpoint before any update arrives: 11 minutes and 1.1 GB of GPU memory beyond the model on Llama-2-7B (33 minutes and 3.2 GB on Qwen-2.5-7B-Instruct; Table 19), and the only thing worth storing is the basis itself, 101–113 MB at 4 bits, if the client keeps it.

Under the protocol we specify for cross-device reproducibility, on an RTX A6000:

Llama-2-7BQwen-2.5-7B-Instruct
Calibration pass over 16,384 public tokens, float32162 s274 s
GPU memory beyond the model during the pass1.1 GB3.2 GB
Basis, exact double-precision eigendecomposition495 s1,700 s
One-time total11 min33 min
Basis kept for later updates, 4 bits (d = 256)101 MB113 MB

The calibration text is public and needs no storage beyond the text itself. The activation statistics are transient: 0.69 GB per layer of Llama-2-7B in FP32, processed eight layers at a time (5.5 GB) and discarded once the basis is built. The client may keep the basis, as above, or rebuild it before a later update. Per update, the only client work is reconstructing the refined weights, about 6 s.

Question 2: choosing between RECAP and a replacement

RECAP turns the trade-off into a dial and the choice into a rule the server applies before sending: refine whenever RECAP reaches the target with fewer bytes than the replacement, which at the 2-bit boundary takes 37–38% of the replacement's bytes for its accuracy and 2–30% for its perplexity on seven of the eight bases.

A replacement is a single point: fixed bytes, fixed quality. RECAP's payload can be any size, and quality rises with it.

The rule. In every setting we tested, RECAP reaches the target with fewer bytes than the replacement:

TargetBytes RECAP needs, as a share of the replacement
2-bit replacement, perplexity2–30% on seven of the eight bases, at most 61% on the eighth
2-bit replacement, seven-task accuracy37–38% (the four 7B bases)
3-bit replacement, seven-task accuracy53–79%

The more quality the target demands, the larger the share.

Who decides. The server holds the residual, so it evaluates RECAP's rate–quality curve on held-out data before sending anything, sends the replacement whenever that is the cheaper way to the target, and can cap the payload to a client's storage budget.

Question 3: payload in Figure 2

It is measured: RECAP's payload is a fixed-length encoding with no entropy coding, so its size is the exact number of bits the decoder reads.

Every transmitted index, codebook entry, scale and header field is counted (Appendix C.5). Nothing is estimated or modeled. Written to disk in a compact format at the crossing payloads, each payload file is 1.1–1.3 MB above the counted payload.


We hope these responses resolve the reviewer's concerns. The revision adds the client-cost and storage measurements, the repeated-update study and the MoE results discussed above. If the concerns are resolved, we would be grateful if the reviewer would consider raising the score, and we are happy to answer any further questions.

Dear Reviewer VhDn,

 

Thank you for taking the time to review our paper and share your feedback. In our rebuttal, we provided a point-by-point response to your comments and clarified the issues you raised.

When you have a chance to read our response, could you please let us know whether it addresses your concerns? If anything remains unclear or you have any further questions, we would be happy to provide additional clarification.

If you feel that our rebuttal has addressed your concerns, could you please consider raising your score? We would greatly appreciate it.

Thank you again for your time and consideration.

 

Best regards,
The Authors

Dear Reviewer VhDn,

 

As the discussion period is coming to a close, we would kindly ask you to review our rebuttal at your earliest convenience. If you have no further questions, we would greatly appreciate it if you could consider raising your score. Should you have any remaining concerns, we would be more than happy to provide further clarification and address them promptly.

Thank you very much for your time and expertise.

 

Best regards,

The Authors

Reviewer_mCCUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RECAP, a method for improving deployed quantized LLMs without transmitting an entirely new model checkpoint. The main idea is to reuse the existing low-bit checkpoint and send a compact correction that recovers part of the quantization error. RECAP combines two components: an activation-subspace code and a low-rank complement. The first uses an activation basis that both the server and client can reconstruct from the cached model and public calibration data, avoiding the need to transmit the basis itself. It also uses water-filling to allocate bits across the correction coefficients. The second component applies a quantized low-rank correction to recover the remaining error. The paper provides theoretical analysis of the subspace selection, bit allocation, and complement fitting. Experiments on Llama-2 and Qwen2.5 models show that RECAP can match the perplexity of corresponding 2-bit replacement checkpoints using only 2–19% of their transmission size in the tested 7B settings. It also generally achieves better perplexity and downstream accuracy than existing reconstruction methods under matched communication budgets.

Critical strengths
  1. An interesting and practical problem. The paper addresses a useful problem: how to improve an already deployed quantized LLM without downloading another full checkpoint. I think this is a meaningful direction, especially for edge devices with limited bandwidth. The idea of reconstructing a shared activation basis at both endpoints, so only the correction coefficients need to be transmitted, is interesting and makes good use of the existing checkpoint.

  2. Fairly comprehensive evaluation with promising results. The authors evaluate multiple model sizes and quantization methods, and compare against several existing reconstruction approaches. RECAP achieves the best or tied-best perplexity in 22 out of 24 matched-payload settings. The paper also includes quite detailed ablations on basis selection, bit allocation, coefficient coding, and the two correction components. These experiments help explain where the improvements come from, rather than only showing better final numbers. I think the experimental effort is a strong point of this work.

Critical weaknesses
  1. I am not really convinced by the main communication-saving result, especially given how poor some of the replacement checkpoints are.

The paper repeatedly highlights that RECAP can reach 2-bit replacement perplexity with only 2–19% of the bytes. But looking at Table 5, the replacement checkpoints have perplexity of 319, 7628, 19702, and 1905. These are extremely poor results. Some of the 1.58-bit starting checkpoints are even worse, with perplexity reaching hundreds of thousands or millions. I find it hard to see why someone would deploy these models in the first place. If the quality target is already this bad, beating it with a small correction doesn't necessarily tell us much about how useful the method really is.

This is also quite different when the replacement is actually stronger. In Table 6, Llama-2-13B SINQ-2 has a perplexity of 13.83, and RECAP still cannot reach it, even with the largest tested payload (17.3). I think this is an important counterexample to the main claim. The paper also switches between perplexity and downstream accuracy when presenting the communication savings. The 2–19% result is for perplexity, while matching seven-task accuracy requires around 37–38% of the replacement bytes. And even then the accuracy is only around 39–42% for the 7B models. So I don't think the impressive transmission numbers necessarily mean a useful model has been recovered.

I also question why the comparison is restricted to same-family 2-bit replacement checkpoints. Table 4 shows that GPTQ-2 can be dramatically better than SINQ-2 or HQQ-2. In an actual deployment, wouldn't we just choose a stronger quantization method for the replacement? The paper should at least compare against the best available replacement checkpoint, rather than only the corresponding PTQ family. Otherwise the measured savings depend heavily on how weak that particular replacement happens to be.

There is also a big gap between the communication result and actual deployment. The paper is motivated by edge devices, but the evaluation uses materialized FP16 weights. If we actually keep the correction factorized on a low-memory device, we have to store the extra factors and shared basis, and pay additional computation during inference. If we merge everything into FP16 weights, we lose much of the memory advantage of having a low-bit model. The existing 1.58-bit checkpoint plus a 37–38%-of-replacement-size update may even take more storage than simply keeping a 2-bit checkpoint. None of these tradeoffs are measured. There is no inference latency, throughput, peak memory, client-side reconstruction time, or energy evaluation. Saving download bytes is useful, but that alone doesn't show this is a good solution for edge deployment.

The zero-bit shared basis also looks less practical than the paper suggests. The client needs to process public calibration data and reproduce the activation eigenspace, which is not free. More importantly, Table 11 shows that changing the eigensolver from GPU to CPU can increase Qwen GPTQ-2 perplexity from 24.00 to 63.96, and without the sign convention it becomes 3084.4. Table 12 shows that the full RECAP is more robust, but still has noticeable differences when calibration or solver settings change. If the method needs tightly matched numerical implementations to decode correctly, that is a serious issue for heterogeneous edge devices. The paper never really measures the cost of making this work across devices.

One other thing is the "successive refinement" framing. All the experiments are essentially one-shot corrections to a fixed checkpoint. There is no experiment showing several upgrades over time, whether the corrections accumulate, or how the total storage and communication evolve after multiple updates. The paper itself leaves repeated refinement as future work. I think this makes the practical motivation somewhat stronger than what is actually demonstrated.

  1. The actual novelty and the source of the improvement are not very clear to me.

The shared activation basis is an interesting idea, but many of the other pieces are already well known. Using top activation eigenvectors is a fairly natural subspace approximation. The bit allocation is standard reverse water-filling, and the complement is based on QERA with CALDERA-style quantization-aware refitting. The three theorems make the method look quite theoretically complete, but I am not sure how much new understanding they provide. Theorem 1 is a minimax result over arbitrary residual orientations, not necessarily the residual from a particular PTQ method. Theorem 2 uses a high-resolution quantization approximation, even though the actual method mainly works with one- or two-bit coefficients. Theorem 3 assumes unquantized corrections and a common activation statistic, while the implementation uses different statistics for the two components and quantizes the factors. So these results explain some idealized cases, but do not really establish that the implemented codec or its budget split is optimal.

The water-filling contribution also seems weaker than the presentation suggests. In Table 7, most of the improvement comes from replacing simple rounding with learned codebooks. Using the combined activation-weighted coefficient energy gives only small additional gains over using activation sensitivity alone, and is even slightly worse in the Llama GPTQ-2 case (20.23 versus 20.15). I don't see strong evidence that the proposed allocation rule is a major reason for RECAP's performance.

The baseline comparison is another concern. RECAP uses 4-bit factors, per-row scaling, and rounding-aware refitting, while LoRC, LQER, and QERA use 8-bit factors, and CALDERA uses per-matrix scaling. I understand that the total transmitted bytes are matched, but this is not enough to isolate the contribution of the shared-basis coding. A 4-bit representation can afford a much higher rank under the same budget, and better scaling and refitting can make a big difference by themselves. I would really like to see QERA with exactly the same 4-bit factor representation and refitting procedure.

Actually, the paper's own ablations already raise this question. In Table 13, complement-only is better than full RECAP on both GPTQ-2 settings: 18.07 versus 18.50 for Llama and 19.75 versus 21.03 for Qwen. Even on SINQ-2, the improvement over complement-only is quite small. Table 8 also shows that the subspace-only code can achieve higher downstream accuracy than full RECAP in some cases. These are not minor details, because the shared basis and the combination of two codes are supposed to be the main technical contribution. If an equally budgeted complement-only method can do better, then what is the real benefit of the new component?

The benefit also seems quite dependent on the base quantization method. Appendix D.6 reports that the shared subspace captures around 64–65% of activation-weighted residual error for SINQ, but only 25–26% for GPTQ. This seems consistent with the cases where adding the shared basis actually hurts. The default quarter-budget split is empirically selected, rather than derived from the proposed analysis, and it is not consistently better across the 2- and 3-bit settings. I think the paper needs a clearer explanation of when shared-basis coding is worth using, and when the budget should simply go to the low-rank complement.

I am also not very impressed by some of the higher-bit results. On 3-bit checkpoints, several perplexity improvements are only around 0.01–0.1, despite transmitting hundreds of MB. Saying RECAP is best or tied-best in 22 of 24 settings sounds strong, but it hides how small many of these differences actually are. There is little analysis of run-to-run variability, and the downstream results are only reported as an average across seven tasks. It is hard to tell whether all these small gains are meaningful.

Finally, the evaluation is limited to Llama-2 and Qwen2.5 models and mostly conventional perplexity and multiple-choice benchmarks. There is no evaluation on more recent model families, realistic instruction-following workloads, or actual deployed inference. The calibration results in Table 9 also show some noticeable sensitivity to the calibration corpus, particularly for GPTQ, but there is not much discussion of what happens when the device's real workload is different from the calibration data.

Overall, I think there is a decent amount of work here, and the shared-basis idea has some value. But the strongest communication result relies on very weak quality targets, and the experiments don't clearly show how much of the improvement actually comes from the new mechanism. Together with the lack of real deployment measurements, these issues make me hesitant to support acceptance.

Questions for authors

I'm a bit confused by the main 2–19% transmission saving result. In Table 5, the 2-bit replacement checkpoints have perplexity ranging from 319 to nearly 20,000. These models are already performing very poorly. How meaningful is it to match such a low quality target? What happens if we use a reasonably good 2-bit checkpoint instead? For example, RECAP cannot match the Llama-2-13B SINQ-2 replacement in Table 6.

Why only compare with same-family replacements? GPTQ-2 is already much better than SINQ-2 and HQQ-2 in several settings. If I were updating a deployed model, I would probably just download a better quantized checkpoint. Would the communication savings still be significant compared with the best available 2-bit replacement?

I'm not sure the baseline comparison is entirely fair. RECAP uses 4-bit factors with per-row scaling and rounding-aware refitting, while QERA and LQER use 8-bit factors. How much of the improvement actually comes from these implementation differences? It would be useful to see QERA with exactly the same factor precision and refitting procedure.

Table 13 is quite interesting. The complement-only method actually beats the full RECAP on both GPTQ-2 settings. Even on SINQ-2, the improvement from adding the shared basis is pretty small. Does this mean the shared-basis component is only useful for certain quantization methods? Why use a fixed quarter-budget split across different bases?

15 comments
AuthorsOpenReview ↗

Response to Reviewer mCCU (1/11): Overall assessment and Weakness 1, first point

We thank the reviewer for the detailed review, and for recognizing the practical value of refining a deployed checkpoint over bandwidth-limited links, the shared activation basis rebuilt at both endpoints, and the breadth of the evaluation and ablations. Table and appendix numbers below refer to the revised PDF, where new material is in blue.


Overall assessment

We answer each of the three concerns with new measurements, and all three answers follow from RECAP's principle: the deployed checkpoint is side information that the client already holds.

Weak targets. The poorer the deployed checkpoint, the less side information the decoder has, so the 1.58-bit study tests RECAP where it is weakest. Against usable targets, the same family's 3-bit checkpoints and the best 2-bit replacement in our study, refinement still costs a fraction of re-transmission, and the 13B case that had not crossed does so one payload doubling later (Weakness 1, first to third points).

Source of the gain. With the factor code held equal, the shared basis carries the correction where the residual follows the activation spectrum, and adds little where a quantizer such as GPTQ has already moved the error out of the leading directions, as our capture analysis predicts. The server now chooses the split on validation data, so RECAP keeps the basis where it pays and gives the budget to the complement code where it does not (Weakness 2, first to fifth points).

Deployment. RECAP minimizes the bytes an update sends, and we now measure what hosting it costs. Served packed, it decodes faster than the two-factor baselines, and its stored total stays close to the replacement's at matched quality. Under a specified numerical protocol every client reproduces the server's correction, after a one-time preparation that never delays an update, and successive updates cost close to a single one (Weakness 1, fourth to sixth points).

Weakness 1, first point: the replacement checkpoints

RECAP aims to exploit the side information in a cached low-precision checkpoint as fully as possible, whether its standalone performance is strong or poor, and against a usable target it needs only a fraction of a replacement's bytes: refining 2-bit Llama-2-7B toward its own 3-bit checkpoint, it passes HQQ-3 on both metrics with 53% of the bytes and matches SINQ-3's accuracy with 79% (Table 1).

Richer and poorer side information. With richer side information, Table 2 (Table 1 in the submission) already shows substantial gains: refining Llama-2-13B SINQ-2 increases seven-task accuracy from 43.8% to 59.0% with a 545 MB update. The 1.58-bit experiments extend this evaluation to severely degraded checkpoints, a stress test by design, demonstrating that RECAP can efficiently exploit even poor side information: it reaches the 2-bit replacement's perplexity with 2–19% of its bytes.

A usable target. Whether the refined model is useful is the complementary question, which we answer with a usable target: we refined 2-bit Llama-2-7B checkpoints toward the same family's 3-bit checkpoint (2,631 MB). WikiText-2 perplexity / seven-task accuracy (%):

Payload (share of the 3-bit checkpoint)SINQ-2 refinedHQQ-2 refined
0 (deployed 2-bit checkpoint)319.0 / 41.47,628.5 / 35.4
696 MB (26%)8.12 / not run8.11 / not run
1,393 MB (53%)6.91 / 60.96.95 / 61.0
2,089 MB (79%)6.28 / 64.06.29 / 63.5
Same-family 3-bit checkpoint (100%)6.13 / 64.17.05 / 60.4
FP165.47 / 66.65.47 / 66.6

The 3-bit target is not a weak one: SINQ-3 is the best 3-bit checkpoint for this model (GPTQ-3: 6.27 / 62.2), and at 79% of its bytes RECAP also comes within 0.15 of its perplexity (6.28 against 6.13). Standalone quality also understates side information: the two 2-bit bases start 24 times apart in perplexity, near chance in accuracy, yet are within 0.04 perplexity of each other at every payload.

The top of the ladder. With the richest side information we tested, 3-bit SINQ and HQQ checkpoints (perplexity 6.13–9.85, Table 7; Table 4 in the submission), about 350 MB, a tenth of a 4-bit checkpoint, delivers 69–92% of the 4-bit checkpoint's perplexity gain at 7B (Table 8; GPTQ is the exception, discussed under Weakness 2, fourth point). The revision states the stress-test design in Section 5 and adds these results (Table 1; Appendix D.2).

AuthorsOpenReview ↗

Response to Reviewer mCCU (2/11): Weakness 1, second and third points

Weakness 1, second point: stronger replacements; perplexity and accuracy

RECAP aims to reach a specified quality target with fewer transmitted bytes, and the required payload depends on the cached checkpoint and the evaluation metric: one more payload doubling passes the stronger Llama-2-13B replacements, SINQ-2 and GPTQ-2, on both corpora, and wherever accuracy is above its floor, the perplexity and accuracy savings agree.

The stronger 13B replacement. For the Llama-2-13B case identified by the reviewer, we extended the payload range: at the previous largest payload (about 1,090 MB, perplexity 17.3/30.1) the trajectory had not flattened, with each payload doubling cutting perplexity by 1.9–2.3×. Refining SINQ-1.58 (perplexity 10,474) with 2,180 MB achieves WikiText-2/C4 perplexity of 9.63/14.78, compared with 13.83/17.43 for the 3,568 MB SINQ-2 replacement. Thus, RECAP now surpasses this stronger replacement on both corpora using 61% of its bytes. It also surpasses GPTQ-2 (3,402 MB, 12.94/23.63) on both corpora using 64% of that replacement's bytes (Table 10 and Figure 2; Table 6 in the submission). Since GPTQ-2 is the best 2-bit checkpoint on WikiText-2 and SINQ-2 on C4, RECAP passes the best replacement on each corpus.

Perplexity versus accuracy. We report the communication results separately (Table 1; Appendix D.2): 2–19% refers to perplexity parity in the reported 7B experiments, while seven-task accuracy parity was demonstrated at 37–38%. The reported 39–42% accuracy describes those operating points, rather than the highest quality attainable through refinement: doubling the payload to 1,393 MB takes Llama HQQ-1.58 from 38.8% to 49.4% (Table 1; Appendix D.2), above every 2-bit Llama-2-7B checkpoint (35.4–41.4%) with fewer bytes than any of them (1,393 MB against 1,737 MB or more). The two shares differ because accuracy has a floor that perplexity does not: all twelve 2-bit checkpoints we test score 35.4–44.2% on the seven tasks, close to the 35.7% of random guessing, while their perplexities span 12.9 to 19,702. Near that floor a seven-task average moves little (its standard error is about 0.5 points), so matching a replacement's accuracy is a coarse test, while perplexity still ranks the models.

A stronger quality target. To evaluate communication savings at a stronger quality target, we additionally refined Llama-2-7B HQQ-2 toward its HQQ-3 replacement. A 1,393 MB update achieves 6.95 WikiText-2 perplexity and 61.0% seven-task accuracy, compared with 7.05 and 60.4% for HQQ-3, while transmitting 53% of its bytes (Table 1). This demonstrates recovery to a substantially higher quality level, with communication savings evaluated at the same operating point on both metrics. With 2,089 MB (79%), refined HQQ-2 and SINQ-2 reach 63.5% and 64.0% accuracy, against 66.6% for FP16.

Weakness 1, third point: the best available replacement

RECAP's saving is measured against what the client would otherwise download, so we also compare with GPTQ-2, the best 2-bit replacement in our study: RECAP passes it on every base, on three of the four 7B bases with 39–40% of its bytes, on the fourth with 80%, and, in perplexity, on all four 13B-class bases with 62–64% (Tables 1 and 10).

GPTQ-2's checkpoints take 1,737 MB (Llama-2-7B) and 1,751 MB (Qwen-2.5-7B-Instruct) over the refined layers (Table 1). Against it, with RECAP at 675–696 MB (WikiText-2 perplexity / seven-task accuracy, %):

BaseRECAP, 675–696 MBGPTQ-2 replacement
Llama SINQ-1.5824.8 / 41.534.98 / 38.9
Llama HQQ-1.5850.0 / 38.834.98 / 38.9
Qwen SINQ-1.5827.8 / 41.247.81 / 36.7
Qwen HQQ-1.5829.3 / 40.847.81 / 36.7

On Llama HQQ the crossing comes at 1,393 MB (11.48 / 49.4), and at the 13B class it takes one more payload doubling. The saving therefore does not hinge on a weak replacement: against the strongest 2-bit checkpoint in our study, RECAP sends less than half its bytes on three of the four 7B bases. We add this comparison to the revision (Table 1, and Table 10 for the 13B class).

AuthorsOpenReview ↗

Response to Reviewer mCCU (3/11): Weakness 1, fourth point

Weakness 1, fourth point: deployment costs

RECAP minimizes the bytes of an update, and the device hosts the refined model as it would host a replacement, so we measured what hosting it costs: served packed, RECAP has the fastest decode step among correction layouts, 1.56 times the 2-bit base against 1.72 for the two-factor baselines, and at matched quality its stored total is within −13% to +6% of the 2-bit replacement (Tables 6, 18 and 19).

Serving. We time the linear projections of each serving form at the same 7B 2-bit operating points (348 MB) on an RTX A6000, relative to the unrefined packed 2-bit base; every packed layout is checked against its unpacked reference, so quality carries over to the factorized form:

Serving formDecode stepPrefill, 2,048 tokensResident correction
Unrefined 2-bit base, packed1.00 (3.81 ms)1.00 (314 ms)none
Two-factor baselines, fastest form (FP16 factors)1.721.19690–706 MB
RECAP, packed1.56 (5.95 ms)1.32 (416 ms)641–649 MB
Refined weights merged into FP165.20.82 (256 ms)12,952 MB in all

Decode is paid for every generated token and is bound by the bytes each step reads, which is why merging the update into FP16 weights costs 5.2 times the base there. Prefill is paid once per prompt and is bound by arithmetic, and there RECAP is slower than the two-factor baselines (1.32 against 1.19).

Storage. The answer depends on the payload, which the server chooses. At the payload where RECAP first passes the 2-bit replacement, with the correction's files measured on disk (payload as transmitted, plus the 4-bit basis):

1.58-bit baseCrossing payload (share of the replacement)Base plus correction2-bit replacement
Llama SINQ348 MB (19%)2.15 GB (+6%)2.02 GB
Llama HQQ87 MB (5%)1.81 GB (−10%)2.02 GB
Qwen SINQ44 MB (2%)1.77 GB (−13%)2.04 GB
Qwen HQQ176 MB (10%)1.95 GB (−4%)2.04 GB

The stored total stays close to the replacement's because the 1.58-bit base is 0.32 GB smaller than a 2-bit checkpoint, and the correction mostly fits into that difference. At the 37–38% payload the reviewer mentions, where RECAP also matches the replacement's accuracy, the total measured on disk is 2.60–2.62 GB, 28–29% above the replacement, for models far better than it on perplexity (24.8–50.0 against 319–19,703). Beyond the crossing, more storage buys more quality, and the server can cap the payload to a client's storage budget.

Memory and client compute. In memory, the serving layout holds about 1.8 times the payload to keep the decode step at 1.56 times. The one-time cost of building the basis on the client is discussed under Weakness 1, fifth point.

These are the trade-offs the reviewer asks about, now measured: hosting RECAP costs a decode step 1.56 times the base's and about a replacement's storage at matched quality, the price of keeping the update factorized; merging it into FP16 instead costs 5.2 times the decode step and 12,952 MB. The revision reports these costs in Section 5 (Table 6) and Appendix D.7 (Tables 18 and 19).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF: stronger targets, deployment costs, controls and robustness

We thank all three reviewers for their careful reading. We have uploaded a revised PDF in which every addition is marked in blue, and the main text still fits within the 10-page limit. Our replies use the revised numbering. New tables shift later numbers: the tables cited in the reviews are now Tables 7, 9, 10, 12, 13, 14, 16, 17 and 20 (formerly 4, 5, 6, 7, 8, 9, 11, 12 and 13), and the former Appendix D.6 is now D.8.

  1. Stronger replacement targets (mCCU W1, W2.6, Q1–Q2). To test usefulness against usable targets, Section 5 adds Table 1: refining 2-bit checkpoints toward the same family's 3-bit checkpoints, RECAP passes HQQ-3 on both metrics with 53% of its bytes and matches SINQ-3's accuracy with 79%, and from all four 1.58-bit bases it passes the stronger GPTQ 2-bit checkpoints with 39–80% of their bytes. Table 10 and Figure 2 add the Llama-2-13B rung that crosses its replacement, and Appendix D.1 reports the 3-bit headroom (Table 8).
  2. Deployment and client cost (mCCU W1.4–W1.5; VhDn W1, Q1). Table 6 states what the client pays: a one-time basis preparation of 11–33 minutes before any update arrives, reconstruction in 6–7 s, a stored total within −13% to +6% of the replacement at the crossing, and a decode step 1.56 times the packed base's, against 1.72 for two-factor baselines. Appendix D.7 adds the serving study and client preparation and storage (Tables 18 and 19).
  3. Numerical robustness (mCCU W1.5; nv7F W2). Appendix C.4 specifies the numerical protocol: an exact double-precision eigendecomposition and a float32 calibration pass at both endpoints, with a fingerprint check and a 4-bit basis as the fallback. Under it every audited client matches the server within 0.004 perplexity (Table 17).
  4. Source of the gain and the budget split (mCCU W2.1–W2.5, Q3–Q4; nv7F W1, Q1–Q2). Table 3 adds QERA at its published setting and the requested control, QERA with exactly RECAP's 4-bit factor code (the complement code alone), which isolates the shared basis: it lowers perplexity by a further 17–71% at 1.58 bits and 1–6% on the naive 2-bit bases (Tables 3 and 20). The server now selects the split on validation data with the complement code alone among the candidates (Appendix C.2; Table 21), Appendix D.8 reports that the codec reaches 92–94% of the distortion reduction Theorem 3's floor allows, and D.3–D.4 add the coverage–precision study and water-filling at one bit.
  5. Repeated updates and scope (mCCU W1.6, W2.7; VhDn W2; nv7F Q3). Table 5 shows four successive updates tracking a single update of about the same size (Appendix D.9). Appendix D.10 adds a mixture-of-experts model and Qwen3-8B (Table 22) and a GlowQ comparison, and D.5 adds calibration on task text (GSM8K).

Figure 1 is redrawn (nv7F Q4). We respond to each reviewer in their own thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.