acceptodds
Under review as a conference paper at ICLR 2027

RESCOM: ACTIVATION RESIDUAL COMPENSATION FOR LARGE LANGUAGE MODEL QUANTIZATION

Abstract

Low-bit weight–activation quantization reduces the memory and computation needed to deploy large language models, but activation rounding can still degrade accuracy after calibration. Reconstruction from rounded inputs cannot distinguish activations within the same quantization bin. We propose ResCom, a low-rank correction that uses the activation rounding residual to compensate for the resulting layer-output error. An output-aware calibration procedure selects residual directions by their contribution to output reconstruction and computes the correction in closed form, without gradient training or updates to the calibrated weights. On Llama-3-8B with four-bit weights and activations, rank-32 attention corrections reduce WikiText-2 perplexity from 7.708 to 7.147, improve MMLU by 2.89 percentage points, and raise five-shot GSM8K accuracy from 23.96 to 36.16 percent. Evaluations on models from 1B to 70B parameters and on held-out C4 test the method beyond this setting. On B200, the evaluated FP32 implementation adds 2.25 percent to 4K prefill time; a separate compact-storage implementation retains a 27.7 percent peak-memory reduction relative to FP16.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training quantization, weight-activation quantization, activation quantization residuals, low-rank compensation, reduced-rank regression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wXchOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ResCom, a method for repairing already-calibrated quantized large language model checkpoints. It feeds the activation quantization residual, d=x−Q(x)d=x-Q(x), into an auxiliary low-rank branch and fits a layer-output correction through rank-constrained ridge regression. The correction factors and intercept admit a closed-form solution, requiring neither gradient-based training nor changes to the host quantized weights or quantizer. In the main SERQ setting, the target is the output difference obtained by applying the same frozen effective linear map to activations before and after quantization, thereby isolating activation-rounding error. The paper evaluates multiple model families ranging from 1B to 70B parameters on language modeling, knowledge, reasoning, and long-context generation. Results show perplexity improvements across several W4A4 settings and notable MMLU and GSM8K gains on Llama-3-8B. The paper also studies branch inputs, rank, repair placement, calibration stability, and transfer across quantization pipelines, and separately measures FP32 simulation, compact storage, and native MXFP4 implementations.

Critical strengths

S1. The method has a clear objective, a complete solution, and a practical attachment point for existing quantized checkpoints. Equations (1)–(5) define activation-rounding error in the main SERQ setting as the missing output under a frozen effective map and derive a closed-form correction through rank-constrained ridge regression. Appendix A provides the derivation, and Algorithm 1 explicitly specifies the order of statistics collection and branch attachment. The method requires neither backpropagation nor requantization of the main weights, making it a potentially useful additional repair step for existing quantized models. S2. Accuracy recovery is supported by reasonably broad and targeted experiments. Tables 2–5 report paired W4A4 perplexity improvements across multiple model sizes and families, including gains on C4, which is not used for fitting. On Llama-3-8B, MMLU improves by 2.89 percentage points and strict GSM8K accuracy rises from 23.96% to 36.16%, indicating benefits beyond language-modeling metrics. Experiments on branch inputs, rank, repair placement, and calibration draws further help identify the source of the improvements. The capacity-matched input comparisons and budget-matched channel-restoration control are particularly useful for assessing the contribution of the residual signal itself.

Critical weaknesses

W1. The additional value of the specific method over the closest prior work and simpler alternatives is not sufficiently established. Under the main target e=Wsde=W_sd, Equation (4) gives C=WsGC=W_sG, closely relating the fit to residual-covariance-weighted low-rank approximation, a basic structure already studied by ARHQ. Section 3.3 and Appendix B usefully explain the deployment distinction between freezing the main path and splitting weights before requantization, but no direct budget-matched experiment quantifies the benefits of these differences. Moreover, the channel-restoration control in Table 8(b) and Appendix A.2 achieves 7.1351 PPL with the same parameter and factor-MAC budgets, compared with 7.1465 for dense rank-32 ResCom. This small difference does not establish statistically significant superiority of the channel alternative, but the proposed low-rank structure does not demonstrate an advantage in this comparison. The evidence therefore supports the usefulness of residual information more strongly than the additional value of the particular low-rank construction over related methods or simpler alternatives. W2. The deployment trade-offs and substantial failure cases limit the practical and reliability conclusions. The principal accuracy gains and the 2.25% prefill overhead come from FP32 simulation; in the same setting, decoding time increases from 76.79 to 86.11 ms/token, approximately 12.14%. The compact-storage implementation uses layerwise dequantization followed by FP32 GEMM. The native MXFP4 experiment instead uses a freshly quantized model and a different repair scope, reducing PPL from 7.865 to 7.733, while its current end-to-end implementation remains substantially slower than BF16. These limitations are disclosed, but the results do not yet jointly demonstrate the main accuracy gains and runtime costs in a competitive native low-bit implementation. Furthermore, Gemma-3-4B W4A4 SAMSum drops from 16.73 to 8.17, and several regressions occur under W4A8. The cross-pipeline experiments also extend the target to include weight-side error. These observations require clearer diagnosis and scope statements to assess the method’s reliability as a general post-calibration repair step.

Questions for authors

1.Under the main setting e=Wsde=W_sd, we obtain C=WsGC=W_sG. Please clarify which additional contributions arise from freezing the main path, directly using residual inputs, centering, or regularization, relative to residual-covariance-weighted low-rank approximation. Is there evidence comparing against ARHQ, or a clearly specified adaptation, using the same model, calibration data, and additional storage budget to quantify the benefits of these design differences? 2.Can accuracy, prefill latency, decoding latency, and peak memory be reported together before and after repair within the same native low-bit implementation, with matched repair sites and factor precision? In particular, how much of the accuracy recovery observed in the main INT4 simulations is retained under native execution, and what is the incremental runtime cost relative to an optimized low-bit baseline? 3.What causes the Gemma-3-4B W4A4 SAMSum drop from 16.73 to 8.17? Can per-example outputs or the validation-error diagnostics in Appendix A.1 help distinguish generation failures, local overcorrection, and changes in activation distributions after repair? Is there a validation signal that does not rely on test outcomes and can identify settings in which repair should not be enabled?

3 comments
AuthorsOpenReview ↗

Response to reviewer wXch — Rebuttal (Part 1/3)

Thank you for your detailed review and for decomposing our claims into separable factors. Your observation on channel restoration also led us to an accounting correction we had missed, which we are grateful for. We address each point below.

W1: With C=WsGC = W_s G, what are the independent contributions of freezing the host, the residual input, centering and the ridge penalty, and is there ARHQ evidence at matched model, calibration and storage?

Thank you for the valuable feedback. We separated all four factors on one frozen host, and we report the outcome including the part that does not favour an elaborate explanation: two of the four account for the gain and two do not.

1. Branch input — rank, sites, target and the relative ridge rule fixed, WikiText-2 perplexity:

BackboneRankUnrepaired hostrounded act. Q(x)Q(x)unrounded act. xxrounding residual dd
Llama-3-8B87.70778.2107.2417.203
Llama-3-8B327.70778.3277.2427.147
Llama-3-8B2567.70778.4177.2297.085
Mistral-7B-v0.3325.66275.7875.6525.617

The ordering is identical at every rank from 8 to 256, and the rounded-activation input is worse than the unrepaired host throughout. The unrounded activation recovers about 83% of what dd recovers at rank 32, with the gap widening monotonically from 0.038 to 0.144 across ranks 8 to 256.

2. Fitting rule — one frozen host, one shared set of FP64 moments, identical storage: the centered ridge fit, the ARHQ-derived weighting, the uncentered fit and the unregularized fit all land within 0.007 PPL of one another. We therefore attribute nothing to centering or regularization. Our original framing left room to read these as contributions, which was not sufficiently rigorous, and the revision states the conclusion directly.

3. Host treatment — the faithful ARHQ arm, same model, same 128 seed-0 windows, same 105.25 MiB budget, actually performing the weight split and GPTQ remainder quantization:

ModelCorpusResComARHQ splitNLL 95% intervalHolm pp
Llama-3-8BWT27.1465447.164656[−0.00502, −0.00005]0.04955
Llama-3-8BC410.34930810.447787[−0.01239, −0.00670]0.00020
Mistral-7BWT25.6167495.647338[−0.00703, −0.00390]0.00020
Mistral-7BC48.1877678.232931[−0.00688, −0.00425]0.00020

4 of 4 after Holm correction, with 20 of 20 lower perplexities across five independently fitted seeds. We keep the limit explicit: the seed-level Holm-adjusted pp is 0.25, because five seeds admit a minimum attainable pp of 0.0625.

4. Deployment cost of that difference — the ARHQ-style construction changes 128 of 128 attention projections, ResCom changes 0 of 128 (verified per projection with torch.equal, archived runs reproducing their one-window NLL to 0.0), and the update package is 48.4–48.9 MiB against 443–452 MiB.

So the answer to your question is that the residual input and the frozen-host attachment carry the result, and the solver details do not. We will present it in these four parts in the revision, which the submission did not do.


AuthorsOpenReview ↗

Response to reviewer wXch — Rebuttal (Part 2/3)

W2: Table 8(b) channel restoration reaches 7.1351 against dense rank-32 at 7.1465, so the low-rank structure shows no advantage.

You are right, and we think this comparison supports the claim we should have made rather than undermining it. Our thesis is that the activation rounding residual is a recoverable signal on a frozen quantized host; it is not a thesis that low-rank is the uniquely correct way to spend the budget, and the revision no longer reads that way.

Two independently constructed parameterizations — a dense rank-32 factorization, and a channel restoration selected by residual energy times the squared column norm of the frozen effective map — land within 0.01 PPL of each other at matched storage, with four paired comparisons nonsignificant under one Holm family and intervals spanning zero. We read that agreement as evidence of a shared underlying correction signal.

Your comment also exposed an accounting gap that our original control genuinely missed: it matched scalar count but did not charge for the stored channel indices that a deployed channel-restoration branch must ship. We added an actual-byte control that does charge for them, at 105.249947 MiB — 56 bytes below the dense branch budget — and it reproduces the same picture, 7.1366 against 7.1465, with no significant difference after Holm correction. The earlier control overstated the channel variant's budget advantage, and we have corrected it.

The revision therefore states the conclusion as residual-conditioned repair under either parameterization. We claim one limited advantage for the dense form, and it is a deployment property rather than an accuracy one: a fixed-rank factorization has a predictable, index-free footprint, so the update package size is known in advance and does not depend on which channels a given host selects.

W3/Q2: The simulated, compact-storage and native measurement scopes are mixed; can accuracy, prefill, decode and memory be reported together inside one native implementation?

Thank you for this — the mixing was a reporting fault on our part, and we have corrected it. We also declare the boundary explicitly rather than letting the three read as one number: the compact-storage figures describe archived coefficients and changed checkpoint bytes for distribution; the runtime table measures live runtime including workspace and KV cache; the CUDA Graph study is an independent measurement in its own tables and is never mixed into the runtime rows.

One frozen Llama-3-8B MXFP4 artifact, 224 natively packed projections under CUTLASS GEMM plus the same FP32 rank-32 factors at 128 attention sites, no branch fusion, shared Triton frontend fusion applied to host and repair alike:

LengthExecutionWT2 PPLPrefill (ms)Decode (ms/tok)Peak mem (GiB)
128Optimized host7.871084160.46152.428.172
128+ ResCom7.733374176.42161.708.275
2048Optimized host7.871084193.79154.139.588
2048+ ResCom7.733374210.52163.299.690
4096Optimized host7.871084324.65148.7411.097
4096+ ResCom7.733374349.53158.0011.200

All 141 complete WikiText-2 test windows — 288,627 scored next-token labels — preserve bitwise-identical full logits across the execution optimization, so the accuracy column is not a kernel artifact. On this artifact the repair costs 7.66% prefill, 6.23% per decode token and 105.25 MiB.

To state the setting unambiguously, the main path here is natively 4-bit, not simulated: E2M1 operands with an E4M3 scale per 16 elements. Simulation appears only where the comparator methods define their own protocol. We do not claim to beat BF16 or a mature serving engine, and the revision says so.

On a separate native NVFP4 artifact, against the same optimized host (identical prefill fusion and packed Q/K/V, differing only in that no branch is attached, all arms in one interleaved rotation), with both branch projections fused, the marginal cost at 4096 tokens is +2.92–3.83% per decode token and +3.34–4.06% end to end for an FP16 branch at 52.625 MiB, and +4.52–4.68% / +4.57–4.69% for an FP32 branch at 105.25 MiB. The two artifacts differ in site count and backend, so we report them separately.


AuthorsOpenReview ↗

Response to reviewer wXch — Rebuttal (Part 3/3)

W4/Q3: Why does SAMSum drop, can a per-example diagnosis distinguish generation failure from local over-correction or an activation-distribution shift, and is there a validation signal not dependent on test outcomes that could decide when to disable the repair?

Thank you for the three-part question; it is the one we spent the most effort on. We reproduced all 400 original Gemma-3-4B W4A4 outputs exactly before intervening.

Every one of the 88 newly blank outputs is token 251 (carriage return) followed by token 107 (newline). SAMSum's newline stopping rule fires on the second token, and the official first-line extractor then returns nothing. The content is generated; the protocol discards it. This distinguishes the three hypotheses you name: it is not generation failure, since the text exists and is well-formed; it is not a shift toward stopping early, since none of the 88 stops on the first token and the mean first-step stop probability decreases by 0.10586 (exact sign test, pp = 1.53e-10); and it is not diffuse over-correction, since the symptom is confined to one two-token prefix in 88 of 88 cases. Under EOS-only stopping with full-text extraction — same model, same frozen factors, same prompts, applied identically to both arms — the comparison reverses, with zero blank ResCom responses against one for the host. We report all six stop-rule × extractor cells rather than a selected one:

Stop ruleExtractionHostResComΔ
EOS + newlineOfficial first line (benchmark protocol)16.7318.167−8.564
EOS + newlineFull text16.4987.133−9.365
EOS onlyOfficial first line15.0448.271−6.773
EOS onlyFull text14.60316.537+1.934

Each rule alone is insufficient; the interaction needs both. Table 4's official numbers are unchanged.

On a validation signal that does not rely on test outcomes: we evaluate the repair on disjoint WikiText-2 raw-v1 windows with no branch fitting and no threshold tuning. Local recovery — percent error-energy reduction at each repaired site — holds at 43–57%, and the end-to-end NLL difference stays negative with intervals excluding zero at every length up to 7,500 tokens, on the Gemma model where the interaction occurs as well as on Llama-3-8B. This is computable before any benchmark is run.

We should report the honest negative on the stronger form of your question. We tested three probability and deviation signals as a per-example gate for disabling the repair, and their AUCs are 0.31–0.63 — not usable. So we can offer a pre-test health check but not a validated per-example switch, and the revision says that rather than implying we have one.

W5: The cross-pipeline experiments extend the target to weight-side error, so the applicable scope changes.

Correct, and we have written the targets out rather than leaving them implicit. With d=x−Q(x)d = x - Q(x) and frozen host output WsQ(x)+cW_s Q(x) + c, the default SERQ target is activation-only:

e=Ws de = W_s\,d

while the PrefixQuant and FlatQuant rows use their pre-rounding weight reference:

e=Wref x−Ws Q(x)=Ws d+(Wref−Ws) xe = W_{\mathrm{ref}}\,x - W_s\,Q(x) = W_s\,d + (W_{\mathrm{ref}} - W_s)\,x

Proposition 1's residual-alignment statement is stated for the activation-only target, and we no longer let it read as covering the cross-pipeline setting; the centered rank-constrained regression and its closed-form solution apply to any fixed target, which is what makes the extension well-defined. Each cross-pipeline row now labels its pipeline, target components, sites, rank and host calibration, so the rows document incremental attachment within each pipeline rather than a cross-host ranking.

On presentation (Presentation 2). We accept this and have a concrete plan rather than a promise to try. The introduction now states the three elements separately — a standard closed-form solver, the rounding-residual target, and the frozen-host attachment — instead of presenting them as one contribution. Table captions are standardized to name the pipeline and the matched quantity ("SERQ W4A4 adaptation, matched 105.25 MiB", "attention-only NVFP4 adaptation"). The regressions and the scope limits move from the appendix into the main text, including the SAMSum protocol finding, the solver negative control, and the gate AUCs above. We would rather the paper be read as a narrower claim with visible limits than as a broad one.

Once again, thank you for your constructive feedback. We hope our response has addressed your concerns.


Reviewer_XBwGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ResCom attaches a low-rank branch to an already-quantized W4A4 linear layer, leaving the host checkpoint frozen. The branch takes the activation rounding residual d = x − Q(x) and adds U(Vd) + b to the quantized output; U, V and b are fitted in closed form by ridge-regularized reduced-rank regression to the output error e = W_s d, where W_s is the host's effective quantized weight, using no gradients and leaving the host unchanged. The experiments mainly cover the W4A4 setting on 1B–70B models, and also assess transfer across quantization pipelines, downstream tasks and runtime overhead.

Critical strengths
  1. The method targets a well-identified signal, and it is correct, cheap and carefully measured. The branch explicitly extracts the information discarded by activation rounding and compensates its contribution to the output, while the quantized backbone is left entirely untouched, so it attaches easily to checkpoints that already exist.
  2. Part of the W4A4 gain is substantial. On Llama-3-8B, WikiText-2 PPL falls from 7.708 to 7.147, MMLU gains 2.89 points and GSM8K improves from 23.96% to 36.16%, which indicates that residual repair recovers downstream capability of real value, not merely a better local reconstruction error.
Critical weaknesses
  1. The increment over existing closed-form compensation is not established, and the selection mechanism reduces to a known construction. The paradigm already exists. QwT (CVPR 2025) already attaches closed-form-fitted linear compensation to a frozen quantized network. ResCom has differences with QwT, but QwT is neither cited nor discussed in this paper.
  • The selection mechanism reduces to a weighted SVD, the best rank-r approximation of Ws in the G^{1/2}-weighted norm — the same construction as ARHQ (arXiv:2605.00140 ), and close to QERA(arXiv:2410.06040). The paper does not compare against neither of them.
  1. The gains and the efficiency claims do not generalize
  • WikiText-2 PPL improves by 7.3% on Llama-3-8B but only 0.4–3.1% on the other backbones; on six of the other eight, zero-shot changes fall in [−0.22, +0.41], with no significance tests.
  • The strongest host gains almost nothing: FlatQuant alone already reaches 6.972, better than SERQ+ResCom's 7.147; on it ResCom contributes only −0.17%, and only with a non-default site set.
  • Regressions are not discussed: Gemma-3-4B SAMSum 16.73 → 8.17 and W4A8 PPL +3.5%, while the conclusion claims consistent improvement.
Questions for authors
  1. What is the key value over QwT and ARHQ?
  2. At what precision does the branch run, what is the bottleneck on the native low-bit path, and what are its marginal prefill and decode costs, separately , in a real INT4/MXFP4 pipeline?
4 comments
AuthorsOpenReview ↗

Response to reviewer XBwG — Rebuttal (Part 1/4)

Thank you for your careful and constructive review. We appreciate the pointers to QwT and QERA, and we address each point below.

X1: QwT is neither cited nor discussed.

Thank you for pointing this out — the omission was ours and we apologize for it. QwT is now cited and reproduced under our own host, following Fu et al. (2025) exactly: the original FP32 teacher block evaluated on the current host block input, affine OLS on centered FP64 moments with a Moore–Penrose inverse, blocks attached sequentially before propagating states.

MethodBranch storageLlama WT2Llama C4
Host (no repair)—7.707711.1922
QwT dense2048.50 MiB7.273510.8916
QwT equal-storage low-rank105.25 MiB8.440013.7988
ResCom105.25 MiB7.146510.3493

Dense QwT carries 537,001,984 FP32 coefficients, 19.5× our storage; constrained to our budget it is worse than the untouched host. We will add QwT to the related work and to the matched-budget table in the revision.

X2: The selection mechanism reduces to a G1/2G^{1/2}-weighted SVD, i.e. the ARHQ construction, with no comparison.

Thank you for the precise reading. We agree on the algebraic connection and claim no priority for it: with e=Wsde = W_s d we have C=WsGC = W_s G, and the closed form is a standard output-aware rank-constrained ridge approximation. Our original framing left room to read the solver as a contribution, which was not sufficiently rigorous on our part, and the revision states that it is not.

We implemented the ARHQ-style arm faithfully rather than approximating it — it performs the actual weight split and GPTQ remainder quantization, on the same host, the same 128 seed-0 calibration windows and the same coefficient budget:

ModelCorpusResComARHQ splitNLL 95% intervalHolm pp
Llama-3-8BWT27.1465447.164656[−0.00502, −0.00005]0.04955
Llama-3-8BC410.34930810.447787[−0.01239, −0.00670]0.00020
Mistral-7BWT25.6167495.647338[−0.00703, −0.00390]0.00020
Mistral-7BC48.1877678.232931[−0.00688, −0.00425]0.00020

Five independently fitted calibration seeds give 20 of 20 lower perplexities. Downstream, MMLU favors ResCom on both backbones over all 57 subjects and 14,042 questions (57.5773% vs 57.0432% on Llama-3-8B; 56.7298% vs 56.5447% on Mistral-7B-v0.3), which we report as descriptive since no ARHQ comparison in the 80-test family reaches corrected significance either way.

We also report the negative control your comment calls for. Isolating centering, the ridge penalty, the uncentered fit and the ARHQ-derived weighting on one frozen host, with one shared set of FP64 moments and identical storage, leaves all four fits within 0.007 PPL of one another. What does separate the two constructions is the host treatment: the ARHQ-style arm changes 128 of 128 attention projections because it requantizes the remainder, ResCom changes 0 of 128 (verified per projection with torch.equal), and the update package is 48.4–48.9 MiB against 443–452 MiB.

We will move this negative control into the main text. It argues against part of our original framing, and we think the paper is more useful with it visible.

X3: The method is close to QERA and no comparison is provided.

Thank you for the suggestion. We added three matched-budget QERA controls on the shared frozen host at rank 32 and 105.25 MiB: exact uncentered autocorrelation and a per-coordinate RMS diagonal, both on Q(x)Q(x), plus exact factors on xx (rows in the Part 2 table). All twelve perplexity comparisons across two models, two corpora and three controls favor ResCom, Holm-adjusted paired-window p≈0.00060p \approx 0.00060.


AuthorsOpenReview ↗

Response to reviewer XBwG — Rebuttal (Part 2/4)

X3, continued.

MethodLlama WT2Llama C4Mistral WT2Mistral C4
Host7.707711.19225.66278.2438
QERA exact, Q(x)Q(x)7.696011.17105.67178.2332
QERA diagonal, Q(x)Q(x)7.680711.16385.66208.2330
QERA exact, xx7.676811.17235.66608.2351
ResCom7.146510.34935.61678.1878

The downstream separation on Llama-3-8B is larger, all significant within one joint 80-test Holm family:

Metricvs QERA exactvs QERA diagvs QERA exact, xx
Eight-task mean+2.018 [+1.39,+2.62]+1.983 [+1.35,+2.62]+2.140 [+1.54,+2.77]
MMLU+3.45 (pp=3.3e-16)+3.06 (1.1e-12)+3.19 (5.2e-14)
GSM8K strict+11.68 (1.1e-15)+11.45 (8.1e-14)+11.90 (3.6e-16)

The mechanism is visible in the third control. QERA's target is Wref−WsW_{\mathrm{ref}} - W_s, so its factors apply to the full activation and recover little once WsW_s already contains the host's linear reconstruction; giving QERA the unrounded activation instead of Q(x)Q(x) does not close the gap (7.6768 against 7.1465), so the difference is the target rather than the input alone.

We state the scope plainly, and will do so in the revision as well: this is a matched-budget adaptation of the QERA target on our shared host, not a reproduction of the full published pipeline, and the Mistral margins are much smaller than the Llama ones — eight-task differences within ±0.25pp with intervals spanning zero.

X4: Only Llama improves; the six-backbone zero-shot changes lie in [−0.22, +0.41] and no significance tests are reported.

You are right on both counts, and we accept the correction. We have added the testing and withdrawn the uniformity claim. We now run 120 paired per-question tests, with families declared and corrected separately:

FamilyDesignResult
Primary downstream2 models × 10 metrics × 4 matched-budget controls = 80 two-sided exact McNemar tests, one joint Holm correction16 significant, all favoring ResCom; none significant against
Host / rebuild-only40 tests, separately declared Holm family12 significant, all favoring ResCom
Original generationall 31 pairable comparisons, joint Holmall directions reported, with intervals and adjusted pp

We also acted on the implication of your comment beyond what it asked: the six original backbone zero-shot means in Tables 1–2 are now presented as descriptive, because matching per-question records for those archived runs are not part of either test family. The cross-backbone claim is instead carried by paired perplexity, where the direction is consistent in 12 of 12 cells across six backbones and two corpora, held-out C4 included:

BackboneSERQ WT2ResCom WT2SERQ C4ResCom C4
Llama-3-8B7.7087.14711.19210.349
LLaMA-2 7B5.9395.8477.4327.315
LLaMA-2 13B5.2275.1486.6936.634
Llama-3.2-3B9.1778.96412.68812.317
Llama-3.2-1B12.46112.07117.22716.712
Mistral-7B-v0.35.6635.6178.2448.188

We keep one limit explicit rather than rounding it in our favour: across five matched calibration seeds ResCom has lower perplexity than the ARHQ adaptation in 5/5 seeds for all four model–corpus pairs, but the seed-level Holm-adjusted pp is 0.25, because five seeds admit a minimum attainable pp of 0.0625. We report that as a limitation of the seed count.

The checklist will be corrected in the same revision so that the significance claim points to the appendix section that actually contains these tests.


AuthorsOpenReview ↗

Response to reviewer XBwG — Rebuttal (Part 3/4)

X5: FlatQuant alone already reaches 6.972, better than SERQ+ResCom at 7.147, and the sites are not the default.

Thank you — we accept both observations. We reproduced 6.9722366 independently, and we agree that a cross-host comparison cannot isolate the increment the repair contributes. We have narrowed the text so these rows document incremental attachment within each pipeline rather than a cross-host ranking, and each row now labels its pipeline, target components, sites, rank and host calibration. We also state that the FlatQuant and PrefixQuant target includes weight-side error and that the sites are the output and down projections, not our default.

Within that scope, the claim holds on the strongest host we could obtain:

FlatQuant W4A4KV4, Llama-3-8BHost+ ResCom
WikiText-2 PPL (141 windows)6.97226.9591
C4 PPL (256 windows)11.183311.1623
MMLU, 57 subjects, 14,042 questions56.26%56.73% (+66 correct)
4K prefill (ms)420.48423.65 (+0.75%)

for 260 seconds of repair installation against 1793 seconds of host calibration.

X6: SAMSum drops from 16.73 to 8.17, and there are W4A8 regressions, which contradicts "consistent improvement".

You are right that the wording was not defensible, and we no longer describe the improvement as consistent. On SAMSum we can now account for the result at the token level. We reproduced all 400 original Gemma-3-4B W4A4 outputs exactly before intervening.

Every one of the 88 newly blank outputs is token 251 (carriage return) followed by token 107 (newline). SAMSum's newline stopping rule fires on the second token, and the official first-line extractor then returns nothing. The content is generated; the protocol discards it. This excludes the two explanations a reader would suspect first: not degraded generation, since the text exists, and not earlier stopping, since none of the 88 stops on the first token and the mean first-step stop probability decreases by 0.10586 (exact sign test, pp = 1.53e-10).

We then varied the two protocol components independently and report all six stop-rule × extractor cells rather than a selected one:

Stop ruleExtractionHostResComΔ
EOS + newlineOfficial first line (benchmark protocol)16.7318.167−8.564
EOS + newlineFull text16.4987.133−9.365
EOS onlyOfficial first line15.0448.271−6.773
EOS onlyFull text14.60316.537+1.934

Each rule alone is insufficient. Under EOS-only stopping with full-text extraction — same model, same frozen factors, same prompts, applied identically to both arms — the comparison reverses, with zero blank ResCom responses against one for the host. Table 4's official numbers are unchanged, and we present this as an explanation of the regression, not a replacement for it.

On the same host the repair improves GSM8K strict by 6.82 (Holm pp = 1.61e-06), QMSum by 4.50 (0.0015) and RepoBench-P by 5.99 (0.0015) in the 31-comparison generation family.

On W4A8: at eight-bit activations the rounding residual the branch feeds on is itself small, so there is little headroom by construction. We have scoped the method to W4A4 in the revision. Across the complete 31-comparison generation family, no W4A8 difference is significant in either direction after Holm correction.


AuthorsOpenReview ↗

Response to reviewer XBwG — Rebuttal (Part 4/4)

X7: What is the branch precision, where is the bottleneck on the native path, and what are the prefill and decode marginal costs on real INT4/MXFP4?

Taking the three parts in order.

Precision. FP32 factors in the submission, with the main path at its native low-bit representation. We now report both precisions. Moving only the branch arithmetic to FP16 — same saved factors, no refitting, bit-identical host — halves coefficient storage from 105.25 to 52.625 MiB, with perplexity relative changes of at most 0.40% across the whole main comparison and 8 of 16 cells lower at FP16.

Bottleneck. The two branch GEMMs, which retained ATen/cuBLAS execution while residual conversion and output/bias fusion already ran in shared Triton/CUDA kernels. At batch one these are rank-32 matrix–vector products, and cuBLAS provides no half-precision GEMV, so a half-precision matmul at M=1M=1 reaches a GEMM kernel that cannot fill a tensor-core MMA tile. That is why an FP16 branch measured slower in eager decode despite moving half the bytes. We then removed the cause rather than only explaining it: both projections are now Triton kernels, Q/K/V down-projections concatenate along the rank axis into one GEMV, and one further kernel performs the up-projection, host addition and bias. Both concatenations are exact, and branch kernels per attention layer drop from 12 to 4.

Marginal costs. MXFP4 artifact (224 packed projections, CUTLASS GEMM), FP32 branch, unfused: 7.66% prefill, 6.23% per decode token, 105.25 MiB, with bitwise-identical full logits across the optimization on all 141 test windows. NVFP4 artifact against the same optimized host — identical prefill fusion and packed Q/K/V, no branch attached, same interleaved rotation — at 4096 tokens:

BranchStoragePrefillDecode / tokenEnd to end
FP16, fused52.625 MiB+4.69–4.77%+2.92–3.83%+3.34–4.06%
FP32, fused105.25 MiB+4.71–4.72%+4.52–4.68%+4.57–4.69%
FP16, unfused52.625 MiB+4.69–4.77%+10.77–11.16%—

With the launch path fixed the FP16 branch is faster than the FP32 branch in 12 of 12 cells across two adaptations, three prompt lengths and two devices, so halving coefficient storage no longer costs decode throughput. The two artifacts differ in site count and backend, so we will label each separately rather than presenting one improvement.

Once again, thank you for your constructive feedback. We hope our response has addressed your concerns.


Reviewer_esMUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ResCom, a correction method for quantized LLMs. It adds a low-rank branch conditioned on the activation rounding error. Experiments across different models demonstrate improvement on W4A4 settings with the added LoRA branch.

Critical strengths
  1. The formulation is clear and the calibration is simple, making it fast to be applied
  2. The ablation study is useful
  3. The method is evaluated on multiple model families and scales
Critical weaknesses
  1. The novelty is limited. Adding an additional lora branch for accuracy recovery is a well-known technique. The novelty relative to existing methods such as SVDQuant[1] is insufficiently established. SVDQuant already use unquantized activations in a high-precision low-rank branch and therefore retain within-bin distinctions. It also provides a efficient kernel implementation to support inference speedup. Therefore, the main problem of "activation rounding residual" seems already solved by SVDQuant.

  2. The benefits seems marginal compared to FlatQuant and PrefixQuant.

  3. The main results do not establish an advantage at a fixed compression ratio. Adding parameters will definitely improve model performance, but it is unclear whether it can outperform alternative precision allocations.

[1] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

Questions for authors

Please see the weakness above

3 comments
AuthorsOpenReview ↗

Response to reviewer esMU — Rebuttal (Part 1/3)

Thank you for your thoughtful and detailed review. We appreciate your feedback and address each point below.

W1: A LoRA-style branch is known, and SVDQuant already retains the within-bin information.

Thank you for the valuable feedback. You are right that using unquantized information in an auxiliary high-precision branch is established, and we do not claim priority for it. We also agree on the algebra: after centering, C=WsGC = W_s G is a standard output-aware rank-constrained ridge approximation. We have revised the text to state both points explicitly rather than leaving the novelty claim broad.

What we do claim is that the input to that branch matters, and that the host must not be rebuilt. We isolated the first factor by holding rank, parameter count, factor-MAC, target and the relative ridge rule fixed, and varying only the branch input. WikiText-2 perplexity on the calibrated SERQ W4A4 host, 128 attention Q/K/V/O sites:

BackboneRankUnrepaired hostrounded act. Q(x)Q(x)unrounded act. xxrounding residual dd
Llama-3-8B87.70778.2107.2417.203
Llama-3-8B327.70778.3277.2427.147
Llama-3-8B1287.70778.3847.2277.110
Llama-3-8B2567.70778.4177.2297.085
Mistral-7B-v0.3325.66275.7875.6525.617

The unrounded activation — the information an SVDQuant-style branch carries — recovers about 83% of what dd recovers at rank 32, and the gap widens monotonically from 0.038 at rank 8 to 0.144 at rank 256. The rounded activation is worse than the unrepaired host at every rank.

On SVDQuant specifically, we re-ran the weight-splitting branch under our deployment constraint rather than arguing from the paper. The calibrated host is retained bit for bit, verified per projection with torch.equal (0 of 128 changed), with identical calibration tokens and an equal or larger factor budget. Under that constraint the two are statistically indistinguishable in 12 of 16 paired comparisons (two-sided exact sign test on per-window NLL, Holm-corrected in a separately declared family). The published construction changes 128 of 128 attention projections because it requantizes the remainder, so we read the published margin as attributable to requantization — the step that is unavailable once a checkpoint is deployed. Where requantization is permitted, that is a different operating point and we report it unchanged.

We will add both the matched-input table and the frozen-host sign-test result to the main text, since at present they appear only in the appendix.

W4: Adding parameters necessarily helps, and no advantage is established at a fixed compression ratio.

Thank you for this point; we think it is the right test to apply, and we answer it with our own controls rather than with an argument. At an identical 105.25 MiB coefficient budget on the same host:

MethodBranch storageLlama WT2Llama C4
Host (no repair)—7.707711.1922
QwT dense2048.50 MiB7.273510.8916
QwT equal-storage low-rank105.25 MiB8.440013.7988
ResCom105.25 MiB7.146510.3493

The equal-storage low-rank adaptation is worse than doing nothing, so capacity is not the operative variable. The table above fixes rank, parameters and factor-MAC; a separate control fixes the actual stored bytes, charging for channel indices at 105.249947 MiB (56 bytes below the dense budget), and varies only the parameterization. Online cost is 0.39% factor-MAC and 105.25 MiB against a 4-bit host, with the main path bit-identical.

We will state in the revision that every comparison we report is budget-matched, and name the matched quantity in each table caption, which the submission did not do consistently.


AuthorsOpenReview ↗

Response to reviewer esMU — Rebuttal (Part 2/3)

W1, continued: the full matched-budget comparison.

Because W1 turns on whether anything survives a matched budget, we report the complete reconstruction study. Shared calibrated SERQ W4A4 host, 128 attention sites, rank 32, FP32 factors, 128 WikiText-2 train windows at seed 0, FP64 statistics, C4 held out, 105.25 MiB for every adaptation unless noted:

MethodBranch storageLlama WT2Llama C4Mistral WT2Mistral C4
Host—7.707711.19225.66278.2438
QwT dense2048.50 MiB7.273510.8916––
QwT equal-storage low-rank105.25 MiB8.440013.7988––
ARHQ split + requantize105.25 MiB7.164710.44785.64738.2329
QERA exact, Q(x)Q(x)105.25 MiB7.696011.17105.67178.2332
QERA diagonal, Q(x)Q(x)105.25 MiB7.680711.16385.66208.2330
QERA exact, xx105.25 MiB7.676811.17235.66608.2351
ResCom105.25 MiB7.146510.34935.61678.1878

The ARHQ arm performs the actual weight split and GPTQ remainder quantization, not an approximation of it. All four paired-window comparisons favor ResCom after Holm correction (Llama WT2 pp = 0.04955; the other three 0.00020), and five independently fitted seeds give 20 of 20 lower perplexities. Downstream, MMLU favors ResCom on both backbones over all 57 subjects and 14,042 questions: 57.5773% vs 57.0432% on Llama-3-8B and 56.7298% vs 56.5447% on Mistral-7B-v0.3. We report the MMLU direction as descriptive, since no ARHQ comparison in the 80-test family reaches corrected significance either way.

We should also report what does not explain the result. On one frozen host with one shared set of FP64 moments and identical storage, the centered ridge fit, the ARHQ-derived weighting, the uncentered fit and the unregularized fit all land within 0.007 PPL of one another. Our original framing left room to read the solver as a contribution; it is not, and the revision says so directly.

W2: SVDQuant has an efficient kernel.

Thank you for the suggestion — this was a genuine gap, and we added the joint measurement rather than a kernel comparison in isolation.

First, to state the setting unambiguously, the main path in these measurements is natively 4-bit, not simulated: E2M1 operands with an E4M3 scale per 16 elements, executed by torch._scaled_mm on all 128 attention projections, with a separate MXFP4 artifact running 224 projections through CUTLASS GEMM. Simulation appears only where the comparator methods define their own protocol.

On one frozen Llama-3-8B MXFP4 artifact, with an FP32 branch and no branch fusion, B200, batch and concurrency one, medians over 32 cached decode steps:

LengthExecutionWT2 PPLPrefill (ms)Decode (ms/tok)Peak mem (GiB)
128Optimized host7.871084160.46152.428.172
128+ ResCom7.733374176.42161.708.275
4096Optimized host7.871084324.65148.7411.097
4096+ ResCom7.733374349.53158.0011.200

All 141 complete WikiText-2 test windows — 288,627 scored next-token labels — preserve bitwise-identical full logits across the execution optimization, for host and repair alike, so the accuracy column is not a kernel artifact. On this artifact the repair costs 7.66% prefill, 6.23% per decode token and 105.25 MiB.

We also ran both adaptations through the same engine. On a shared native NVFP4 backend with the same Q/K/V fusion and CUDA Graph decoding, two B200 GPUs, eight interleaved matched requests, ResCom's end-to-end latency with an FP16 branch is within 0.25–1.31% of SVDQuant-style weight splitting at 128–4096 tokens, and the two are at parity downstream (PIQA 79.76 vs 79.65, ARC-Challenge 52.30 vs 52.30, BoolQ 81.71 vs 80.86).


AuthorsOpenReview ↗

Response to reviewer esMU — Rebuttal (Part 3/3)

W2, continued: marginal cost against the same optimized host, at both branch precisions.

We realized while preparing this response that our earlier latency statements did not say which branch precision they referred to, which made them hard to compare. We now report both, against a host that receives identical prefill fusion and packed Q/K/V and differs only in that no branch is attached, running in the same interleaved rotation (seven arms, 8 requests × 7 repeats per GPU so every arm occupies every position exactly eight times). With both branch projections fused, at 4096 tokens:

BranchStoragePrefillDecode / tokenEnd to end
FP16, fused52.625 MiB+4.69–4.77%+2.92–3.83%+3.34–4.06%
FP32, fused105.25 MiB+4.71–4.72%+4.52–4.68%+4.57–4.69%
FP16, unfused52.625 MiB+4.69–4.77%+10.77–11.16%—

The two artifacts differ in site count and backend, so we do not present the MXFP4 and NVFP4 percentages as a single improvement, and we will label each accordingly in the revision.

Finally, a cost axis a kernel cannot address: because ResCom rewrites no main-path bytes, distributing the repair to a recipient who already holds the quantized base costs 48.4–48.9 MiB, against 443–478 MiB for ARHQ- and SVDQuant-style updates on the same artifacts.

W3: The gains are marginal relative to FlatQuant and PrefixQuant.

We agree that on a strong host the gains are small, and we have narrowed the claim rather than defending the original wording. The claim these rows support is a deployment claim: a checkpoint whose calibration cost is already sunk can still be improved by one gradient-free pass that rewrites no main-path weight.

PipelineTarget componentsSitesRankHost PPL+ Repair
SERQactivation-onlyQ/K/V/O327.70777.1465
PrefixQuantactivation + weightoutput647.8607.6046
FlatQuantactivation + weightoutput + down326.97226.9600

We reload the completed 15-epoch FlatQuant transformation — 1793 seconds of calibration, independently reproduced at 6.9722366 — and install the repair in 260 seconds, roughly 15% of that cost. An independent re-run extends this to held-out text and to a downstream task, with the base logits restored bit for bit when the branch is disabled:

FlatQuant W4A4KV4, Llama-3-8BHost+ ResCom
WikiText-2 PPL (141 windows)6.97226.9591
C4 PPL (256 windows)11.183311.1623
MMLU, 57 subjects, 14,042 questions56.26%56.73% (+66 correct)
4K prefill (ms)420.48423.65 (+0.75%)

We do not propose ResCom as a replacement for these pipelines, and the revision states this in the text rather than only in the table captions.

On the soundness and informativeness scores. We would be grateful if these could be reconsidered against the specific technical points. In this revision we withdraw the solver as a novelty claim and present the negative control ourselves; we add faithful QwT, ARHQ and QERA arms at matched budget, a native 4-bit runtime measurement with both branch precisions, and 120 paired per-question significance tests in separately declared Holm families, of which 28 are significant, all favoring ResCom and none against. We have also narrowed every claim the evidence does not carry, including presenting the six-backbone zero-shot means as descriptive.

Once again, thank you for your constructive feedback. We hope our response has addressed your concerns.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions and New Experiments

Thank you for the reviews and discussion. Since submission, we have revised the exposition and organization of the paper and added experiments in response to the reviews. The revised manuscript has ten pages of main text. Blue text highlights the rebuttal additions. Other post-submission edits appear in regular type. The OpenReview PDF diff shows the full set of changes from the original submission.

For readers using the original table numbers, the channel-restoration control in Table 8(b) is now Table 7(b) (p. 9). Its original parameter- and compute-matched comparison is retained. Table 48(a–b) (p. 39) compares storage and perplexity after counting channel indices and gives paired uncertainty estimates. Table 48(c) on the same page reports prefill and decode latency for the byte-matched repairs on a shared eager PyTorch backend with FP32 simulation of SERQ W4A4. Table 49 (p. 40) reports perplexity, prefill, decode, and peak memory for the optimized native MXFP4 host with and without ResCom.

Other additions include matched-budget comparisons with ARHQ and QERA, paired downstream analyses, branch-precision measurements, and an investigation of the Gemma SAMSum regression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.