- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ResCom, a method for repairing already-calibrated quantized large language model checkpoints. It feeds the activation quantization residual, , into an auxiliary low-rank branch and fits a layer-output correction through rank-constrained ridge regression. The correction factors and intercept admit a closed-form solution, requiring neither gradient-based training nor changes to the host quantized weights or quantizer. In the main SERQ setting, the target is the output difference obtained by applying the same frozen effective linear map to activations before and after quantization, thereby isolating activation-rounding error. The paper evaluates multiple model families ranging from 1B to 70B parameters on language modeling, knowledge, reasoning, and long-context generation. Results show perplexity improvements across several W4A4 settings and notable MMLU and GSM8K gains on Llama-3-8B. The paper also studies branch inputs, rank, repair placement, calibration stability, and transfer across quantization pipelines, and separately measures FP32 simulation, compact storage, and native MXFP4 implementations.
Critical strengths
S1. The method has a clear objective, a complete solution, and a practical attachment point for existing quantized checkpoints. Equations (1)–(5) define activation-rounding error in the main SERQ setting as the missing output under a frozen effective map and derive a closed-form correction through rank-constrained ridge regression. Appendix A provides the derivation, and Algorithm 1 explicitly specifies the order of statistics collection and branch attachment. The method requires neither backpropagation nor requantization of the main weights, making it a potentially useful additional repair step for existing quantized models. S2. Accuracy recovery is supported by reasonably broad and targeted experiments. Tables 2–5 report paired W4A4 perplexity improvements across multiple model sizes and families, including gains on C4, which is not used for fitting. On Llama-3-8B, MMLU improves by 2.89 percentage points and strict GSM8K accuracy rises from 23.96% to 36.16%, indicating benefits beyond language-modeling metrics. Experiments on branch inputs, rank, repair placement, and calibration draws further help identify the source of the improvements. The capacity-matched input comparisons and budget-matched channel-restoration control are particularly useful for assessing the contribution of the residual signal itself.
Critical weaknesses
W1. The additional value of the specific method over the closest prior work and simpler alternatives is not sufficiently established. Under the main target , Equation (4) gives , closely relating the fit to residual-covariance-weighted low-rank approximation, a basic structure already studied by ARHQ. Section 3.3 and Appendix B usefully explain the deployment distinction between freezing the main path and splitting weights before requantization, but no direct budget-matched experiment quantifies the benefits of these differences. Moreover, the channel-restoration control in Table 8(b) and Appendix A.2 achieves 7.1351 PPL with the same parameter and factor-MAC budgets, compared with 7.1465 for dense rank-32 ResCom. This small difference does not establish statistically significant superiority of the channel alternative, but the proposed low-rank structure does not demonstrate an advantage in this comparison. The evidence therefore supports the usefulness of residual information more strongly than the additional value of the particular low-rank construction over related methods or simpler alternatives. W2. The deployment trade-offs and substantial failure cases limit the practical and reliability conclusions. The principal accuracy gains and the 2.25% prefill overhead come from FP32 simulation; in the same setting, decoding time increases from 76.79 to 86.11 ms/token, approximately 12.14%. The compact-storage implementation uses layerwise dequantization followed by FP32 GEMM. The native MXFP4 experiment instead uses a freshly quantized model and a different repair scope, reducing PPL from 7.865 to 7.733, while its current end-to-end implementation remains substantially slower than BF16. These limitations are disclosed, but the results do not yet jointly demonstrate the main accuracy gains and runtime costs in a competitive native low-bit implementation. Furthermore, Gemma-3-4B W4A4 SAMSum drops from 16.73 to 8.17, and several regressions occur under W4A8. The cross-pipeline experiments also extend the target to include weight-side error. These observations require clearer diagnosis and scope statements to assess the method’s reliability as a general post-calibration repair step.
Questions for authors
1.Under the main setting , we obtain . Please clarify which additional contributions arise from freezing the main path, directly using residual inputs, centering, or regularization, relative to residual-covariance-weighted low-rank approximation. Is there evidence comparing against ARHQ, or a clearly specified adaptation, using the same model, calibration data, and additional storage budget to quantify the benefits of these design differences? 2.Can accuracy, prefill latency, decoding latency, and peak memory be reported together before and after repair within the same native low-bit implementation, with matched repair sites and factor precision? In particular, how much of the accuracy recovery observed in the main INT4 simulations is retained under native execution, and what is the incremental runtime cost relative to an optimized low-bit baseline? 3.What causes the Gemma-3-4B W4A4 SAMSum drop from 16.73 to 8.17? Can per-example outputs or the validation-error diagnostics in Appendix A.1 help distinguish generation failures, local overcorrection, and changes in activation distributions after repair? Is there a validation signal that does not rely on test outcomes and can identify settings in which repair should not be enabled?
3 comments
Response to reviewer wXch — Rebuttal (Part 2/3)
W2: Table 8(b) channel restoration reaches 7.1351 against dense rank-32 at 7.1465, so the low-rank structure shows no advantage.
You are right, and we think this comparison supports the claim we should have made rather than undermining it. Our thesis is that the activation rounding residual is a recoverable signal on a frozen quantized host; it is not a thesis that low-rank is the uniquely correct way to spend the budget, and the revision no longer reads that way.
Two independently constructed parameterizations — a dense rank-32 factorization, and a channel restoration selected by residual energy times the squared column norm of the frozen effective map — land within 0.01 PPL of each other at matched storage, with four paired comparisons nonsignificant under one Holm family and intervals spanning zero. We read that agreement as evidence of a shared underlying correction signal.
Your comment also exposed an accounting gap that our original control genuinely missed: it matched scalar count but did not charge for the stored channel indices that a deployed channel-restoration branch must ship. We added an actual-byte control that does charge for them, at 105.249947 MiB — 56 bytes below the dense branch budget — and it reproduces the same picture, 7.1366 against 7.1465, with no significant difference after Holm correction. The earlier control overstated the channel variant's budget advantage, and we have corrected it.
The revision therefore states the conclusion as residual-conditioned repair under either parameterization. We claim one limited advantage for the dense form, and it is a deployment property rather than an accuracy one: a fixed-rank factorization has a predictable, index-free footprint, so the update package size is known in advance and does not depend on which channels a given host selects.
W3/Q2: The simulated, compact-storage and native measurement scopes are mixed; can accuracy, prefill, decode and memory be reported together inside one native implementation?
Thank you for this — the mixing was a reporting fault on our part, and we have corrected it. We also declare the boundary explicitly rather than letting the three read as one number: the compact-storage figures describe archived coefficients and changed checkpoint bytes for distribution; the runtime table measures live runtime including workspace and KV cache; the CUDA Graph study is an independent measurement in its own tables and is never mixed into the runtime rows.
One frozen Llama-3-8B MXFP4 artifact, 224 natively packed projections under CUTLASS GEMM plus the same FP32 rank-32 factors at 128 attention sites, no branch fusion, shared Triton frontend fusion applied to host and repair alike:
| Length | Execution | WT2 PPL | Prefill (ms) | Decode (ms/tok) | Peak mem (GiB) |
|---|---|---|---|---|---|
| 128 | Optimized host | 7.871084 | 160.46 | 152.42 | 8.172 |
| 128 | + ResCom | 7.733374 | 176.42 | 161.70 | 8.275 |
| 2048 | Optimized host | 7.871084 | 193.79 | 154.13 | 9.588 |
| 2048 | + ResCom | 7.733374 | 210.52 | 163.29 | 9.690 |
| 4096 | Optimized host | 7.871084 | 324.65 | 148.74 | 11.097 |
| 4096 | + ResCom | 7.733374 | 349.53 | 158.00 | 11.200 |
All 141 complete WikiText-2 test windows — 288,627 scored next-token labels — preserve bitwise-identical full logits across the execution optimization, so the accuracy column is not a kernel artifact. On this artifact the repair costs 7.66% prefill, 6.23% per decode token and 105.25 MiB.
To state the setting unambiguously, the main path here is natively 4-bit, not simulated: E2M1 operands with an E4M3 scale per 16 elements. Simulation appears only where the comparator methods define their own protocol. We do not claim to beat BF16 or a mature serving engine, and the revision says so.
On a separate native NVFP4 artifact, against the same optimized host (identical prefill fusion and packed Q/K/V, differing only in that no branch is attached, all arms in one interleaved rotation), with both branch projections fused, the marginal cost at 4096 tokens is +2.92–3.83% per decode token and +3.34–4.06% end to end for an FP16 branch at 52.625 MiB, and +4.52–4.68% / +4.57–4.69% for an FP32 branch at 105.25 MiB. The two artifacts differ in site count and backend, so we report them separately.
Response to reviewer wXch — Rebuttal (Part 3/3)
W4/Q3: Why does SAMSum drop, can a per-example diagnosis distinguish generation failure from local over-correction or an activation-distribution shift, and is there a validation signal not dependent on test outcomes that could decide when to disable the repair?
Thank you for the three-part question; it is the one we spent the most effort on. We reproduced all 400 original Gemma-3-4B W4A4 outputs exactly before intervening.
Every one of the 88 newly blank outputs is token 251 (carriage return) followed by token 107 (newline). SAMSum's newline stopping rule fires on the second token, and the official first-line extractor then returns nothing. The content is generated; the protocol discards it. This distinguishes the three hypotheses you name: it is not generation failure, since the text exists and is well-formed; it is not a shift toward stopping early, since none of the 88 stops on the first token and the mean first-step stop probability decreases by 0.10586 (exact sign test, = 1.53e-10); and it is not diffuse over-correction, since the symptom is confined to one two-token prefix in 88 of 88 cases. Under EOS-only stopping with full-text extraction — same model, same frozen factors, same prompts, applied identically to both arms — the comparison reverses, with zero blank ResCom responses against one for the host. We report all six stop-rule × extractor cells rather than a selected one:
| Stop rule | Extraction | Host | ResCom | Δ |
|---|---|---|---|---|
| EOS + newline | Official first line (benchmark protocol) | 16.731 | 8.167 | −8.564 |
| EOS + newline | Full text | 16.498 | 7.133 | −9.365 |
| EOS only | Official first line | 15.044 | 8.271 | −6.773 |
| EOS only | Full text | 14.603 | 16.537 | +1.934 |
Each rule alone is insufficient; the interaction needs both. Table 4's official numbers are unchanged.
On a validation signal that does not rely on test outcomes: we evaluate the repair on disjoint WikiText-2 raw-v1 windows with no branch fitting and no threshold tuning. Local recovery — percent error-energy reduction at each repaired site — holds at 43–57%, and the end-to-end NLL difference stays negative with intervals excluding zero at every length up to 7,500 tokens, on the Gemma model where the interaction occurs as well as on Llama-3-8B. This is computable before any benchmark is run.
We should report the honest negative on the stronger form of your question. We tested three probability and deviation signals as a per-example gate for disabling the repair, and their AUCs are 0.31–0.63 — not usable. So we can offer a pre-test health check but not a validated per-example switch, and the revision says that rather than implying we have one.
W5: The cross-pipeline experiments extend the target to weight-side error, so the applicable scope changes.
Correct, and we have written the targets out rather than leaving them implicit. With and frozen host output , the default SERQ target is activation-only:
while the PrefixQuant and FlatQuant rows use their pre-rounding weight reference:
Proposition 1's residual-alignment statement is stated for the activation-only target, and we no longer let it read as covering the cross-pipeline setting; the centered rank-constrained regression and its closed-form solution apply to any fixed target, which is what makes the extension well-defined. Each cross-pipeline row now labels its pipeline, target components, sites, rank and host calibration, so the rows document incremental attachment within each pipeline rather than a cross-host ranking.
On presentation (Presentation 2). We accept this and have a concrete plan rather than a promise to try. The introduction now states the three elements separately — a standard closed-form solver, the rounding-residual target, and the frozen-host attachment — instead of presenting them as one contribution. Table captions are standardized to name the pipeline and the matched quantity ("SERQ W4A4 adaptation, matched 105.25 MiB", "attention-only NVFP4 adaptation"). The regressions and the scope limits move from the appendix into the main text, including the SAMSum protocol finding, the solver negative control, and the gate AUCs above. We would rather the paper be read as a narrower claim with visible limits than as a broad one.
Once again, thank you for your constructive feedback. We hope our response has addressed your concerns.
Response to reviewer wXch — Rebuttal (Part 1/3)
Thank you for your detailed review and for decomposing our claims into separable factors. Your observation on channel restoration also led us to an accounting correction we had missed, which we are grateful for. We address each point below.
W1: With C=WsG, what are the independent contributions of freezing the host, the residual input, centering and the ridge penalty, and is there ARHQ evidence at matched model, calibration and storage?
Thank you for the valuable feedback. We separated all four factors on one frozen host, and we report the outcome including the part that does not favour an elaborate explanation: two of the four account for the gain and two do not.
1. Branch input — rank, sites, target and the relative ridge rule fixed, WikiText-2 perplexity:
The ordering is identical at every rank from 8 to 256, and the rounded-activation input is worse than the unrepaired host throughout. The unrounded activation recovers about 83% of what d recovers at rank 32, with the gap widening monotonically from 0.038 to 0.144 across ranks 8 to 256.
2. Fitting rule — one frozen host, one shared set of FP64 moments, identical storage: the centered ridge fit, the ARHQ-derived weighting, the uncentered fit and the unregularized fit all land within 0.007 PPL of one another. We therefore attribute nothing to centering or regularization. Our original framing left room to read these as contributions, which was not sufficiently rigorous, and the revision states the conclusion directly.
3. Host treatment — the faithful ARHQ arm, same model, same 128 seed-0 windows, same 105.25 MiB budget, actually performing the weight split and GPTQ remainder quantization:
4 of 4 after Holm correction, with 20 of 20 lower perplexities across five independently fitted seeds. We keep the limit explicit: the seed-level Holm-adjusted p is 0.25, because five seeds admit a minimum attainable p of 0.0625.
4. Deployment cost of that difference — the ARHQ-style construction changes 128 of 128 attention projections, ResCom changes 0 of 128 (verified per projection with
torch.equal, archived runs reproducing their one-window NLL to 0.0), and the update package is 48.4–48.9 MiB against 443–452 MiB.So the answer to your question is that the residual input and the frozen-host attachment carry the result, and the solver details do not. We will present it in these four parts in the revision, which the submission did not do.