acceptodds
Under review as a conference paper at ICLR 2027

When Quantization Helps: Corrective Computation in Low-Bit Language Models

Abstract

Low-bit quantization reduces deployment costs by approximating high-precision weights, introducing changes that can improve some predictions. We investigate which activation changes support lower target-task loss than the high-precision reference and whether their useful structure can be shared across inputs. Experiments on Llama (GPTQ, AWQ, NF4) and Qwen (GPTQ) reveal systematic input-level improvements despite lower average performance over the full evaluation set. Improved inputs exhibit stronger loss-reducing contributions per unit activation displacement. Stronger first-order loss reduction can outweigh comparable or larger loss increases beyond the linear approximation. Targeted interventions link these changes to predictions: replacing selected quantized activations with their high-precision values can reverse naturally corrected answers; strengthening favorable changes or suppressing harmful ones repairs errors. We construct useful local adjustments from quantization-induced changes and find contrasting shared structure: a rank-four subspace retains 95.3% of their cross-entropy improvement on HANS, whereas MMLU retains 47.8% even at rank 128. Mechanism-guided activation adaptation raises accuracy above the matched high-precision model by 7.41 percentage points on HANS test data and 2.42 on MMLU development data. Its benefit over ordinary supervised adaptation is larger on HANS, consistent with its stronger shared structure. Together, these results connect the loss-reducing contributions of quantization-induced changes to their effects on predictions and relate their shared structure to the benefits of mechanism-guided adaptation.

Primary area:
interpretability and explainable AI
Keywords:
Low-bit Quantization, Large Language Models, Mechanistic Interpretability, Causal Intervention, Parameter-Efficient Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WiHXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why quantization, despite degrading overall model performance, can improve predictions on some individual examples. The authors attribute these improvements to quantization-induced activation changes that align with directions reducing the target loss, rather than to smaller perturbation magnitude. They further study these changes through activation interventions and shared-subspace analysis. Finally, they train a lightweight activation modifier to predict beneficial adjustments without requiring target-loss information at inference time.

Critical strengths
  1. The paper investigates an interesting phenomenon. Quantization can improve individual predictions despite degrading aggregate performance. It offers an input-level examination of such cases, including an analysis of the direction and functional effect of quantization-induced activation changes.
Critical weaknesses
  1. The analysis in Sections 5–7 is overly fragmented and notation-heavy. A relatively simple central observation is expanded into many overlapping decompositions, metrics, and intervention variants, without clearly stating the distinct insight provided by each. Consolidating the main argument and moving secondary diagnostics to the appendix would substantially improve readability.

  2. The empirical validation is too limited to establish the generality of the reported findings. The study evaluates only two models and a small number of tasks, while most mechanistic and adaptation experiments are conducted on a single 4-bit GPTQ model. More recent representative methods, such as OmniQuant and FlatQuant, are not evaluated, and no evidence is provided for more aggressive 2-bit or 3-bit settings. It therefore remains unclear whether the findings generalize beyond the specific models, tasks, and quantization configuration studied.

Questions for authors
  1. Can the authors provide results on additional models, quantization methods, or bit-widths to demonstrate the generality of the findings?
3 comments
AuthorsOpenReview ↗

Clarifications and Generalization Across Quantizers, Models, and Tasks

Thank you for the suggestions. We clarify Sections 5–7 and report added tasks, quantizers and Qwen analyses.

W1. Organization and interpretation of Sections 5–7

We will retain the sequence and add this roadmap at the start of Section 5:

  • Section 5 — Characterization. What distinguishes improved inputs? We compare displacement magnitude, signed loss contribution and nonlinear effects. Improved inputs show stronger loss-reducing movement per unit displacement, not necessarily smaller nonlinear cost. At the final block, CC is fitted gradient-parallel reduction and DD orthogonal quadratic cost.
  • Section 6 — Functional intervention. Do these changes affect predictions? We restore/amplify activation channels. Same-sign random channels test whether sign and count alone explain reversals; contribution-matched selections test whether the total signed loss contribution explains the effect without requiring the same channel identities.
  • Section 7 — Cross-input sharing. Can useful local adjustments be shared? We fit the original uncentered S=[c1,...,cN,r1,...,rN]S=[c_1,...,c_N,r_1,...,r_N] on source inputs (Eq. 17), then evaluate retained CE reduction after projection (Eq. 22).

We will clarify the reference points: Sections 5–6 compare complete FP/Q executions, including propagated changes, using FP-reference gradients. Section 7 instead fixes the Q gate input and studies local error e=(WQ−WFP)xQe=(W_Q-W_{FP})x_Q, using the target-loss gradient evaluated at the quantized gate output. Shared energy measures the fraction of adjustment energy in the learned subspace (Eq. 21); retained effect compares total CE reductions from projected and full adjustments (Eq. 22), not accuracy. Full and projected changes use the same scale (Eq. 19). Section 7 uses the correct answer to compute diagnostic gradients; the Section 8 modifier requires neither the answer nor its gradient at inference. We will consolidate notation, specify denominators in captions, and move fitting details to the appendix while retaining the definitions.

W2 / Q1. Generality across quantizers, precision, models and tasks

Quantization configurations. We evaluated (i) Llama-3-8B GPTQ W3, an actual packed 3-bit checkpoint (group128, activation-order enabled); (ii) Llama OmniQuant W4 with learned weight clipping (LWC, 20 epochs, 128 FineWeb calibration documents); and (iii) Qwen2.5-7B-Instruct GPTQ4 with expanded mechanism tests.

We evaluated five-shot MMLU questions and FineWeb documents, followed by Top-128 restoration, same-sign random and contribution-matched controls:

ConfigurationFP / Q accuracyNatural W2CTop-128 reversalRandomContribution matched
Llama GPTQ W365.40 / 41.36%1,19039.06%3.91%36.20%
Llama OmniQuant W465.40 / 63.45%66680.08%19.66%78.65%
Qwen GPTQ W474.33 / 73.09%41274.60%4.76%74.51%

On FineWeb, W3/OmniQuant/Qwen improved 7/48/418 documents; restoration reversed 5/7, 42/48, 361/418 improvements, respectively. W3 has few improved documents.

Additional tasks. With Llama GPTQ4, we evaluated BoolQand ARC-Challenge. Despite lower overall Q accuracy, quantization corrected 164/49 predictions on BoolQ/ARC (with 237/94 newly wrong answers). Top-128 restoration reversed 157/164 (95.73%) / 44/49 (89.80%), versus 22.36%/15.65% for random and 95.12%/86.39% for contribution-matched controls.

Sharing across models and tasks. We also completed the original Eq. (17) rank-eight SS-basis diagnostic on Qwen BoolQ/ARC, in addition to HANS/MMLU. Retained fractions of the full CE reduction are:

ModelHANSMMLUBoolQARC
Llama96.06%24.77%69.09%21.06%
Qwen88.21%18.89%53.57%52.95%

These are gold-gradient diagnostics, not modifier accuracies. Sharing depends on task and model: Qwen MMLU and ARC are both four-choice, yet retain 18.89% and 52.95%.

Thank you for the detailed rebuttal and the additional experiments. The new results significantly strengthen the evidence and improve the generality of the findings. I also appreciate the clarification of section 5–7. The rebuttal addresses my main concerns. I will increase my rating to 3.

Thank you for your recognition of our work!

Reviewer_gti7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores the inputs where models that are quantized to 4-bits outperform their original non-quantized version. They first 250k FineWeb scan documents and they find that improved examples constantly show up under GPTQ, AWQ, and NF4 on Llama and GPTQ for Qwen, although the loss goes up overall. Furthermore, they find that improved inputs tend to have a larger loss-reducing contribution. Moreover, they explore whether loss-reducing changes affect predictions and show that the useful local changes tend to be concentrated in a rank-eight shared component on HANS rather than on MMLU. Lastly, they build an activation modifier that beats a baseline by 1.28 points on HANS.

Critical strengths
  • The paper shows evidence that the improvement is structured, the difference across all rows in Table 1 is consistent, as well as Figure 2 that shows contribution matched sets can recover most of the reversal effect. Section 5.1 also shows that improvement is structured across quantizer with a 41x enrichment when looking at GPTQ and AWQ strongly hinting toward the fact that the improvement is structured rather than random.
  • The design is thoughtful and the authors evaluate when quantization helps across several dimensions. They are also open, for example showing that the CI touches 0 for MMLU in subsection 8.2.
Critical weaknesses
  • No results for non-quantized null controls, for example in Section 5.1 there are no results for noise added to the layers as a baseline to see if the structure actually comes from quantization. Furthermore, editing channels that are directly tied to the right answer is more likely to lead to answers flipping regardless if quantization was the reason behind it, potentially invalidating some of the results presented in the paper.
  • The gap in shared low-rank structure between HANS and MMLU might stem from the fact HANS has a random choice accuracy of 50% which is close to what the models are scoring whereas MMLU has a random sits at 25%. This could mean that on HANS the gradients of the right answer point in just a few directions toward one label or the other while for MMLU, with four options per task, they point in more directions.
Questions for authors
  • Can the authors extend the experiments to one or two more datasets beyond HANS and MMLU?
  • Can the authors run experiments with random weight noise as control to check whether the insights actually come from quantization?
  • Can the authors run Table 6 with at least 3 random seeds to be able to differentiate from random noise?
4 comments
AuthorsOpenReview ↗

Noise Controls, Cross-Task Analysis, and Multi-Seed Results

Thank you for the review. We added noise, answer-choice, task and seed controls.

W1 / Q2. Quantization versus generic weight perturbations

Quantization error can be viewed as a weight perturbation. Noise Perturbation Fine-tuning (NPFT), for example, injects weight noise during fine-tuning to improve robustness to subsequent quantization. We test whether error placement matters and whether generic noise supports the same interventions.

Controls. For 224 Llama-3-8B GPTQ-Int4 matrices, E=WQ−WFPE=W_Q-W_{FP} uses dequantized weights. Controls: GmatG_{mat}, Gaussian noise matched to each matrix's error Frobenius norm; ProwP_{row}, actual errors shuffled within output rows (across groups allowed); GgroupG_{group}, Gaussian matched to error mean/SD per row/group; UgroupU_{group}, same with uniform samples; and PgroupP_{group}, actual errors shuffled only within each row/group. With group_size=128 (up to 128 input weights per row/group), g_idx resolves reordered groups. PgroupP_{group} preserves each group's error values before FP16 writeback, changing only positions.

Mean NLL increase over FP on five-shot MMLU questions: 0.0500 (Q) versus 0.1474 (PgroupP_{group}) (paired difference +0.0974, 95% CI [0.0758,0.1190]). On FineWeb documents: 0.0726/0.2091 (difference +0.1365, CI [0.1354,0.1375]). Error placement matters beyond group distributions.

Intervention. Gaussian/row-permuted noise at scale 0.5 improved 0.684%/0.833% FineWeb documents (lower NLL), versus 0.247% for Q; its average degradation was also lower at this scale. Thus improvement counts depend on perturbation strength. We also applied our original target-gradient intervention to generic noise. For naturally wrong-to-correct MMLU questions selected separately for each model (FP wrong, perturbed model correct), we restored the 128 activation channels with the largest estimated loss-reducing contributions to their FP reference values. Consequently, 58.18% of Q, 50.91% of Gaussian-noise, and 49.48% of row-permutation predictions became incorrect again. Hence the intervention identifies functional effects of measured quantization-induced changes, but similar effects also occur under generic perturbations. We will clarify this distinction.

W2. Does answer count explain the HANS–MMLU sharing gap?

HANS has two choices; MMLU has four. We test if two-candidate MMLU source gradients recover HANS-like sharing under the paper's original method.

We keep the four-choice MMLU prompts, Q activations and local error e=(WQ−WFP)xQe=(W_Q-W_{FP})x_Q unchanged. Source gradients: g4g_4 uses four-choice CE; g2g_2 uses CE over gold and an ID-fixed distractor. For k∈{4,2}k\in\{4,2\}, we construct ck=e⊙1[gk⊙e<0]c_k=e\odot\mathbf{1}[g_k\odot e<0] and rk=−e⊙1[gk⊙e>0]r_k=-e\odot\mathbf{1}[g_k\odot e>0]. Using the original Sec. 7.2, Eq. (17), we form Sk=[ck,1,...,ck,N,rk,1,...,rk,N]S_k=[c_{k,1},...,c_{k,N},r_{k,1},...,r_{k,N}] and take its top eight left singular vectors.

Held-out evaluation remains four-choice: both bases project the same adjustment d4=c4+r4d_4=c_4+r_4, use the same scale, and are scored by the original four-choice CE. Only the source gradient changes.

Rank-8 basis (Eq. 17)Retained CE reduction
HANS, native two-choice96.06%
MMLU, four-choice gradient24.77%
MMLU, two-candidate source gradient24.74%

The two-candidate result has a paired-input 95% CI of [23.00%, 26.44%]. This controls the number of candidates used to construct the source gradients, without changing the original MMLU task. The two-candidate basis does not recover HANS-like sharing.

Q1. Additional datasets and cross-model evidence

On BoolQ and ARC-Challenge , Top-128 restoration reversed 95.73% and 89.80% natural corrections, versus 22.36%/15.65% for same-sign random controls and 95.12%/86.39% for contribution-matched controls.

Using the original Eq. (17) basis, rank-8 retained CE effects are:

ModelHANSMMLUBoolQARC
Llama96.06%24.77%69.09%21.06%
Qwen88.21%18.89%53.57%52.95%

Qwen MMLU and ARC both have four choices yet differ markedly. These are diagnostic effects, not modifier accuracy.

Q3. Training seeds

Paired BASE/C/D use frozen Q4, matched architecture/data order and 800 steps, varying initialization. C/D minus BASE accuracy (percentage points):

  • HANS (seeds 17/59/88/99/43): +1.28/+7.24/+4.14/+6.10/−1.39.
  • MMLU (same five as HANS): +0.06/+0.20/+0.17/+0.07/+0.06, mean +0.112.
  • BoolQ (same five as HANS): +0.10/+0.10/−0.12/+0.15/−0.17, mean +0.012.

Overall, C/D consistently yields small positive gains across the five reported MMLU runs and achieves larger improvements in four of the five displayed HANS runs. On BoolQ, C/D provides no consistent additional benefit over BASE.

AuthorsOpenReview ↗

Response for reviewer gti7: Algorithmic Clarification for W2

We provide further details of the algorithm used in the two-candidate source-gradient control. The pseudocode is given below.

Algorithm 1: Two-Candidate Source-Gradient Control

Input:

  • Frozen Llama-3-8B FP and GPTQ-Int4 models.
  • Source set: MMLU questions, with two answer-order views per question.
  • Evaluation set: 512 fixed MMLU questions, with two answer-order views per question.
  • Quantized gate: layer 11, pre-SiLU, final input position.
  • Shared rank: r = 8.

Output:

  • Two-candidate shared basis U2U_2.
  • Retained four-choice CE reduction R2R_2.

Step 1: Construct the two-candidate source basis

For each source question and its two answer-order views:

  1. Keep the original four-choice prompt. Run Q and capture the gate input xQx_Q, gate output uQu_Q, and four answer logits.

  2. Select one incorrect option deterministically using SHA256(question ID, seed = 17). Keep the same semantic distractor across both views.

  3. Compute the gradient g2g_2 using cross-entropy over only the gold answer and the selected distractor, with the gold answer as the target.

  4. Compute the local quantization error:

    e=(WQ−WFP)xQe = (W_Q - W_FP) x_Q

  5. Construct the useful components following the original paper:

    c=e⊙1[g2⊙e<0]c = e ⊙ 1[g_2 ⊙ e < 0]

    r=−e⊙1[g2⊙e>0]r = -e ⊙ 1[g_2 ⊙ e > 0]

  6. Form the original uncentered matrix:

    S2=[c1,...,cN,r1,...,rN]S_2 = [c_1, ..., c_N, r_1, ..., r_N]

    Do not center or normalize individual columns. Extract the top eight left singular vectors of S2S_2 to obtain U2U_2.

Step 2: Evaluate on the original four-choice task

For each of the 512 held-out questions and its two answer-order views:

  1. Load the frozen four-choice Q state, full adjustment d4=c4+r4d_4 = c_4 + r_4, and baseline four-choice CE.

  2. Determine the intervention scale using the full adjustment:

    alpha=min(1,0.05∗∣∣uQ∣∣/(∣∣d4∣∣+epsilon))alpha = min(1, 0.05 * ||u_Q|| / (||d_4|| + epsilon))

  3. Project the same four-choice adjustment onto the two-candidate basis:

    dshared=U2U2Td4d_{shared} = U_2 U_2^T d_4

  4. Evaluate both the full and projected adjustments using the original four-choice CE, and accumulate their loss reductions.

Step 3: Calculate retained effect (Eq. 22)

R2=100∗sum(Lshared)/sum(Lfull)R_2 = 100 * sum(L_{shared}) / sum(L_{full})

Step 4: Compare with U4U_4, the original basis constructed using four-candidate source gradients, under exactly the same evaluation protocol.

Key clarification. The two-candidate setting changes only the candidate set used to construct source gradients. The original four-choice prompts, quantized model states, evaluation adjustments, intervention scales, and four-choice evaluation objective remain unchanged. Thus, the comparison isolates the effect of reducing the candidate set during source-basis construction, rather than changing the MMLU task or evaluation objective.

Reviewer_gti7OpenReview ↗

Reviewer Response

Thank you for the detailed response, I've increased my rating by one point.

Thank you so much for your positive feedback on our work! We also really appreciate your helpful suggestions and comments!

Reviewer_8GjjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores why quantization improves some individual predictions despite the overall degraded performance. The authors seek to find relationship between quantization-induced activation changes and target loss

Critical strengths
  1. the research question is interesting. the first paper i'm aware of to study seperate predictions caused by quantization.

  2. the comparison is fair, the arthitecture, parameter, data protocol are aligned.

Critical weaknesses
  1. The specific affect of quantization is not sufficiently established. The reason why quantization generates beneficial changes are not well explained. Do quantization-induced activation changes differ meaningfully form generic perturbations?

  2. The activation modifier is also applicable to a full-precision backbone. We can also use full-preicsion model to obtain gradient information. Exceeding an unadapted full-precision model after task-supervised training therefore does not isolate a benefit attributable to quantization.

Questions for authors

Please see the weakness above

1 comment
AuthorsOpenReview ↗

Generic Perturbation Controls and Matched FP Adaptation

Thank you for the review. We added controls for the effects of generic perturbations and matched FP adaptation.

W1. Quantization versus generic weight perturbations

Quantization error can be viewed as a weight perturbation. Noise Perturbation Fine-tuning (NPFT), for example, injects weight noise during fine-tuning to improve robustness to subsequent quantization. We ask whether the values and placement of quantization errors matter for input-level changes.

Experimental Construction. We use Llama-3-8B FP/GPTQ-Int4 and perturb only its 224 quantized linear matrices, with E=WQ−WFPE=W_Q-W_{FP} from dequantized weights. We compare:

  1. GmatG_{mat}: draw Gaussian noise for each matrix and match the Frobenius norm of EE.
  2. ProwP_{row}: permute actual error values within each output row, allowing movement across quantization groups.
  3. GgroupG_{group}: for each output row and quantization group, center/scale Gaussian samples to match the actual errors' sample mean and standard deviation.
  4. UgroupU_{group}: match the same group statistics, starting from uniform rather than Gaussian samples.
  5. PgroupP_{group}: permute actual error values only within the same output row and quantization group, never across groups, rows or layers.

GPTQ uses group_size=128 up to 128 input-channel weights per row/group. We recover the true group assignments from checkpoint g_idx because columns may be reordered. Unlike Ggroup/UgroupG_{group}/U_{group}, PgroupP_{group} creates no new error values; each group's entire error-value set is preserved before FP16 writeback, while its assignment to weights changes.

Results. Mean NLL increase relative to FP on five-shot MMLU questions (answer NLL) and FineWeb documents (teacher-forced next-token NLL):

WeightsMMLUFineWeb
Q0.05000.0726
GmatG_{mat} / ProwP_{row}0.1396 / 0.13940.1919 / 0.1618
GgroupG_{group} / UgroupU_{group}0.1337 / 0.14610.2037 / 0.2061
PgroupP_{group}0.14740.2091

Although PgroupP_{group} preserves the actual errors in each group, changing their positions increases NLL relative to Q by 0.0974 on MMLU (paired 95% CI [0.0758, 0.1190]) and 0.1365 on FineWeb ([0.1354, 0.1375]). Error placement matters beyond magnitude or group-wise distributions. Lower degradation alone does not establish a unique corrective mechanism.

Generic noise also helps some inputs. Gaussian/row-permuted noise at scale 0.5 improved 0.684%/0.833% FineWeb documents (lower NLL), versus 0.247% for Q; its average degradation was also lower at this scale. Thus improvement counts depend on perturbation strength. We also applied our original target-gradient intervention to generic noise. For naturally wrong-to-correct MMLU questions selected separately for each model (FP wrong, perturbed model correct), we restored the 128 activation channels with the largest estimated loss-reducing contributions to their FP reference values. Consequently, 58.18% of Q, 50.91% of Gaussian-noise, and 49.48% of row-permutation predictions became incorrect again. Hence the intervention identifies functional effects of measured quantization-induced changes, but similar effects also occur under generic perturbations. We will clarify this distinction.

W2. Matched FP adaptation

We agree that adapted Q4 outperforming unadapted FP does not establish a quantization-specific gain. Our goal is a frozen low-bit backbone; adapted FP is an important control.

We trained FP/Q4×BASE/C/DFP/Q4 × BASE/C/D with both backbones frozen. All use the same FP32 modifier at layer-11 pre-SiLU gate (final input position; 604,192 parameters), paired initialization/data order and 800 AdamW steps (lr 3e-4, 40 warmup steps, gradient clip 1; 16 questions × two answer orders per update). BASE includes answer CE, additional-damage and adjustment-magnitude penalties. C/D adds LCD=LBASE−0.1(C−Cb)+0.1(D−Db)L_{CD}=L_{BASE}-0.1(C-C_b)+0.1(D-D_b), with fixed unmodified-backbone references. As defined in Sec. 5.3, CC measures fitted gradient-parallel loss-reducing movement; DD is the fitted orthogonal quadratic cost. Both can also be defined for FP modifier-induced changes.

HANS

BackboneOriginal+BASE+C/D
FP50.27%66.26%69.41%
Q450.00%56.40%57.68%

C/D improves over BASE by 3.15 pp on FP and 1.28 pp on Q4. Its effectiveness is therefore not restricted to quantized models. We will distinguish supervision from the C/D increment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Reviewer-Specific Responses (Part 1/2)

Summary of Revisions and Reviewer-Specific Responses

We sincerely thank all three reviewers for their thoughtful and constructive feedback. Their comments helped us clarify the scope of our contribution, strengthen the experimental evidence, and improve the organization of the manuscript.

The main concerns raised by the reviewers were: (1) whether the observed corrective effects are specific to quantization or can also arise under generic perturbations; (2) whether the difference in shared low-rank structure between HANS and MMLU can be explained by their different numbers of answer candidates; (3) whether the adaptation results demonstrate a quantization-specific advantage, given that the activation modifier can also be applied to a full-precision backbone, and whether the results are robust across random initializations; and (4) the clarity of the analysis and the generality of the findings across models, quantization methods, and tasks.

We have addressed these concerns through additional controlled experiments, cross-model and cross-task evaluations, multi-seed experiments, and revisions to the manuscript.

What changed, why, and where

All locations below refer to the revised 23-page manuscript.

Reviewer concern / UpdateWhat we changed and whyWhere to find it
8Gjj, gti7: Quantization versus generic perturbationsAdded five controlled weight-perturbation settings (Gaussian, row permutation, group Gaussian, group Uniform, and group permutation) to examine the roles of error magnitude, distribution, and placement.Appendix E.3–E.4, pp. 21–22; Table 13, p. 22
8Gjj, gti7: Whether corrective intervention effects are specific to quantizationEvaluated input-level improvements and target-gradient-based activation restoration under generic weight perturbations to determine whether similar functional effects also arise beyond quantization.Appendix E.5, pp. 22–23
8Gjj: Applicability of mechanism-guided adaptation to full-precision modelsAdded a matched adaptation experiment on a frozen full-precision backbone, comparing BASE and C/D under the same modifier architecture and training protocol as the Q4 setting.Appendix D.4, pp. 19–20; Table 9, p. 20
gti7: Whether answer-choice count explains the HANS–MMLU sharing gapAdded a two-candidate MMLU source-gradient control while keeping the original four-choice evaluation unchanged.Appendix C.3, p. 18; Table 7, p. 18
gti7: Additional tasks and modelsExtended activation restoration to BoolQ and ARC-Challenge, and rank-eight sharing analysis to Llama and Qwen across HANS, MMLU, BoolQ, and ARC-Challenge.Section 6.2, p. 6; Section 7.3, p. 9; Appendix C.2, pp. 17–18; Table 6, p. 18; Table 12, p. 21
gti7: Robustness across random initializationsAdded five paired initialization seeds for BASE/C/D adaptation on HANS, MMLU, and BoolQ, reporting per-seed accuracy differences and variability.Section 8.2, p. 10; Appendix D.5, p. 20; Table 10, p. 20
WiHX: Generality across quantizers, bit-widths, models, and tasksAdded Llama GPTQ W3, Llama OmniQuant W4, and Qwen GPTQ W4 experiments, together with additional task evaluations and restoration controls.Sections 5.1 and 6.2, pp. 4–6; Appendix E.1–E.2, pp. 20–21; Tables 11–12
WiHX: Fragmented analysis and notation-heavy presentationAdded an analysis roadmap and clarified the distinct roles of characterization, intervention, sharing, and adaptation. We also clarified the reference points used in Sections 5–6 and Section 7, the sharing metrics, and the interpretation of C and D as directional diagnostics rather than a complete decomposition of the total nonlinear loss change.Section 4, pp. 3–4; Section 5.3, p. 5; Sections 5–8, pp. 4–10
AuthorsOpenReview ↗

Additional Experimental Evidence and Overall Contributions ( (Part 2/2)

Additional Experimental Evidence and Overall Contributions

What these updates establish

1. Quantization error placement matters, although corrective effects are not unique to quantization.

The group-permutation control preserves the actual error values within each quantization group while changing their assignments to weights. Relative to GPTQ, this increases mean NLL by 0.0974 on MMLU (paired 95% CI [0.0758, 0.1190]) and 0.1365 on FineWeb ([0.1354, 0.1375]). Thus, error placement matters beyond error magnitude and group-wise distributions.

Generic perturbations also produce input-level improvements. Target-gradient-based restoration reverses 58.18% of GPTQ, 50.91% of Gaussian-noise, and 49.48% of row-permutation natural MMLU corrections. Our contribution is therefore to characterize the functional effects of quantization-induced changes, not to claim that corrective effects are exclusive to quantization.

2. The HANS–MMLU sharing gap cannot be explained solely by the number of answer candidates.

Reducing MMLU source gradients from four candidates to two yields 24.74% retained cross-entropy reduction at rank eight, compared with 24.77% using four-choice gradients. Both remain below HANS (96.06%), while the original four-choice MMLU evaluation remains unchanged.

Cross-task and cross-model results further show that shared structure varies with both task and model:

ModelHANSMMLUBoolQARC-Challenge
Llama-3-8B96.06%24.77%69.09%21.06%
Qwen2.5-7B88.21%18.89%53.57%52.95%

These values measure retained cross-entropy reduction from diagnostic adjustments, not prediction accuracy. Qwen's MMLU and ARC-Challenge results differ substantially despite both being four-choice tasks.

3. The adaptation results clarify the applicability and robustness of mechanism guidance.

We added matched full-precision adaptation to examine whether mechanism guidance applies beyond quantized backbones. On HANS, C/D improves over BASE by 3.15 percentage points on FP and 1.28 points on Q4. Thus, the objective is not restricted to quantized models; gains over an unmodified FP model are not evidence of a quantization-specific advantage.

Across five paired initialization seeds on the frozen Q4 backbone, average C/D-minus-BASE gains are:

  • HANS: +3.47 ± 3.54 percentage points (positive in four of five seeds).
  • MMLU: +0.112 ± 0.068 percentage points.
  • BoolQ: +0.012 ± 0.146 percentage points.

HANS shows larger but variable gains; MMLU shows consistently small gains; and BoolQ shows no consistent additional benefit. These results characterize the variability of mechanism-guided adaptation.

4. The functional observations extend beyond the original experimental settings.

Under additional quantization configurations, Top-128 restoration reverses 39.06% of natural MMLU corrections for GPTQ W3 and 80.08% for OmniQuant W4, versus 3.91% and 19.66% for same-sign random controls.

On BoolQ and ARC-Challenge, reversal rates are 95.73% and 89.80%, versus 22.36% and 15.65% for random controls. Contribution-matched controls recover most Top-128 effects, supporting the relationship between accumulated signed loss contribution and intervention outcomes.

These experiments broaden the evidence across quantization configurations, models, and tasks, while full adaptation analysis remains focused on matched GPTQ W4.

Overall Contribution and Scope

The revised manuscript separates four questions:

  • Section 5 — Characterization: What distinguishes beneficial quantization-induced activation changes?
  • Section 6 — Functional intervention: Do these changes affect model predictions?
  • Section 7 — Cross-input sharing: When can useful local adjustments be shared across inputs?
  • Section 8 — Mechanism-guided adaptation: Can loss-geometry guidance improve lightweight, input-conditioned activation adaptation?

Together, the experiments support our central conclusion: the consequences of quantization-induced changes depend not only on their magnitude, but also on their relationship to target-loss geometry and the extent to which useful adjustments can be shared across inputs.

We also clarify the scope and limitations: corrective effects are not unique to quantization, adaptation gains vary across tasks and initializations, and cross-input sharing is task- and model-dependent.

We sincerely thank the reviewers and the Area Chair for their constructive feedback, which helped strengthen the empirical evidence and improve the clarity and precision of the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.