acceptodds
Under review as a conference paper at ICLR 2027

The Butterfly Effect in Large Language Model Quantization

Abstract

Quantization has been widely used to reduce the memory footprint and inference latency of large language models (LLMs). Low-bit quantizers have typically been evaluated using average metrics such as perplexity and task accuracy. Nevertheless, these average metrics do not reveal failures on individual inputs, such as correct answers becoming wrong after quantization. Focusing on tail reliability, we propose a unified source-to-decision analysis, which measures rare output failures and traces each input's quantization errors from individual modules through downstream amplification to changes in output distributions, token decisions, and autoregressive generation. Our analysis reveals the quantization butterfly effect: High-gain computation paths amplify small rounding errors on particular inputs into large output shifts and token changes. These amplified errors form an upper tail hidden by average metrics, and an early token change redirects the autoregressive continuation. Our experiments demonstrate the quantization butterfly effect on Qwen2.5-3B, where 4-bit weight-only GPTQ increases mean perplexity by only 7.3%, yet the 99th-percentile (p99) per-token KL divergence is 8.9 times the mean, 14.8% of top-1 predictions flip, and almost every greedy continuation leaves the full-precision trajectory. The same pattern recurs across model families and zero-shot, reasoning, code, and instruction-following tasks. Similar average accuracy scores do not show how often correct predictions become wrong. Our perturbation–propagation theory factorizes each module's impact into local error and downstream gain along the error's direction. We prove non-asymptotic KL bounds, derive an exact propagation-aware decision radius and a sharp multiclass KL barrier for top-1 changes, and establish guarantees for preserving greedy trajectories; finite-perturbation bounds extend Fisher-based control to every quantile and CVaR. Our theory-guided diagnostic score identifies intervention targets for causal validation. Restoring only the four highest-scoring layers to full precision while keeping all other layers at 3 bits reduces the p99 divergence gap by up to 83%. To improve tail reliability, we introduce two complementary mitigation methods: Propagation-Aware Precision Allocation (PAPA) protects high-impact modules at a fixed bit budget, while Butterfly-Aware Reliability Tuning (BART) learns a lightweight correction at a fixed bit-width. By updating under 2% of parameters, BART raises 3-bit GSM8K accuracy from 0.030 to 0.350. Tail reliability must therefore be measured and reported alongside average performance, and optimized as a quantization objective.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM quantization, butterfly effect, tail reliability, error propagation, decision robustness, mixed precision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xNKmOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies example-level reliability under low-bit quantization of large language models, emphasizing that average perplexity and accuracy can conceal upper-tail output divergence, harmful prediction flips, and branching autoregressive trajectories. The authors introduce a perturbation–propagation framework connecting module-level quantization errors to downstream gain, and derive non-asymptotic KL bounds, decision radii within a linearized propagation model, an exact KL barrier for losing the reference top-1 decision, and conditions for preserving greedy trajectories. Experiments span multiple model families, weight and KV-cache quantization, and downstream tasks, with numerical-noise controls and layer-restoration interventions. The paper also proposes PAPA, which allocates precision using single-module output KL on a hard subset, and BART, which improves fixed-bit models through low-rank corrections, learnable clipping, and output distillation. The results support supplementing average quantization metrics with example-level and tail measurements and demonstrate effective mitigation in selected settings. However, the realized directional-propagation mechanism and the independent contribution of the tail-specific design remain insufficiently established.

Critical strengths

S1. The reliability evaluation is broad and includes evidence of actual task regressions. The paper goes beyond changes in token distributions and generated text by reporting harmful flips on zero-shot tasks, GSM8K, code execution, and instruction following. Tables 19 and 21 clearly distinguish changes in average accuracy from example-level regressions, while Table 14 extends the analysis to deployment-relevant tasks. The FP numerical-perturbation controls in Table 6 further help distinguish quantization effects from ordinary numerical sensitivity. These results give the reliability problem practical significance beyond disagreement with a reference model. S2. The theoretical connection between output distributions and discrete decisions is clear. Theorem 2 gives the exact minimum forward KL required for the reference model to lose its unique top-1 decision, correctly distinguishing preservation below the barrier from the absence of a guaranteed flip above it. Theorem 4 and Corollaries 5–6 further connect linearized source perturbations, decision margins, and greedy trajectories. The derivations in Appendix F provide concrete analytical tools linking continuous distributional discrepancies to discrete behavior and constitute a valuable aspect of the paper. S3. The interventions demonstrate practical value, and the paper reports relevant limitations. Table 1 shows that restoring four highly ranked layers in Qwen2.5-3B RTN-W3 recovers 83.1% of the p99 KL gap. Table 11 shows that BART increases GSM8K accuracy from 0.030 to 0.350 at the same 3-bit precision. The authors also report negative results, including competitive random layer selection on Pythia, PAPA’s lack of advantage on RTN, and degradation from BART in some W4 settings. These experiments demonstrate practical value for the proposed diagnostics and interventions while providing useful evidence about their scope.

Critical weaknesses

W1. Validation of realized directional error propagation remains indirect, weakening the claimed end-to-end mechanistic account. The central quantity in Eq. (1) is Gℓ,mδℓ,mG_{\ell,m}\delta_{\ell,m}, propagated along the realized quantization-error direction for each input. However, Appendix H.4 explains that the layer-restoration score uses gain averaged over random directions at the block input. It neither preserves the realized error direction nor matches the injection site of the block-output residual. The Fisher checks in Tables 7–8 use the measured true logit difference, validating an output-level approximation rather than the source-level propagation prediction and its remainder. Table 1 supports the intervention utility of the ranking score, but does not by itself establish the proposed directional explanation. Consequently, the paper’s central empirical connection from realized error sources to decisions remains insufficiently direct, particularly for tail examples and harmful flips. W2. The independent contribution of the tail-specific design over general output-fidelity optimization is not sufficiently isolated. Tables 16–18 compare several local and curvature-based scores, but omit the most direct PAPA control: single-module mean output KL over all calibration inputs, with every other setting unchanged. The current results therefore do not separate the benefit of output sensitivity from that of tail conditioning. For BART, the authors explicitly acknowledge that most gains come from the mean-KD core with λ=0\lambda=0. The additional CVaR term produces small and inconsistent benefits in Table 9; for example, on Qwen2.5-3B W3, CVaR5_5 changes from 3.31 with BART to 3.36 with BARTtail_{\mathrm{tail}}. Given existing work on output-aware quantization, the paper needs to identify more clearly what the tail-specific design adds beyond general output-fidelity optimization. W3. The connection between distributional-tail improvement and recovery of task reliability is not systematically established. The harmful-flip analyses are valuable, but the main layer-restoration and PAPA intervention results are evaluated using KL metrics. For BART, the distributional and task metrics in Table 9 come from different datasets, and their changes sometimes disagree. For example, Mistral-7B W4 shows substantial task-accuracy recovery while its WikiText-2 p99 increases from 0.40 to 0.52; BART on Qwen2.5-3B W4 increases both p99 and the harmful-flip rate. These results do not invalidate the methods’ utility, but they show that improving the KL tail cannot simply be equated with reducing task failures. Supporting the complete reliability chain requires examining risk predictions, intervention-induced distributional changes, and recovery from harmful flips on the same labeled examples.

Questions for authors

1.Could you evaluate the propagation prediction along actual module quantization-error directions in one representative W4 setting, reporting its agreement with the true logit/KL changes and the magnitude of the remainder separately for ordinary and tail examples? Under the same layer-restoration budget, how does this realized-direction score compare with the current random-direction proxy measured at block inputs? This would directly affect my assessment of the mechanistic contribution. 2.With the same proxy quantizer, calibration sweep, and achieved bit budget, how does PAPA compare with single-module mean KL computed over all calibration inputs? In light of the inconsistent gains of BARTtail_{\mathrm{tail}} over mean KD, which results can be attributed specifically to the tail-oriented design rather than general output-KL optimization? 3.Are OMNIQ’s correction-magnitude projection, clipping floor, and weight decay also applied to BART? If not, does BART retain its advantage in Table 10 when both objectives use the same constraints? Please specify which training settings are identical and which are exclusive to the local-reconstruction control.

4 comments

Thank you for specifying the decisive experiments. The added tests connect actual internal errors → predicted decision risk → recovery of correct answers. Preserving the actual error direction predicts output changes and token flips; selecting high-risk calibration inputs preserves more correct words at identical bits; and optimizing output KL improves tail and task reliability under matched training constraints. Together, these controlled experiments establish how the paper's propagation and decision framework guides effective quantization repair.

W1 / Q1. Actual module errors predict output changes and decisions

We directly predict the output effect of an actual internal quantization error. For each module, δm=(WmQ−Wm)hm\delta_m=(W_m^Q-W_m)h_m measures its output error on the FP input. We propagate this error from the module output, preserving its direction and injection site, using the remaining network's derivative: vm=Gmδmv_m=G_m\delta_m. Adding it to FP logits zz predicts softmax⁡(z+vm)\operatorname{softmax}(z+v_m). An independent single-module quantization run measures Δzm\Delta z_m and the remainder rm=Δzm−vmr_m=\Delta z_m-v_m. Thus the internal error supplies the prediction and the quantized run tests it. This directly complements the submitted output-level Fisher checks in Tables 7–8 and restoration evidence in Table 1.

The Qwen2.5-3B GPTQ-W4 test covers q/k/v/o/gate/up/down projections at layers 0, 1, 12, 24, and 35: 35 modules × 32 held-out 128-token windows × 63 positions = 70,560 cases. Computation uses FP32/eager attention, with constant logit shifts removed. The tail is the largest 5% of actual single-source KL; relative RMS error is RMS remainder divided by RMS actual change.

Evaluation subsetCasesKL rank correlationLogit cosineRelative RMS errorPredicted / actual mean KL
Ordinary 95%67,0320.9980.9930.2781.004
Upper 5%3,5280.9600.9860.2440.973

The predictor finds 516 of 538 actual flips, with 30 false positives: 94.5% precision / 95.9% recall. Tail precision/recall is 93.3%/94.3%. It predicts the exact new winner in 98 of 104 harmful source flips.

At fixed norm and injection site, three random directions give median within-module tail correlation 0.536, versus 0.976 for actual directions. The predicted top 5% retrieves 96.3% of the actual tail [95% CI 95.0, 96.8], versus 27.6% for calibrated local energy. Whole-window resampling preserves token/module dependence. Tail correlations remain 0.972/0.981 at 512/2,048-token contexts.

The all-block composition test measures how source effects combine in a complete quantized network: summed FP-linear predictions have overall/tail correlations 0.910/0.537 and relative RMS remainders 0.735/0.873. These remainders quantify departures from additive source predictions when errors coexist, as retained explicitly in the theory. PAPA uses measured finite single-module output effects for scoring, and its allocation gains are evaluated on complete quantized models with interactions present. The experiments therefore test source prediction, composition error, and full-model repair separately.

We also add a controlled comparison of the submitted proxy and the realized-direction score. All restoration rankings use the same eight disjoint C4 windows and GPTQ-W4 block residuals. The original proxy averages three random block-input directions; the realized score propagates the block-output residual. Each restores four blocks to BF16 at identical storage. Evaluation uses 128 held-out WikiText-2-256 windows.

Selection ruleToken p99 KL ↓Token CVaR5 ↓
Uniform GPTQ-W40.8680.687
Original random-input gain proxy0.7290.592
Local residual magnitude0.6770.551
Realized-direction propagation0.6700.550

At identical restoration cost, actual-direction selection improves both tail metrics over the submitted proxy: 0.059 p99 KL [paired-window 95% CI 0.037, 0.089] and 0.042 CVaR5 [0.028, 0.058]. It also gives the lowest point estimates in the table; the p99 difference from local magnitude is 0.007 [−0.019, 0.034]. On the accompanying 512 labeled W4 prefixes, all three rules reduce harm 30→29. These experiments connect the fixed-norm direction ablation to a measurable restoration benefit. The paired W3 study below extends this test to recovery of correct answers, where realized-direction selection leaves 118 harmful decisions versus 133 for both the original proxy and local magnitude at the same budget.

W2 / Q2. Tail conditioning has an isolated, measurable effect

The added ablation isolates the contribution of selecting high-risk calibration inputs. It extends the submitted local-error and curvature comparisons with the direct all-input component-KL control requested by the reviewer (Kong et al., https://arxiv.org/abs/2604.13440). Only the aggregation of the same single-module RTN-W3 output-KL array changes; proxy quantizer, calibration sweep, deployed candidates, allocator, and achieved 3.500-bit decoder-weight budget are fixed. Any resulting allocation difference therefore comes from which inputs determine module importance.

The experiment covers seven rotated-GPTQ models—Qwen2.5-1.5B/3B/7B, Llama-3.2-1B/3B, Llama-3.1-8B, and Mistral-7B—with three independent draws each, 48 C4-512 calibration windows, and 160 held-out WikiText-2 windows. The implemented subset rule selects max⁡(8,round⁡(0.15N))\max(8,\operatorname{round}(0.15N)) windows, or eight of 48; we have corrected Section 4.1's “10%” description.

CVaR5/CVaR1 averages the worst 5%/1% of token KL. Confident tokens have FP top-1 probability at least 0.9. Entries are relative changes with 95% t-intervals across seven model means, each averaging its three paired draws.

PAPA compared withToken CVaR5 changeToken CVaR1 changeConfident-FP token flip-rate change
All-input mean KL (FP→Q)−3.34% [−7.07, +0.39]−4.69% [−8.59, −0.79]−8.34% [−15.19, −1.50]
All-input mean KL (Q→FP)−2.83% [−6.02, +0.37]−3.85% [−6.95, −0.75]−7.46% [−13.63, −1.30]

At matched resources, tail conditioning reduces extreme output errors and preserves more confident FP decisions. Both forward- and reverse-KL controls show this gain in worst-1% KL and confident-token flips. Against forward mean KL, mean KL changes +0.20% [−1.58, +1.98], while the broader CVaR5 interval includes zero. The controlled result identifies a reliability benefit that the near-zero change in average KL does not reveal.

An equal-size subset ablation separates input selection from subset size using 48 C4-256 windows on rotated Qwen2.5-3B. CVaR5 at context 512 is 1.597±0.020 for all inputs, 1.574±0.020 for random eight-input subsets, and 1.511±0.031 for the hardest eight. At context 2,048: 1.309±0.023 / 1.294±0.022 / 1.250±0.010. Values are mean ± sample SD over three draws, averaging four random subsets per draw. Selecting the hardest inputs improves over random selection in all three draws at both contexts. The separate plain-GPTQ 1.5B two-draw control gives 2.155 versus 2.222 CVaR5 for mean KL versus PAPA. Together with the submitted cross-base comparisons, these controls identify both the additional value of input selection on rotated GPTQ and the role of the quantizer in choosing an allocation rule.

BART tests the theory's correction principle: use limited correction capacity to remove errors that matter at the output. The Q3 ablation holds capacity, training constraints, and auxiliary supervision fixed while changing the correction objective. CVaR is disabled by design to isolate this mechanism. Output KL improves both the KL tail and correct-word preservation under these controls, establishing the reliability value of the propagated correction target.

A separate ablation measures the incremental effect of adding CVaR to that core: 3.470±0.071→3.462±0.053 token CVaR5 and 21.35±0.07%→20.60±1.41% harm over two seeds. The harm-reduction interval [−0.40, 1.95] points includes zero; logs show the threshold can lag falling KL. This separates the established benefit of the output objective from the additional effect of explicitly reweighting the tail.

W3. Risk, distributional change, and recovered answers on the same items

The paired tests connect predicted risk to actual loss and recovery of correct answers. We measure risk, distributional change, and correctness on all 1,000 fixed LAMBADA words and all 1,283 target tokens. Correctness requires every token of a word to be correct; harm is FP-correct→quantized-wrong divided by all 1,000 words. Word KL averages forward KL over that word's target prefixes; word CVaR5 averages the worst 5% of words.

The risk score measures how large the predicted output change is relative to the original decision's safety margin. A close contest between tokens can be overturned by a small change; a clear winner can withstand more. For each token, we divide the predicted KL change by the minimum KL needed to remove the FP winner's lead, then sum over the word:

R(x)=∑t,in targetD^tκ(pt).R(x) = \sum_{t\\,\text{in target}} \frac{\widehat{D}_t}{\kappa(p_t)}.

Here D^t\widehat{D}_t is the KL predicted from internal errors and κ(pt)\kappa(p_t) is the FP decision barrier. The score is fixed before evaluating outcomes and uses only FP derivatives and local errors—no measured W4 logits or fitted classifier. Of 673 FP-correct words, W4 loses 52:

Frozen predictorHarmful-word AUROC [95% CI]
Inverse FP barrier0.824 [0.757, 0.882]
Local energy / FP barrier0.837 [0.779, 0.888]
Source-predicted KL alone0.822 [0.765, 0.875]
Source-predicted KL / FP barrier0.916 [0.888, 0.941]

Among FP-correct words, the score ranks a lost answer above a retained answer in about 91.6% of randomly selected pairs. This is the meaning of AUROC 0.916. Combining predicted change with decision tolerance gains 0.092 AUROC [paired 95% CI 0.043, 0.150] over the barrier alone; either ingredient alone scores about 0.82. The internal-error prediction therefore adds substantial information about which correct answers quantization will lose.

Allocation and training also use these same 1,000 item IDs. The W3 runs use BF16/SDPA, whose FP reference gets 674 words correct. Values are mean ± sample SD over two seeds.

InterventionDecoder bitsMean word KL ↓Word CVaR5 ↓Word accuracy ↑Harmful words ↓
Uniform GPTQ-W33.00.655 ± 0.0052.753 ± 0.01144.40 ± 0.14%26.00 ± 0.14%
All-input mean-KL allocation3.50.402 ± 0.0152.046 ± 0.09351.75 ± 0.07%19.05 ± 0.21%
PAPA, same allocation budget3.50.358 ± 0.0031.924 ± 0.14553.80 ± 1.13%17.40 ± 0.85%
Local reconstruction + 0.1 NLL3.00.723 ± 0.0133.263 ± 0.01346.00 ± 1.84%25.70 ± 1.98%
Output KL + 0.1 NLL, same constraints3.00.568 ± 0.0152.752 ± 0.18150.45 ± 0.21%21.50 ± 0.42%

PAPA preserves more correct answers at the same 3.5-bit budget: harm falls by 1.65 points [paired-item 95% CI 0.30, 3.00], or 8.7%, and accuracy rises by 2.05 points [0.50, 3.60]. Harm falls 192→180 / 189→168 across seeds; rescues/new harms are 57/45 / 52/31. Mean word KL also falls; the word-CVaR5 reduction interval includes zero. At identical W3 budget and NLL, output KL reduces harm by 4.20 points [2.70, 5.70]. These are direct measurements of correct-word preservation on the same items. Intervals jointly resample word IDs across fitted runs; SD separately describes run variation.

An HQQ-W3 experiment tracks both outcomes on 512 identical LAMBADA first-token prefixes. Every layer set is frozen on disjoint C4 text and transferred from RTN-W3:

Four-layer W3→W4 selectionMean decision KL ↓Decision CVaR5 ↓Accuracy ↑Harmful cases ↓Rescued / newly harmed
No restoration3.2219.15716.80%267—
Original random-input proxy0.7132.71245.31%133145 / 11
Local residual magnitude0.7313.01545.51%133145 / 11
Realized-direction propagation0.6422.53749.41%118158 / 9

Realized-direction repair removes 55.8% of original harms, counting new harms, at 3.000→3.111 decoder bits. Its accuracy gain over the original proxy is 4.10 points [paired-item 95% CI 0.98, 7.42]. It captures 65.1% [58.2, 71.8] of uniform-W4's harm reduction using 11.1% of its added weight bits. On the fixed original worst-5% subset, KL falls 9.157→0.574 and harm 25→2; recovered cases stay in the evaluation.

The paired W4 repairs also make the role of the decision barrier concrete. On the exact model/population used for risk prediction, C4-selected four-layer BF16 repairs change harm 52→48/51/53 for original-proxy/realized-direction/local-magnitude selection; mean word KL changes 0.0899→0.0734/0.0754/0.0816. All three lower average KL, while their retained correct answers differ, with the original proxy giving the largest harm reduction here. Measuring error relative to the decision barrier adds the information needed to assess whether an output change threatens the answer. The risk test above quantifies that information, and the paired repairs measure the resulting correctness directly.

A controlled input-format test also clarifies the submitted Mistral discrepancy. Fixing GPTQ-W4 weights and 64 WikiText-2 windows while adding the zero-shot prompts' BOS token changes p99 0.42→6.27 and disagreement 8.9%→34.8%. Including BOS in half the calibration windows at the same budget reduces zero-shot harm 38.3%→14.5% in the rerun. This identifies a consequential format sensitivity and an effective calibration change that BOS-free evaluation would miss.

For Qwen2.5-3B W4, a separate GPTQ-start BART refinement with held-out-C4 learning-rate selection retains bits, capacity, and 500 updates. CVaR5 falls 0.685→0.655 (−4.39% [paired-window 95% CI −5.58, −3.17]), p99 0.848→0.811, and mean task harm 4.19%→3.82%. This joint test of initialization/grid and learning-rate selection demonstrates improved tail and task reliability while preserving the deployment and training budgets.

The common logic is output error relative to the same input's decision barrier. The source score predicts which correct answers are at risk; the controlled allocation and training experiments measure how many correct answers each intervention preserves. Testing both on the same inputs gives the propagation framework a concrete reliability outcome.

Q3. Controlled ablations isolate the reliability gain from BART's output objective

We add a factorial ablation designed to separate the correction objective from auxiliary supervision and stabilization settings. The submitted Table 10 matched correction capacity, data, and update budget; the local control additionally used the stabilization settings listed in the appendix. The new experiment applies those settings identically to both objectives and crosses local reconstruction/output KL with NLL weights 0 and 0.1. NLL trains on observed next tokens; matching it tests the output objective at the same level of auxiliary supervision. Both objectives use:

  • Qwen2.5-3B W3, group size 128, rank 16, alpha 32, and 30,965,760 trained parameters.
  • The same 256 C4 sequences × 512 tokens, initialization, sample order within each seed, and 500 updates.
  • AdamW with betas (0.9, 0.95), identical learning rates and cosine schedule, gradient clipping 1.0, and weight decay 0.001.
  • The same correction-row norm cap (0.2× base-row norm) and clipping floor (0.5). CVaR is disabled.

Within each NLL setting, only the correction objective changes: matching intermediate hidden states versus matching final token probabilities. Disabling CVaR isolates this difference from extra tail weighting. Values are mean ± sample SD over two seeds; word metrics use the 1,000 complete targets above.

ObjectiveNLL weightPPL ↓Token CVaR5 ↓Word accuracy ↑Harmful words ↓
Cumulative hidden-state reconstruction016.581 ± 0.0104.223 ± 0.06522.65 ± 2.33%46.10 ± 2.12%
Output KL013.108 ± 0.1243.459 ± 0.05048.80 ± 0.28%22.35 ± 0.64%
Cumulative hidden-state reconstruction0.114.541 ± 0.1873.862 ± 0.01146.00 ± 1.84%25.70 ± 1.98%
Output KL0.113.141 ± 0.0363.529 ± 0.15950.45 ± 0.21%21.50 ± 0.42%

The output objective improves both the KL tail and correct-word preservation in both supervision settings. It reduces harmful words by 23.75 points [paired-item 95% CI 21.25, 26.20] without NLL and 4.20 points [2.70, 5.70] with the same 0.1 NLL anchor. Holding the other training choices fixed isolates the contribution of optimizing the downstream output effect.

A second controlled experiment tests the role of accumulated input error by reconstructing each block on its exact FP input. This gives the local objective a clean input at every block. All objectives share the learning-rate search and disjoint C4 validation rule, selecting 2×10−42 \times 10^{-4}; rank, constraints, data, and updates remain fixed, with NLL/CVaR disabled. On 96 WikiText-2 windows of 2,048 tokens and the same 512 LAMBADA first-token prefixes, two-seed mean ± sample SD is:

ObjectivePPL ↓Token CVaR5 ↓First-token accuracy ↑Harmful decisions ↓
Reconstruction on FP block inputs14.697 ± 0.2963.552 ± 0.02440.92 ± 0.97%29.69 ± 0.28%
Cumulative hidden-state reconstruction16.759 ± 0.1763.889 ± 0.04121.97 ± 1.80%46.09 ± 1.38%
Output KL12.864 ± 0.0523.022 ± 0.04648.44 ± 0.83%22.56 ± 0.41%

Output KL improves on FP-input reconstruction by 14.9% CVaR5 and 7.13 percentage points in harm [paired-item 95% CI 4.39, 9.86]. These controlled experiments validate the paper's correction principle: at fixed correction capacity, optimizing the errors' final output effect preserves more correct decisions. The result holds with matched constraints and auxiliary supervision, and against local reconstruction supplied with clean FP inputs and the same learning-rate search. The revised manuscript presents these controls in main-text Table 5 and Appendix G.5 (Table 17), and same-item risk/recovery evidence in Section 3.3 and Table 4. The original Table 10 is retained as a scale/capacity comparison in revised Table 15.

Reviewer_hsPwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors show that small quantization errors can be amplified through downstream layers, which can produce large changes in the output distribution and result in token flips that substantially affect autoregressive generation. Aggregate downstream task performance metrics, such as accuracy or perplexity, can hide this effect. To mitigate this quantization-induced behavioral divergence, the authors propose two approaches: one identifies modules that contribute most to the butterfly effect and recovers them with higher precision, and another learns lightweight corrections via distillation.

Critical strengths
  • The paper addresses a critical phenomenon: quantized models that have close performance (measured using accuracy or perplexity) to their corresponding full-precision models do not necessarily share the same behavior and can produce a strong behavioral shift. The authors characterize the quantization butterfly effect, in which small local errors can be amplified through downstream layers, causing large distribution shifts and token changes that strongly affect sequential generation.

  • The authors propose two mitigation strategies to mitigate divergence between full-precision and quantized models: (i) PAPA assigns higher precision to modules where the quantization errors have the greatest downstream impact, while respecting a fixed total bit budget. (ii) BART learns lightweight corrections via distillation while maintaining the deployment bit-width.

  • The authors provide extensive empirical evaluation that spans various model families, sizes, quantization settings, and language, reasoning, code, and instruction-following tasks.

Critical weaknesses
  • W1: I believe that the presentation of the paper is poor, and it is hard to follow. This is due to dense notation and insufficiently defined operators and symbols (Section 2). Also, many central experiments appear only in the appendix, which requires jumping back and forth between the main text and the appendix. Could you please (i) add a brief notation section before Section 2? (ii) Summarize the important tables/figures with clear takeaways in Section 3? This would substantially improve the paper's readability.

  • W2: The authors propose two certificates; however, it is unclear how often these certificates provide useful guarantees and how expensive they are to compute.

  • W3: The flips (top-1disagreement) and the KL divergence between base and quantized models were already studied in previous works [1,2,3] under different reliability and trustworthiness dimensions. Can the authors specify the contribution of their empirical evaluation/metrics to these papers? And what additional findings do their empirical assessments provide?

  • W4: The empirical evaluation is limited to RTN and GPTQ, which are outdated. Many recent SOTA quantization methods are missing (AWQ, HQQ, Quanto, etc.).

[1] Dutta, Abhinav, et al. "Accuracy is not all you need." Advances in Neural Information Processing Systems 37 (2024): 124347-124390.

[2] Stanton, Samuel, et al. "Does knowledge distillation really work?." Advances in neural information processing systems 34 (2021): 6906-6919

[3] Ayadi, Sirine, et al. "Reliability Scaling Laws for Quantized Large Language Models." arXiv preprint arXiv:2607.10855 (2026).

Questions for authors
  • Q1: For the experiment in Table 1: How does the recovery look if the layers are recovered to 4-bit instead of full precision? Also, these results are limited to RTN. Could you please present results for recent quantization methods? Also, I believe that it is important to include the total number of model bits for all variants in the table.

  • Q2: For the recovery, the authors always select exactly 4 layers to recover to FP regardless of the setting. Could you clarify this choice?

  • Q3: Could the authors provide an evaluation of the observed butterfly effect if they simply add Gaussian noise to the full-precision model weights instead of performing weight quantization?

8 comments

Thank you for the concrete requests. Our central contribution is to connect which internal errors occur, which decisions they threaten, and which repairs preserve correct answers. The added experiments test each link: actual error directions predict token changes; predicted output change relative to the decision's safety margin identifies lost correct words with 0.916 AUROC; and promoting four HQQ layers from W3 to W4 reduces harmful decisions 267→118. Below we explain the guarantees and their practical coverage, identify what the controlled experiments add to prior work, and report computation and full storage costs.

W1. Define the notation and explain each experiment beside its claim

We have introduced a short notation guide before Section 2, with a plain-language explanation: a module produces an error δm\delta_m; the remaining network propagates it through GmG_m; GmδmG_m\delta_m predicts the change in final token scores; and rr records the difference between prediction and reality. The decision tolerance κ(p)\kappa(p) is the smallest KL change that can remove the FP token's lead.

The main text now follows local error → output change → decision preservation → repair. The submitted Table 1 already demonstrates the repair step: across four models, our four-layer proxy recovers about 53% of the p99 KL gap, versus 38%/37%/22% for Hessian/local-error/random selection at the same per-model storage. We have placed each experiment beside its claim, identified submitted and new evidence, shortened the abstract, and given the realized-direction diagnostic, original layer proxy, and PAPA allocation score distinct names.

W2. Two guarantees answer two different questions

The output guarantee certifies that the next greedy token stays the same when the output change is below the decision's tolerance. The source guarantee gives the internal error budget for preserving that decision in the linearized network, combining the token's score gap with the network's sensitivity to internal errors.

Output guarantee: a measurable tolerance for preserving the FP token

For an FP distribution pp with a unique winner, Theorem 2 computes κ(p)\kappa(p) from the winner and runner-up probabilities. Any quantized distribution qq satisfying

DKL(p,∥,q)<κ(p)D_{\mathrm{KL}}(p \\,\Vert\\, q) < \kappa(p)

must keep the FP winner. This covers the entire vocabulary and applies to any quantizer. The threshold is exact: at the boundary, a competing token can tie the winner; above it, a change is possible but not required. It can be used with measured KL or with any independently certified KL upper bound.

On the same 512 labeled Qwen2.5-3B prefixes, the exact threshold certifies more decisions than Pinsker's (pc−pa)2/2(p_c-p_a)^2/2 threshold, where cc and aa are the FP winner and runner-up:

QuantizationPinsker-certifiedExact-barrier-certifiedFalse certificates
GPTQ-W344/512 (8.6%)64/512 (12.5%)0
GPTQ-W4230/512 (44.9%)270/512 (52.7%)0

At W4, this gives 40 additional certified decisions, or +7.8 percentage points [paired 95% CI 5.7, 10.2]. Once probabilities and KL are available, the test takes 6.13 μs per decision in batches of 32 on an A100. This is the additional audit cost; model forward passes and KL computation are separate.

The guarantee also measures how many correct answers a repair protects. A complete word is certified when every target token passes. Among 674 FP-correct words in a fixed 1,000-item LAMBADA set, using the BF16/SDPA reference:

AllocationAverage decoder weight bitsCertified complete words ↑
Uniform GPTQ-W33.021.96 ± 1.05%
All-input mean-KL allocation3.534.50 ± 0.94%
PAPA3.538.35 ± 1.36%

Values are mean ± sample SD across two calibration seeds. At identical 3.5-bit budgets, PAPA certifies 3.86 percentage points more correct words [paired-item 95% CI 2.08, 5.71], with zero false word certificates. The interval resamples word IDs jointly across the two fitted runs. Because these words are FP-correct, their gold prefixes coincide with FP greedy prefixes: passing every target position guarantees that the whole correct word is retained. Thus the certificate evaluates a concrete benefit of allocating precision.

The same stepwise rule applies to longer greedy continuations. Across 128 GPTQ-W4 prompts, coverage at horizons 1/4/8/16/48 is 43.8/7.0/0.8/0/0%, with zero violations. Every step must pass for exact trajectory preservation; one uncertified step prevents that guarantee even if the generated token stays unchanged.

Source guarantee: an internal error budget for a decision

Theorem 4 gives the smallest weighted internal error that can close any competing token's score gap in the linearized network. It combines the size of that gap with how strongly the network propagates errors toward it. A forward pass plus a vector–Jacobian product for one competitor over all 252 module outputs takes 119.41 ms / 14.01 GB on a GH200 (128-token FP32/eager input; median of 20 repetitions); covering all competitors requires further computation. A nonlinear source certificate additionally requires a bound on the prediction remainder. Theorem 1's conservative range bound certifies 0% in these full-quantization runs. The experiments establish two practical uses: measured output KL certifies preservation directly, with the coverage reported above, and source information predicts which correct answers are at risk, tested in W3.

W3. The contribution is to predict decision loss from internal errors and test targeted recovery

Dutta et al. (https://arxiv.org/abs/2407.09141) establish KL/answer-flip audits, harmful and beneficial transitions, and an empirical association with small answer margins. Stanton et al. (https://arxiv.org/abs/2106.05945) distinguish predictive fidelity from generalization. Ayadi et al. (https://arxiv.org/abs/2607.10855) study uncertainty, calibration, robustness, and mean token-KL scaling with model bits. Building on these measurements, our framework makes and tests predictions about the internal errors responsible for output changes and the decisions those changes threaten.

1. Predict the output effect of the actual error, including its direction

Chang et al. (https://arxiv.org/abs/2506.12044) connect large quantization errors on particular inputs to residual magnitudes and MLP gates; QEP (https://arxiv.org/abs/2504.09629) compensates accumulated input error; RaanA (https://arxiv.org/abs/2504.03717) uses a downstream-Jacobian norm for allocation; and Zhou et al. (https://arxiv.org/abs/2608.08188) analyze source noise and cross-layer accumulation. Our test retains the per-input error vector in GmδmG_m\delta_m and predicts its resulting distribution and token choice. Across 35 GPTQ-W4 modules, quantized one at a time, KL rank correlation is 0.998 in the ordinary 95% / 0.960 in the upper 5%, with flip precision/recall 94.5%/95.9%. At the same injection site and error norm, replacing the actual direction with random directions reduces median within-module tail correlation 0.976→0.536. This isolates information carried by the realized direction beyond magnitude and location.

2. Predict lost correct answers by comparing the expected change with the decision's safety margin

A close contest between tokens can be overturned by a small change; a clear winner can withstand more. The fixed risk score therefore divides source-predicted KL by the minimum KL needed to remove the FP winner's lead, then sums over the word's target positions. It uses only FP derivatives and local errors, without measured W4 logits or a fitted classifier. On 1,000 fixed LAMBADA words under the FP32/eager protocol, 673 are FP-correct and 52 become wrong under W4:

Frozen predictorHarmful-word AUROC [95% CI]
Inverse FP barrier0.824 [0.757, 0.882]
Local energy / FP barrier0.837 [0.779, 0.888]
Source-predicted KL alone0.822 [0.765, 0.875]
Source-predicted KL / FP barrier0.916 [0.888, 0.941]

Among FP-correct words, the score ranks a lost answer above a retained answer in about 91.6% of randomly selected pairs. This is the meaning of AUROC 0.916. Either ingredient alone scores about 0.82; combining them gains 0.092 AUROC [paired 95% CI 0.043, 0.150] over the barrier alone. Thus internal-error prediction adds substantial information about which correct answers quantization will lose. Wu et al. (https://arxiv.org/abs/2608.06564) forecast decision loss using fitted margin changes; our experiment tests information extracted directly from internal quantization errors.

The same exact output-KL boundary also appears in Sharma's speculative-decoding analysis, Corollary 4.7 (https://arxiv.org/abs/2606.30265); we now cite this relationship beside Theorem 2 and in the related-work discussion. Our contribution here is its connection to internal quantization-error prediction, measured harmful decisions, and precision interventions that increase certified correctness, tested above and in W2.

3. Connect average and extreme output errors with the same finite-perturbation result

The submitted Fisher theorem bounds actual KL above and below by explicit multiples of Fisher-weighted logit-error energy. With a common finite bound on logit-error ranges, the same multipliers control the mean, every quantile, and CVaR. This gives a mathematical basis for studying the largest errors as well as average fidelity, without assuming a tail shape or taking an infinitesimal-error limit.

4. Test which allocation choice improves reliability at fixed resources

Kong et al. (https://arxiv.org/abs/2604.13440) rank components using their output KL. Our new PAPA ablation holds the component-KL measurements, quantizer, allocator, and achieved 3.500-bit budget fixed, changing only which calibration inputs determine importance. Across seven rotated-GPTQ models and three draws per model, selecting high-risk inputs lowers worst-1% token KL by 4.69% [95% CI 0.79, 8.59] relative to all-input forward KL, and 3.85% [0.75, 6.95] relative to all-input reverse KL. In the separate LAMBADA comparison, at the same 3.5-bit budget, PAPA reduces harmful words 19.05%→17.40%, a 1.65-point gain [paired-item 95% CI 0.30, 3.00], and increases certified correct words as shown in W2.

The direction-based ranking also transfers from RTN calibration to an HQQ intervention. Promoting four HQQ-W3 layers to W4 reduces harmful decisions 267→118 on 512 labeled prefixes: 158 rescues / 9 new harms. The original proxy and local-error selection each leave 133 harms at the same four-layer budget. Accuracy rises 16.80%→49.41% at 3.000→3.111 decoder bits. Rankings are frozen on disjoint C4 text without task labels. These experiments establish the practical value of the extra information: better prediction of lost answers and more correct behavior preserved per bit.

W4. Modern quantizers retain the tail, and HQQ also supports targeted repair

We add the actual AWQ implementation, upstream HQQ, Quanto qint4, and bitsandbytes NF4. These extend the submitted RTN, AWQ-style, GPTQ, rotated-GPTQ, and KIVI-style KV-cache experiments.

For Qwen2.5-3B, token metrics use the same 128 WikiText-2 windows of length 512; PPL uses a common 32-window, 2,048-token set. Each row is a fixed W4 configuration:

QuantizerPPL increase over FPMean token KLToken p99 KLp99 / meanTop-1 disagreement
GPTQ6.84%0.09690.87739.05×14.50%
AWQ5.81%0.08530.70538.27×13.78%
HQQ10.64%0.14001.15078.22×17.55%
Quanto13.27%0.19151.67018.72×19.30%
NF46.03%0.08680.68017.83×13.93%

Integer quantizers use group size 128; NF4 uses block size 64. All five implementations retain a pronounced upper tail: p99 KL is 7.8–9.1× the mean. These fixed configurations extend the observation across implementations under a common protocol. The HQQ intervention in Q1 adds the practical test: a ranking selected on RTN identifies layers whose W4 restoration improves both the tail and task correctness.

Q1. Four-layer W4 recovery captures most BF16 recovery at much lower added storage

Yes: restoring four layers to W4 retains most of the BF16 benefit with one-thirteenth of the added weight payload. The new ablation keeps the four selected layers fixed and changes only their restored precision. Eight disjoint C4 sequences select the realized-direction RTN-W3 ranking, which transfers unchanged to HQQ. Evaluation uses 128 held-out WikiText-2 sequences. As in submitted Table 1, p99 is over per-sequence mean KL.

BaseRestorationAverage decoder bitsTotal model bitsSequence p99 KL ↓Recovery [95% CI]
RTN-W3None3.00013.653 Gbit8.760—
RTN-W3Four layers → W43.11113.961 Gbit2.56470.7% [66.6, 77.7]
RTN-W3Four layers → BF164.44417.622 Gbit1.38784.2% [83.2, 85.2]
HQQ-W3None3.00014.693 Gbit3.883—
HQQ-W3Four layers → W43.11115.002 Gbit0.96075.3% [71.8, 84.0]
HQQ-W3Four layers → BF164.44418.547 Gbit0.75980.4% [76.6, 88.1]

Intervals resample paired evaluation sequences for the fixed models and rankings. Total bits include quantized weights, unchanged BF16 tensors, and quantizer metadata: one 16-bit RTN scale or an FP32 HQQ scale and zero point per group. Caches, file headers, and kernel padding are excluded.

W4 adds 0.308 Gbit (38.5 MB) of weight payload versus 4.008 Gbit for BF16, retaining 84.0%/93.6% of RTN/HQQ's BF16 p99 recovery for one-thirteenth of the added payload. Keeping the submitted Table 1's original Qwen2.5-3B set {0, 1, 2, 6} also recovers 72.8%/84.4% under RTN/HQQ on a separate 160-window evaluation.

The benefit also reaches task correctness. On the 512 HQQ prefixes above, four W4 layers recover 65.1% [paired 95% CI 58.2, 71.8] of the harmful-decision reduction achieved by uniform W4, using 11.1% of its additional weight bits.

Reviewer_vCFfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

[Post-rebuttal update]: The authors’ controlled experiments and revised framing have addressed most of my concerns, so I have raised my score to 3.

This paper studies input-level reliability issues in low-bit LLM quantization that are often overlooked by average metrics. The authors develop a framework linking local quantization errors, downstream propagation, output distribution shifts, token decisions, and decoding trajectories. They derive KL bounds and decision stability conditions, and propose PAPA for mixed precision allocation and BART for low rank quantization correction. Experiments cover multiple models, quantizers, and downstream tasks.

Critical strengths
  1. The paper highlights that preserving average performance does not necessarily preserve individual model behaviors. This is particularly relevant for quantized LLMs deployed in applications where failures on specific inputs can be costly.

  2. By connecting local quantization errors to output changes and generation behavior, the paper offers a more comprehensive view of why certain inputs are vulnerable to quantization. This perspective could inform future research on quantization robustness and reliability.

  3. Rather than only characterizing quantization failures, the paper explores how the analysis can guide precision allocation and lightweight correction. The results suggest that targeting high impact components can improve reliability in some low bit settings without uniformly increasing model precision.

Critical weaknesses
  1. Limited direct validation of error propagation. Table 7 evaluates the Fisher approximation using measured logit differences rather than testing whether propagated local errors predict those differences. The layer gain proxy also differs from the theoretical propagation operator. The results support the proposed diagnostics but provide limited direct evidence for the full propagation mechanism.

  2. Unclear incremental contribution of PAPA. [1] have already explored component-level output KL for precision allocation. The paper lacks a controlled comparison between KL averaged over all calibration samples and KL measured on the high risk subset, making the specific benefit of tail conditioning difficult to assess.

  3. Missing comparison with closely related methods. LR-QAT [2] also uses trainable low rank corrections that can be folded into quantized weights without additional inference branches. A matched comparison would help clarify the benefits of BART's design.

  4. Inconsistent tail reliability improvements. PAPA does not consistently beat simpler baselines:

  • It underperforms local-error baselines on RTN and only ties them on AWQ-style quantization (Table 16).
  • On plain GPTQ, it trails Hessian by about 10% in CVaR5 (Figure 6b).
  • Hessian is also better on Llama-3.2-1B and in p99 at 7B (Table 18).

For BART, most gains come from standard distillation. The CVaR term gives small and inconsistent improvements, and BART slightly degrades the Qwen2.5 models at W4 (Table 9). Together, these results weaken the evidence that explicitly targeting tail risk is generally effective.

  1. Presentation. The paper is hard to follow.
  • Much of the key evidence is deferred to the appendix (e.g., Tables 4, 6, 9, 16, 19).
  • Similarly named diagnostics (butterfly score, butterfly-impact, QBI, gain-weighted local proxy) are easy to confuse.
  • The abstract is lengthy and lists most theoretical results and numbers, which obscures the paper’s core message.

[1] Kong, Jason, et al. "A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models." arXiv preprint arXiv:2604.13440 (2026).

[2] Bondarenko, Yelysei, Riccardo Del Chiaro, and Markus Nagel. "Low-rank quantization-aware training for llms." arXiv preprint arXiv:2406.06385 (2024).

Questions for authors
  1. The paper states that PAPA’s scoring cost is comparable to GPTQ Hessian collection and that BART keeps the PTQ storage and inference kernel. Could the authors report wall-clock time and peak memory for PAPA scoring and BART training (compared with EfficientQAT), as well as the actual inference latency of PAPA’s mixed 3/4-bit models?

  2. The trajectory analysis primarily uses greedy decoding. How would the observed divergence patterns change under temperature sampling or other decoding strategies commonly used in deployment?

  3. Table 21 suggests that on GSM8K most diverged trajectories still reach the same final answer (100% divergence vs. 22.3% answer changes). How often do diverged generations remain correct or semantically equivalent on open-ended continuations, HumanEval, and IFEval? This would clarify the practical significance of the high divergence rates.

8 comments

Thank you for identifying the key controls. The paper connects local quantization error → downstream output change → decision stability → targeted repair through theory and interventions. The added controlled experiments isolate the contributions of actual error direction, high-risk-input selection, and the correction objective. Together with deployment-cost and sampling/task evaluations, they establish the framework's practical value: predict consequential output changes, preserve more correct answers at matched resources, and measure failures that average accuracy hides.

W1. Actual local errors predict output changes and token flips

Submitted evidence. Eq. (1) models local-error propagation; Tables 7–8 evaluate the output-level Fisher approximation using measured logit changes; Table 1 demonstrates recovery through layer restoration.

New direct test. For this rebuttal, we additionally predict output changes from the actual local error, preserving both its direction and its module-output injection site:

δm=(WmQ−Wm)hm,vm=Gmδm.\delta_m=(W_m^Q-W_m)h_m,\qquad v_m=G_m\delta_m.

Using the FP input hmh_m, δm\delta_m measures the module's output error. A Jacobian–vector product through the remaining network gives its predicted logit effect vmv_m, hence the predicted distribution softmax⁡(z+vm)\operatorname{softmax}(z+v_m) from FP logits zz. A separate single-module quantization run measures Δzm\Delta z_m; the remainder Δzm−vm\Delta z_m-v_m is the prediction error. The source predicts; the quantized run checks.

On Qwen2.5-3B GPTQ-W4, we test seven projection types in five early/middle/late layers: 35 modules × 32 held-out 128-token windows × 63 positions = 70,560 cases. We use FP32/eager attention and remove constant logit shifts. The tail is the largest 5% of actual single-module KL values; relative RMS error is RMS remainder divided by RMS actual change.

Evaluation subsetKL rank correlationLogit cosineRelative RMS errorPredicted / actual mean KL
Ordinary 95%0.9980.9930.2781.004
Upper 5%0.9600.9860.2440.973

The predicted flips have 94.5% precision and 95.9% recall, including 93.3%/94.3% in the tail. The highest predicted 5% retrieves 96.3% of the actual tail [95% paired-window CI 95.0, 96.8], versus 27.6% for the strongest calibrated local-error control.

New direction ablation. Holding both error norm and module-output injection site fixed, we replace the actual direction with three random directions. Median within-module tail correlation falls from 0.976 to 0.536. This controlled intervention isolates the predictive information carried by the realized direction, independently of error size and injection site.

The all-block composition test measures how source effects combine: summed FP-linear predictions give overall/tail correlations 0.910/0.537 and relative RMS remainders 0.735/0.873. These remainders quantify departures from additive source predictions when errors coexist, as retained explicitly in the theory. PAPA scores modules using measured finite single-module KL, and its allocation gains are evaluated on complete quantized models with interactions present. The source, composition, and allocation tests therefore measure each step of the framework directly.

W2. Selecting high-risk inputs improves the deepest tail at the same bit budget

PAPA's distinctive allocation choice is which calibration inputs determine component importance. The submission compared PAPA against local-error and curvature-based scores. We now add two controlled experiments isolating high-risk-input selection beyond component output-KL sensitivity (Kong et al., https://arxiv.org/abs/2604.13440). The first changes only the aggregation of the same single-module KL measurements; proxy quantizer, calibration sweep, candidate weights, allocator, and achieved 3.500-bit decoder-weight budget are identical. This directly attributes allocation differences to input selection.

We evaluate seven rotated-GPTQ models: Qwen2.5-1.5B/3B/7B, Llama-3.2-1B/3B, Llama-3.1-8B, and Mistral-7B, with three independent rotation/calibration draws each. Each run uses an RTN-W3 proxy, 48 C4-512 calibration windows, and 160 disjoint WikiText-2 windows. PAPA selects the eight highest-risk windows using the implemented rule max⁡(8,round⁡(0.15N))\max(8,\operatorname{round}(0.15N)); we have replaced Section 4.1's “10%” description with this rule.

CVaR5/CVaR1 means the average KL among the worst 5%/1% of tokens. Confident tokens have FP top-1 probability at least 0.9. Entries are relative changes with 95% t-intervals across seven model means, each averaging its three paired draws; negative is better.

PAPA compared withToken CVaR5 changeToken CVaR1 changeConfident-FP token flip-rate change
All-input mean KL (FP→Q)−3.34% [−7.07, +0.39]−4.69% [−8.59, −0.79]−8.34% [−15.19, −1.50]
All-input mean KL (Q→FP)−2.83% [−6.02, +0.37]−3.85% [−6.95, −0.75]−7.46% [−13.63, −1.30]

At the same resources, tail conditioning reduces extreme output errors and preserves more confident FP decisions. Both KL directions show this gain in worst-1% KL and confident-token flips. Mean KL changes +0.20% [−1.58, +1.98], while the broader CVaR5 interval includes zero. This isolates a reliability benefit that the near-zero change in average KL does not reveal. The reverse-KL control covers Kong et al.'s preferred direction under our common allocator.

The second experiment is an equal-size subset ablation, separating which inputs are selected from how many. On rotated Qwen2.5-3B, all subset scores use eight of the same 48 C4-256 windows. Values are mean ± sample SD across three calibration draws; random selection averages four prespecified subsets within each draw.

Calibration aggregationToken CVaR5, context 512 ↓Token CVaR5, context 2,048 ↓
All 48 inputs1.597 ± 0.0201.309 ± 0.023
Random eight inputs1.574 ± 0.0201.294 ± 0.022
Lowest-risk eight inputs1.583 ± 0.0511.304 ± 0.045
PAPA: highest-risk eight inputs1.511 ± 0.0311.250 ± 0.010

PAPA improves on random eight-input selection in all three draws. The paired CVaR5 reductions are 0.063 ± 0.026 at context 512 and 0.044 ± 0.013 at context 2,048. Together, the two ablations isolate high-risk-input selection from both output-KL scoring and calibration-set size, establishing its additional value in these rotated-GPTQ experiments.

W3. The propagation-based correction objective improves BART and transfers to LR-QAT

The contribution is a theory of which errors a limited-rank correction should remove, connected to decision reliability and tested with matched objectives. The submitted Theorem 9 characterizes the optimal correction after downstream propagation and Fisher weighting, with both fixed; Fisher weighting measures logit changes by their effect on output probabilities. BART implements this output-based correction principle within a foldable low-rank parameterization. For this rebuttal, we add LR-QAT (https://arxiv.org/abs/2406.06385), an established foldable low-rank QAT method, as both a matched-budget comparator and a test of the objective's transfer across parameterizations.

QERA (https://arxiv.org/abs/2410.06040) optimizes layer-output reconstruction using input second moments; our objective also accounts for the remaining network. Consider orthogonal errors of size 2 and 1, amplified by 1 and 10. A rank-one local correction leaves output error 10; a propagated correction leaves error 2. Our submitted proposition shows that this gap can grow arbitrarily large and identifies equal weighted downstream gains as the case where local reconstruction is optimal too. The theory therefore specifies both when propagation changes the best correction and when local reconstruction suffices.

The Qwen2.5-3B W3 comparison shares group size 128, 256 C4-512 sequences, 500 updates, and independent C4 validation for learning-rate selection. LR-QAT follows the published quantizer, scales, and clipping before INT8 conversion. Rank 4 closely matches BART's trained capacity; rank 32 tests larger capacity. CE trains on observed tokens; KL preserves FP probabilities. Each row is one selected run under this common budget, shorter than LR-QAT's published schedule. Harm uses the same 1,000 LAMBADA first-token decisions.

Method and objectiveTrained parametersPPL ↓Token CVaR5 ↓Harmful decisions ↓
LR-QAT, rank 4, CE29.16M14.4164.48630.0%
LR-QAT, rank 32, CE81.54M14.7764.55830.4%
BART, rank 16, output KL + 0.1 NLL30.97M12.9853.42120.9%
LR-QAT, rank 4, same KL/NLL objective29.16M10.6762.30314.4%
LR-QAT, rank 32, same KL/NLL objective81.54M10.6832.25814.7%

BART improves on CE-trained LR-QAT at comparable capacity. Changing LR-QAT's objective at fixed rank further improves LR-QAT, which achieves the strongest results with the same KL/NLL objective. This transfer establishes practical value for the correction target beyond BART's implementation. On 300 Qwen2.5-3B-Instruct GSM8K problems with the same 500-update budget, replacing LR-QAT's CE with KL + 0.1 NLL raises accuracy 38.7%→55.7%, a paired 17.0-point gain [95% CI 10.7, 22.7], extending the benefit to reasoning accuracy.

W4. Controlled ablations identify where the gains come from

The BART ablation tests a specific prediction of the theory: limited correction capacity is more effective when directed toward errors that matter at the output. The Fisher analysis explains how output KL weights internal errors by their downstream probability effects. W2 isolates PAPA's input selection; here a factorial ablation separates the correction objective from auxiliary supervision by crossing local reconstruction/output KL with NLL weights 0 and 0.1. NLL trains on observed next tokens, so each comparison holds this supervision fixed.

Both methods share rank, initialization, sample order, optimizer, correction projection, clipping floor, weight decay, and 500 updates. Within each NLL setting, only the correction objective changes. CVaR is disabled by design to isolate that effect from extra tail weighting. Values are mean ± sample SD over two training seeds. PPL and token CVaR5 use WikiText-2; word accuracy and harm use the same 1,000 complete LAMBADA targets. A word is correct only if all target tokens are correct; harm is FP-correct→quantized-wrong as a percentage of all words.

ObjectiveNLL weightPPL ↓Token CVaR5 ↓Word accuracy ↑Harmful words ↓
Cumulative hidden-state reconstruction016.581 ± 0.0104.223 ± 0.06522.65 ± 2.33%46.10 ± 2.12%
Output KL013.108 ± 0.1243.459 ± 0.05048.80 ± 0.28%22.35 ± 0.64%
Cumulative hidden-state reconstruction0.114.541 ± 0.1873.862 ± 0.01146.00 ± 1.84%25.70 ± 1.98%
Output KL0.113.141 ± 0.0363.529 ± 0.15950.45 ± 0.21%21.50 ± 0.42%

At matched capacity, constraints, and supervision, the output objective improves both tail and word reliability. With the same NLL anchor, output KL reduces harmful words by 4.20 points [paired-item 95% CI 2.70, 5.70], or 16.3%, improving in both seeds. A second control supplies exact FP inputs to each block, removing accumulated input error from local reconstruction, and uses an equal learning-rate search. Output KL also improves on this control: CVaR5 3.552±0.024→3.022±0.046.

The objective ablations establish the reliability value of directing correction toward propagated output effects. A separate ablation measures the additional effect of explicit CVaR weighting: token CVaR5 3.470±0.071→3.462±0.053 and harmful words 21.35±0.07%→20.60±1.41% across two seeds. Its harm-reduction interval [−0.40, 1.95] points includes zero; logs record a tail threshold lagging the falling KL. These experiments separately identify the output objective's established gain and measure the extra tail penalty's contribution. Paired intervals jointly resample word IDs across fitted runs; SD separately describes run variation.

The cross-base comparisons also make the allocation choice concrete: local/Hessian scores lead on RTN and some plain-GPTQ settings, while the controlled rotated-GPTQ experiments isolate gains from high-risk-input selection. The theory characterizes when local and propagated objectives agree, and the fixed-norm direction ablation identifies predictive information beyond error magnitude. Together, the analysis and controls explain which information each rule uses and measure its value for allocating a fixed budget.

We also add a targeted test of the submitted W4 training configuration: GPTQ initialization plus held-out-C4 learning-rate selection, retaining bits, capacity, and 500 updates. All columns use a shared new protocol; RTN-start BART reproduces the submitted-style configuration. Entries are token CVaR5.

W4 modelGPTQRTN-start BARTGPTQ-start BART + LR selectionChange versus GPTQ [95% CI]
Qwen2.5-1.5B0.7450.9040.718−3.68% [−5.24, −2.08]
Qwen2.5-3B0.6850.8960.655−4.39% [−5.58, −3.17]
Qwen2.5-7B0.5730.5920.542−5.53% [−6.76, −4.30]

The refined protocol improves the tail in all three tested W4 models while preserving their bit budget and training capacity. Token p99 also decreases in all three models, and mean harmful-flip rates across six tasks change 5.65/4.19/4.99%→5.24/3.82/4.17%. Paired-window intervals quantify the improvement for these fixed runs; the tested change combines initialization/grid and learning-rate selection. This provides a concrete way to build on a strong PTQ starting point and improve both its tail and task reliability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.