- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies example-level reliability under low-bit quantization of large language models, emphasizing that average perplexity and accuracy can conceal upper-tail output divergence, harmful prediction flips, and branching autoregressive trajectories. The authors introduce a perturbation–propagation framework connecting module-level quantization errors to downstream gain, and derive non-asymptotic KL bounds, decision radii within a linearized propagation model, an exact KL barrier for losing the reference top-1 decision, and conditions for preserving greedy trajectories. Experiments span multiple model families, weight and KV-cache quantization, and downstream tasks, with numerical-noise controls and layer-restoration interventions. The paper also proposes PAPA, which allocates precision using single-module output KL on a hard subset, and BART, which improves fixed-bit models through low-rank corrections, learnable clipping, and output distillation. The results support supplementing average quantization metrics with example-level and tail measurements and demonstrate effective mitigation in selected settings. However, the realized directional-propagation mechanism and the independent contribution of the tail-specific design remain insufficiently established.
Critical strengths
S1. The reliability evaluation is broad and includes evidence of actual task regressions. The paper goes beyond changes in token distributions and generated text by reporting harmful flips on zero-shot tasks, GSM8K, code execution, and instruction following. Tables 19 and 21 clearly distinguish changes in average accuracy from example-level regressions, while Table 14 extends the analysis to deployment-relevant tasks. The FP numerical-perturbation controls in Table 6 further help distinguish quantization effects from ordinary numerical sensitivity. These results give the reliability problem practical significance beyond disagreement with a reference model. S2. The theoretical connection between output distributions and discrete decisions is clear. Theorem 2 gives the exact minimum forward KL required for the reference model to lose its unique top-1 decision, correctly distinguishing preservation below the barrier from the absence of a guaranteed flip above it. Theorem 4 and Corollaries 5–6 further connect linearized source perturbations, decision margins, and greedy trajectories. The derivations in Appendix F provide concrete analytical tools linking continuous distributional discrepancies to discrete behavior and constitute a valuable aspect of the paper. S3. The interventions demonstrate practical value, and the paper reports relevant limitations. Table 1 shows that restoring four highly ranked layers in Qwen2.5-3B RTN-W3 recovers 83.1% of the p99 KL gap. Table 11 shows that BART increases GSM8K accuracy from 0.030 to 0.350 at the same 3-bit precision. The authors also report negative results, including competitive random layer selection on Pythia, PAPA’s lack of advantage on RTN, and degradation from BART in some W4 settings. These experiments demonstrate practical value for the proposed diagnostics and interventions while providing useful evidence about their scope.
Critical weaknesses
W1. Validation of realized directional error propagation remains indirect, weakening the claimed end-to-end mechanistic account. The central quantity in Eq. (1) is , propagated along the realized quantization-error direction for each input. However, Appendix H.4 explains that the layer-restoration score uses gain averaged over random directions at the block input. It neither preserves the realized error direction nor matches the injection site of the block-output residual. The Fisher checks in Tables 7–8 use the measured true logit difference, validating an output-level approximation rather than the source-level propagation prediction and its remainder. Table 1 supports the intervention utility of the ranking score, but does not by itself establish the proposed directional explanation. Consequently, the paper’s central empirical connection from realized error sources to decisions remains insufficiently direct, particularly for tail examples and harmful flips. W2. The independent contribution of the tail-specific design over general output-fidelity optimization is not sufficiently isolated. Tables 16–18 compare several local and curvature-based scores, but omit the most direct PAPA control: single-module mean output KL over all calibration inputs, with every other setting unchanged. The current results therefore do not separate the benefit of output sensitivity from that of tail conditioning. For BART, the authors explicitly acknowledge that most gains come from the mean-KD core with . The additional CVaR term produces small and inconsistent benefits in Table 9; for example, on Qwen2.5-3B W3, CVaR changes from 3.31 with BART to 3.36 with BART. Given existing work on output-aware quantization, the paper needs to identify more clearly what the tail-specific design adds beyond general output-fidelity optimization. W3. The connection between distributional-tail improvement and recovery of task reliability is not systematically established. The harmful-flip analyses are valuable, but the main layer-restoration and PAPA intervention results are evaluated using KL metrics. For BART, the distributional and task metrics in Table 9 come from different datasets, and their changes sometimes disagree. For example, Mistral-7B W4 shows substantial task-accuracy recovery while its WikiText-2 p99 increases from 0.40 to 0.52; BART on Qwen2.5-3B W4 increases both p99 and the harmful-flip rate. These results do not invalidate the methods’ utility, but they show that improving the KL tail cannot simply be equated with reducing task failures. Supporting the complete reliability chain requires examining risk predictions, intervention-induced distributional changes, and recovery from harmful flips on the same labeled examples.
Questions for authors
1.Could you evaluate the propagation prediction along actual module quantization-error directions in one representative W4 setting, reporting its agreement with the true logit/KL changes and the magnitude of the remainder separately for ordinary and tail examples? Under the same layer-restoration budget, how does this realized-direction score compare with the current random-direction proxy measured at block inputs? This would directly affect my assessment of the mechanistic contribution. 2.With the same proxy quantizer, calibration sweep, and achieved bit budget, how does PAPA compare with single-module mean KL computed over all calibration inputs? In light of the inconsistent gains of BART over mean KD, which results can be attributed specifically to the tail-oriented design rather than general output-KL optimization? 3.Are OMNIQ’s correction-magnitude projection, clipping floor, and weight decay also applied to BART? If not, does BART retain its advantage in Table 10 when both objectives use the same constraints? Please specify which training settings are identical and which are exclusive to the local-reconstruction control.
4 comments
W2 / Q2. Tail conditioning has an isolated, measurable effect
The added ablation isolates the contribution of selecting high-risk calibration inputs. It extends the submitted local-error and curvature comparisons with the direct all-input component-KL control requested by the reviewer (Kong et al., https://arxiv.org/abs/2604.13440). Only the aggregation of the same single-module RTN-W3 output-KL array changes; proxy quantizer, calibration sweep, deployed candidates, allocator, and achieved 3.500-bit decoder-weight budget are fixed. Any resulting allocation difference therefore comes from which inputs determine module importance.
The experiment covers seven rotated-GPTQ models—Qwen2.5-1.5B/3B/7B, Llama-3.2-1B/3B, Llama-3.1-8B, and Mistral-7B—with three independent draws each, 48 C4-512 calibration windows, and 160 held-out WikiText-2 windows. The implemented subset rule selects windows, or eight of 48; we have corrected Section 4.1's “10%” description.
CVaR5/CVaR1 averages the worst 5%/1% of token KL. Confident tokens have FP top-1 probability at least 0.9. Entries are relative changes with 95% t-intervals across seven model means, each averaging its three paired draws.
| PAPA compared with | Token CVaR5 change | Token CVaR1 change | Confident-FP token flip-rate change |
|---|---|---|---|
| All-input mean KL (FP→Q) | −3.34% [−7.07, +0.39] | −4.69% [−8.59, −0.79] | −8.34% [−15.19, −1.50] |
| All-input mean KL (Q→FP) | −2.83% [−6.02, +0.37] | −3.85% [−6.95, −0.75] | −7.46% [−13.63, −1.30] |
At matched resources, tail conditioning reduces extreme output errors and preserves more confident FP decisions. Both forward- and reverse-KL controls show this gain in worst-1% KL and confident-token flips. Against forward mean KL, mean KL changes +0.20% [−1.58, +1.98], while the broader CVaR5 interval includes zero. The controlled result identifies a reliability benefit that the near-zero change in average KL does not reveal.
An equal-size subset ablation separates input selection from subset size using 48 C4-256 windows on rotated Qwen2.5-3B. CVaR5 at context 512 is 1.597±0.020 for all inputs, 1.574±0.020 for random eight-input subsets, and 1.511±0.031 for the hardest eight. At context 2,048: 1.309±0.023 / 1.294±0.022 / 1.250±0.010. Values are mean ± sample SD over three draws, averaging four random subsets per draw. Selecting the hardest inputs improves over random selection in all three draws at both contexts. The separate plain-GPTQ 1.5B two-draw control gives 2.155 versus 2.222 CVaR5 for mean KL versus PAPA. Together with the submitted cross-base comparisons, these controls identify both the additional value of input selection on rotated GPTQ and the role of the quantizer in choosing an allocation rule.
BART tests the theory's correction principle: use limited correction capacity to remove errors that matter at the output. The Q3 ablation holds capacity, training constraints, and auxiliary supervision fixed while changing the correction objective. CVaR is disabled by design to isolate this mechanism. Output KL improves both the KL tail and correct-word preservation under these controls, establishing the reliability value of the propagated correction target.
A separate ablation measures the incremental effect of adding CVaR to that core: 3.470±0.071→3.462±0.053 token CVaR5 and 21.35±0.07%→20.60±1.41% harm over two seeds. The harm-reduction interval [−0.40, 1.95] points includes zero; logs show the threshold can lag falling KL. This separates the established benefit of the output objective from the additional effect of explicitly reweighting the tail.
W3. Risk, distributional change, and recovered answers on the same items
The paired tests connect predicted risk to actual loss and recovery of correct answers. We measure risk, distributional change, and correctness on all 1,000 fixed LAMBADA words and all 1,283 target tokens. Correctness requires every token of a word to be correct; harm is FP-correct→quantized-wrong divided by all 1,000 words. Word KL averages forward KL over that word's target prefixes; word CVaR5 averages the worst 5% of words.
The risk score measures how large the predicted output change is relative to the original decision's safety margin. A close contest between tokens can be overturned by a small change; a clear winner can withstand more. For each token, we divide the predicted KL change by the minimum KL needed to remove the FP winner's lead, then sum over the word:
Here is the KL predicted from internal errors and is the FP decision barrier. The score is fixed before evaluating outcomes and uses only FP derivatives and local errors—no measured W4 logits or fitted classifier. Of 673 FP-correct words, W4 loses 52:
| Frozen predictor | Harmful-word AUROC [95% CI] |
|---|---|
| Inverse FP barrier | 0.824 [0.757, 0.882] |
| Local energy / FP barrier | 0.837 [0.779, 0.888] |
| Source-predicted KL alone | 0.822 [0.765, 0.875] |
| Source-predicted KL / FP barrier | 0.916 [0.888, 0.941] |
Among FP-correct words, the score ranks a lost answer above a retained answer in about 91.6% of randomly selected pairs. This is the meaning of AUROC 0.916. Combining predicted change with decision tolerance gains 0.092 AUROC [paired 95% CI 0.043, 0.150] over the barrier alone; either ingredient alone scores about 0.82. The internal-error prediction therefore adds substantial information about which correct answers quantization will lose.
Allocation and training also use these same 1,000 item IDs. The W3 runs use BF16/SDPA, whose FP reference gets 674 words correct. Values are mean ± sample SD over two seeds.
| Intervention | Decoder bits | Mean word KL ↓ | Word CVaR5 ↓ | Word accuracy ↑ | Harmful words ↓ |
|---|---|---|---|---|---|
| Uniform GPTQ-W3 | 3.0 | 0.655 ± 0.005 | 2.753 ± 0.011 | 44.40 ± 0.14% | 26.00 ± 0.14% |
| All-input mean-KL allocation | 3.5 | 0.402 ± 0.015 | 2.046 ± 0.093 | 51.75 ± 0.07% | 19.05 ± 0.21% |
| PAPA, same allocation budget | 3.5 | 0.358 ± 0.003 | 1.924 ± 0.145 | 53.80 ± 1.13% | 17.40 ± 0.85% |
| Local reconstruction + 0.1 NLL | 3.0 | 0.723 ± 0.013 | 3.263 ± 0.013 | 46.00 ± 1.84% | 25.70 ± 1.98% |
| Output KL + 0.1 NLL, same constraints | 3.0 | 0.568 ± 0.015 | 2.752 ± 0.181 | 50.45 ± 0.21% | 21.50 ± 0.42% |
PAPA preserves more correct answers at the same 3.5-bit budget: harm falls by 1.65 points [paired-item 95% CI 0.30, 3.00], or 8.7%, and accuracy rises by 2.05 points [0.50, 3.60]. Harm falls 192→180 / 189→168 across seeds; rescues/new harms are 57/45 / 52/31. Mean word KL also falls; the word-CVaR5 reduction interval includes zero. At identical W3 budget and NLL, output KL reduces harm by 4.20 points [2.70, 5.70]. These are direct measurements of correct-word preservation on the same items. Intervals jointly resample word IDs across fitted runs; SD separately describes run variation.
An HQQ-W3 experiment tracks both outcomes on 512 identical LAMBADA first-token prefixes. Every layer set is frozen on disjoint C4 text and transferred from RTN-W3:
| Four-layer W3→W4 selection | Mean decision KL ↓ | Decision CVaR5 ↓ | Accuracy ↑ | Harmful cases ↓ | Rescued / newly harmed |
|---|---|---|---|---|---|
| No restoration | 3.221 | 9.157 | 16.80% | 267 | — |
| Original random-input proxy | 0.713 | 2.712 | 45.31% | 133 | 145 / 11 |
| Local residual magnitude | 0.731 | 3.015 | 45.51% | 133 | 145 / 11 |
| Realized-direction propagation | 0.642 | 2.537 | 49.41% | 118 | 158 / 9 |
Realized-direction repair removes 55.8% of original harms, counting new harms, at 3.000→3.111 decoder bits. Its accuracy gain over the original proxy is 4.10 points [paired-item 95% CI 0.98, 7.42]. It captures 65.1% [58.2, 71.8] of uniform-W4's harm reduction using 11.1% of its added weight bits. On the fixed original worst-5% subset, KL falls 9.157→0.574 and harm 25→2; recovered cases stay in the evaluation.
The paired W4 repairs also make the role of the decision barrier concrete. On the exact model/population used for risk prediction, C4-selected four-layer BF16 repairs change harm 52→48/51/53 for original-proxy/realized-direction/local-magnitude selection; mean word KL changes 0.0899→0.0734/0.0754/0.0816. All three lower average KL, while their retained correct answers differ, with the original proxy giving the largest harm reduction here. Measuring error relative to the decision barrier adds the information needed to assess whether an output change threatens the answer. The risk test above quantifies that information, and the paired repairs measure the resulting correctness directly.
A controlled input-format test also clarifies the submitted Mistral discrepancy. Fixing GPTQ-W4 weights and 64 WikiText-2 windows while adding the zero-shot prompts' BOS token changes p99 0.42→6.27 and disagreement 8.9%→34.8%. Including BOS in half the calibration windows at the same budget reduces zero-shot harm 38.3%→14.5% in the rerun. This identifies a consequential format sensitivity and an effective calibration change that BOS-free evaluation would miss.
For Qwen2.5-3B W4, a separate GPTQ-start BART refinement with held-out-C4 learning-rate selection retains bits, capacity, and 500 updates. CVaR5 falls 0.685→0.655 (−4.39% [paired-window 95% CI −5.58, −3.17]), p99 0.848→0.811, and mean task harm 4.19%→3.82%. This joint test of initialization/grid and learning-rate selection demonstrates improved tail and task reliability while preserving the deployment and training budgets.
The common logic is output error relative to the same input's decision barrier. The source score predicts which correct answers are at risk; the controlled allocation and training experiments measure how many correct answers each intervention preserves. Testing both on the same inputs gives the propagation framework a concrete reliability outcome.
Q3. Controlled ablations isolate the reliability gain from BART's output objective
We add a factorial ablation designed to separate the correction objective from auxiliary supervision and stabilization settings. The submitted Table 10 matched correction capacity, data, and update budget; the local control additionally used the stabilization settings listed in the appendix. The new experiment applies those settings identically to both objectives and crosses local reconstruction/output KL with NLL weights 0 and 0.1. NLL trains on observed next tokens; matching it tests the output objective at the same level of auxiliary supervision. Both objectives use:
- Qwen2.5-3B W3, group size 128, rank 16, alpha 32, and 30,965,760 trained parameters.
- The same 256 C4 sequences × 512 tokens, initialization, sample order within each seed, and 500 updates.
- AdamW with betas (0.9, 0.95), identical learning rates and cosine schedule, gradient clipping 1.0, and weight decay 0.001.
- The same correction-row norm cap (0.2× base-row norm) and clipping floor (0.5). CVaR is disabled.
Within each NLL setting, only the correction objective changes: matching intermediate hidden states versus matching final token probabilities. Disabling CVaR isolates this difference from extra tail weighting. Values are mean ± sample SD over two seeds; word metrics use the 1,000 complete targets above.
| Objective | NLL weight | PPL ↓ | Token CVaR5 ↓ | Word accuracy ↑ | Harmful words ↓ |
|---|---|---|---|---|---|
| Cumulative hidden-state reconstruction | 0 | 16.581 ± 0.010 | 4.223 ± 0.065 | 22.65 ± 2.33% | 46.10 ± 2.12% |
| Output KL | 0 | 13.108 ± 0.124 | 3.459 ± 0.050 | 48.80 ± 0.28% | 22.35 ± 0.64% |
| Cumulative hidden-state reconstruction | 0.1 | 14.541 ± 0.187 | 3.862 ± 0.011 | 46.00 ± 1.84% | 25.70 ± 1.98% |
| Output KL | 0.1 | 13.141 ± 0.036 | 3.529 ± 0.159 | 50.45 ± 0.21% | 21.50 ± 0.42% |
The output objective improves both the KL tail and correct-word preservation in both supervision settings. It reduces harmful words by 23.75 points [paired-item 95% CI 21.25, 26.20] without NLL and 4.20 points [2.70, 5.70] with the same 0.1 NLL anchor. Holding the other training choices fixed isolates the contribution of optimizing the downstream output effect.
A second controlled experiment tests the role of accumulated input error by reconstructing each block on its exact FP input. This gives the local objective a clean input at every block. All objectives share the learning-rate search and disjoint C4 validation rule, selecting ; rank, constraints, data, and updates remain fixed, with NLL/CVaR disabled. On 96 WikiText-2 windows of 2,048 tokens and the same 512 LAMBADA first-token prefixes, two-seed mean ± sample SD is:
| Objective | PPL ↓ | Token CVaR5 ↓ | First-token accuracy ↑ | Harmful decisions ↓ |
|---|---|---|---|---|
| Reconstruction on FP block inputs | 14.697 ± 0.296 | 3.552 ± 0.024 | 40.92 ± 0.97% | 29.69 ± 0.28% |
| Cumulative hidden-state reconstruction | 16.759 ± 0.176 | 3.889 ± 0.041 | 21.97 ± 1.80% | 46.09 ± 1.38% |
| Output KL | 12.864 ± 0.052 | 3.022 ± 0.046 | 48.44 ± 0.83% | 22.56 ± 0.41% |
Output KL improves on FP-input reconstruction by 14.9% CVaR5 and 7.13 percentage points in harm [paired-item 95% CI 4.39, 9.86]. These controlled experiments validate the paper's correction principle: at fixed correction capacity, optimizing the errors' final output effect preserves more correct decisions. The result holds with matched constraints and auxiliary supervision, and against local reconstruction supplied with clean FP inputs and the same learning-rate search. The revised manuscript presents these controls in main-text Table 5 and Appendix G.5 (Table 17), and same-item risk/recovery evidence in Section 3.3 and Table 4. The original Table 10 is retained as a scale/capacity comparison in revised Table 15.
Thank you for specifying the decisive experiments. The added tests connect actual internal errors → predicted decision risk → recovery of correct answers. Preserving the actual error direction predicts output changes and token flips; selecting high-risk calibration inputs preserves more correct words at identical bits; and optimizing output KL improves tail and task reliability under matched training constraints. Together, these controlled experiments establish how the paper's propagation and decision framework guides effective quantization repair.
W1 / Q1. Actual module errors predict output changes and decisions
We directly predict the output effect of an actual internal quantization error. For each module, δm=(WmQ−Wm)hm measures its output error on the FP input. We propagate this error from the module output, preserving its direction and injection site, using the remaining network's derivative: vm=Gmδm. Adding it to FP logits z predicts softmax(z+vm). An independent single-module quantization run measures Δzm and the remainder rm=Δzm−vm. Thus the internal error supplies the prediction and the quantized run tests it. This directly complements the submitted output-level Fisher checks in Tables 7–8 and restoration evidence in Table 1.
The Qwen2.5-3B GPTQ-W4 test covers q/k/v/o/gate/up/down projections at layers 0, 1, 12, 24, and 35: 35 modules × 32 held-out 128-token windows × 63 positions = 70,560 cases. Computation uses FP32/eager attention, with constant logit shifts removed. The tail is the largest 5% of actual single-source KL; relative RMS error is RMS remainder divided by RMS actual change.
The predictor finds 516 of 538 actual flips, with 30 false positives: 94.5% precision / 95.9% recall. Tail precision/recall is 93.3%/94.3%. It predicts the exact new winner in 98 of 104 harmful source flips.
At fixed norm and injection site, three random directions give median within-module tail correlation 0.536, versus 0.976 for actual directions. The predicted top 5% retrieves 96.3% of the actual tail [95% CI 95.0, 96.8], versus 27.6% for calibrated local energy. Whole-window resampling preserves token/module dependence. Tail correlations remain 0.972/0.981 at 512/2,048-token contexts.
The all-block composition test measures how source effects combine in a complete quantized network: summed FP-linear predictions have overall/tail correlations 0.910/0.537 and relative RMS remainders 0.735/0.873. These remainders quantify departures from additive source predictions when errors coexist, as retained explicitly in the theory. PAPA uses measured finite single-module output effects for scoring, and its allocation gains are evaluated on complete quantized models with interactions present. The experiments therefore test source prediction, composition error, and full-model repair separately.
We also add a controlled comparison of the submitted proxy and the realized-direction score. All restoration rankings use the same eight disjoint C4 windows and GPTQ-W4 block residuals. The original proxy averages three random block-input directions; the realized score propagates the block-output residual. Each restores four blocks to BF16 at identical storage. Evaluation uses 128 held-out WikiText-2-256 windows.
At identical restoration cost, actual-direction selection improves both tail metrics over the submitted proxy: 0.059 p99 KL [paired-window 95% CI 0.037, 0.089] and 0.042 CVaR5 [0.028, 0.058]. It also gives the lowest point estimates in the table; the p99 difference from local magnitude is 0.007 [−0.019, 0.034]. On the accompanying 512 labeled W4 prefixes, all three rules reduce harm 30→29. These experiments connect the fixed-norm direction ablation to a measurable restoration benefit. The paired W3 study below extends this test to recovery of correct answers, where realized-direction selection leaves 118 harmful decisions versus 133 for both the original proxy and local magnitude at the same budget.