- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why quantization, despite degrading overall model performance, can improve predictions on some individual examples. The authors attribute these improvements to quantization-induced activation changes that align with directions reducing the target loss, rather than to smaller perturbation magnitude. They further study these changes through activation interventions and shared-subspace analysis. Finally, they train a lightweight activation modifier to predict beneficial adjustments without requiring target-loss information at inference time.
Critical strengths
- The paper investigates an interesting phenomenon. Quantization can improve individual predictions despite degrading aggregate performance. It offers an input-level examination of such cases, including an analysis of the direction and functional effect of quantization-induced activation changes.
Critical weaknesses
-
The analysis in Sections 5–7 is overly fragmented and notation-heavy. A relatively simple central observation is expanded into many overlapping decompositions, metrics, and intervention variants, without clearly stating the distinct insight provided by each. Consolidating the main argument and moving secondary diagnostics to the appendix would substantially improve readability.
-
The empirical validation is too limited to establish the generality of the reported findings. The study evaluates only two models and a small number of tasks, while most mechanistic and adaptation experiments are conducted on a single 4-bit GPTQ model. More recent representative methods, such as OmniQuant and FlatQuant, are not evaluated, and no evidence is provided for more aggressive 2-bit or 3-bit settings. It therefore remains unclear whether the findings generalize beyond the specific models, tasks, and quantization configuration studied.
Questions for authors
- Can the authors provide results on additional models, quantization methods, or bit-widths to demonstrate the generality of the findings?
3 comments
Thank you for the detailed rebuttal and the additional experiments. The new results significantly strengthen the evidence and improve the generality of the findings. I also appreciate the clarification of section 5–7. The rebuttal addresses my main concerns. I will increase my rating to 3.
Thank you for your recognition of our work!
Clarifications and Generalization Across Quantizers, Models, and Tasks
Thank you for the suggestions. We clarify Sections 5–7 and report added tasks, quantizers and Qwen analyses.
W1. Organization and interpretation of Sections 5–7
We will retain the sequence and add this roadmap at the start of Section 5:
We will clarify the reference points: Sections 5–6 compare complete FP/Q executions, including propagated changes, using FP-reference gradients. Section 7 instead fixes the Q gate input and studies local error e=(WQ−WFP)xQ, using the target-loss gradient evaluated at the quantized gate output. Shared energy measures the fraction of adjustment energy in the learned subspace (Eq. 21); retained effect compares total CE reductions from projected and full adjustments (Eq. 22), not accuracy. Full and projected changes use the same scale (Eq. 19). Section 7 uses the correct answer to compute diagnostic gradients; the Section 8 modifier requires neither the answer nor its gradient at inference. We will consolidate notation, specify denominators in captions, and move fitting details to the appendix while retaining the definitions.
W2 / Q1. Generality across quantizers, precision, models and tasks
Quantization configurations. We evaluated (i) Llama-3-8B GPTQ W3, an actual packed 3-bit checkpoint (group128, activation-order enabled); (ii) Llama OmniQuant W4 with learned weight clipping (LWC, 20 epochs, 128 FineWeb calibration documents); and (iii) Qwen2.5-7B-Instruct GPTQ4 with expanded mechanism tests.
We evaluated five-shot MMLU questions and FineWeb documents, followed by Top-128 restoration, same-sign random and contribution-matched controls:
On FineWeb, W3/OmniQuant/Qwen improved 7/48/418 documents; restoration reversed 5/7, 42/48, 361/418 improvements, respectively. W3 has few improved documents.
Additional tasks. With Llama GPTQ4, we evaluated BoolQand ARC-Challenge. Despite lower overall Q accuracy, quantization corrected 164/49 predictions on BoolQ/ARC (with 237/94 newly wrong answers). Top-128 restoration reversed 157/164 (95.73%) / 44/49 (89.80%), versus 22.36%/15.65% for random and 95.12%/86.39% for contribution-matched controls.
Sharing across models and tasks. We also completed the original Eq. (17) rank-eight S-basis diagnostic on Qwen BoolQ/ARC, in addition to HANS/MMLU. Retained fractions of the full CE reduction are:
These are gold-gradient diagnostics, not modifier accuracies. Sharing depends on task and model: Qwen MMLU and ARC are both four-choice, yet retain 18.89% and 52.95%.