- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates capability-specific failures under low-bit weight quantization and proposes Pinpoint Quantization (PinQ), a mixed-precision allocation method guided by intervention-based sensitivity scores. The authors localize quantization-sensitive modules, characterize perturbations in attention and MLP outputs, and use a knapsack formulation to assign matrix-level precision under a global budget.
Critical strengths
- The paper addresses an important limitation of aggregate evaluation: small changes in average accuracy can conceal substantial degradation in individual capabilities.
- The bidirectional intervention experiments provide useful evidence that the identified modules contribute to quantization-induced failures. Comparisons with alternative ranking criteria strengthen the localization results.
- The evaluation spans multiple model scales and both dense and MoE architectures. The inclusion of custom inference kernels and serving measurements is also valuable.
- The paper is well-written and the presentation is clear.
Critical weaknesses
- The practical advantage over optimized INT4 remains unproven. Reducing nominal average precision slightly below four bits does not necessarily yield a meaningful deployment benefit. For example, 3.6 bits represents only a nominal 10% reduction in weight storage relative to four bits, before accounting for overhead. Whether this saving justifies mixed-precision execution depends on actual memory usage, kernel efficiency, and serving performance. The reported efficiency comparison is against FP16, rather than optimized INT4, and therefore does not establish the practical value of the proposed allocation.
- The precision accounting is incomplete for deployment-oriented comparisons. The reported average bit widths exclude packing padding, quantization metadata, and parameters retained in FP16. The custom 3-bit format also requires 3.2 packed bits per weight before scales and zero-points. These costs are particularly consequential when the claimed savings amount to a fraction of a bit. Actual checkpoint size and resident memory should be reported consistently across methods.
- The mechanistic conclusions require further validation. The interventions establish that selected modules causally affect predictions, but do not fully establish attention direction shifts and MLP magnitude suppression as the explanations for capability loss. LogitLens offers an approximate interpretation of intermediate representations, and changes in decoded tokens do not uniquely identify a module’s computational role. The direction- and magnitude-repair experiments are helpful, but broader quantitative evidence is needed to support the proposed distinction.
- The diagnostic protocol limits generalization. The mechanistic analysis focuses on prediction-flipped instances generated from twenty templates, with an instance-level calibration/test split. This supports conclusions about selected quantization failures, but does not establish transfer to unseen template families or naturally occurring benchmark errors. Furthermore, “tail” denotes chosen stress-test settings rather than measured rarity in the training distribution; the broader head–tail interpretation should therefore be qualified.
- The connection between the mechanistic analysis and the allocation method is indirect. PinQ recomputes sensitivity scores at matrix granularity and optimizes an additive KL-divergence surrogate. It does not directly incorporate the distinction between attention direction shifts and MLP magnitude suppression. The contribution of the mechanistic findings to the final algorithm is therefore unclear beyond motivating sensitivity-based allocation. In addition, the surrogate does not capture interactions among jointly quantized matrices.
- Aggregate accuracy does not establish preservation of fragile capabilities. Although the average results are encouraging, individual regressions remain. For example, Qwen3-235B-A22B drops from 35.0 to 33.0 on LongBenchV2. Given the paper’s emphasis on failures obscured by aggregate metrics, the proposed method should be evaluated more explicitly in terms of capability-specific regressions and statistical uncertainty.
Questions for authors
- How does PinQ compare with optimized INT4 in actual checkpoint size, resident memory, throughput, and latency under matched conditions? Do the fractional-bit savings enable deployment on fewer devices or another concrete benefit?
- Do the sensitivity rankings and proposed perturbation mechanisms generalize to unseen templates, natural benchmark failures, and stronger underlying quantizers?
- What additional benefit does the mechanistic characterization provide beyond matrix-level sensitivity scoring, and how well does the additive objective predict the effects of joint quantization?
4 comments
Response to Reviewer G6KH Part2
Generalization across unseen templates, natural failures, and stronger quantizers (W4, Q2)
Thank you for these thoughtful questions on generalization.
- Transfer of localized components to unseen templates. Complementing the original instance-level split, we evaluate four additional diagnostic templates excluded from module localization (Table 3). We keep the originally localized components fixed and apply repairs without re-localization on the new templates. Attention-direction repair and MLP-magnitude repair recover an average of 25.0% and 27.0% of the FP16–INT3 NLL gap, respectively. These results support the cross-template utility of the localized components beyond held-out instances of the original templates.
- Transfer to an unseen natural benchmark. We further evaluate on GSM8K, which was not used for module localization, keeping the originally localized components fixed without re-localization. Attention-direction repair recovers 34.7% of the FP16–INT3 NLL gap, compared with 8.5% for attention-magnitude repair; MLP-magnitude repair recovers 29.8%, compared with 5.2% for MLP-direction repair. These results support transfer of the localized components to a natural benchmark beyond the diagnostic templates, with the same direction–magnitude repair contrast.
- Operational scope of “tail capabilities.” We clarify that “tail capabilities” serves strictly as an operational label for the evaluated stress-test conditions (niche languages, long contexts, complex facts, and higher-hop reasoning) rather than measured pretraining-corpus rarity.
- Evidence and scope across quantizers. Our experiments compared PinQ under different underlying quantizers (main-paper Table 5a), providing evidence that its downstream effectiveness is not restricted to a single quantizer. However, this comparison does not directly establish that sensitivity rankings transfer across quantizers or that the same direction–magnitude repair contrast persists. We will distinguish the cross-quantizer performance evidence from the scope of the mechanistic analysis.
Table 3. Transfer to four unseen diagnostic templates using repair targets localized solely on the original templates (% of FP16–INT3 NLL gap recovered).
| Transferred repair | NLL gap recovered (%) |
|---|---|
| Attention direction | 25.0 |
| MLP magnitude | 27.0 |
| Both | 47.0 |
Role of mechanistic analysis and scope of the additive allocation surrogate (W5, Q3)
PinQ indeed uses a unified matrix-level allocation objective rather than separate rules for attention and MLPs. However, the mechanistic findings in Section 4 directly inform our scoring criterion and the resulting bit distribution in Section 5, while our joint-intervention experiments validate the effectiveness of the additive surrogate:
- How the mechanistic findings shape PinQ’s scoring criterion. Section 4 yields two actionable insights that directly guide PinQ: (1) quantization sensitivity is highly localized to a small fraction of modules (<10% of modules drive over 80% of the accuracy drop; Figure 6), motivating selective precision protection; and (2) because attention degrades via norm-preserving direction shifts while MLPs degrade via magnitude suppression, magnitude-based proxies (e.g., activation norms: 44.7% recovery in main-paper Table 2) fail to capture both modes, whereas intervention-based output KL evaluates both on a common scale (80.9% recovery). PinQ directly transfers this causal scoring principle to bit allocation.
- What PinQ simplifies for deployment. Although attention- and MLP-level patching explains how internal representations degrade, sensitivity also shifts across tasks and layers (Figures 3 and 5e) and does not prescribe a fixed module-type rule. For hardware-friendly deployment (supporting vLLM GEMM kernels), PinQ elevates the intervention unit from heads/neurons to complete weight matrices and optimizes the global bit allocation via dynamic programming.
Response to Reviewer G6KH Part3
- PinQ naturally assigns different bit-widths to attention and MLPs. Even without separate allocation rules for each module type, the causal-effect scores naturally assign different precisions to attention and MLP matrices. As shown in Appendix G.1 (Figures 12–13), PinQ predominantly assigns 6–8 bits to attention projections across layers while compressing many middle-layer MoE expert projections to 2–3 bits and protecting boundary layers.
- How well the additive surrogate predicts joint quantization. Unlike local weight-error proxies, each score measures the output KL after propagating the quantization error of matrix at its assigned bit width through the full model. While summing omits higher-order cross-matrix interactions so that DP can solve the global budget, our bidirectional joint interventions (Figure 6 and main-paper Table 1) show that isolated scores accurately capture joint criticality: jointly restoring the top-10% modules ranked by single-module scores recovers 80.9%–91.3% of the full-model INT3 drop across 4B–235B models, and jointly quantizing only that subset reproduces 82.2%–93.8% of the total loss. Moreover, globally optimizing this additive objective via DP outperforms greedy allocation by +5.34 MMLU points at 3.0 bits (main-paper Table 5b).
Capability-level preservation, 235B LongBenchV2 breakdown, and statistical scope (W6)
We thank the reviewer for emphasizing capability-resolved evaluation and statistical uncertainty. Following Figure 2’s protocol, Tables 4–5 below break down recovery across individual languages, context lengths, ARC subsets, and reasoning hops, complemented by a LongBenchV2 subset analysis and five-run uncertainty estimates:
- Fine-grained recovery across all four fragile capabilities (Tables 4–5). PinQ (3.8 bits) consistently outperforms uniform INT4 across all evaluated sub-regimes on Qwen3-4B: (1) it narrows the FP16 gap across all 10 MGSM languages (e.g., from 10.8 to 3.3 points on Vietnamese and 5.6 to 2.2 on Japanese); (2) it tracks FP16 across RULER lengths up to 128k (65.3% → 70.3%, vs. 71.6% FP16); (3) it restores ARC-Challenge (69.8% → 75.0%, vs. 75.6% FP16); and (4) it nearly halves the INT4 gap on 2-hop (5.0 → 2.6 points) and 3-hop (5.9 → 2.9 points) TriviaQA.
- Additional LongBenchV2 analysis. Under an aggressive 3.3-bit budget (0.7 bits below INT4), overall LongBenchV2 is 33.0% vs. 34.3% (INT4) and 35.0% (FP16), but evaluating its official Long subset (longest context regime) yields 29.6% (FP16), 28.0% (INT4), and 28.7% (PinQ). Thus, while the tight 3.3-bit budget trades off accuracy on shorter/medium subsets, PinQ still better preserves the longest context regime.
- Statistical uncertainty across five repeated runs. Evaluating Qwen3-4B across five repeated runs over the five benchmarks covering the fragile capabilities, PinQ’s mean improvements consistently exceed run-to-run standard deviations (SD ≤ 0.28% across tasks), raising the macro-average from 63.16 ± 0.28% (INT4) to 65.92 ± 0.20% (PinQ), compared with 67.01 ± 0.18% for FP16 (72% gap recovery). Mean gains are largest on RULER-128K (+5.12 points) and ARC-Challenge (+4.16), followed by MGSM (+2.44), TriviaQA (+1.06), and ARC-Easy (+0.98), with modest residual gaps to FP16 on MGSM (1.74) and TriviaQA (1.58).
Table 4. Language-specific accuracy gaps (percentage points relative to FP16, ↓) on MGSM.
| Metric | Vietnamese | Korean | Japanese | Portuguese | French | Arabic | Thai | Spanish | Chinese | English |
|---|---|---|---|---|---|---|---|---|---|---|
| INT4 gap | 10.8 | 5.6 | 5.6 | 5.2 | 4.4 | 4.0 | 4.0 | 3.2 | 0.8 | 0.8 |
| PinQ gap | 3.3 | 2.8 | 2.2 | 2.0 | 1.9 | 1.4 | 0.9 | 0.6 | 0.2 | 0.1 |
Table 5. Capability-resolved accuracy (%) across context lengths, ARC difficulty subsets, and TriviaQA reasoning hops.
| Benchmark | Condition | FP16 | INT4 | PinQ |
|---|---|---|---|---|
| RULER | 4k | 90.2 | 89.7 | 90.0 |
| RULER | 8k | 85.6 | 85.6 | 85.6 |
| RULER | 16k | 84.3 | 83.8 | 84.1 |
| RULER | 32k | 83.3 | 81.6 | 82.9 |
| RULER | 64k | 77.9 | 75.4 | 77.3 |
| RULER | 128k | 71.6 | 65.3 | 70.3 |
| ARC | Easy | 84.3 | 83.3 | 84.5 |
| ARC | Challenge | 75.6 | 69.8 | 75.0 |
| TriviaQA | 1-hop | 45.6 | 44.3 | 44.5 |
| TriviaQA | 2-hop | 37.7 | 32.7 | 35.1 |
| TriviaQA | 3-hop | 29.1 | 23.2 | 26.2 |
Dear Reviewer G6KH,
We really appreciate your supportive feedback and thoughtful comments on our work. With the discussion deadline (Nov 18 AoE) coming up soon, we wanted to check whether our responses (Parts 1–3) and the new results in Appendix A have resolved your concerns. We are happy to clarify any remaining points before the window closes!
Best regards,
The Authors
Response to Reviewer G6KH Part1
Thank you for your constructive review and for requesting matched deployment comparisons against optimized INT4, complete storage accounting, template-transfer validation, and a formal discussion of our additive objective.
Serving performance and complete storage accounting vs. optimized INT4 (W1, W2, Q1)
Thank you for highlighting these practical deployment considerations. We supplement the original FP16 comparison with a direct comparison against optimized INT4. Table 1 reports checkpoint size, memory footprint, and serving performance for Qwen3-235B-A22B under matched vLLM conditions: eight NVIDIA A100 80GB GPUs, 512-token inputs, 128-token outputs, and concurrency 128. All storage and memory footprints are reported in GiB and include packing padding, quantization metadata, and parameters retained in FP16.
Table 1. Storage, memory, and serving performance of Qwen3-235B-A22B under matched conditions (8×A100 80GB, 512-token input, 128-token output, concurrency 128). Checkpoint size and memory are reported in GiB and include padding, quantization metadata, and FP16 parameters.
Quantitative validation of perturbation mechanisms and role of LogitLens (W3)
We agree that establishing a module’s causal relevance is not sufficient to establish which aspect of its perturbation contributes to capability loss. We therefore strengthen Section 4 with symmetric quantitative measurements, module-type-specific repairs, and a clearer statement of the role and limitations of LogitLens:
Table 2. Bold entries denote newly added symmetric measurements; relative norm change is normalized by each vector’s FP16 norm.