acceptodds
Under review as a conference paper at ICLR 2027

Pinpoint Quantization: A Mechanistic Account of Tail-Capability Failures in Quantized LLMs

Abstract

Weight quantization is essential for deploying large language models (LLMs), yet quantization below bits can sharply degrade accuracy. Prior work has primarily characterized this degradation through model behavior or weight statistics, offering limited insight into its internal causes or how to enable targeted mitigation. Instead, we provide a *mechanistic account* of this failure by tracing capability-specific degradation to internal model components. We find that the damage is strikingly selective: it disproportionately impairs ”tail” capabilities (e.g., niche languages, long-context processing, complex factual queries, and multi-hop reasoning) while largely preserving ”head” capabilities. To explain this selectivity, we analyze module function and discover two predominant perturbation mechanisms: attention heads undergo *semantic direction shifts*, whereas MLP neurons undergo *activation-magnitude suppression*. Crucially, the mechanistic fragility is concentrated in a sparse set of key modules: restoring their weight precision recovers most of the accuracy loss on the degraded tasks, suggesting that precise bit allocation, rather than bit width alone, is critical to sub--bit quantization. As a practical application of these findings, we introduce Pinpoint Quantization, which uses causal-effect scores to guide group-knapsack bit allocation. Across benchmarks and dense and MoE models ranging from B to B parameters, it achieves near-lossless accuracy at an average weight precision below bits.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Post-Training Quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_G6KHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates capability-specific failures under low-bit weight quantization and proposes Pinpoint Quantization (PinQ), a mixed-precision allocation method guided by intervention-based sensitivity scores. The authors localize quantization-sensitive modules, characterize perturbations in attention and MLP outputs, and use a knapsack formulation to assign matrix-level precision under a global budget.

Critical strengths
  1. The paper addresses an important limitation of aggregate evaluation: small changes in average accuracy can conceal substantial degradation in individual capabilities.
  2. The bidirectional intervention experiments provide useful evidence that the identified modules contribute to quantization-induced failures. Comparisons with alternative ranking criteria strengthen the localization results.
  3. The evaluation spans multiple model scales and both dense and MoE architectures. The inclusion of custom inference kernels and serving measurements is also valuable.
  4. The paper is well-written and the presentation is clear.
Critical weaknesses
  1. The practical advantage over optimized INT4 remains unproven. Reducing nominal average precision slightly below four bits does not necessarily yield a meaningful deployment benefit. For example, 3.6 bits represents only a nominal 10% reduction in weight storage relative to four bits, before accounting for overhead. Whether this saving justifies mixed-precision execution depends on actual memory usage, kernel efficiency, and serving performance. The reported efficiency comparison is against FP16, rather than optimized INT4, and therefore does not establish the practical value of the proposed allocation.
  2. The precision accounting is incomplete for deployment-oriented comparisons. The reported average bit widths exclude packing padding, quantization metadata, and parameters retained in FP16. The custom 3-bit format also requires 3.2 packed bits per weight before scales and zero-points. These costs are particularly consequential when the claimed savings amount to a fraction of a bit. Actual checkpoint size and resident memory should be reported consistently across methods.
  3. The mechanistic conclusions require further validation. The interventions establish that selected modules causally affect predictions, but do not fully establish attention direction shifts and MLP magnitude suppression as the explanations for capability loss. LogitLens offers an approximate interpretation of intermediate representations, and changes in decoded tokens do not uniquely identify a module’s computational role. The direction- and magnitude-repair experiments are helpful, but broader quantitative evidence is needed to support the proposed distinction.
  4. The diagnostic protocol limits generalization. The mechanistic analysis focuses on prediction-flipped instances generated from twenty templates, with an instance-level calibration/test split. This supports conclusions about selected quantization failures, but does not establish transfer to unseen template families or naturally occurring benchmark errors. Furthermore, “tail” denotes chosen stress-test settings rather than measured rarity in the training distribution; the broader head–tail interpretation should therefore be qualified.
  5. The connection between the mechanistic analysis and the allocation method is indirect. PinQ recomputes sensitivity scores at matrix granularity and optimizes an additive KL-divergence surrogate. It does not directly incorporate the distinction between attention direction shifts and MLP magnitude suppression. The contribution of the mechanistic findings to the final algorithm is therefore unclear beyond motivating sensitivity-based allocation. In addition, the surrogate does not capture interactions among jointly quantized matrices.
  6. Aggregate accuracy does not establish preservation of fragile capabilities. Although the average results are encouraging, individual regressions remain. For example, Qwen3-235B-A22B drops from 35.0 to 33.0 on LongBenchV2. Given the paper’s emphasis on failures obscured by aggregate metrics, the proposed method should be evaluated more explicitly in terms of capability-specific regressions and statistical uncertainty.
Questions for authors
  1. How does PinQ compare with optimized INT4 in actual checkpoint size, resident memory, throughput, and latency under matched conditions? Do the fractional-bit savings enable deployment on fewer devices or another concrete benefit?
  2. Do the sensitivity rankings and proposed perturbation mechanisms generalize to unseen templates, natural benchmark failures, and stronger underlying quantizers?
  3. What additional benefit does the mechanistic characterization provide beyond matrix-level sensitivity scoring, and how well does the additive objective predict the effects of joint quantization?
4 comments
AuthorsOpenReview ↗

Response to Reviewer G6KH Part1

Thank you for your constructive review and for requesting matched deployment comparisons against optimized INT4, complete storage accounting, template-transfer validation, and a formal discussion of our additive objective.

Serving performance and complete storage accounting vs. optimized INT4 (W1, W2, Q1)

Thank you for highlighting these practical deployment considerations. We supplement the original FP16 comparison with a direct comparison against optimized INT4. Table 1 reports checkpoint size, memory footprint, and serving performance for Qwen3-235B-A22B under matched vLLM conditions: eight NVIDIA A100 80GB GPUs, 512-token inputs, 128-token outputs, and concurrency 128. All storage and memory footprints are reported in GiB and include packing padding, quantization metadata, and parameters retained in FP16.

  • Higher score with greater throughput and lower latency at equal cost. Compared with optimized uniform INT4, PinQ increases throughput from 1,228.0 to 1,472.4 tokens/s (19.9% higher), reduces TTFT from 1,820.5 to 1,625.5 ms (10.7% lower), and reduces TPOT from 78.9 to 74.7 ms (5.3% lower). These results demonstrate a concrete deployment benefit: on the same eight-GPU configuration, PinQ serves 19.9% more tokens per second while reducing both TTFT and TPOT. Thus, its practical value extends beyond nominal storage savings to improved serving capacity and responsiveness, without requiring additional devices. On the same model, PinQ also increases the reported 18-benchmark average score from 78.5 to 79.5 (main-paper Table 3), providing higher aggregate accuracy alongside improved serving efficiency.
  • Lower storage and memory requirements after accounting for overheads. The checkpoint size decreases from 116.0 GiB for INT4 to 96.2 GiB for PinQ, a 17.1% reduction. The corresponding memory footprint decreases from 116.4 to 96.8 GiB, a 16.9% reduction. These results demonstrate that the nominal precision reduction translates into practical storage and model-memory savings with quantization overheads included.

Table 1. Storage, memory, and serving performance of Qwen3-235B-A22B under matched conditions (8×A100 80GB, 512-token input, 128-token output, concurrency 128). Checkpoint size and memory are reported in GiB and include padding, quantization metadata, and FP16 parameters.

MethodAvg. bitsMemory (GiB)Checkpoint (GiB)Throughput (tok/s) ↑TTFT (ms) ↓TPOT (ms) ↓
FP1616.0437.9436.91149.03413.583.5
INT44.0116.4116.01228.01820.578.9
PinQ3.396.896.21472.41625.574.7

Quantitative validation of perturbation mechanisms and role of LogitLens (W3)

We agree that establishing a module’s causal relevance is not sufficient to establish which aspect of its perturbation contributes to capability loss. We therefore strengthen Section 4 with symmetric quantitative measurements, module-type-specific repairs, and a clearer statement of the role and limitations of LogitLens:

  • Symmetric, scale-normalized measurements. Using identical metrics for both module types on Qwen3-4B (Table 2), attention-head contributions exhibit a cosine distance of 0.62 and a relative norm change of −5.2%, whereas MLP-neuron contributions exhibit a cosine distance of 0.07 and a relative norm change of −27.3%. These measurements quantify the contrast beyond individual LogitLens examples: attention contributions change more in direction, while MLP contributions decrease more in relative magnitude.
  • Module-type-specific repairs across tasks. We separately apply direction-only and magnitude-only repairs to selected contributions of each module type. On GSM8K, direction-only repair of attention-head contributions recovers 34.7% of the FP16–INT3 NLL gap, compared with 8.5% for magnitude-only repair. For MLP-neuron contributions, magnitude-only repair recovers 29.8%, compared with 5.2% for direction-only repair. These repairs provide evidence not only that the selected modules affect predictions, but that directional changes in attention heads and magnitude suppression in MLP neurons contribute to the observed prediction degradation.
  • Interpretive role and limitations of LogitLens. We agree that decoded token changes alone cannot establish a module’s computational role. We use LogitLens to illustrate how quantization changes the vocabulary-space projections of selected contributions. The proposed direction–magnitude distinction is evaluated through the quantitative measurements and component-specific repairs above, not inferred from LogitLens token changes alone.

Table 2. Bold entries denote newly added symmetric measurements; relative norm change is normalized by each vector’s FP16 norm.

ModuleCosine distanceRelative norm change (%)
Attention heads0.62−5.2
MLP neurons0.07−27.3
AuthorsOpenReview ↗

Response to Reviewer G6KH Part2

Generalization across unseen templates, natural failures, and stronger quantizers (W4, Q2)

Thank you for these thoughtful questions on generalization.

  • Transfer of localized components to unseen templates. Complementing the original instance-level split, we evaluate four additional diagnostic templates excluded from module localization (Table 3). We keep the originally localized components fixed and apply repairs without re-localization on the new templates. Attention-direction repair and MLP-magnitude repair recover an average of 25.0% and 27.0% of the FP16–INT3 NLL gap, respectively. These results support the cross-template utility of the localized components beyond held-out instances of the original templates.
  • Transfer to an unseen natural benchmark. We further evaluate on GSM8K, which was not used for module localization, keeping the originally localized components fixed without re-localization. Attention-direction repair recovers 34.7% of the FP16–INT3 NLL gap, compared with 8.5% for attention-magnitude repair; MLP-magnitude repair recovers 29.8%, compared with 5.2% for MLP-direction repair. These results support transfer of the localized components to a natural benchmark beyond the diagnostic templates, with the same direction–magnitude repair contrast.
  • Operational scope of “tail capabilities.” We clarify that “tail capabilities” serves strictly as an operational label for the evaluated stress-test conditions (niche languages, long contexts, complex facts, and higher-hop reasoning) rather than measured pretraining-corpus rarity.
  • Evidence and scope across quantizers. Our experiments compared PinQ under different underlying quantizers (main-paper Table 5a), providing evidence that its downstream effectiveness is not restricted to a single quantizer. However, this comparison does not directly establish that sensitivity rankings transfer across quantizers or that the same direction–magnitude repair contrast persists. We will distinguish the cross-quantizer performance evidence from the scope of the mechanistic analysis.

Table 3. Transfer to four unseen diagnostic templates using repair targets localized solely on the original templates (% of FP16–INT3 NLL gap recovered).

Transferred repairNLL gap recovered (%)
Attention direction25.0
MLP magnitude27.0
Both47.0

Role of mechanistic analysis and scope of the additive allocation surrogate (W5, Q3)

PinQ indeed uses a unified matrix-level allocation objective rather than separate rules for attention and MLPs. However, the mechanistic findings in Section 4 directly inform our scoring criterion and the resulting bit distribution in Section 5, while our joint-intervention experiments validate the effectiveness of the additive surrogate:

  • How the mechanistic findings shape PinQ’s scoring criterion. Section 4 yields two actionable insights that directly guide PinQ: (1) quantization sensitivity is highly localized to a small fraction of modules (<10% of modules drive over 80% of the accuracy drop; Figure 6), motivating selective precision protection; and (2) because attention degrades via norm-preserving direction shifts while MLPs degrade via magnitude suppression, magnitude-based proxies (e.g., activation norms: 44.7% recovery in main-paper Table 2) fail to capture both modes, whereas intervention-based output KL evaluates both on a common scale (80.9% recovery). PinQ directly transfers this causal scoring principle to bit allocation.
  • What PinQ simplifies for deployment. Although attention- and MLP-level patching explains how internal representations degrade, sensitivity also shifts across tasks and layers (Figures 3 and 5e) and does not prescribe a fixed module-type rule. For hardware-friendly deployment (supporting vLLM GEMM kernels), PinQ elevates the intervention unit from heads/neurons to complete weight matrices and optimizes the global bit allocation via dynamic programming.
AuthorsOpenReview ↗

Response to Reviewer G6KH Part3

  • PinQ naturally assigns different bit-widths to attention and MLPs. Even without separate allocation rules for each module type, the causal-effect scores naturally assign different precisions to attention and MLP matrices. As shown in Appendix G.1 (Figures 12–13), PinQ predominantly assigns 6–8 bits to attention projections across layers while compressing many middle-layer MoE expert projections to 2–3 bits and protecting boundary layers.
  • How well the additive surrogate predicts joint quantization. Unlike local weight-error proxies, each score si(bi)s_i(b_i) measures the output KL after propagating the quantization error of matrix ii at its assigned bit width through the full model. While summing si(bi)s_i(b_i) omits higher-order cross-matrix interactions so that DP can solve the global budget, our bidirectional joint interventions (Figure 6 and main-paper Table 1) show that isolated scores accurately capture joint criticality: jointly restoring the top-10% modules ranked by single-module scores recovers 80.9%–91.3% of the full-model INT3 drop across 4B–235B models, and jointly quantizing only that subset reproduces 82.2%–93.8% of the total loss. Moreover, globally optimizing this additive objective via DP outperforms greedy allocation by +5.34 MMLU points at 3.0 bits (main-paper Table 5b).

Capability-level preservation, 235B LongBenchV2 breakdown, and statistical scope (W6)

We thank the reviewer for emphasizing capability-resolved evaluation and statistical uncertainty. Following Figure 2’s protocol, Tables 4–5 below break down recovery across individual languages, context lengths, ARC subsets, and reasoning hops, complemented by a LongBenchV2 subset analysis and five-run uncertainty estimates:

  • Fine-grained recovery across all four fragile capabilities (Tables 4–5). PinQ (3.8 bits) consistently outperforms uniform INT4 across all evaluated sub-regimes on Qwen3-4B: (1) it narrows the FP16 gap across all 10 MGSM languages (e.g., from 10.8 to 3.3 points on Vietnamese and 5.6 to 2.2 on Japanese); (2) it tracks FP16 across RULER lengths up to 128k (65.3% → 70.3%, vs. 71.6% FP16); (3) it restores ARC-Challenge (69.8% → 75.0%, vs. 75.6% FP16); and (4) it nearly halves the INT4 gap on 2-hop (5.0 → 2.6 points) and 3-hop (5.9 → 2.9 points) TriviaQA.
  • Additional LongBenchV2 analysis. Under an aggressive 3.3-bit budget (0.7 bits below INT4), overall LongBenchV2 is 33.0% vs. 34.3% (INT4) and 35.0% (FP16), but evaluating its official Long subset (longest context regime) yields 29.6% (FP16), 28.0% (INT4), and 28.7% (PinQ). Thus, while the tight 3.3-bit budget trades off accuracy on shorter/medium subsets, PinQ still better preserves the longest context regime.
  • Statistical uncertainty across five repeated runs. Evaluating Qwen3-4B across five repeated runs over the five benchmarks covering the fragile capabilities, PinQ’s mean improvements consistently exceed run-to-run standard deviations (SD ≤ 0.28% across tasks), raising the macro-average from 63.16 ± 0.28% (INT4) to 65.92 ± 0.20% (PinQ), compared with 67.01 ± 0.18% for FP16 (72% gap recovery). Mean gains are largest on RULER-128K (+5.12 points) and ARC-Challenge (+4.16), followed by MGSM (+2.44), TriviaQA (+1.06), and ARC-Easy (+0.98), with modest residual gaps to FP16 on MGSM (1.74) and TriviaQA (1.58).

Table 4. Language-specific accuracy gaps (percentage points relative to FP16, ↓) on MGSM.

MetricVietnameseKoreanJapanesePortugueseFrenchArabicThaiSpanishChineseEnglish
INT4 gap10.85.65.65.24.44.04.03.20.80.8
PinQ gap3.32.82.22.01.91.40.90.60.20.1

Table 5. Capability-resolved accuracy (%) across context lengths, ARC difficulty subsets, and TriviaQA reasoning hops.

BenchmarkConditionFP16INT4PinQ
RULER4k90.289.790.0
RULER8k85.685.685.6
RULER16k84.383.884.1
RULER32k83.381.682.9
RULER64k77.975.477.3
RULER128k71.665.370.3
ARCEasy84.383.384.5
ARCChallenge75.669.875.0
TriviaQA1-hop45.644.344.5
TriviaQA2-hop37.732.735.1
TriviaQA3-hop29.123.226.2

Dear Reviewer G6KH,

We really appreciate your supportive feedback and thoughtful comments on our work. With the discussion deadline (Nov 18 AoE) coming up soon, we wanted to check whether our responses (Parts 1–3) and the new results in Appendix A have resolved your concerns. We are happy to clarify any remaining points before the window closes!

Best regards,

The Authors

Reviewer_seHqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why low-bit quantization disproportionately degrades certain capabilities of large language models. The authors first identify a disparity between relatively robust “head” capabilities and fragile “tail” capabilities, including low-resource languages, long-context processing, complex factual knowledge, and multi-hop reasoning. They then introduce a quantization-patching approach to localize modules whose quantization substantially affects model predictions. Through an analysis of attention heads and MLP neurons, they identify two predominant perturbation patterns: semantic direction shifts in attention-head outputs and magnitude suppression in MLP contributions.

Based on these observations, the authors propose Pinpoint Quantization (PinQ), a mixed-precision quantization method that measures the output-distribution changes caused by quantizing individual weight matrices and uses a group-knapsack formulation to allocate precision under a memory budget. Experiments across multiple dense and MoE models, including Qwen3 models up to 235B parameters, demonstrate competitive performance at average precision below four bits. The paper also provides intervention-based validation of the identified modules and reports inference-efficiency improvements.

Critical strengths

The paper is very well written, with clear organization and presentation from observations to applications and results.

The paper studies an important problem, which is how low-bit quantization degrades the model performance. Using mechanistic interpretability approaches such as activation patching and logit lens, it identifies different patterns of degradation in different modules in the model, grounded in individual tasks. Discussions are informative.

An intervention-guided low-bit quantization approach PinQ is proposed, which correlates parameter intervention scores with bit width allocation and determines mixed-precision quantization with a dynamic program formulated as a group knapsack problem. Comprehensive results over a wide range of benchmarks demonstrate the competitiveness of the proposed quantization approach, including ablation studies covering different models, different sizes, and different quantization baselines.

Overall, it is a well written paper with clear presentations and experimental evidence.

Critical weaknesses

There is a bit of a disconnection from the attention and MLP activation patching analysis to the application with the proposed quantization approach. To my understanding, The proposed PinQ method does not concern whether a parameter matrix is in attention or in the MLP. The method itself resembles a natural application of sensitivity-based allocation with a knapsack optimizer. This creates a slight weakness on the cohesiveness of the paper.

I'm not sure I'm fully sold on the conclusions from the attention and MLP analysis, where the effects from the low-bit quantization on attention are mostly semantic drift, and on MLP are mostly norm reduction. It is usually observed that the attention outputs have smaller magnitudes, and the MLP outputs would have larger magnitudes, also shown in Figure 4. So, relatively it may be less clear that the effect on MLP is magnitude reduction since on the absolute values it is more likely that any intervention would result in MLP output magnitude reduction.

The experiments do not provide details of the pretraining-style calibration corpus applied for different tests. This leaves me wondering whether one calibration corpus works for all the tasks and all the models, or whether the performance still largely depends on the calibration set or its relation with the downstream testing data.

This makes the proposed quantization approach a trade-off compared with existing approaches such as uniform bit allocation or data-independent quantization. With the knowledge of the data to be applied, it is expected to get better translation performance (e.g. from information theory, if you know the data distribution you can get lower compression rate, but generally we do not know the data distribution). But a data-driven approach also induces additional cost for the approach. There is no discussion on the overhead in storage and computation when comparing different approaches. And the proposed quantization approach may not generalize well. The paper can benefit from showing more controlled comparison considering the overall computational and storage cost, as well as generalization results using one calibration set applied to different downstream tasks.

Questions for authors

Question: for the four difficult tasks that you have tried, it seems the performance degradation from quantization increases with the complexity of the questions, such as increased hops, context lengths, etc. Could this be a result of error compounding? For example, quantization would create a small drift in the generation for one-hub reasoning, and if a task requires more hubs, the error would be accumulated, resulting in worse performance that can be observed.

Suggestion for minor edit: Related work, Post-Training Quantization for LLMs: when you first mention PTQ, better expand the full name alongside it for easier readability.

Suggestion for revision on presentation: For numerical values in Tables, right align the same width of decimal digits for better visibility and fair comparison, see Table 9 (e.g. 83.5 vs. 74.68, etc.) and Table 8 (Appendix).

3 comments
AuthorsOpenReview ↗

Response to Reviewer seHq Part1

Thank you for your positive assessment and constructive questions regarding the link between our mechanistic analysis and PinQ, scale-normalized perturbation metrics, calibration generalization, and computational overhead.

Connection between the mechanistic patching analysis and PinQ allocation (W1)

Thank you for raising this point. PinQ indeed uses a unified allocation objective across weight matrices rather than separate rules for attention and MLPs. However, the mechanistic findings in Section 4 directly inform the choice of our scoring criterion and the resulting bit distribution in Section 5:

  • What the analysis motivates. Section 4 provides two actionable findings that directly shape PinQ: (1) quantization fragility is concentrated in a small subset of components (<10% of modules drive over 80% of the accuracy loss; Figure 6), motivating selective precision protection; and (2) because attention degrades via norm-preserving direction shifts while MLPs degrade via magnitude suppression, magnitude-based proxies (e.g., activation norms: 44.7% recovery in main-paper Table 2) fail to capture both modes, whereas intervention-based output KL evaluates both on a common scale (80.9% recovery). PinQ directly transfers this causal scoring principle to bit allocation.
  • What PinQ simplifies for deployment. While attention- and MLP-level patching explains how internal computations fail, fragility also varies across tasks and layers (Figures 3 and 5e) and does not prescribe a fixed module-type rule. For hardware-friendly deployment (supporting vLLM GEMM kernels), PinQ elevates the intervention unit from heads/neurons to complete weight matrices and solves the global bit budget via dynamic programming.
  • PinQ naturally assigns different bit-widths to attention and MLPs. Even without separate allocation rules for each module type, the causal-effect scores automatically assign different precisions to attention and MLP matrices. As shown in Appendix G.1 (Figures 12–13), PinQ predominantly assigns 6–8 bits to attention projections across layers while compressing many middle-layer MoE expert projections to 2–3 bits and protecting boundary layers.

Scale-normalized comparison of attention and MLP perturbations (W2)

To address the concern that the observed magnitude differences may reflect different baseline activation scales, we supplement main-paper Table 1 with a symmetric comparison on Qwen3-4B (FP16 vs. INT3), reporting both cosine distance and relative L2L_2 norm change for attention-head and MLP-neuron contribution vectors (Table 1 below).

Magnitude suppression persists after scale normalization. Measuring magnitude changes relative to each contribution vector’s FP16 norm, MLP neurons exhibit a mean relative norm change of −27.3%, compared with −5.2% for attention heads. Thus, the stronger magnitude suppression in MLP neurons persists after normalization and is not explained solely by their larger FP16 contribution magnitudes in Figure 4.

Symmetric metrics and targeted repairs support distinct patterns. Attention-head contributions exhibit substantially larger cosine distance than MLP-neuron contributions (0.62 vs. 0.07), consistent with stronger directional changes. Additional repair experiments on GSM8K show a corresponding contrast: for selected attention-head contributions, direction-only repair recovers 34.7% of the FP16–INT3 NLL gap, compared with 8.5% for magnitude-only repair; for selected MLP-neuron contributions, magnitude-only repair recovers 29.8%, compared with 5.2% for direction-only repair.

Together, these results address the baseline-scale concern: the stronger magnitude suppression in MLP neurons remains evident after normalization by their own FP16 norms, and magnitude-only repair yields substantially greater recovery than direction-only repair for these neurons. Thus, the observed MLP magnitude suppression is not merely an artifact of their larger baseline contribution magnitudes.

Table 1. Symmetric comparison of FP16 and INT3 contribution vectors on Qwen3-4B. Bold entries denote newly added symmetric measurements; relative norm change is normalized by each vector’s FP16 norm.

ModuleCosine distanceRelative norm change (%)
Attention heads0.62−5.2
MLP neurons0.07−27.3

Calibration corpus details and cross-task transfer from a single calibration set (W3)

Thank you for this insightful question. We will make the corpus composition and its use across experiments explicit in the main text. As detailed in Appendix F and manuscript Table 11, the calibration corpus contains 2,048 sequences of 512 tokens drawn from mixed-domain sources spanning general text, multilingual text, instruction following, mathematics, and code.

AuthorsOpenReview ↗

Response to Reviewer seHq Part2

Shared calibration data, model-specific allocation. For the main experiments, we use the same calibration sequences across the evaluated models, but recompute bit-dependent intervention scores and precision assignments separately for each target model and bit budget. For each resulting quantized checkpoint, all downstream benchmarks are evaluated without changing the calibration corpus or recomputing the allocation for individual tasks. Thus, the results do not rely on task-specific calibration or benchmark-specific precision assignments.

Scope of the evidence on calibration dependence. These results show that one fixed mixed-domain calibration corpus can support a single allocation per model and bit budget across diverse downstream tasks. They do not, however, establish that allocation is insensitive to the calibration distribution. Because PinQ averages intervention effects over calibration examples, its scores can depend on the capabilities represented in the corpus. The current evidence therefore supports cross-task effectiveness with a shared corpus, rather than independence from calibration data.

Computational and storage trade-offs of data-driven allocation (W4)

  • Offline calibration cost and reuse. PinQ incurs an offline calibration cost to obtain data-driven sensitivity scores and determine the allocation. The calibration of our largest model, Qwen3-235B-A22B, takes approximately 5 hours (Appendix E.1, lines 1382–1383). Importantly, each model’s allocation is computed once and reused across downstream tasks without recalibration. We will report the hardware and timing scope explicitly, separating offline allocation cost from deployment-time storage and execution costs.
  • Measured storage and serving costs. For Qwen3-235B-A22B, the checkpoint size decreases from 116.0 GiB for INT4 to 96.2 GiB for PinQ, a 17.1% reduction. The corresponding model-tensor memory footprint also decreases from 116.4 to 96.8 GiB. These measurements include packing padding, quantization metadata, and parameters retained in FP16, rather than relying only on nominal average bit-width. Under matched serving conditions on the same model, PinQ increases throughput by 19.9% and reduces TTFT and TPOT by 10.7% and 5.3%, respectively, relative to optimized INT4. Thus, the measured deployment benefits remain after accounting for mixed-precision storage and execution overheads. We will specify the shared hardware, serving configuration, and workload used for this comparison. Table 2 below provides these storage and serving measurements.
  • Cross-task reuse of a shared calibration dataset. Our evaluation does not assume access to each downstream task’s data distribution for allocation. All models use the same calibration dataset, and each model’s fixed allocation is evaluated across downstream tasks without task-specific recalibration, as detailed above. The downstream evaluations therefore test reuse of a shared calibration source rather than separately tuned allocations. As a concrete long-context example, PinQ achieves 70.3 on RULER-128k for Qwen3-4B, compared with 65.3 for INT4, providing evidence of transfer to substantially longer contexts.

Table 2. Storage, memory, and serving performance of Qwen3-235B-A22B under matched conditions (8×A100 80GB, 512-token input, 128-token output, concurrency 128). Checkpoint size and memory are reported in GiB and include padding, quantization metadata, and FP16 parameters.

MethodAvg. bitsMemory (GiB)Checkpoint (GiB)Throughput (tok/s) ↑TTFT (ms) ↓TPOT (ms) ↓
FP1616.0437.9436.91149.03413.583.5
INT44.0116.4116.01228.01820.578.9
PinQ3.396.896.21472.41625.574.7

Error compounding across question complexity (Q1)

We agree that error compounding across reasoning steps and sequence positions is a plausible explanation for the steeper degradation on complex tasks: small directional or magnitude perturbations introduced in single-hop reasoning propagate and accumulate across subsequent hops and long-range context dependencies, causing larger accuracy losses on 3-hop TriviaQA and 128k RULER. Importantly, our causal interventions show that these compounding errors originate disproportionately in a sparse set of sensitive modules, explaining why targeted protection in PinQ mitigates multi-hop and long-context degradation.

Abbreviations and numerical alignment in tables (Minor Suggestions 1–2)

Thank you for both suggestions. We have expanded “Post-Training Quantization (PTQ)” at its first occurrence in lines 123–124 and standardized decimal precision with right-aligned numerical columns in lines 1604–1607.

Dear Reviewer seHq,

Thank you again for your positive review and helpful suggestions. Since the discussion period ends soon (Nov 18 AoE), we wanted to briefly check if our responses (Parts 1–2) have addressed all of your questions and concerns. Please let us know if anything needs further explanation!

Best regards,

The Authors

Reviewer_2Uy7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how quantization damages long-tail tasks. It identifies sensitive components for this, examines changes in attention and MLP outputs, and proposes PinQ, which assigns precision using component sensitivity. Experiments on dense and MoE models show improvements over uniform MinMax INT4.

Critical strengths
  • The paper studies capability-specific failures that average benchmark scores can hide and focuses on tail capabilities.

  • Restoration and selective quantization experiments identify components responsible for substantial performance degradation.

  • The evaluation covers multiple model sizes, including large MoE models, across 18 benchmarks.

Critical weaknesses
  • The paper does not show whether PinQ recovers the specific languages and higher-hop questions identified as fragile in Figure 2. It mainly reports overall benchmark scores. There is recovery on RULER-128k, but results are not uniformly positive: LongBenchV2 on the 235B model falls from 35.0 in FP16 to 34.3 with INT4 and 33.0 with PinQ.

  • The claimed attention-direction versus MLP-magnitude distinction needs stronger evidence. Appendix C shows that direction and magnitude repairs improve NLL, but does not separate their effects by module type on the four diagnostic tasks. Comparing whole attention heads with individual MLP neurons may also affect the finding. Both direction and magnitude should be evaluated for both module types at comparable granularity.

  • The main Qwen/MoE experiments primarily compare against simple MinMax INT4. Stronger baselines are evaluated on Llama3-8B, while the GPTQ ablation only covers MMLU and perplexity. These comparisons do not establish whether PinQ improves over strong existing methods on the models and tail tasks motivating the paper.

  • The allocator’s novelty is unclear. Isolated-component KL scoring appears in A KL Lens on Quantization (https://arxiv.org/abs/2604.13440), budgeted precision allocation appears in HIGGS (https://arxiv.org/abs/2411.17525), and KL scoring followed by knapsack allocation appears in MixFrag (https://arxiv.org/abs/2607.28589). The authors should explain what PinQ adds beyond applying these ideas to large LLMs and MoE models.

Questions for authors
  • Can the authors repeat Figure 2 with PinQ included?

  • Can direction-only and magnitude-only repairs be tested separately for attention and MLP outputs?

  • How does PinQ compare with strong quantization and allocation baselines under matched settings?

  • What technical differences distinguish PinQ’s allocator from the cited methods?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2Uy7 Part1

Thank you for the thorough review and constructive feedback. Following your suggestions, we have (1) added capability-resolved evaluations matching every panel of Figure 2 and analyzed the LongBenchV2 trade-off, (2) conducted matched-granularity direction-only and magnitude-only repairs across all four diagnostic tasks, (3) expanded strong-baseline comparisons (GPTQ, HIGGS, KL-lens, and GPTQ+PinQ) on dense and MoE Qwen models, and (4) clarified our positioning relative to recent sensitivity-based allocation methods.

Capability-resolved recovery on fragile tasks and updated Figure 2 (W1, Q1)

Thank you for this insightful comment. We provide the complete capability-resolved recovery results of PinQ. Tables 1 and 2 below quantify recovery for the specific languages and higher-hop questions identified as fragile, as well as across context lengths and ARC difficulty subsets. We have repeated Figure 2 with PinQ included alongside FP16 and uniform INT4 in the revised manuscript Figure 7 (lines 932–940).

  • Language-specific recovery. PinQ narrows the accuracy gap relative to FP16 across all ten evaluated MGSM languages (Table 1). On the most degraded niche languages, the gap drops from 10.8 to 3.3 percentage points for Vietnamese and from 5.6 to 2.2 points for Japanese, while high-resource languages (Chinese and English) stay within 0.2 points of FP16.
  • Long-context recovery on RULER. Across RULER context lengths from 4k to 128k (Table 2), PinQ closely tracks FP16, raising 128k accuracy from 65.3% (INT4) to 70.3% (FP16: 71.6%).
  • ARC-Challenge recovery. On ARC-Challenge, PinQ improves accuracy from 69.8% (INT4) to 75.0%, approaching FP16 (75.6%), while preserving ARC-Easy (84.5% vs. 84.3% FP16).
  • Higher-hop recovery. On TriviaQA, PinQ improves 2-hop accuracy from 32.7% to 35.1% and 3-hop accuracy from 23.2% to 26.2%, cutting the INT4 gaps of 5.0 and 5.9 percentage points down to 2.6 and 2.9 points, respectively.
  • Additional LongBenchV2 analysis. PinQ operates at an aggressive 3.3-bit budget (0.7 bits/weight below uniform INT4) on Qwen3-235B-A22B. While the overall LongBenchV2 score across all length categories is 33.0% vs. 34.3% (INT4) and 35.0% (FP16), evaluating LongBenchV2’s official Long subset (the longest context regime) yields 29.6% (FP16), 28.0% (INT4), and 28.7% (PinQ). This suggests that while the tight 3.3-bit budget introduces a trade-off on shorter and medium-length subsets, PinQ still better preserves accuracy in the long context regime.

Table 1. Language-specific accuracy gaps relative to FP16 on MGSM.

MetricVietnameseKoreanJapanesePortugueseFrenchArabicThaiSpanishChineseEnglish
INT4 gap10.85.65.65.24.44.04.03.20.80.8
PinQ gap3.32.82.22.01.91.40.90.60.20.1

Table 2. Capability-resolved accuracy (%) across context lengths, ARC difficulty subsets, and TriviaQA reasoning hops.

BenchmarkConditionFP16INT4PinQ
RULER4k90.289.790.0
RULER8k85.685.685.6
RULER16k84.383.884.1
RULER32k83.381.682.9
RULER64k77.975.477.3
RULER128k71.665.370.3
ARCEasy84.383.384.5
ARCChallenge75.669.875.0
TriviaQA1-hop45.644.344.5
TriviaQA2-hop37.732.735.1
TriviaQA3-hop29.123.226.2

Decoupled direction-only and magnitude-only repairs at matched granularity (W2, Q2)

We thank you for this constructive suggestion. To address this concern, we decouple direction-only and magnitude-only interventions by module type across all four diagnostic tasks, operating on complete attention-branch and MLP-branch outputs to ensure matched granularity. The results confirm a functional asymmetry: directional restoration primarily drives recovery in the attention branch, whereas magnitude restoration dominates in the MLP branch, demonstrating that our mechanistic distinction holds independently of head-versus-neuron granularity.

AuthorsOpenReview ↗

Response to Reviewer 2Uy7 Part2

  • Matched-granularity experimental protocol. To eliminate the head-versus-neuron dimensionality mismatch, we elevate the intervention unit to the complete layer-wise attention-branch output uattn∈Rdu_{\mathrm{attn}} \in \mathbb{R}^d and MLP-branch output umlp∈Rdu_{\mathrm{mlp}} \in \mathbb{R}^d added to the shared residual stream (with input states and token positions held fixed on Qwen3-4B). For a quantized branch output uqu_q and its FP16 reference ufpu_{\mathrm{fp}}, direction-only repair restores the reference unit vector while retaining the quantized norm: u~dir=∥uq∥2ufp∥ufp∥2.\widetilde{u}_{\mathrm{dir}} = \lVert u_q \rVert_2 \frac{u_{\mathrm{fp}}}{\lVert u_{\mathrm{fp}} \rVert_2}.

  • Magnitude-only repair restores the reference norm while keeping the quantized direction: u~mag=∥ufp∥2uq∥uq∥2.\widetilde{u}_{\mathrm{mag}} = \lVert u_{\mathrm{fp}} \rVert_2 \frac{u_q}{\lVert u_q \rVert_2}.

  • Consistent module-type asymmetry across all four diagnostic tasks. As shown in Table 3, the attention branch is direction-dominated. Restoring direction recovers 36.0%–43.5% of the FP16–INT3 NLL gap, whereas restoring magnitude recovers only 8.1%–13.9%. The MLP branch is magnitude-dominated. Restoring magnitude recovers 32.5%–39.7% of the NLL gap, whereas restoring direction yields only 1.9%–6.8% recovery.

  • Alignment with capability-specific failure patterns. These decoupled results also validate our error taxonomy in Section 4.2 (Figure 5e). For instance, niche languages, which exhibited the highest proportion of MLP-only failures, achieve the strongest MLP magnitude recovery (39.7%), while multi-hop reasoning shows minimal sensitivity to MLP direction (1.9%) but strong reliance on attention direction (36.0%). This confirms that the attention-direction versus MLP-magnitude distinction reflects intrinsic module computations rather than an artifact of comparing heads with individual neurons. We incorporated this table and analysis into lines 980–986 in the revised manuscript.

Table 3. Branch-level repair across the four diagnostic tasks (% of FP16–INT3 NLL gap recovered), evaluated on complete attention and MLP branch outputs.

TaskAttention: directionAttention: magnitudeMLP: directionMLP: magnitude
Niche languages43.513.96.839.7
Long contexts40.810.26.734.6
Complex facts40.58.34.533.3
Multi-hop reasoning36.08.11.932.5

Comparisons with strong quantization and allocation baselines on Qwen models (W3, Q3)

We evaluate PinQ against strong baselines on the Qwen family across seven additional benchmarks covering all four motivating tail capabilities: multilingual reasoning (MGSM), long-context retrieval (RULER-128K), complex factual/logical reasoning (ARC-Challenge, GSM8K), multi-hop recall (TriviaQA), and general tasks (ARC-Easy, WinoGrande). Specifically, we compare against strong quantizers (GPTQ, HIGGS), the mixed-precision allocation baseline KL-lens, and the combined GPTQ+PinQ under matched bit budgets (Table 4):

  • Consistent advantages on general and tail-capability benchmarks. Even with a simple MinMax base quantizer, standalone PinQ achieves a 69.31% average score on Qwen3-4B, surpassing GPTQ (68.03%, +1.28 pp), KL-lens (68.50%, +0.81 pp), and HIGGS (68.59%, +0.72 pp) on every benchmark. Gains are most pronounced on the fragile tail tasks motivating our analysis—improving over GPTQ / KL-lens / HIGGS by +2.7/+1.5/+1.9 points on RULER-128K, +2.9/+1.4/+0.9 points on ARC-Challenge, and +1.2/+0.8/+0.6 points on MGSM.
  • Complementary gains when paired with GPTQ (GPTQ+PinQ). Pairing PinQ’s causal bit allocation with GPTQ weight updates further lifts performance across all seven benchmarks to a 69.90% average (+1.87 pp over standalone GPTQ and +1.31 pp over HIGGS). Notably, GPTQ+PinQ recovers 88.7% of the average INT4 quantization drop, narrowing the overall gap to full-precision FP16 (70.29%) to just 0.39 points (and within 0.2–0.5 points on ARC-Challenge, GSM8K, and RULER-128K).
AuthorsOpenReview ↗

Response to Reviewer 2Uy7 Part3

Table 4. Quantization performance comparison on Qwen3-4B across seven benchmarks under a matched bit budget (%). Bold indicates the best quantized result per row (GPTQ+PinQ).

BenchmarkFP16INT4GPTQKL-lensHIGGSPinQGPTQ+PinQ
MGSM62.0058.0059.3059.7059.9060.5061.30
RULER-128K71.6065.3067.6068.8068.4070.3071.10
ARC-Easy84.3083.3083.8084.0084.1084.5084.60
ARC-Challenge75.6069.8072.1073.6074.1075.0075.40
TriviaQA41.9039.1039.9039.7039.8040.2041.10
GSM8K89.8087.6088.2088.5088.3088.9089.40
WinoGrande66.8064.7065.3065.2065.5065.8066.40
Average70.2966.8368.0368.5068.5969.3169.90

A parallel evaluation on the MoE model Qwen3-30B-A3B under matched settings exhibits the exact same performance hierarchy: PinQ consistently outperforms uniform INT4, GPTQ, KL-lens, and HIGGS across both general and fragile tail tasks, while GPTQ+PinQ further improves over standalone GPTQ on every benchmark (e.g., lifting multi-task average accuracy from 73.00% to 73.50%, including a +1.1-point gain on ARC-Challenge). We incorporated these expanded comparisons into lines 1029–1039 in the revised manuscript.

Allocator positioning and technical distinctions from the cited methods (W4, Q4)

We thank the reviewer for pointing out these outstanding studies. Table 5 summarizes the specific technical differences between PinQ’s allocator and the cited methods.

Scoring and optimization differences relative to KL-Lens [1] and HIGGS [2]. Regarding the allocator itself, PinQ differs from KL-Lens [1]’s rank/threshold assignment by jointly selecting matrix precisions under explicit parameter-bit costs ci(b)=nibc_i(b) = n_i b using group-knapsack DP, where nin_i is the number of weights in matrix ii and bb is its assigned bit width. Unlike HIGGS [2]’s reconstruction-error-based surrogate, PinQ directly measures end-to-end output KL under actual candidate-bit quantization.

Relationship to concurrent MixFrag [3] and allocation for autoregressive LLMs. MixFrag (arXiv, July 30, 2026) pairs isolated-component KL scores with multiple-choice knapsack optimization for Vision Transformers. We explicitly acknowledge this shared high-level allocation structure. The distinction lies in what the scores measure and what the allocation seeks to preserve: MixFrag evaluates changes in classification-output distributions, whereas PinQ measures forward KL over full-vocabulary next-token distributions across valid prediction positions. Furthermore, PinQ’s bidirectional interventions validate the sensitivity ranking through capability recovery and failure reproduction. This provides empirical grounding for the scoring principle used by PinQ, beyond its role as an allocation objective.

Within our Identify–Analyze–Allocate framework, the main focus is identifying and analyzing capability-specific quantization failures. Precision allocation is the practical application of the intervention-based sensitivity principle developed and validated in this investigation. Beyond sensitivity ranking, we examine how direction and magnitude perturbations in attention and MLP outputs contribute to these failures through controlled repairs. Accordingly, the paper’s novelty extends beyond whether KL scoring or knapsack optimization is new in isolation.

Table 5. Comparison of scoring, allocation, and intervention evidence in the cited works (✓: reported; ×: not reported in the cited paper).

MethodAllocation scorePrecision allocationBidirectional ranking validationDirection/magnitude mechanistic analysis
KL Lens [1]Output KLRank/threshold××
HIGGS [2]Reconstruction-error PPL surrogateBudgeted allocation××
MixFrag [3]Classification output KLMultiple-choice knapsack DP××
OursNext-token forward KLGroup-knapsack DP✓✓

References

[1] Kong et al. A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM–Transformer Models. arXiv:2604.13440, 2026.

[2] Malinovskii et al. HIGGS: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem. NAACL, 2025

[3] Opi et al. MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers. arXiv:2607.28589, July 30, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.