acceptodds
Desk rejected at ICLR 2027

Maximum Activations in Open Large Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

The dynamic range of activations is a first-order constraint for low-bit quantization, activation scaling, and stable LLM inference. Prior work characterized outlier features and massive activations on pre-2024 LLaMA-style models, and the downstream activation-quantization stack inherits that picture without revisiting it for the post-LLaMA open-model boom. We ask the deployment-oriented question: how large can activations get in modern open LLMs, and how does this magnitude vary across families, generations, and training stages? Under a unified pipeline (5000-sample multi-domain corpus, family-specific tokenization, identical hooks across embeddings, hidden states, attention, MLP/MoE, SwiGLU gates, and final norm), we measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants. We find that (i) global maxima span over nearly four orders of magnitude at comparable parameter counts, with Qwen3.5 and MoE checkpoints in the – range and Gemma3-27B-it reaching ; (ii) cross-family and cross-generation comparisons break simple monotonic scaling; and (iii) in two matched MoE–dense pairs, MoE checkpoints exhibit – lower peaks, while the residual stream carries the global maximum in 22/24 main-analysis checkpoints. A lightweight INT-8 sanity check shows that measured maxima co-vary with low-bit reconstruction error via activation-scale selection. We conclude that maximum activation magnitude is a protocol-conditioned model statistic associated with family, architecture, and training stage—not a simple byproduct of size—and should be measured and reported alongside any open-weight release before low-bit deployment. An anonymous code repository is linked at https://anonymous.4open.science/r/Max_act_llm-CB63/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wASTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically measures maximum activation magnitudes across multiple open LLM families under a unified protocol, examines their variation with model size, architecture, and training stage, and provides preliminary evidence of their relevance to low-precision quantization.

Critical strengths
  1. Broad empirical coverage. The study covers multiple model families, parameter scales, and model variants, including dense models, MoE models, vision-language models, and checkpoints from different training stages. This breadth provides a useful comparative overview of activation magnitudes.

  2. Practical relevance. The results demonstrate that models with similar parameter counts can exhibit substantially different activation ranges. This observation motivates model-specific numerical precision and quantization strategies and provides useful guidance for pre-deployment assessment and quantization calibration.

Critical weaknesses
  1. Limited insight beyond descriptive observations. The paper primarily presents an empirical survey of activation statistics. Although the measurements are useful, the analysis offers limited explanations for the observed differences or actionable insights beyond recommending that activation peaks be measured. The contribution would be stronger if the reported patterns supported more specific, testable conclusions.

  2. Insufficient justification of absolute peak magnitude as an indicator of quantization difficulty. The paper does not clearly establish how reliably absolute activation peaks predict quantization error. A large peak may reflect either a broadly elevated activation scale or a small number of extreme outliers, which can have different quantization implications. The reported results also include low-peak models with worse reconstruction quality than higher-peak models, leaving the predictive value of the proposed statistic unclear.

  3. An incomplete connection between measurement locations and quantization sites. Most global maxima occur in the residual stream, whereas residual connections and normalization operations remain in BF16 in the W8A8 experiment. It is therefore unclear how much of the extreme magnitude persists after normalization in the tensors actually quantized to A8, and which quantization sites mediate the observed degradation. The current evidence does not directly connect the measured residual-stream peaks to the resulting quantization error.

Questions for authors
  1. Improve presentation and sharpen the contribution. Several figures, particularly Figures 2 and 4, occupy substantial space relative to their information content. A more compact layout and a tighter narrative would improve readability. The introduction should distinguish the principal new findings from supporting observations and state more clearly what the study adds beyond existing work.

  2. Strengthen the validation of the proposed metric. Compare absolute peak magnitude against scale-normalized alternatives, such as peak-to-RMS and peak-to-percentile ratios, across a broader set of checkpoints. Measure these statistics at the actual A8 quantization sites and assess their association with both reconstruction error and end-to-end performance degradation.

1 comment

Thank you for the review and for emphasizing the distinction between measuring an extreme activation range and establishing its actual quantization consequence. We agree that the submitted version did not sufficiently separate these questions. We therefore added analyses at the actual A8 quantization sites, compared absolute and normalized range statistics, and measured end-to-end W8A8 degradation.

W1. Beyond descriptive observations.

We measured the inputs to the GEMMs that are actually quantized to A8, rather than relying only on residual-stream maxima.

For three representative checkpoints, the largest A8 activation occurs at a down_proj input:

  • Qwen3-30B-A3B: residual M 1,512, A8 M 1,736;
  • Qwen3-32B: 35,328 to 24,960;
  • Gemma-3-27B-it: 696,320 to 25,856.

Thus, residual peaks do not propagate uniformly to quantized tensors, but extreme values can persist at actual A8 sites.

The matched Qwen Dense-MoE difference also becomes more localized: the Dense/MoE gap decreases from 23.4x in global M to 14.4x in global A8 M, while median A8 inputs show little separation. This suggests that the difference is concentrated in a small number of extreme sites, especially down_proj, rather than being uniformly distributed across the model.

W2. Absolute M versus normalized statistics.

We agree that absolute peak magnitude alone cannot distinguish a broadly elevated distribution from a sparse outlier.

We therefore measured M, RMS, P99.9, M/RMS, and M/P99.9 at every actual A8 input tensor. Within every evaluated model, M/RMS is more strongly associated with static reconstruction SQNR than raw M:

  • Qwen3-30B-A3B: -0.79 vs. -0.46;
  • Qwen3-32B: -0.94 vs. -0.77;
  • Gemma-3-27B-it: -0.81 vs. -0.38;
  • Qwen3.5-0.8B: -0.97 vs. -0.13.

The low-peak Qwen3.5-0.8B checkpoint is a useful counterexample. Its full-corpus A8 maximum is only 74, compared with 1,736 for Qwen3-30B-A3B, yet static W8A8 gives delta NLL = +1.217 versus only +0.032 for A3B.

Even model-level normalized maxima do not fully explain this cross-model difference. What differs more directly is the low tail of reconstruction quality: 26/96 distinct A8 inputs in Qwen3.5-0.8B have static SQNR below 10 dB, compared with 3/192 for A3B. We treat this as a descriptive four-model observation rather than a universal law.

We therefore refine the claim: M measures worst-case activation range, while normalized statistics describe the sparsity of that range. Neither is a standalone model-level predictor of end-to-end degradation.

W3. Residual maxima versus actual A8 sites.

We directly instrumented q/k/v/o, gate/up/down, and routed-expert GEMM inputs.

The actual A8 maxima are:

  • Qwen3-30B-A3B: 1,736 at layer_47.experts.down_proj;
  • Qwen3-32B: 24,960 at layer_63.down_proj;
  • Gemma-3-27B-it: 25,856 at layer_0.down_proj;
  • Qwen3.5-0.8B: 74 at layer_14.down_proj.

For Gemma-3-27B-it, the residual maximum attenuates by 26.9x before the largest A8 site. We therefore agree that residual M should not be interpreted as the value directly seen by the A8 quantizer.

Q1. Presentation and contribution.

We agree that the narrative can be tighter. We will reduce the space devoted to less informative figures, bring the key matched and quantization results closer to the main text, and distinguish more clearly between direct measurements, matched observational comparisons, and deployment implications.

Q2. Reconstruction and end-to-end degradation.

We additionally evaluated simulated W8A8 using identical per-output-channel INT8 weights and compared static per-tensor with dynamic per-token activation scaling.

Static to dynamic delta NLL:

  • Qwen3-30B-A3B: +0.032 to +0.004;
  • Qwen3-32B: +1.483 to +0.024;
  • Gemma-3-27B-it: +6.222 to +0.017;
  • Qwen3.5-0.8B: +1.217 to +0.011.

Dynamic per-token scaling recovers 87.5-99.7% of the static degradation. Static saturation is negligible (at most 1.6e-7 of evaluation activations), indicating that the static failure is mainly resolution loss rather than clipping: sparse extremes enlarge the per-tensor step size and reduce resolution for typical values.

These results narrow the deployment claim. We no longer interpret M as a standalone predictor of quantization difficulty. Rather, M characterizes worst-case range, normalized local statistics characterize how sparse that range is, and actual degradation depends strongly on the quantization policy.

These are QDQ/fake-quantization results and do not measure INT8 kernel latency or throughput.

We thank the reviewer for motivating these analyses. They sharpen the deployment interpretation while preserving the paper's main contribution: a unified characterization of activation range across modern open checkpoints.

Reviewer_Xh9hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an observational study measuring the global maximum activation magnitude, M=max⁡∣a∣M = \max |a|, across modern open LLMs. Using a fixed 5,000-sample multi-domain corpus (≈19.5M\approx 19.5\text{M} tokens, re-tokenized per family) and PyTorch forward hooks across key module outputs and pre-activations, the authors measure MM and layerwise peaks across 24 checkpoints from 8 families (Qwen2.5, Qwen2.5-VL, Qwen3, Qwen3.5, Gemma2, Gemma3, Ling-mini, GPT-OSS) alongside 3 Qwen2.5-Instruct checkpoints.

The authors find that MM spans nearly four orders of magnitude across the suite (from 122 in Qwen3.5-0.8B to ≈7×105\approx 7 \times 10^5 in Gemma3-27B-it) and roughly 2.5 orders of magnitude at matched parameter counts. Within a given family, MM generally grows with model size, though cross-family trends are non-monotonic. The post-residual hidden state carries the global peak in 22 of the 24 checkpoints. In two matched MoE-dense comparisons, the MoE models exhibit a 14×14\times--23×23\times lower peak. Finally, downstream relevance is examined via static INT8 per-tensor SQNR on 8 models and W8A8 perplexity on 3 checkpoints. The authors release their code and summary statistics.

Critical strengths
  • Unified and reproducible measurement protocol on modern models. Prior studies of massive activations (such as Dettmers et al. 2022 and Sun et al. 2024) focused on older architectures like OPT, BLOOM, and LLaMA. Applying a standardized measurement pipeline—with a consistent corpus, hook locations, and per-family tokenization—across modern families like Qwen2.5/3/3.5, Gemma2/3, Ling, and GPT-OSS is a valuable empirical contribution. Documenting that MM spans from ≈102\approx 10^2 (Qwen3.5-0.8B) to ≈7×105\approx 7 \times 10^5 (Gemma3-27B-it), with differences of ≈2.5\approx 2.5 orders of magnitude even at matched parameter counts (Qwen3.5-27B at ≈1.8K\approx 1.8\text{K} vs Gemma3-27B-it at ≈696K\approx 696\text{K}), provides useful reference data.

  • Carefully scoped claims and stability checks. The authors appropriately frame small-sample comparisons (n=2n=2 MoE-dense, n=2n=2 VL-text, n=3n=3 Base-Instruct) as empirical associations rather than causal effects. The inclusion of subsample stability checks (CoV≤10.1%\text{CoV} \le 10.1\%), domain stratification, and routing statistics that challenge simple explanations (such as negative median ARCres\mathrm{ARC}_{\mathrm{res}}) demonstrates good experimental rigor. The MoE versus dense W8A8 perplexity comparison (+3.4%+3.4\% vs +282%+282\% at ≈30B\approx 30\text{B} scale) is also a concrete, interesting observation.

Critical weaknesses
  • The paper is primarily a descriptive catalogue, and it lacks the analyses needed to substantiate its deployment claims. That massive activations are widespread across LLM families was already established by Sun et al. (2024). This paper frames MM as a first-order constraint for practical quantization, but two essential analyses are missing:

    First, token position and identity are never reported. Prior work and the attention-sink literature show that outlier activations often concentrate on the BOS or delimiter tokens. The fact that Top-100 values are within a few percent of Top-1 across 5,000 sequences strongly hints at a systematic per-sequence positional artifact. If MM is predominantly carried by initial tokens, the practical solution is prefix handling (as in PrefixQuant) rather than adjusting per-tensor scales for the whole sequence. Without position data, the paper cannot evaluate this.

    Second, the quantization evaluation relies exclusively on static per-tensor INT8. Real-world serving frameworks (such as TensorRT-LLM and vLLM) typically use per-token dynamic quantization, which is largely robust to isolated outlier tokens. Without comparing against per-token dynamic baselines, the claim that MM poses a major bottleneck for modern deployment stacks remains unsupported.

  • Inconsistency in Table 1 regarding the massive activation criterion. The text states that passing models must have a ratio of Top-1 to Median exceeding 1,000. However, the reported values for gemma-2-2b (2,992/50.5≈592{,}992 / 50.5 \approx 59) and Qwen3.5-27B (167/0.225≈742167 / 0.225 \approx 742) both fall well below 1,000, yet neither is marked as failing. It is unclear whether these numbers reflect different layers, whether the criterion was applied differently than described, or if the headline figure of 20/24 passing checkpoints is inaccurate. Because Table 1 is the sole operationalization of this criterion in the paper, this discrepancy makes the 20/24 prevalence claim unverified.

Questions for authors
  1. Table 1 discrepancy: Could you clarify the Table 1 entries for gemma-2-2b (ratio≈59\text{ratio} \approx 59) and Qwen3.5-27B (ratio≈742\text{ratio} \approx 742)? Are these marked as passing because a different layer or token meets the criterion? Please clarify what Top-1 and Median represent here and confirm the 20/24 count.

  2. Token positions: At what token position and token identity does MM typically occur (e.g., BOS/first token, delimiter, or content tokens)? What fraction of the per-layer peaks in Figure 3 occur at position 0? If MM is predominantly an initial-token artifact, how does that affect the practical deployment implications?

  3. Dynamic quantization: Does the relationship between MM and quantization degradation (SQNR / perplexity) hold under per-token dynamic INT8, which is supported in standard serving runtimes like TensorRT-LLM and vLLM?

  4. Comprehensive table of MM: Could you include a complete table reporting global MM, along with the carrying component and layer index, for all evaluated checkpoints in the main text or appendix? Currently, several models (e.g., Qwen2.5-VL-3B, gemma-2-9b, gpt-oss-20b) lack explicit numerical values in the text.

  5. Base Gemma 3 checkpoints: Why were base (pretrained) Gemma 3 checkpoints omitted given they are public? Adding them would help isolate whether the high magnitudes in Gemma 3 stem from pretraining or instruction tuning.

  6. Numerical precision and hook locations: Were all forward passes executed in BF16? Could any extreme values be influenced by the precision of the hook capture or the choice of pre- versus post-norm hook placement in Gemma (which scales embeddings by d\sqrt{d})?

  7. Scope of the MoE magnitude claim: The abstract notes that Qwen3.5 and MoE checkpoints fall in the 10210^2--10310^3 range, but GPT-OSS-20B (M≈43,008M \approx 43{,}008) and Ling-mini-2.0 (7.6K7.6\text{K}--11.8K11.8\text{K}) are also MoEs that exceed this range. Should this claim be specifically restricted to the Qwen MoE-dense pairs?

Additional suggestions:

  • The corpus citation currently points to RedPajama (Shen et al., 2024), but that paper introduces SlimPajama-DC; please correct the citation.
  • Several important matched-design comparisons currently appear only in Appendix C and E. Consider bringing key findings, particularly the MoE-dense and W8A8 results, into the main body.
1 comment

Thank you for the detailed review and for recognizing the value of the unified protocol and matched comparisons. We agree that the deployment interpretation should distinguish token-position effects from quantization-policy effects. We therefore performed additional token-position and dynamic-W8A8 analyses and re-audited Table 1.

Q1. Table 1 and the 20/24 count.

We rechecked the Sun-style massive-activation criterion over all 24 main checkpoints, and the reported 20/24 count is correct.

The confusion comes from the fact that the Top-1 and Median values shown in Table 1 describe the token carrying the reported representative peak, whereas the pass/fail criterion is existential over all eligible token vectors and requires both absolute activation > 100 and same-token max/median > 1000 for the same token.

For gemma-2-2b, 5,455 token vectors satisfy both conditions although the displayed peak token has 2992/50.5 approximately 59. Qwen3.5-27B similarly has 29 jointly passing token vectors although the displayed row gives 167/0.225 approximately 742. We will clarify this distinction in the table caption.

Q2. Token positions and identities.

We ran a full 5,000-sample position analysis on three representative checkpoints.

  • Qwen3-30B-A3B: global M is at position 0; 91.3% of sample-by-layer peaks and 96% of Top-100 token-level extremes are at position 0.
  • Qwen3-32B: global M is at position 1199; 70.5% of sample-by-layer peaks are at position 0, but 0% of its Top-100 token-level extremes are.
  • Gemma-3-27B-it: only 0.2% of sample-by-layer peaks and 13% of Top-100 extremes occur at position 0.

Importantly, position 0 is not BOS or another special token: for each model, the position-0 BOS and special-token fractions are both 0/5,000. We do observe delimiter-like ordinary tokens among some extreme events in Qwen3-32B and Gemma.

Thus, positional structure matters, but initial-token concentration is model-dependent rather than a universal BOS artifact.

Q3. Dynamic per-token quantization.

We agree that static per-tensor INT8 alone is insufficient. We therefore compared static per-tensor and dynamic per-token activation scaling under simulated W8A8, using identical per-output-channel INT8 weights.

Static to dynamic delta NLL:

  • Qwen3-30B-A3B: +0.032 to +0.004
  • Qwen3-32B: +1.483 to +0.024
  • Gemma-3-27B-it: +6.222 to +0.017
  • Qwen3.5-0.8B: +1.217 to +0.011

Dynamic scaling recovers 87.5-99.7% of the static degradation and leaves only 0.4-2.4% PPL degradation. We therefore agree that per-token scaling largely mitigates this failure mode and will narrow the deployment claim accordingly. We do not claim that M is a universal bottleneck under dynamic per-token quantization.

These are QDQ/fake-quantization results and do not measure INT8 kernel speed.

Q4. Complete table of M.

We agree and will add a complete table with global M, carrying component, and layer index for all evaluated checkpoints.

Q5. Base Gemma 3.

The current suite contains Gemma3 instruction-tuned checkpoints but not a matched base checkpoint comparison. We agree that such a comparison would help isolate pretraining versus instruction-tuning effects. We therefore will not attribute Gemma3's large magnitudes specifically to instruction tuning, and will describe the Gemma2-Gemma3 result as an observational generation-level comparison rather than a controlled ablation.

Q6. Precision and hook locations.

All model forwards are executed in BF16; captured activations are converted to FP32 only for statistics. Hooks record embedding outputs, post-residual hidden states, attention outputs before the outer residual, MLP/MoE outputs before the outer residual, and final normalization outputs.

Gemma3's global maxima occur at layer_29_hidden for 4B-it and layer_46_hidden for 27B-it, not at the embedding output. Thus, embedding scaling by sqrt(d) does not explain these maxima.

Q7. Scope of the MoE claim.

We agree that the abstract wording is too broad. We will restrict the claim to what is directly supported:

In the two matched Qwen Dense-MoE comparisons, the evaluated MoE checkpoints have 14.0-23.4x lower global maxima than their nearby Dense counterparts.

We will not generalize this observation to MoE architectures as a whole.

We will also correct the SlimPajama-DC citation and move the key matched Dense-MoE and quantization findings closer to the main text.

Overall, the new results support a narrower interpretation: token-position effects are model-dependent, dynamic per-token scaling largely mitigates static-per-tensor degradation, and M should be viewed as a worst-case range descriptor rather than a universal quantization-difficulty score.

Reviewer_aG7mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper measures activation maxima across 27 open-model checkpoints using a shared 5,000-sample text corpus. It compares model families, parameter scales, dense and MoE architectures, and training variants. The reported maxima range from 122 to 696,320, with residual hidden states carrying the largest values in most checkpoints. Two dense–MoE pairs show 14–23-fold differences. The authors also examine subsampling stability and layerwise patterns, and use INT8 reconstruction and three-checkpoint W8A8 experiments to investigate quantization relevance.

Critical strengths

The survey covers a useful range of recent checkpoints and shows substantial activation differences that parameter count alone does not explain. The released code and statistics, together with subsampling checks, make the measurements easier to examine.

Critical weaknesses
  1. For Qwen3.5-0.8B, Table 1 reports Top-100 = 3.23, while the released results give 103.5 at the reported peak component. The published tables need to be reconciled with the saved outputs.

  2. Section 3.1 includes SwiGLU gate pre-activations in the global maximum, but the measurement code explicitly excludes gate outputs. The scope of the main statistic needs clarification.

  3. Qwen3.5-0.8B's favorable 29.1 dB SQNR result comes from final_layernorm, although the experiment is described as a peak-hidden-layer comparison. Comparing different normalization locations confounds the interpretation.

  4. Across the eight max-abs reconstruction results, the peak–SQNR rank correlation is approximately +0.119, providing little support for larger peaks predicting lower SQNR. Comparisons with peak-to-RMS statistics at the actual quantized tensors would better establish the value of absolute maxima.

Questions for authors

N/A

1 comment

Thank you for the careful review and for cross-checking the manuscript against our released code and statistics. We performed a complete audit of the relevant implementation and saved outputs. The review identified two presentation inconsistencies, while the main maximum-activation measurements remain unchanged. We address each point below.

W1. Qwen3.5-0.8B Top-100 discrepancy.

We confirm that the Top-100 value of 3.23 in Table 1 is incorrect. We traced this to the table-generation logic: the global diagnostic aggregated the per-component Top-5 lists instead of reading the Top-100 values at the representative peak component. For Qwen3.5-0.8B, the global peak component is final_layernorm, where the correct Top-10 and Top-100 values are 112 and 103.5, respectively.

We will correct Table 1 and the corresponding aggregation code. Importantly, this error affects the Top-k diagnostic table only; the reported global maximum M = 122, its location, and the cross-model maximum-activation analyses are unchanged.

W2. Scope of the global maximum.

We agree that Section 3.1 is imprecise. The implementation records SwiGLU gate and router activations for auxiliary analyses, but these tensors are excluded from the reported global M. The actual statistic is computed over embedding outputs, post-residual hidden states, attention outputs, MLP/MoE outputs, and final-normalization outputs.

We will revise the definition in the manuscript so that it exactly matches the released implementation. This is a scope-description error rather than a measurement error.

W3. Qwen3.5-0.8B SQNR location.

The reviewer is correct. The 29.1 dB reconstruction result for Qwen3.5-0.8B was measured at final_layernorm, because this is its global peak location. The other seven checkpoints in that experiment were evaluated at hidden-state peak locations. We will therefore replace the inaccurate phrase "peak hidden layer" with "global peak activation location" and explicitly report the evaluated component.

We additionally measured the tensors that are actually quantized to A8. On the full 5,000-sample corpus, Qwen3.5-0.8B has an A8-site maximum of only M = 74, located at the layer-14 down_proj input, distinct from the final-normalization maximum used in the original sanity check.

W4. Absolute M, normalized statistics, and quantization difficulty.

We agree with the reviewer that the original eight-checkpoint rank correlation (Spearman rho approximately +0.119) does not support treating M as a standalone cross-model predictor of SQNR. We therefore performed the suggested analysis directly at the tensors actually quantized to A8.

For four representative checkpoints, we measured M, RMS, P99.9, M/RMS, and M/P99.9 at every A8 input tensor and compared them with reconstruction SQNR. Within every model, M/RMS is more strongly associated with static per-tensor SQNR than raw M:

  • Qwen3-30B-A3B: -0.79 vs. -0.46;
  • Qwen3-32B: -0.94 vs. -0.77;
  • Gemma-3-27B-it: -0.81 vs. -0.38;
  • Qwen3.5-0.8B: -0.97 vs. -0.13.

The Qwen3.5-0.8B result is particularly informative: its full-corpus A8 maximum is only 74, yet static W8A8 produces delta NLL = +1.217, whereas Qwen3-30B-A3B has a much larger A8 maximum of 1,736 but only delta NLL = +0.032. Thus, absolute magnitude alone does not order end-to-end quantization degradation across models.

We also evaluated dynamic per-token activation quantization using identical per-output-channel INT8 weights. Dynamic scaling recovers 87.5-99.7% of the static-per-tensor delta NLL across the four checkpoints, leaving only 0.4-2.4% PPL degradation. Static saturation is negligible (at most 1.6e-7 of evaluation values), indicating that the failure of static scaling is primarily resolution loss: sparse extremes enlarge the per-tensor scale and reduce the effective resolution for typical activations.

These results lead us to refine, rather than strengthen, the original claim. M is a descriptor of the worst-case activation range, but it is not a sufficient predictor of quantization difficulty. Normalized distributional statistics and the activation-scaling policy provide complementary information. We will revise the manuscript accordingly.

We thank the reviewer again for identifying the inconsistencies and for suggesting the normalized A8-site analysis. The audit corrects the Top-k table and measurement description, while the main cross-model M measurements remain unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.