- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper systematically measures maximum activation magnitudes across multiple open LLM families under a unified protocol, examines their variation with model size, architecture, and training stage, and provides preliminary evidence of their relevance to low-precision quantization.
Critical strengths
-
Broad empirical coverage. The study covers multiple model families, parameter scales, and model variants, including dense models, MoE models, vision-language models, and checkpoints from different training stages. This breadth provides a useful comparative overview of activation magnitudes.
-
Practical relevance. The results demonstrate that models with similar parameter counts can exhibit substantially different activation ranges. This observation motivates model-specific numerical precision and quantization strategies and provides useful guidance for pre-deployment assessment and quantization calibration.
Critical weaknesses
-
Limited insight beyond descriptive observations. The paper primarily presents an empirical survey of activation statistics. Although the measurements are useful, the analysis offers limited explanations for the observed differences or actionable insights beyond recommending that activation peaks be measured. The contribution would be stronger if the reported patterns supported more specific, testable conclusions.
-
Insufficient justification of absolute peak magnitude as an indicator of quantization difficulty. The paper does not clearly establish how reliably absolute activation peaks predict quantization error. A large peak may reflect either a broadly elevated activation scale or a small number of extreme outliers, which can have different quantization implications. The reported results also include low-peak models with worse reconstruction quality than higher-peak models, leaving the predictive value of the proposed statistic unclear.
-
An incomplete connection between measurement locations and quantization sites. Most global maxima occur in the residual stream, whereas residual connections and normalization operations remain in BF16 in the W8A8 experiment. It is therefore unclear how much of the extreme magnitude persists after normalization in the tensors actually quantized to A8, and which quantization sites mediate the observed degradation. The current evidence does not directly connect the measured residual-stream peaks to the resulting quantization error.
Questions for authors
-
Improve presentation and sharpen the contribution. Several figures, particularly Figures 2 and 4, occupy substantial space relative to their information content. A more compact layout and a tighter narrative would improve readability. The introduction should distinguish the principal new findings from supporting observations and state more clearly what the study adds beyond existing work.
-
Strengthen the validation of the proposed metric. Compare absolute peak magnitude against scale-normalized alternatives, such as peak-to-RMS and peak-to-percentile ratios, across a broader set of checkpoints. Measure these statistics at the actual A8 quantization sites and assess their association with both reconstruction error and end-to-end performance degradation.
Thank you for the review and for emphasizing the distinction between measuring an extreme activation range and establishing its actual quantization consequence. We agree that the submitted version did not sufficiently separate these questions. We therefore added analyses at the actual A8 quantization sites, compared absolute and normalized range statistics, and measured end-to-end W8A8 degradation.
W1. Beyond descriptive observations.
We measured the inputs to the GEMMs that are actually quantized to A8, rather than relying only on residual-stream maxima.
For three representative checkpoints, the largest A8 activation occurs at a
down_projinput:Thus, residual peaks do not propagate uniformly to quantized tensors, but extreme values can persist at actual A8 sites.
The matched Qwen Dense-MoE difference also becomes more localized: the Dense/MoE gap decreases from 23.4x in global M to 14.4x in global A8 M, while median A8 inputs show little separation. This suggests that the difference is concentrated in a small number of extreme sites, especially
down_proj, rather than being uniformly distributed across the model.W2. Absolute M versus normalized statistics.
We agree that absolute peak magnitude alone cannot distinguish a broadly elevated distribution from a sparse outlier.
We therefore measured M, RMS, P99.9, M/RMS, and M/P99.9 at every actual A8 input tensor. Within every evaluated model, M/RMS is more strongly associated with static reconstruction SQNR than raw M:
The low-peak Qwen3.5-0.8B checkpoint is a useful counterexample. Its full-corpus A8 maximum is only 74, compared with 1,736 for Qwen3-30B-A3B, yet static W8A8 gives delta NLL = +1.217 versus only +0.032 for A3B.
Even model-level normalized maxima do not fully explain this cross-model difference. What differs more directly is the low tail of reconstruction quality: 26/96 distinct A8 inputs in Qwen3.5-0.8B have static SQNR below 10 dB, compared with 3/192 for A3B. We treat this as a descriptive four-model observation rather than a universal law.
We therefore refine the claim: M measures worst-case activation range, while normalized statistics describe the sparsity of that range. Neither is a standalone model-level predictor of end-to-end degradation.
W3. Residual maxima versus actual A8 sites.
We directly instrumented q/k/v/o, gate/up/down, and routed-expert GEMM inputs.
The actual A8 maxima are:
layer_47.experts.down_proj;layer_63.down_proj;layer_0.down_proj;layer_14.down_proj.For Gemma-3-27B-it, the residual maximum attenuates by 26.9x before the largest A8 site. We therefore agree that residual M should not be interpreted as the value directly seen by the A8 quantizer.
Q1. Presentation and contribution.
We agree that the narrative can be tighter. We will reduce the space devoted to less informative figures, bring the key matched and quantization results closer to the main text, and distinguish more clearly between direct measurements, matched observational comparisons, and deployment implications.
Q2. Reconstruction and end-to-end degradation.
We additionally evaluated simulated W8A8 using identical per-output-channel INT8 weights and compared static per-tensor with dynamic per-token activation scaling.
Static to dynamic delta NLL:
Dynamic per-token scaling recovers 87.5-99.7% of the static degradation. Static saturation is negligible (at most 1.6e-7 of evaluation activations), indicating that the static failure is mainly resolution loss rather than clipping: sparse extremes enlarge the per-tensor step size and reduce resolution for typical values.
These results narrow the deployment claim. We no longer interpret M as a standalone predictor of quantization difficulty. Rather, M characterizes worst-case range, normalized local statistics characterize how sparse that range is, and actual degradation depends strongly on the quantization policy.
These are QDQ/fake-quantization results and do not measure INT8 kernel latency or throughput.
We thank the reviewer for motivating these analyses. They sharpen the deployment interpretation while preserving the paper's main contribution: a unified characterization of activation range across modern open checkpoints.