acceptodds
Under review as a conference paper at ICLR 2027

Dissecting Outlier Dynamics in LLM NVFP4 Pretraining

Abstract

Training large language models using 4-bit arithmetic enhances throughput and memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP4 mitigates quantization error via hierarchical microscaling, a persistent loss gap remains compared to BF16. This study conducts a longitudinal analysis of outlier dynamics across architecture during NVFP4 pretraining, focusing on where they localize, why they occur, and how they evolve. We find that, compared with Softmax Attention (SA), Linear Attention (LA) reduces per-tensor heavy tails but still exhibits block-level spikes under block quantization. Our analysis attributes outliers to specific architectural components: Softmax in SA, gating in LA, and SwiGLU in FFN, with “post-QK” operations exhibiting higher sensitivity. Notably, outliers evolve from transient spikes early in training to a small set of persistent hot channels (i.e., channels with persistently large magnitudes) in later stages. Based on these findings, we introduce Hot-Channel Patch (HCP), an online compensation mechanism that identifies hot channels and reinjects residuals using hardware-efficient kernels. We then develop CHON, an NVFP4 training recipe integrating HCP with post-QK operation protection. On GLA-1.3B trained for 60B tokens, CHON reduces the loss gap to BF16 from 0.94% to 0.58% while maintaining downstream accuracy.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
NVFP4, low-precision training, large language models, outlier dynamics, quantization, efficient training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_dv3QOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper focuses on the outlier phenomenon in LLM training. First, they identify that the outlier localizes at the softmax in softmax-attention (SA), gating in linear attention (LA), and SwiGLU in Feed-Forward Network (FFN). Regarding the evolution of the outlier, they suggest the outliers evolve from early transient spikes to later persistent outlier channels. They propose a recipe that combines forward-only Hot-Channel Patch (HCP), which injects weight and activation residual channels into an NVFP4 GEMM, with BF16 protection of sensitive post-QK projections.

Critical strengths
  1. The dynamic outlier analysis is practically insightful. Rather than only analyzing the post-training outlier phenomenon, they observe the outlier evolution across the whole training process for the SA / LA architecture.

  2. The method is hardware-friendly, the overhead is controllable, and the method provides meaningful improvement. It adds residual NVFP4-GEMM channels rather than using additional high-precision GEMM to compensate for the precision.

Critical weaknesses
  1. The advantage of the design can be supported by more evidence. The authors can show that the proposed method achieves better performance at the same overhead cost, or comparable accuracy at lower overhead. Possible baselines can be other FP4 training recipes with mixed higher-precision components/channels (e.g., OutControl that keeps some channels at higher precision [1]; Or, keeping the last 15% of layers at higher precision [2])

  2. The writing can be improved. For example, Sections 3.1/3.2 provide less direct support for HCP than Section 3.3, and the evidence supports descriptive observations rather than causal explanations. From my perspective, Section 3.3 gives a more valuable observation and supports the method better. I think the evolution of the outliers can be more detailed in the main text. Overall, I think the presentation should focus on stronger parts of the findings, rather than listing all the results and figures.

[1] TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control, ICML 2026
[2] What Matters for NVFP4 Training? A Scaling Study of Low-Precision Pre-Training Recipes, ICML 2026 AdaptFM Workshop

Questions for authors

Questions

  1. Does the proposed method outperform previous FP4-training methods that also set higher-precision components in overhead / performance? (See Weakness 1)

  2. Why is there still a 1,000-step refresh interval? Since the author states that the hot channels are persistent in later training. How much do the selected channels change during later training?

Suggestions For Presentation:

  • The writing needs to be clearer.

    • “Sensitive operations” are mentioned in the abstract and introduction without a clear definition. How is sensitivity defined, and how different are these operations from the others? Define the term when it first appears.
    • What does “a second-order MSE upper bound for idealized two-sided compensation” bound? The matrix multiplication output error? Please specify this clearly.
  • The figures do not clearly highlight the relevant evidence or support the accompanying claims.

    • In Figure 3, the weight-kurtosis panel does not use the same scale multiplier as the activation panel. What is the purpose of showing weight kurtosis, and what does its comparison with activation kurtosis establish?
    • Figures 6 and 7 show trends consistent with increasing outliers, but why do these establish Softmax and SwiGLU as the causes? Do other modules not show similar trends?
    • What do the two columns in Figure 8 represent? Are they different tensors? Please label and explain them.
  • I find the evidence in Section 3.3 (e.g., Figure 8) is stronger and easier to understand, and more directly relevant to the method than that in Sections 3.1–3.2. This appears to be the more solid contribution. Sections 3.1–3.2 mainly present observations without sufficiently convincing findings.

    • For the evidence in Section 3.3, I think the authors can also provide stronger aggregate statistics, rather than relying on examples of outlier-channel evolution in two tensors.
  • Please distinguish textual citations (\citet) from parenthetical citations (\citep). For example, some citations in lines 098–107 should use \citep; check similar cases throughout.

Overall, I appreciate the contribution and the method, so I recommend this work as a "weak accept"; however, the writing needs improvement before publication.

Reviewer_bhUGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes outlier dynamics during NVFP4 language-model pretraining and proposes HCP to compensate quantization residuals on selected channels. The compensation is incorporated into an expanded NVFP4 forward matrix multiplication. The full CHON recipe combines HCP with BF16 protection for selected sensitive projections to improve training quality and stability.

Critical strengths
  1. The paper studies how outliers develop during low-precision pretraining across architectures and training stages. This provides useful diagnostic context for understanding why NVFP4 training can retain a quality gap relative to higher-precision training.

  2. The proposed compensation has a concrete implementation that incorporates selected residual contributions into an NVFP4 forward matrix multiplication. The experiments include component ablations, multiple-seed results, and downstream evaluations; the full recipe produces a substantial improvement in the reported Qwen3-1.7B downstream evaluation.

Critical weaknesses
  1. The fixed compensation ratio of 9.09% is insufficiently justified. The paper does not establish why this value was selected or how sensitive training quality, stability, and actual runtime are to it. The synthetic patched-channel analysis does not substitute for a training-level budget sweep. Without this characterization, it is difficult to assess whether the selected ratio offers a robust quality–cost tradeoff or whether substantially smaller budgets would obtain similar benefits.

  2. The independent value of HCP, the more distinctive component, is not convincingly established by the available component-level evidence. In the GLA-1.3B ablation, HCP alone and high-precision protection alone yield similarly small improvements: loss decreases from 2.3919 to 2.3887 and 2.3884, respectively. Combining them gives 2.3869, so HCP adds only 0.0015 beyond protection alone. The much larger Qwen downstream improvement is reported for the combined recipe without a corresponding component decomposition. Thus, the strongest result cannot currently be attributed to HCP, and the evidence for its incremental benefit relative to its overhead remains limited.

Questions for authors
  1. How was the 9.09% compensation ratio selected? Please clarify whether existing training experiments characterize loss, stability, and measured runtime across compensation budgets.

  2. In the Qwen setting with the strongest downstream improvement, how much of the gain comes from HCP versus high-precision protection? Please provide the corresponding component results if available.

  3. What is the end-to-end quality–runtime tradeoff relative to protection alone and BF16 training, including channel selection, quantization, residual processing, and mixed-precision overhead?

Reviewer_fYUMOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at how outliers change when training language models with NVFP4. The authors propose Hot-Channel Patch (HCP), which picks a small set of channels with large quantization errors and adds corrections for both weights and activations. They pack these corrections into the same 4-bit matrix multiplication. Their full recipe, CHON, also runs some sensitive operations in higher precision.

Critical strengths
  1. I find the analysis of how outliers change during training useful. The observation that they increasingly concentrate in a small set of channels gives a sensible reason to focus correction on those channels. This could also help guide other FP4 training methods.

  2. HCP has an appealing implementation: it adds corrections for both weights and activations within one expanded 4-bit matrix multiplication. The ablation in Table 11 also separates HCP from the higher-precision protection, providing evidence that HCP itself contributes to the improvement.

Critical weaknesses
  1. The practical systems benefit is still unclear. Given the paper’s focus on infrastructure and systems area, I would expect a more complete hardware evaluation. This should include peak and persistent GPU memory, temporary buffers, additional FLOPs, and end-to-end throughput for BF16, baseline NVFP4, and CHON under the same settings. The paper describes the 2k/n increase in GEMM arithmetic, but this does not capture the full cost of the compensation path. In particular, I could not find measured memory comparisons. If the extra buffers and higher-precision operations make CHON’s peak memory comparable to or higher than BF16 or FP8, the authors need to show what throughput advantage remains and why the trade-off is worthwhile. FP8 is also an important alternative: Table 10 reports a loss of 2.1320 for FP8, compared with 2.1433 for CHON and 2.1337 for BF16. Since FP8 already achieves better quality in this setting, the paper should show whether CHON offers enough speed or memory savings to justify the added compensation.

  2. The baseline comparisons need stronger checks and a matched-budget control. Some BF16 baselines are surprisingly weak. In Table 1, Gated-DeltaNet-340M has an average score of 39.5 with BF16, compared with 53.2 for NVFP4 and 53.3 for CHON. For Qwen3-1.7B, the corresponding BF16 and CHON scores are 32.1 and 38.5. These are substantial gaps, and I do not see enough diagnostics to explain them. Training curves, checkpoint details, precision settings, and relevant hyperparameter checks would help establish that these baselines are working as intended. The appendix’s learning-rate and seed studies do not resolve these particular cases. Also, CHON uses approximately 9%–15% more GPU-hours than baseline NVFP4. A key missing comparison is to train the baseline NVFP4 recipe longer using the same GPU-hour budget, with an appropriately adjusted learning-rate schedule. Without this control, it is unclear whether compensation is a better use of the extra budget than simply spending it on more baseline training.

Questions for authors
  1. Could you report peak and persistent GPU memory, temporary buffer usage, and end-to-end throughput for BF16, baseline NVFP4, and CHON under the same settings? Beyond the stated 2k/n GEMM expansion, it would help to account for the extra computation in the full compensation path. These measurements are needed to understand how much of NVFP4’s systems advantage remains after adding HCP and BF16 protection. Could you also include FP8 in these hardware measurements and in the GLA-1.3B loss comparison in Table 3? In settings where FP8 matches or outperforms CHON, such as Mamba-2 in Table 10, what speed or memory advantage does CHON provide?

  2. Could you explain the unusually weak BF16 results in Table 1? For Gated-DeltaNet-340M, the average score is 39.5 with BF16, versus 53.2 with NVFP4 and 53.3 with CHON. For Qwen3-1.7B, BF16 scores 32.1 versus 38.5 with CHON. Could you show convergence curves and describe how the baselines were tuned and checked for these particular settings?

  3. Since CHON takes approximately 9%–15% more GPU-hours, could you compare it with baseline NVFP4 trained longer using the same GPU-hour budget, with a learning-rate schedule appropriate for the longer run? This would help establish whether compensation is a better use of the extra time than simply training the baseline longer.

  4. Could you enlarge the plots and make the captions more informative? I had trouble reading the labels and legends in several figures, including Figures 8 and 12. The captions also sometimes left me unsure what exactly was being measured or which comparison supported the claim in the text. For the diagnostic plots, please clarify which tensors and layers are shown, how the plotted values are calculated, and what the reader should learn from the figure. This made parts of the analysis difficult to follow.

Reviewer_Z1R6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates outlier dynamics during NVFP4 LLM pretraining, observing a transition from transient spikes to persistent hot channels. It proposes CHON, which combines two-sided channel residual compensation with higher-precision protection of sensitive operators. On GLA-1.3B trained for 60B tokens, CHON reduces the loss gap relative to BF16 from 0.95% to 0.59%. Cross-architecture evaluations, component ablations, and three-seed experiments provide additional evidence of effectiveness.

Critical strengths

The paper studies how outliers evolve during low-precision pretraining and distinguishes tensor-level heavy tails from block-level extremes. This distinction is relevant to microscaling quantization. The persistence of hot channels also provides an empirical motivation for periodically refreshing the selected channels.

HCP implements two-sided weight and activation residual compensation through a concatenated GEMM, with a clear engineering motivation. The GLA-1.3B experiments demonstrate consistent loss improvements, while component ablations, three-seed experiments, and cross-architecture evaluations strengthen the evidence. The theoretical analysis appropriately distinguishes idealized compensation identities from actual execution errors.

Critical weaknesses

The empirical analysis is extensive, but the causal explanations for outlier formation and persistence are less firmly established. Several interpretations rely on observed correlations across training stages and architectures; controlled interventions would strengthen the connection between these observations and the proposed mechanisms.

Questions for authors

Q1: With a global batch size of 4M tokens, 60B tokens correspond to approximately 15k optimizer updates, whereas Figure 12 shows roughly 30k–40k steps. Please clarify the step definition and actual training budgets.

Q2: In Table 1, Gated-DeltaNet-340M improves from an average score of 39.5 under BF16 to 53.3 under CHON. Were training budgets, checkpoint selection, and evaluation settings identical? What explains this substantial difference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.