- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper focuses on the outlier phenomenon in LLM training. First, they identify that the outlier localizes at the softmax in softmax-attention (SA), gating in linear attention (LA), and SwiGLU in Feed-Forward Network (FFN). Regarding the evolution of the outlier, they suggest the outliers evolve from early transient spikes to later persistent outlier channels. They propose a recipe that combines forward-only Hot-Channel Patch (HCP), which injects weight and activation residual channels into an NVFP4 GEMM, with BF16 protection of sensitive post-QK projections.
Critical strengths
-
The dynamic outlier analysis is practically insightful. Rather than only analyzing the post-training outlier phenomenon, they observe the outlier evolution across the whole training process for the SA / LA architecture.
-
The method is hardware-friendly, the overhead is controllable, and the method provides meaningful improvement. It adds residual NVFP4-GEMM channels rather than using additional high-precision GEMM to compensate for the precision.
Critical weaknesses
-
The advantage of the design can be supported by more evidence. The authors can show that the proposed method achieves better performance at the same overhead cost, or comparable accuracy at lower overhead. Possible baselines can be other FP4 training recipes with mixed higher-precision components/channels (e.g., OutControl that keeps some channels at higher precision [1]; Or, keeping the last 15% of layers at higher precision [2])
-
The writing can be improved. For example, Sections 3.1/3.2 provide less direct support for HCP than Section 3.3, and the evidence supports descriptive observations rather than causal explanations. From my perspective, Section 3.3 gives a more valuable observation and supports the method better. I think the evolution of the outliers can be more detailed in the main text. Overall, I think the presentation should focus on stronger parts of the findings, rather than listing all the results and figures.
[1] TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control, ICML 2026
[2] What Matters for NVFP4 Training? A Scaling Study of Low-Precision Pre-Training Recipes, ICML 2026 AdaptFM Workshop
Questions for authors
Questions
-
Does the proposed method outperform previous FP4-training methods that also set higher-precision components in overhead / performance? (See Weakness 1)
-
Why is there still a 1,000-step refresh interval? Since the author states that the hot channels are persistent in later training. How much do the selected channels change during later training?
Suggestions For Presentation:
-
The writing needs to be clearer.
- “Sensitive operations” are mentioned in the abstract and introduction without a clear definition. How is sensitivity defined, and how different are these operations from the others? Define the term when it first appears.
- What does “a second-order MSE upper bound for idealized two-sided compensation” bound? The matrix multiplication output error? Please specify this clearly.
-
The figures do not clearly highlight the relevant evidence or support the accompanying claims.
- In Figure 3, the weight-kurtosis panel does not use the same scale multiplier as the activation panel. What is the purpose of showing weight kurtosis, and what does its comparison with activation kurtosis establish?
- Figures 6 and 7 show trends consistent with increasing outliers, but why do these establish Softmax and SwiGLU as the causes? Do other modules not show similar trends?
- What do the two columns in Figure 8 represent? Are they different tensors? Please label and explain them.
-
I find the evidence in Section 3.3 (e.g., Figure 8) is stronger and easier to understand, and more directly relevant to the method than that in Sections 3.1–3.2. This appears to be the more solid contribution. Sections 3.1–3.2 mainly present observations without sufficiently convincing findings.
- For the evidence in Section 3.3, I think the authors can also provide stronger aggregate statistics, rather than relying on examples of outlier-channel evolution in two tensors.
-
Please distinguish textual citations (
\citet) from parenthetical citations (\citep). For example, some citations in lines 098–107 should use\citep; check similar cases throughout.
Overall, I appreciate the contribution and the method, so I recommend this work as a "weak accept"; however, the writing needs improvement before publication.