- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper replaces RMSNorm with an online least-squares affine approximation through a scheduled interpolation. Experiments on three GPT-OSS-style MoE models show that late removal preserves much of the downstream performance, although quality and stability deteriorate at larger scales.
Critical strengths
- Experiments extend normalization removal to billion-parameter MoE models and include downstream evaluation.
- Schedule sweeps document unsuccessful transitions as well as successful endpoints, providing useful evidence about the limitations of scalar affine replacements.
Critical weaknesses
- A matched TaperNorm baseline is missing. Since gated removal already exists, the contribution rests on simplifying its replacement branch and removing auxiliary regularization. Without testing these components separately under the same setup, it is unclear whether failures arise from the scalar surrogate, missing scale anchoring, or schedule choice.
- Successful schedules reach only at the final training step. This demonstrates a usable inference checkpoint, but provides little evidence for stable training after removal. One continuation ablation instead enters a sustained high-loss regime. These results do not establish that normalization serves primarily an optimization role.
- Scale dependence is not cleanly isolated. Width, depth, expert count, token budget, learning rate, and removal schedule all change across scales. Seed sensitivity is measured only for the Small normalized baseline; homotopy results and downstream comparisons use one seed. Robustness of the selected schedules therefore remains uncertain.
- During training, affine coefficients continue to depend on batch statistics computed from RMSNorm, including at . The output is statistics-free only once those coefficients are frozen. A continuation experiment with frozen coefficients and the RMSNorm branch removed would clarify what has actually been eliminated.
Questions for authors
Typos
- Section 4.2 says Medium inherits Small’s 0.94/0.06 schedule, whereas Section 4.3 and Table 5 specify 0.96/0.04.
- “summerizes” in Appendix B should be “summarizes.”
2 comments
During training, affine coefficients continue to depend on batch statistics computed from RMSNorm, including at (g=0).
This is correct, and we clarify this distinction in the revised manuscript (Section 3.2 and Appendix A). We deliberately retain RMSNorm evaluation during training to continuously update the affine coefficients through online least-squares regression. This gives the linear endpoint the best possible opportunity to approximate the normalized computation using current activation statistics, rather than relying on a potentially stale, frozen calibration.
At (g=0), the model output itself is computed exclusively from the affine branch and no longer depends on per-token RMS statistics. However, RMSNorm is still evaluated during training to update the regression coefficients. Importantly, even with this continued assistance, our post-transition experiment shows that further optimization at the affine endpoint enters a sustained high-loss regime (Figure 4). This suggests that the limitation is not simply the absence of updated normalization statistics, but potentially the inability of a scalar affine transformation to provide the nonlinear, bounded scale control needed for stable optimization.
This motivates our complementary investigation of nonlinear sample-independent transformations. In particular, tanh(20x) trains from initialization without explicit normalization statistics (Section 4.5), suggesting that bounded nonlinearities may provide a more suitable alternative. We have not yet established that these alternatives fully recover RMSNorm performance, but consider this an important direction for future work.
Section 4.2 says Medium inherits Small's 0.94/0.06 schedule, whereas Section 4.3 and Table 5 specify 0.96/0.04. Also, “summerizes” should be “summarizes.”
Both issues have been addressed in the revised manuscript. Medium consistently uses the 0.94/0.06 schedule, and the spelling is corrected to “summarizes.” We thank the reviewer for identifying these inconsistencies and for helping us improve the clarity and accuracy of the manuscript.
We respectfully thank reviewer ERP8 for the careful evaluation of our work. In the following, we will address the reviewer's concerns.
We have added a reproduction of the original TaperNorm protocol in the same GPT-OSS-style MoE setting (Section 4.2). This experiment retains the learnable per-feature scaling and auxiliary scale-anchoring objective. Small and Medium diverge early, while Large becomes unstable before reaching (g=0). Thus, none of the three configurations demonstrates a stable normalization-free endpoint. These results provide an empirical comparison with the original framework.
We agree that reaching (g=0) only at the final step does not establish continued trainability. We have also explicitly tested continued training after reaching the affine endpoint. In the failure case shown in Figure 4, we continue training for approximately 1,000 steps at (g=0). The loss rapidly increases and enters a sustained high-loss regime.
Importantly, our implementation continues evaluating RMSNorm during this experiment solely to update the online least-squares coefficients. The affine branch therefore receives continuously refreshed regression estimates rather than relying on frozen calibration coefficients. Even with this assistance, continued optimization becomes unstable. This suggests that a scalar affine transformation lacks the nonlinear (and possibly bounded) scale-control mechanism required for stable optimization.
This observation motivates exploring bounded, sample-independent nonlinear alternatives. Our tanh experiment (Section 4.5) demonstrates that tanh(20x) can train for the full training horizon without explicit normalization statistics from initialization, although it remains below the RMSNorm baseline. Other bounded transformations, such as arctan, are potential directions for future investigation. We therefore distinguish between obtaining a normalization-free inference checkpoint and demonstrating a stable normalization-free training regime.
We clarify that the experiments are designed as a scale study using scale-appropriate model and training configurations, rather than as an intervention that changes parameter count while holding every other variable fixed. We evaluate models with 1.031B, 3.352B, and 6.090B parameters, trained on 8.39B, 16.78B, and 33.55B tokens, respectively. This corresponds to known optimal scaling law regimes for LLMs. All configurations use the same ClimbMix data, 32K tokenizer, sequence length of 4,096, and global batch size of 256. Peak learning rates are adjusted across scales (6e-4, 4e-4, and 3e-4), following scale-aware LLM training practice (Appendix B, Tables 5–6).
Crucially, within each scale, the normalized and homotopy configurations use the same architecture, training budget, data, and optimization setup; the normalization treatment is the experimental intervention. Thus, the within-scale comparisons are controlled, while the across-scale comparisons reflect appropriately scaled training configurations. We do not claim to isolate parameter count as the sole causal variable. We also now report three independent seeds for the principal training configurations at all three scales (Table 1). The downstream benchmark table reports standard errors where available, rather than across-seed standard deviations.