acceptodds
Under review as a conference paper at ICLR 2027

Normalization removal at scale: RMSNorm-to-affine transition in GPT-OSS-style MoE transformers

Abstract

Although normalization is nearly universal in modern pre-norm Transformers, it is still unclear whether input-dependent normalization statistics are required by the learned computation or if they primarily serve to stabilize optimization. To answer this question, we examined three GPT-OSS-style mixture-of-experts (MoE) transformer variants with 1 billion, 3 billion, and 6 billion total parameters each. These models were trained on 8 billion, 16 billion, and 32 billion processed tokens, respectively, and evaluated on 8 common NLP benchmarks. Our main intervention is a homotopy, or convex combination, from RMSNorm to a scalar affine map at each layer. This combination is parameterized by a coefficient that decreases from 1 (pure RMSNorm) to 0 (pure affine function) during training. The optimal slope and intercept of the affine function were chosen to minimize the mean squared error between RMSNorm and its best linear approximation. Across scales, the normalization-free endpoints retain most of the zero-shot downstream performance of their normalized baselines. Nevertheless, the performance gap becomes more pronounced at larger scales. We also trained learnable element-wise alternatives from step zero. Among the trained variants, the variant initialized as tanh(20x) achieved the best performance, though it remained behind the RMSNorm baseline. The results demonstrate that a simple affine endpoint and a carefully scheduled gate can eliminate the need for normalization statistics. However, the results also demonstrate that the schedule's effectiveness diminishes as the scale increases. Altogether, these results suggest treating normalization as primarily an optimization-stability mechanism and searching for richer, self-normalizing non-linearities that can replace its stabilizing role without the need for explicit, sample-dependent normalization statistics. The code and checkpoints to reproduce this study are available at: URL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
normalization removal, RMSNorm, Transformers, mixture-of-experts, large language models, normalization-free networks, normalization, GPT-OSS, training stability, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ERP8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper replaces RMSNorm with an online least-squares affine approximation through a scheduled interpolation. Experiments on three GPT-OSS-style MoE models show that late removal preserves much of the downstream performance, although quality and stability deteriorate at larger scales.

Critical strengths
  • Experiments extend normalization removal to billion-parameter MoE models and include downstream evaluation.
  • Schedule sweeps document unsuccessful transitions as well as successful endpoints, providing useful evidence about the limitations of scalar affine replacements.
Critical weaknesses
  • A matched TaperNorm baseline is missing. Since gated removal already exists, the contribution rests on simplifying its replacement branch and removing auxiliary regularization. Without testing these components separately under the same setup, it is unclear whether failures arise from the scalar surrogate, missing scale anchoring, or schedule choice.
  • Successful schedules reach g=0g=0 only at the final training step. This demonstrates a usable inference checkpoint, but provides little evidence for stable training after removal. One continuation ablation instead enters a sustained high-loss regime. These results do not establish that normalization serves primarily an optimization role.
  • Scale dependence is not cleanly isolated. Width, depth, expert count, token budget, learning rate, and removal schedule all change across scales. Seed sensitivity is measured only for the Small normalized baseline; homotopy results and downstream comparisons use one seed. Robustness of the selected schedules therefore remains uncertain.
  • During training, affine coefficients continue to depend on batch statistics computed from RMSNorm, including at g=0g=0. The output is statistics-free only once those coefficients are frozen. A continuation experiment with frozen coefficients and the RMSNorm branch removed would clarify what has actually been eliminated.
Questions for authors

Typos

  • Section 4.2 says Medium inherits Small’s 0.94/0.06 schedule, whereas Section 4.3 and Table 5 specify 0.96/0.04.
  • “summerizes” in Appendix B should be “summarizes.”
2 comments

We respectfully thank reviewer ERP8 for the careful evaluation of our work. In the following, we will address the reviewer's concerns.

A matched TaperNorm baseline is missing.

We have added a reproduction of the original TaperNorm protocol in the same GPT-OSS-style MoE setting (Section 4.2). This experiment retains the learnable per-feature scaling and auxiliary scale-anchoring objective. Small and Medium diverge early, while Large becomes unstable before reaching (g=0). Thus, none of the three configurations demonstrates a stable normalization-free endpoint. These results provide an empirical comparison with the original framework.

Successful schedules reach (g=0) only at the final training step. This demonstrates a usable inference checkpoint, but provides little evidence of stable training after removal.

We agree that reaching (g=0) only at the final step does not establish continued trainability. We have also explicitly tested continued training after reaching the affine endpoint. In the failure case shown in Figure 4, we continue training for approximately 1,000 steps at (g=0). The loss rapidly increases and enters a sustained high-loss regime.

Importantly, our implementation continues evaluating RMSNorm during this experiment solely to update the online least-squares coefficients. The affine branch therefore receives continuously refreshed regression estimates rather than relying on frozen calibration coefficients. Even with this assistance, continued optimization becomes unstable. This suggests that a scalar affine transformation lacks the nonlinear (and possibly bounded) scale-control mechanism required for stable optimization.

This observation motivates exploring bounded, sample-independent nonlinear alternatives. Our tanh experiment (Section 4.5) demonstrates that tanh(20x) can train for the full training horizon without explicit normalization statistics from initialization, although it remains below the RMSNorm baseline. Other bounded transformations, such as arctan, are potential directions for future investigation. We therefore distinguish between obtaining a normalization-free inference checkpoint and demonstrating a stable normalization-free training regime.

Scale dependence is not cleanly isolated, and seed sensitivity remains uncertain.

We clarify that the experiments are designed as a scale study using scale-appropriate model and training configurations, rather than as an intervention that changes parameter count while holding every other variable fixed. We evaluate models with 1.031B, 3.352B, and 6.090B parameters, trained on 8.39B, 16.78B, and 33.55B tokens, respectively. This corresponds to known optimal scaling law regimes for LLMs. All configurations use the same ClimbMix data, 32K tokenizer, sequence length of 4,096, and global batch size of 256. Peak learning rates are adjusted across scales (6e-4, 4e-4, and 3e-4), following scale-aware LLM training practice (Appendix B, Tables 5–6).

Crucially, within each scale, the normalized and homotopy configurations use the same architecture, training budget, data, and optimization setup; the normalization treatment is the experimental intervention. Thus, the within-scale comparisons are controlled, while the across-scale comparisons reflect appropriately scaled training configurations. We do not claim to isolate parameter count as the sole causal variable. We also now report three independent seeds for the principal training configurations at all three scales (Table 1). The downstream benchmark table reports standard errors where available, rather than across-seed standard deviations.

During training, affine coefficients continue to depend on batch statistics computed from RMSNorm, including at (g=0).

This is correct, and we clarify this distinction in the revised manuscript (Section 3.2 and Appendix A). We deliberately retain RMSNorm evaluation during training to continuously update the affine coefficients through online least-squares regression. This gives the linear endpoint the best possible opportunity to approximate the normalized computation using current activation statistics, rather than relying on a potentially stale, frozen calibration.

At (g=0), the model output itself is computed exclusively from the affine branch and no longer depends on per-token RMS statistics. However, RMSNorm is still evaluated during training to update the regression coefficients. Importantly, even with this continued assistance, our post-transition experiment shows that further optimization at the affine endpoint enters a sustained high-loss regime (Figure 4). This suggests that the limitation is not simply the absence of updated normalization statistics, but potentially the inability of a scalar affine transformation to provide the nonlinear, bounded scale control needed for stable optimization.

This motivates our complementary investigation of nonlinear sample-independent transformations. In particular, tanh(20x) trains from initialization without explicit normalization statistics (Section 4.5), suggesting that bounded nonlinearities may provide a more suitable alternative. We have not yet established that these alternatives fully recover RMSNorm performance, but consider this an important direction for future work.

Section 4.2 says Medium inherits Small's 0.94/0.06 schedule, whereas Section 4.3 and Table 5 specify 0.96/0.04. Also, “summerizes” should be “summarizes.”

Both issues have been addressed in the revised manuscript. Medium consistently uses the 0.94/0.06 schedule, and the spelling is corrected to “summarizes.” We thank the reviewer for identifying these inconsistencies and for helping us improve the clarity and accuracy of the manuscript.

Reviewer_hfibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether input-dependent RMSNorm operations can be removed from pretrained-style mixture-of-experts (MoE) Transformers through a gradual transition to sample-independent affine transformations.

The authors propose a simplified homotopy-based normalization removal procedure. Each RMSNorm module is interpolated with a scalar affine approximation, (A(x)=ax+b), where the coefficients are estimated through online least-squares regression using exponentially averaged activation statistics. A shared gate gradually decreases from one to zero, ultimately replacing RMSNorm with its affine approximation.

The approach is evaluated on three GPT-OSS-style MoE architectures containing approximately 1.03B, 3.35B, and 6.09B total parameters, trained with 8.39B, 16.78B, and 33.55B tokens, respectively.

The experiments investigate different gate schedules, training stability, validation loss, and zero-shot performance on seven downstream benchmarks and WikiText. The authors also examine pointwise nonlinear normalization replacements based on learnable tanh transformations.

The main observations are that:

  1. Late and sufficiently gradual homotopy transitions can produce normalization-free final checkpoints with relatively limited performance degradation at smaller scales.

  2. Successful transitions become increasingly sensitive to the gate schedule at larger model sizes, particularly when the gate approaches zero.

  3. The resulting normalization-free checkpoints consistently underperform their RMSNorm baselines, with larger performance gaps observed at the largest evaluated scale.

  4. Pointwise nonlinear replacements can be trained without normalization from initialization, but their performance remains below that of RMSNorm.

The paper concludes that normalization may primarily facilitate stable optimization rather than constitute an indispensable computation at inference time.

Critical strengths
  1. A useful empirical investigation of normalization removal in relatively large MoE architectures.

A meaningful strength is the extension of normalization-removal experiments to GPT-OSS-style MoE architectures with up to 6.09B total parameters and approximately 1.23B active parameters.

Previous normalization-removal studies have primarily focused on smaller or dense Transformer architectures. Investigating whether similar behavior persists in MoE models is relevant because normalization affects the inputs to attention modules, expert networks, and routing computations.

The extensive gate-schedule experiments, particularly at the Small and Large scales, provide useful observations about the difficulty of transitioning away from input-dependent normalization.

For example, the Small-scale experiments identify a relatively narrow region of successful late transitions, whereas the Large-scale experiments show that several schedules can remain apparently stable until the gate approaches zero, at which point the loss increases sharply.

These results are consequential because they expose practical limitations of normalization-removal procedures that may not be apparent from experiments restricted to smaller architectures.

  1. Transparent reporting of negative results and failure modes.

The paper does not merely report successful normalization-free endpoints. It explicitly documents unsuccessful gate schedules, loss spikes, increasing performance degradation with scale, and the inferior performance of alternative pointwise transformations.

The reported validation-loss gaps increase from approximately 0.034 at Small scale to 0.078 at Large scale, while the average downstream accuracy gap increases from 0.61 to 3.30 percentage points.

The authors also acknowledge that the current implementation retains RMSNorm computations during training and therefore does not demonstrate an actual training-throughput benefit.

This transparency improves the credibility of the empirical observations and makes the study potentially useful for researchers investigating normalization-free architectures.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparison with the most relevant existing normalization-removal methods.

The principal methodological contribution is closely related to existing approaches, particularly TaperNorm (Kanavalau et al., 2026). TaperNorm already introduces a gated transition from standard normalization to a sample-independent linear or affine transformation, using calibration statistics and a scheduled gate. It also investigates the importance of scale anchoring when removing the final normalization layer.

The proposed method primarily simplifies that framework by removing trainable per-feature scaling and auxiliary regularization, while selecting a scalar affine approximation through ordinary least squares.

Although this simplification is reasonable, the paper does not establish that it offers a meaningful advantage over the original formulation. In particular, there is no direct comparison against TaperNorm or the sequential normalization-removal procedure of Baroni et al. (2025) under matched model architectures, training budgets, and evaluation protocols.

This omission is especially important because the proposed method is substantially less expressive than TaperNorm's learned per-feature transformation, and the experiments demonstrate persistent performance degradation relative to RMSNorm.

The paper also removes the final normalization without investigating the scale-anchoring mechanisms identified in prior work. Consequently, the observed instability may partly result from deliberately excluding components already known to be important for successful normalization removal.

Furthermore, the claimed practical simplicity is not accompanied by measured inference-speed or memory benefits. The reported training throughput is approximately 6% lower than that of the RMSNorm baseline.

Overall, the paper provides an interesting ablation and scale-extension study, but its incremental methodological contribution is not sufficiently substantiated by comparisons with existing methods.

This substantially limits its significance.

  1. Insufficient evidence for stable and generalizable normalization-free endpoints, particularly at larger scales.

The central empirical claim is that input-dependent normalization can be eliminated while retaining most of the model's predictive capability. However, the experimental protocol does not convincingly establish the stability or reproducibility of the resulting normalization-free models.

First, successful homotopy transitions reach a gate value of zero only at the final training step. The models therefore spend almost their entire training trajectories using RMSNorm, and the resulting checkpoints are evaluated immediately after normalization removal. More importantly, Section 4.3 reports that continuing training after reaching the normalization-free endpoint can lead to a sustained high-loss regime.

This raises an important distinction between obtaining an apparently functional checkpoint at the exact transition endpoint and demonstrating a stable normalization-free model that can support continued optimization.

Second, the reported homotopy results rely on a single training seed for each principal configuration. Appendix F provides three-seed variability estimates for the Small RMSNorm baseline, but corresponding multi-seed evaluations are not provided for the selected homotopy configurations.

Given the pronounced sensitivity to gate schedules, particularly at Large scale, single-seed experiments are insufficient to establish that the reported successful transitions are reliable.

Third, the claim that normalization removal becomes more difficult with model scale is potentially confounded. The three model configurations differ simultaneously in width, depth, learning rate, training duration, token budget, and selected gate schedule. These differences prevent a clean attribution of the observed degradation specifically to model scale.

Finally, the interpretation that normalization primarily serves as optimization scaffolding is stronger than the presented evidence warrants. The experiments show that certain trained networks can tolerate replacing normalization with an affine approximation, but with measurable performance degradation. This does not establish that normalization's input-dependent behavior is unimportant to the representational or predictive properties of the original network.

The lack of robust post-transition training experiments and multi-seed validation is particularly consequential because stable normalization removal is the central objective of the study.

Questions for authors
  1. Comparison against TaperNorm and the role of the final normalization layer (Critical Weakness 1).

Can the authors provide a direct comparison with TaperNorm under the same GPT-OSS-style MoE architecture and training budget?

In particular, how does the proposed scalar least-squares affine endpoint compare with TaperNorm's learned per-feature scaling, with and without its auxiliary scale-anchoring mechanism?

It would also be important to distinguish the effects of removing internal RMSNorm modules from those of removing the final output normalization.

Does retaining the final RMSNorm, or introducing an explicit output-scale anchor, substantially improve the stability and predictive quality of the affine endpoints?

These comparisons could materially change my assessment of both the novelty and practical value of the proposed simplification.

  1. Stability and reproducibility after reaching the normalization-free endpoint (Critical Weakness 2).

Can the authors provide multi-seed evaluations of the selected homotopy schedules, particularly at Large scale, and investigate what happens when training continues for a meaningful number of steps after the gate reaches zero?

The observation in Section 4.3 that extended training at zero gate can result in sustained high loss raises concerns about whether the resulting endpoints are genuinely stable.

I would like to understand whether the selected successful checkpoints remain trainable, whether their performance degrades under continued optimization, and how frequently successful transitions occur across independent training seeds.

Evidence that the normalization-free endpoints remain stable beyond the exact final transition step would significantly strengthen the paper.

  1. Scale-dependent behavior and practical benefits (Critical Weaknesses 1 and 2).

Can the authors clarify how much of the reported scale-dependent degradation remains when the same gate-schedule family and comparable transition budgets are used across model sizes?

In particular, Appendix B states that Large-H resumes from the normalized checkpoint at step 31,000, whereas the reported staircase schedule begins its transition at approximately step 30,720. How is the gate initialized when resuming from this checkpoint, and does the implementation introduce an immediate discontinuity?

Additionally, can the authors report the actual inference latency and memory consumption after replacing the zero-gate homotopy modules with their affine endpoints?

These results would clarify whether the reported scale sensitivity is intrinsic to normalization removal and whether the proposed approach provides a practical advantage beyond producing a normalization-free final checkpoint.

2 comments

We would like to express our gratitude to reviewer hfib for his insightful commentary. We provide our responses to the comments and questions below.

The methodological contribution is incremental, and a matched TaperNorm baseline is missing.

We have substantially expanded the experimental study in response to this concern. First, Section 4.2 reproduces the original TaperNorm protocol in our GPT-OSS-style MoE setting, retaining its learnable per-feature scaling and auxiliary scale-anchoring objective. In this setting, Small and Medium exhibit early loss divergence, while Large becomes unstable before reaching (g=0).

Beyond this reproduction, our main contribution is a systematic ablation of the normalization-removal mechanism: we remove the learnable per-feature scaling and auxiliary regularization, replace the taper branch with a scalar least-squares affine approximation, and investigate multiple gate schedules, including linear, polynomial, sinusoidal, and staircase schedules (Appendix D). This schedule-sensitivity analysis was not performed in the same relatively large MoE setting in the original TaperNorm study, which focused on smaller dense models and identified larger-scale experiments as an open direction.

Our findings are primarily negative but informative: although some schedules can reach a normalization-free endpoint, the linear replacement becomes increasingly brittle with scale, incurs measurable downstream quality degradation, and does not provide a broad, robust transition window at Large scale. Even continued optimization at the affine endpoint becomes unstable. We therefore position the contribution as a scale-extension and empirical ablation study that exposes the limitations of linear normalization removal, rather than proposing a fundamentally new homotopy mechanism.

The role of the final normalization layer and output-scale anchoring remains unclear.

In our early experiments (not reported here) at the evaluated scales, removing the final RMSNorm did not appear to have a substantial impact on the overall results. This suggests that its contribution may be less critical in the relatively deep networks considered here, although we do not claim that this generalizes to all architectures. We therefore chose to simplify the intervention by targeting all normalization layers, including the final one, rather than retaining a normalization-dependent component in an otherwise normalization-free model. The TaperNorm reproduction, in contrast, retains the final RMSNorm and its auxiliary scale-anchoring objective (Section 4.2). Our objective is to investigate the limits of fully normalization-free endpoints, rather than optimize a partially normalized configuration.

There is insufficient evidence for stable and reproducible normalization-free endpoints, particularly at larger scales.

We now report three-seed results for the principal normalized and homotopy configurations at all three scales (Table 1), including standard deviations for training and validation losses. Furthermore, the continuation experiment shows that continuing training at (g=0) for approximately 1,000 steps leads to a sustained high-loss regime (Figure 4). Thus, our results demonstrate that selected schedules can produce normalization-free final checkpoints, not that these checkpoints support stable continued optimization.

Scale dependence is not cleanly isolated, and seed sensitivity remains uncertain.

We clarify that the experiments are designed as a scale study using scale-appropriate model and training configurations, rather than as an intervention that changes parameter count while holding every other variable fixed. We evaluate models with 1.031B, 3.352B, and 6.090B parameters, trained on 8.39B, 16.78B, and 33.55B tokens, respectively. This corresponds to known optimal scaling law regimes for LLMs. All configurations use the same ClimbMix data, 32K tokenizer, sequence length of 4,096, and global batch size of 256. Peak learning rates are adjusted across scales (6e-4, 4e-4, and 3e-4), following scale-aware LLM training practice (Appendix B, Tables 5–6).

Importantly, within each scale, the normalized and homotopy configurations use the same architecture, training budget, data, and optimization setup; the normalization treatment is the experimental intervention. We therefore consider the within-scale comparisons controlled, while the across-scale comparisons evaluate the behavior of appropriately scaled model configurations. We do not claim to isolate parameter count as the sole causal variable. Regarding seed sensitivity, the principal training configurations now use three independent seeds. Table 3 additionally reports benchmark standard errors.

The interpretation that normalization primarily serves as optimization scaffolding is stronger than the presented evidence warrants.

We agree that this interpretation must be qualified. Although the (g=0) endpoint no longer requires per-token RMS statistics in its evaluated affine computation, it consistently underperforms the normalized baseline, and successful transitions remain highly schedule-sensitive. The tanh experiments also show that removing normalization statistics does not automatically preserve the original model's performance. We therefore interpret our findings as evidence that some effects of normalization can be transferred to the surrounding network, while acknowledging that normalization may also have representational effects. We explicitly discuss this limitation in Sections 5 and 6.

Can the authors provide a direct comparison with TaperNorm under the same GPT-OSS-style MoE architecture and training budget?

We have added a reproduction of the original TaperNorm protocol across the Small, Medium, and Large configurations (Section 4.2). It retains the learnable per-feature scaling and auxiliary scale-anchoring objective. In this setting, none of the three runs reaches a stable (g=0) endpoint: Small and Medium diverge early, while Large becomes unstable at approximately (g=0.043).

How does the scalar affine endpoint compare with TaperNorm's learned per-feature scaling, with and without scale anchoring? Does retaining the final RMSNorm improve stability?

The new TaperNorm reproduction retains both the learnable per-feature scaling and the auxiliary scale-anchoring objective. It does not reach a stable normalization-free endpoint in our MoE setting. Our simplified HomotopyNorm can reach selected endpoints, but with increasing schedule sensitivity and performance degradation.

Can the authors provide multi-seed evaluations and investigate continued training after reaching the normalization-free endpoint?

We now report three independent seeds for the principal normalized and homotopy configurations at all three scales (Table 1).

We have also explicitly tested continued training after reaching (g=0). In the failure case shown in Figure 4, training continues for approximately 1,000 additional steps, but the loss rapidly increases and enters a sustained high-loss regime. Importantly, our implementation continues evaluating RMSNorm during this experiment solely to update the online least-squares coefficients. Thus, the affine endpoint does not rely on frozen calibration coefficients; it receives refreshed regression estimates from the current activations. Even with this assistance, continued optimization becomes unstable.

This result suggests that the scalar affine endpoint lacks the nonlinear scale-control mechanism needed for stable optimization. It motivates exploring bounded, sample-independent nonlinear transformations. Our tanh experiment (Section 4.5) provides an initial example: tanh(20x) trains for the full training horizon without explicit normalization statistics, although it still underperforms RMSNorm. Other bounded functions, such as arctan, are promising candidates for future investigation. We therefore distinguish between obtaining a normalization-free checkpoint and establishing a stable normalization-free training regime.

Can the authors clarify how much of the scale-dependent degradation remains under comparable gate schedules?

We use the same nominal 0.94/0.06 transition window across all three scales, meaning that normalization is retained for the first 94% of training and removed over the final 6%. The only difference is the schedule shape: Small and Medium use a linear decay, whereas Large uses an eight-step staircase within the same transition window. Importantly, within each scale, normalized and homotopy models use identical architectures, data, training budgets, and optimization settings, making these controlled comparisons. We therefore consider the increasing degradation across scales meaningful evidence of the difficulty of linear normalization removal under appropriately scaled training configurations.

Appendix B states that Large-H resumes from step 31,000, whereas the staircase schedule begins at approximately step 30,720...?

We corrected the configuration references. Large-H resumes from the normalized checkpoint at step 30,000 and uses the eight-step staircase with a 0.94/0.06 terminal window (Section 4.3 and Appendix B, Table 5).

Can the authors report actual inference latency and memory consumption after replacing the normalization modules?

We clarify that the reported throughput overhead (approximately 6% across scales) comes from continuing to evaluate RMSNorm during training to update the online regression statistics. It is not an optimized inference measurement. Table 1 reports training throughput and peak training memory, not inference latency or memory. We therefore make no quantitative claim about inference efficiency.

Reviewer_7w1MOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The study suggests an alternative to RMSNorm normalization based on a homotopy approach that searches for a trade-off between normalization and an affine map. The experimental evaluation covers three model scales and downstream tasks. Based on the results, the authors provide a detailed discussion of the strengths and weaknesses of the proposed approach.

Critical strengths
  1. The fair discussion of the pros and cons of the proposed technique
  2. Different model scales used in evaluation
Critical weaknesses
  1. No variances for the presented numbers are reported.
  2. The motivation of the study is confusing since RMSNorm is used in state-of-the-art models, and this layer is typically not a bottleneck
  3. The homotopy schedule requires intensive fine-tuning, which prevents straightforward incorporation into existing models.
  4. H-versions of the models lead to the loss of quality for the downstream tasks.
Questions for authors
  1. What particular measurable issue of the classical normalization layer are you going to solve with HomotopyNorm?
  2. How do you recommend selecting the multiple hyperparameters that appear in (9)-(11) ?
  3. What empirical motivation leads to the homotopy approach? The choice of framework for replacing normalization is unclear.
2 comments

We sincerely thank the reviewer 7w1M for their detailed feedback. We provide our responses to the comments and questions below.

No variances for the presented numbers are reported.

We now report the mean ± standard deviation across three independent seeds for the main training and runtime metrics (Table 1). Downstream benchmark results additionally include standard errors (Table 3).

The motivation is confusing since RMSNorm is used in state-of-the-art models and is typically not a bottleneck.

Our objective is not to address a computational bottleneck, but to investigate whether input-dependent normalization can be replaced by a simple sample-independent transformation. Specifically, we ask whether a linear affine map can replace RMSNorm while preserving the learned computation. Our results show that this is possible under certain conditions, but becomes increasingly brittle as model scale increases. This suggests that a linear replacement may be insufficient at larger scales. We therefore also investigate richer nonlinear alternatives in Section 4.5. In particular, learnable tanh transformations can be trained stably without normalization statistics from initialization, although they still underperform RMSNorm. This complementary experiment motivates exploring nonlinear alternatives rather than insisting on linear normalization removal.

The homotopy schedule requires intensive fine-tuning...

We agree, and consider this an important finding rather than an implementation detail. Our schedule sweeps show that the transition must be delayed until very late in training. Even then, the acceptable transition window becomes increasingly narrow at larger scales. At Large scale, the eight-step staircase is the best completed configuration, but several other schedules remain stable until the final steps before diverging as (g) approaches zero. We therefore do not propose HomotopyNorm as a straightforward drop-in replacement. The difficulty of finding a reliable transition schedule is itself one of the main negative results of this study (Sections 4.3-4.4, Appendix D).

H-versions of the models lead to a loss of quality on downstream tasks.

We agree. This is precisely one of the central findings of our study: linear normalization removal comes with a measurable quality cost, which increases with model scale. The mean downstream accuracy gap increases from 0.61 percentage points at Small scale to 2.05 at Medium and 3.30 at Large scale (Figure 5). Moreover, our reproduction of the original TaperNorm protocol fails to reach a stable normalization-free endpoint in any of the three evaluated MoE configurations (Section 4.2). Finally, the complementary tanh experiments (Section 4.5) show that nonlinear (and bounded), sample-independent alternatives can train successfully from initialization, although they also remain slightly below the RMSNorm baseline. Together, these results expose the limitations of existing linear replacement strategies and motivate investigating richer nonlinear alternatives.

What particular measurable issue of the classical normalization layer are you going to solve with HomotopyNorm?

We do not target a specific computational bottleneck in RMSNorm. Our research question is whether input-dependent normalization statistics are necessary for the final computation of a trained Transformer or whether their effects can be transferred to a linear sample-independent transformation. We measure this through training stability, validation loss, and downstream performance. The results show that a linear affine replacement is possible in some regimes, but increasingly difficult and detrimental to performance at larger scales. The tanh experiments further investigate whether nonlinear sample-independent transformations provide a more viable alternative (Sections 4.3--4.6).

How do you recommend selecting the multiple hyperparameters that appear in (9)-(11)?

We select them empirically through schedule sweeps (Appendix D), rather than relying on a general selection rule. The main observation is that the gate should remain close to one for most of training and decrease only during a late transition window. At Small and Medium scales, a 0.94/0.06 linear schedule works. At Large scale, an eight-step staircase over the same terminal window is required for the best completed run. However, even schedules that remain stable throughout almost the entire transition can exhibit a sharp loss increase as (g) approaches zero. We therefore do not find a broad, robust sweet spot at Large scale. For the nonlinear transformation in (10), we also test several initializations, with tanh(20x) performing best among the completed runs (Section 4.5).

What empirical motivation leads to the homotopy approach?

The approach builds on sequential normalization removal by Baroni et al. (2025) and the gated removal framework of Kanavalau et al. (2026). Our motivation is to investigate whether a simple linear replacement can work in larger GPT-OSS-style MoE architectures. In the revised manuscript, we additionally reproduce the original TaperNorm protocol in our setting. Its failure to reach stable normalization-free endpoints motivates investigating the limitations of linear replacements and exploring richer nonlinear alternatives (Sections 2 and 4.2-4.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.