acceptodds
Under review as a conference paper at ICLR 2027

RhoNorm: Learning Channel-Energy Weights in Transformer Normalization

Abstract

In Transformers, shared normalization lets one channel's energy affect every channel's scale. RMSNorm fixes this aggregation rule to uniform weights. We learn the rule under ordinary predictive training and evaluate prediction and sparse-spike tolerance. Building on weighted divisive normalization, RhoNorm learns an exponentially parameterized positive diagonal metric, initialized to RMSNorm with one vector per site. Weighted energy shares link its task gradients to the exact shared-radius response under finite spikes. Across four decoder sizes, the learned rule improves clean prediction and reduces spike-induced extra loss in middle and late layers. In a 2.82B-parameter model, a spike in 1% of block-input coordinates at 75% depth adds 15.0 to the loss with RMSNorm and 6.3 with RhoNorm. Controls support relative energy weighting beyond output gains; the diagonal metric approaches dense-metric mean quality at much lower parameter cost. Quality gains extend to image classification and multimodal fine-tuning. On nanochat's time-to-GPT-2 target, a 1.38B-parameter model exceeds the GPT-2 reference in mean full CORE after 88 minutes of training on eight H100 GPUs, compared with 99 minutes for the fastest published Autoresearch entry.

Primary area:
foundation or frontier models, including LLMs
Keywords:
norm,transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WCxrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RhoNorm, a modification of RMSNorm that replaces uniform channel-energy aggregation with a learnable, positive diagonal metric. The method parameterizes channel weights exponentially, initializes them to recover standard RMSNorm, and introduces one learnable vector per normalization site.

The paper presents an analytical characterization of RhoNorm, including its scale-invariance properties, gradients with respect to the learned metric, and exact responses to finite multiplicative activation spikes. The analysis identifies weighted channel-energy shares as a common quantity governing metric gradients and normalization responses to perturbations.

Empirically, the authors evaluate RhoNorm on decoder-only language models ranging from 0.29B to 2.82B parameters. The main results demonstrate consistent improvements in validation bits per byte (bpb) compared with RMSNorm. Controlled ablations investigate scalar and vector output gains, relative metric weights, alternative positive parameterizations, and higher-capacity metric formulations.

The authors further report reduced prediction loss under artificially injected activation spikes, improvements on ImageNet classification and multimodal fine-tuning, and a practical time-to-GPT-2 benchmark using nanochat.

The central contribution is an empirically effective and parameter-efficient extension of RMSNorm that learns how individual channel energies contribute to the shared normalization denominator.

Critical strengths
  1. Consistent improvements across model scales, supported by meaningful controls that isolate denominator learning.

The strongest aspect of the paper is the empirical evidence that learning channel-energy weights can improve predictive performance beyond conventional RMSNorm.

In the principal 1.38B-parameter language-model experiment (Table 1), RhoNorm reduces validation bpb from 0.717272 to 0.714820, with improvements across all five paired seeds. The improvements also persist across the four evaluated model sizes, including the 2.82B-parameter model, where the mean reduction reaches approximately 0.00373 bpb.

Importantly, the paper does not rely solely on comparisons against parameter-free RMSNorm. Table 2 includes controls for scalar gains, vector gains, and zero-mean metric parameterizations. The Block-scope comparison shows that RhoNorm outperforms an equal-parameter vector-gain baseline, with a paired difference of approximately 0.00049 bpb and a confidence interval excluding zero. The zero-mean metric control further supports the claim that relative channel-energy weighting contributes beyond global scaling. These controls are consequential because they provide evidence that the observed improvements are not merely explained by adding a conventional learned output gain.

  1. A parameter-efficient formulation with useful analytical characterization and complementary stress-test evidence.

RhoNorm introduces a simple diagonal parameterization that adds approximately 0.006% parameters in the principal language-model setting. Table 3 shows that its observed mean validation quality is close to that of a substantially more expensive dense-metric formulation, although statistical equivalence is not established.

The analytical treatment is also reasonably clear. The derivations connect learned energy shares to parameter gradients and finite-spike normalization responses, helping explain what the additional degrees of freedom control.

The activation-spike experiments provide complementary evidence about the behavior of the resulting trained networks. For example, in the 2.82B-parameter model, perturbations at 75% depth produce approximately 15.0 nats of additional cross-entropy under RMSNorm, compared with 6.3 nats under RhoNorm.

Although these experiments do not establish general robustness or a complete causal mechanism, they demonstrate a potentially useful property beyond clean validation quality.

Overall, the combination of multi-scale improvements, parameter efficiency, targeted ablations, and analytical characterization makes the contribution more substantive than a straightforward architectural substitution.

Critical weaknesses
  1. The conceptual novelty is limited by existing divisive-normalization formulations, and the proposed explanation of spike tolerance lacks direct causal validation.

The mathematical operator underlying RhoNorm is closely related to established weighted divisive-normalization formulations. As the authors acknowledge in Section 4 and Appendix A.3, the proposed normalization corresponds to a restricted weighted L2 form within the generalized divisive normalization (GDN) family.

Consequently, the main mathematical identities, including the finite-spike radius response in Equation (5), follow directly from the weighted quadratic denominator. These results provide a useful characterization of the operator but do not, by themselves, establish a new normalization principle.

The more distinctive contribution is therefore the parameterization and empirical application of this operator to modern Transformers. However, the experiments do not fully establish why the learned metric improves resistance to activation spikes.

The stress tests compare separately trained RMSNorm and RhoNorm checkpoints. Differences in downstream loss could reflect the learned normalization weights, differences in the surrounding network parameters, or other consequences of the training trajectories. Furthermore, although Table 2 provides useful clean-quality controls, the corresponding vector-gain, scalar-gain, and zero-mean-metric configurations are not evaluated under the same spike perturbations. The authors explicitly acknowledge that their controls do not attribute spike tolerance to individual design choices.

The robustness analysis is also restricted to a single fixed evaluation batch and perturbation mask, with a primary spike magnitude of 100 and approximately 1% randomly selected tensor elements.

These limitations matter because improved spike tolerance is one of the paper's central empirical claims, while the proposed connection between learned energy shares and the observed robustness advantage remains largely indirect.

A more convincing analysis would distinguish the effect of the learned denominator from general changes in network representations and optimization.

  1. Several practical and cross-domain conclusions rely on incompletely matched comparisons, limiting their evidential strength.

The principal language-model improvements are reasonably well controlled, but important additional results involve differences beyond the normalization operator.

First, the ImageNet experiments use size-specific configurations selected using validation performance. In particular, the DeiT-Base result compares a more strongly regularized RhoNorm configuration against a standard-recipe LayerNorm baseline. Appendix D shows that under the original standard recipe, RhoNorm achieves lower validation accuracy than LayerNorm (81.45% versus 81.59%). The reported improvement to 82.03% follows changes to drop-path and mixup settings, without a corresponding stronger-regularization LayerNorm control.

Similarly, the DeiT-Tiny comparison introduces additional Q/K normalization sites, making it difficult to attribute the full reported improvement specifically to learned channel-energy weighting.

Second, the multimodal experiments on Qwen2.5-VL-7B use only one training run per configuration and nonstandard sampled evaluation subsets. Although the results are encouraging, they provide limited evidence of reproducible downstream improvements.

Third, the practical efficiency claim requires stronger control. Appendix E reports approximately 5.1% greater training-step time for RhoNorm in its systems comparison. However, the paper does not establish whether the clean-quality advantage persists when RMSNorm and RhoNorm are compared under equal wall-clock training budgets.

The time-to-GPT-2 result is also not a normalization-only comparison: the 88-minute RhoNorm result is compared with published configurations using different recipes. Moreover, its reported 95% interval for mean CORE, [0.2483, 0.2681], includes the GPT-2 reference score of 0.256525.

The authors appropriately disclose many of these limitations, but the headline results still risk conveying stronger evidence for cross-domain generalization and practical efficiency than the controlled experiments establish.

These issues are consequential because they affect whether RhoNorm should be regarded as a broadly preferable normalization replacement rather than a promising modification with demonstrated benefits under specific training configurations.

Questions for authors
  1. Attribution of the learned metric's contribution to spike tolerance (Critical Weakness 1).

Can the authors provide activation-spike evaluations for the matched output-gain and relative-metric controls from Table 2, ideally under multiple independently sampled batches, masks, and perturbation magnitudes?

In particular, does RhoNorm retain a meaningful robustness advantage over an equal-parameter vector-gain baseline, rather than only over parameter-free RMSNorm?

It would also be informative to examine whether the learned channel-energy shares quantitatively predict the observed differences in spike sensitivity.

This evidence could materially strengthen the paper's central claim that learning the denominator, rather than merely modifying the trained network, is responsible for improved perturbation tolerance.

  1. Compute-matched predictive quality and practical efficiency (Critical Weakness 2).

Does RhoNorm retain its validation-quality advantage over RMSNorm when both models are trained for the same wall-clock time, rather than the same number of optimization steps or tokens?

Relatedly, can the authors provide a time-to-GPT-2 comparison against an otherwise identical RMSNorm nanochat configuration, using the same evaluation protocol and comparable random seeds?

These results would clarify whether the reported efficiency benefit is attributable to RhoNorm itself and whether the additional per-step computational cost is justified by faster improvement in predictive quality.

  1. Normalization-only improvements on vision tasks (Critical Weakness 2).

Can the authors provide a matched-recipe DeiT-Base LayerNorm baseline using the stronger regularization settings selected for RhoNorm?

Additionally, for DeiT-Tiny and DeiT-Small, how much of the reported improvement remains when both baselines use identical normalization locations, particularly with respect to Q/K normalization?

These comparisons are important for determining whether the ImageNet gains are attributable to learned channel-energy weights rather than changes in normalization placement, regularization, or configuration selection.

2 comments
AuthorsOpenReview ↗

Response to Reviewer WCxr (Part 1)

Conceptual novelty (W1). We agree. The abstract, introduction and Section 2 now state that the operator is a scale-invariant special case of weighted divisive normalization and that the identities follow from the weighted quadratic denominator (Appendix A.3). The contribution is the Transformer parameterization, the analysis and the controlled evaluation.

Q1. Spike evaluations for the matched controls, and attribution to the learned metric.

Does RhoNorm keep a robustness advantage over an equal-parameter vector gain? Under Full replacement and late, strong spikes, yes, but it is a small part of the gap to RMSNorm; under Block replacement, no.

d24, block 18, 100× spikes (n=3; 8 batches × 5 masks per seed)Extra loss (nats/token)
RMSNorm12.853
Full vector gain (equal parameters)7.337
Full RhoNorm6.384
Vector gain − RhoNorm0.953 [0.749, 1.157]

The grid uses three seeds, eight batches, five masks, blocks 6/12/18 and strengths 10/30/100 (Table 30). An output gain removes 5.5 of the 6.5 nats between RMSNorm and RhoNorm. RhoNorm's further advantage is 0.953 at block 18 and 1.007 [0.821, 1.194] at block 12 for 100×, and is absent or reversed for weaker or shallower spikes.

Controls from the clean-quality table, evaluated under spikes:

  • Full zero-mean metric vs. vector gain (six batches, four masks): +0.754 [0.546, 0.961] at block 18, 100×, and positive in all four tested conditions (Table 7).
  • Block scalar gain, vector gain and RhoNorm (eight batches, five masks; block 18, 100×): scalar − Rho = +1.130 [0.527, 1.734]; vector − Rho = −3.014 [−3.403, −2.625]. In the Block cohort RhoNorm has better clean bpb than vector gain but worse spike tolerance (Table 45).
  • Not run: a Full-scope scalar-gain spike grid; scalar-gain checkpoints exist at Block scope only.

Do differences in separately trained checkpoints reflect the learned denominator? To address this we intervene at fixed checkpoints: at all native block sites from the injection block onward, RhoNorm's perturbed-to-clean radius ratio is replaced by the uniform one, keeping each token's clean scale and all weights.

Full configuration / referenceReference − Rho gap, originalRho extra loss, CF − originalReference − Rho gap after CF
d24, block 18 / equal-parameter vector gain0.979 [0.781, 1.177]+0.851 [0.724, 0.977]0.128 [−0.027, 0.284]
d32, block 24 / RMSNorm8.724 [6.998, 10.450]+0.763 [0.621, 0.905]7.961 [6.100, 9.821]

Values are nats/token at 100×, six batches and four masks per seed (Table 6, Figure 3, Appendix F.10). The learned response accounts for most of the margin over vector gain at d24 and for less than a tenth of the margin over RMSNorm at d32. At half depth the effect is reversed at d24 (−0.273 [−0.297, −0.248]) and unresolved at d32. The 0.979 and 0.953 d24 gaps are the same contrast on the six-batch and eight-batch evaluation sets.

Do the learned energy shares quantitatively predict spike sensitivity? No. The shares give the radius ratio exactly, sqrt(1 + (k² − 1) q_M) (Equation 7; predicted and measured ratios agree to four figures in Table 32). They do not give a calibrated predictor of downstream loss: under a uniform random mask the mean share equals the mask fraction for any weights, and the path-separated tests show that the model difference arises downstream of the injection site, not at it (Table 39). The intervention above is the direct test we ran. It does not explain why training produces the tolerance, and Q/K, embedding and final sites were not intervened on separately.

AuthorsOpenReview ↗

Response to Reviewer WCxr (Part 2)

Q2a. Does the quality advantage persist at equal wall-clock training time?

Yes at matched training-step time, by a smaller margin.

Method, n=3UpdatesTraining-step time (min)Final bpb
RMSNorm5,67499.030.715935
RhoNorm5,37499.370.715258

RhoNorm minus RMSNorm is −0.000677 bpb [−0.001138, −0.000216], with all three pairs improving (Table 4, Table 26). Both arms use graph replay, eight H100 GPUs and a complete learning-rate schedule; pilot timing fixed the horizons. The fixed-update gain is 0.002323 bpb; in the matched budget RMSNorm receives 5.6% more updates. The clock sums training-step durations and excludes graph capture, evaluation and logging; total job times are 104.43 (RMSNorm) and 104.13 (RhoNorm) minutes. The runs are not FLOP-matched.

Q2b. Is there a time-to-GPT-2 comparison against an otherwise identical RMSNorm configuration?

No. That experiment was not run. What the available data show:

  • Full 22-task CORE on the matched-time checkpoints, at about 99 minutes: 0.2586 (RMSNorm) and 0.2615 (RhoNorm), a difference of +0.0029 [−0.0379, +0.0437]. Both arms exceed the GPT-2 reference of 0.256525 in mean, and the difference is unresolved (Appendix F.2).
  • The 88-minute result is a complete-configuration comparison with five seeds per horizon and no RMSNorm counterpart. As the reviewer notes, the 95% interval of its mean, [0.2483, 0.2681], includes the reference (Table 25).

Section 3.4 and the abstract now state that a time saving attributable to the normalizer is not established. The matched-time study supports a bpb advantage at equal training-step time; it does not support an efficiency claim in CORE.

Q3a. A matched-recipe DeiT-Base LayerNorm baseline.

Provided. RhoNorm remains ahead by 0.3–0.5 points, depending on how runs are selected.

ConfigurationSitesnFinal top-1 (%)
Base / LayerNorm25381.642 ± 0.178
Base / RhoNorm25382.107 ± 0.136
Base / RhoNorm, all completed runs25381.951 ± 0.300
Tiny / LayerNorm25372.065 ± 0.138
Tiny / RhoNorm25372.599 ± 0.068
Tiny / LayerNorm + free Q/K49374.018 ± 0.068
Tiny / RhoNorm + free Q/K49373.898 ± 0.241
Tiny / RhoNorm + learned Q/K49374.313 ± 0.145

Both Base methods use the 25 native sites, drop path 0.2 and mixup 1.0, with five full-length runs each.

Q3b. How much of the Tiny and Small improvement remains with identical normalization locations?

Tiny: 0.3–0.5 points of the 2.248. Small: not tested in the selected cohort.

  • Tiny, same 25 native sites: RhoNorm is 0.534 points above LayerNorm.
  • Tiny, same 49 sites: adding parameter-free Q/K normalization raises LayerNorm by 1.953 points. RhoNorm with learned Q/K weights is 0.295 points above that configuration, and RhoNorm with free Q/K is 0.120 points below it. These are three-seed configuration means without paired intervals (Appendix F.6).
  • Small: no matched-site control was run in the selected timm/EMA cohort. The only same-site evidence is the DeiT-runner native-site diagnostic, +0.14 points in best-checkpoint accuracy (Table 20).

Multimodal and headline claims (W2). The multimodal study still has one run per configuration on sampled subsets; the text now states that the differences cannot be separated from seed variation (Section 3.7). The abstract no longer presents the DeiT, OLMoE and time-to-GPT-2 results as normalizer-only gains: it reports 0.3–0.5 ImageNet points with matched recipes and sites and labels the other two as a single run against published baselines and a complete-configuration comparison.

Statistics: every decoder comparison uses three training seeds common to its arms; values in brackets are unadjusted paired 95% Student intervals over seeds, with repeated batches and masks averaged within a seed (Section 3.1, Appendix F.1). An interval that contains zero does not establish equivalence.

Reviewer_H3knOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper replaces the uniform squared-feature weighting in RMSNorm with learned positive per-channel weights in the denominator. This operation is not new; the paper itself shows that it is a restricted special case of prior GDN-style normalization.

The method gives small but consistent improvements in language-model validation bpb across four model sizes (<1%). The paper also reports a much larger improvement on a synthetic stress test where 1% of activation entries are multiplied by 100.

The gains are very small, the method adds parameters and training cost, and the key resource-matched comparison is missing.

Critical strengths

S1. The synthetic spike experiment shows a large effect. The extra loss drops from 15.0 to 6.3 nats. However, the test is synthetic, and the paper does not explain what exactly (mechanistically) produces this robustness.

Critical weaknesses

W1. The clean gains are very small, while the reported RhoNorm implementation is about 5.1% slower per training step. The paper does not show whether RhoNorm still wins in a controlled comparison at matched training compute or wall-clock time. The time-to-target comparison in Section 3.7 uses published leaderboard runs and does not establish whether changing only the normalizer improves performance at matched training cost. The equal-parameter output-gain control does not test whether the additional capacity could be used more effectively elsewhere in the model.

W2. The underlying normalization operator is not new; the novelty is mainly its application to Transformer normalization. Much of Section 2 derives immediate properties of this operator, but does not explain the main empirical phenomenon: why ordinary training produces much better tolerance to the synthetic spikes. The analysis shows that RhoNorm and learned output scaling can give different outputs for the same input, but leaves open whether it increases the expressivity of the full Transformer.

Questions for authors

Q1. Given the reported 5.1% increase in training-step time, does RhoNorm still outperform RMSNorm at equal total training compute or equal wall-clock time?

Q2. At equal total parameter count, does RhoNorm still outperform RMSNorm when the baseline spends the additional parameter budget elsewhere in the model rather than specifically as an output gain?

Q3. Can the authors provide evidence for why ordinary predictive training produces the large improvement on the synthetic spike test? The current analysis describes how the normalizer responds to spikes, but I do not see a causal or mechanistic explanation for why the learned weights improve full-network spike tolerance.

4 comments
AuthorsOpenReview ↗

Response to Reviewer H3kn (Part 1)

Q1. Does RhoNorm still outperform RMSNorm at equal training compute or wall-clock time?

Yes at matched training-step time, by a smaller margin. Pilot timing fixed the update horizons so that both arms train for about 99 minutes on eight H100 GPUs, each with a complete learning-rate schedule.

Method, n=3UpdatesTraining-step time (min)Final bpb
RMSNorm5,67499.030.715935
RhoNorm5,37499.370.715258

RhoNorm minus RMSNorm is −0.000677 bpb [−0.001138, −0.000216], and all three pairs improve (Table 4, Table 26). The fixed-update gain is 0.002323 bpb; in the matched budget RMSNorm receives 5.6% more updates.

Scope. The clock sums training-step durations and excludes graph capture, evaluation and logging; the two budgets differ by 0.35%. Recorded total job times are 104.43 (RMSNorm) and 104.13 (RhoNorm) minutes. The runs are not FLOP-matched. Full CORE on the same checkpoints differs by +0.0029 [−0.0379, +0.0437], so no downstream capability difference is resolved (Appendix F.2).

The 5.1% figure is the training-step overhead in the fixed-update timing cohort, whose two arms use different execution modes; with graph replay in both arms, the matched-time study implies 6.0% more time per update (Section 3.4).

We agree that the time-to-GPT-2 result does not isolate the normalizer. No RMSNorm run of our configuration exists at 88 minutes, and in the matched-time study both arms exceed the GPT-2 reference in mean full CORE at about 99 minutes (0.2586 and 0.2615). Section 3.4 and the abstract now describe that result as a complete-configuration comparison.

Q2. At equal parameter count, does RhoNorm still win when the baseline spends the budget elsewhere?

Yes for the two allocations we tested. Neither of them improved on RMSNorm at all. We added a bias-free low-rank branch B·GELU(Az) to the FFN of an RMSNorm backbone, either at rank 27 in block 12 or with total rank 27 spread over all 24 blocks. Each adds exactly 82,944 trainable parameters, the same as Full RhoNorm, and starts from the baseline function.

d24, n=3, 5,568 updatesValidation bpbRhoNorm − row
RMSNorm0.717246 ± 0.000260−0.002255 [−0.002689, −0.001821]
RMSNorm + single-block FFN branch0.717219 ± 0.000177−0.002227 [−0.002465, −0.001990]
RMSNorm + distributed FFN branch0.717246 ± 0.000107−0.002254 [−0.002327, −0.002181]
RhoNorm0.714991 ± 0.000086

The gain is therefore not a generic return on 0.006% more parameters (Table 5, Table 4, Appendix F.3).

Scope. The test covers two FFN allocations under the native optimizer routing (Muon for the branch matrices, AdamW for the log-weights); the branch layers compute in BF16 next to FP8 backbone projections. Other placements of the budget, for example in attention, were not tried. The output-gain control answers a different question, whether one vector per site is better spent on the denominator than on an output gain: at the 48 native block sites RhoNorm is better than an equal-parameter vector gain by 0.000360 bpb [0.000240, 0.000481], and the vector gain itself recovers 69% of RhoNorm's improvement there (Table 1, Table 3).

AuthorsOpenReview ↗

Response to Reviewer H3kn (Part 2)

Q3. Why does ordinary predictive training produce the large improvement on the spike test?

We can explain it only in part, and the paper no longer suggests otherwise. The new experiments establish four things.

  1. Most of the improvement is not specific to the denominator. At d24, block 18, 100× spikes, extra loss is 12.853 nats/token for RMSNorm, 7.337 for an equal-parameter Full vector gain and 6.384 for Full RhoNorm (three seeds, eight batches, five masks; Table 30). An output gain removes 5.5 of the 6.5 nats.
  2. RhoNorm's further advantage is conditional. It is 0.953 [0.749, 1.157] at block 18 and 1.007 [0.821, 1.194] at block 12 for 100× spikes, and absent or reversed for weaker or shallower spikes. Under Block replacement the ranking reverses: vector gain is better than RhoNorm by 3.014 [2.625, 3.403] (Table 45).
  3. The difference does not arise at the injection-site normalizer. Routing the spike only into the first normalized attention branch costs RhoNorm 0.036 nats, against 6.391 through the residual bypass. The gain-minus-Rho contrast is 0.944 [0.755, 1.133] bypass-only and −0.014 [−0.029, 0.001] branch-only (Table 31, Table 39).
  4. The learned radius response at downstream sites contributes under one nat. With all weights fixed, we replace RhoNorm's perturbed-to-clean radius ratio by the uniform one at all native block sites from the injection block onward, keeping each token's clean scale.
Full configuration / referenceReference − Rho gap, originalRho extra loss, CF − originalReference − Rho gap after CF
d24, block 18 / equal-parameter vector gain0.979 [0.781, 1.177]+0.851 [0.724, 0.977]0.128 [−0.027, 0.284]
d32, block 24 / RMSNorm8.724 [6.998, 10.450]+0.763 [0.621, 0.905]7.961 [6.100, 9.821]

Values are nats/token at 100× (Table 6, Figure 3, Appendix F.10). The intervention accounts for most of RhoNorm's margin over vector gain at d24 and for less than a tenth of its margin over RMSNorm at d32. At half depth the effect is reversed at d24 (−0.273 [−0.297, −0.248]) and unresolved at d32 (−0.080 [−0.241, 0.081]).

What remains open. The interventions are oracle diagnostics that use the clean pass; they measure test-time effects in jointly trained networks and do not partition training co-adaptation. Q/K, embedding and final sites were not intervened on separately, and a Full scalar-gain stress control was not run. Section 2.3 shows how predictive gradients can change relative weights, which is a route by which such responses can be learned, not evidence that this is why the tolerance arises.

Does the analysis show greater expressivity of the full Transformer (W2)?

No, and the revised text says so. Proposition 1 states only that, at fixed input coordinates, a non-isotropic metric cannot be folded into the following linear map as a static output gain can. If the linear maps on both sides of the normalizer are free, f_P(x) = P^(−1/2) f_0(P^(1/2) x) lets them absorb the metric, so the proposition does not show that RhoNorm enlarges a Transformer's function class or speeds convergence (Section 2.2, Appendix A.1). The case for learning the denominator rests on the controls in Q1 and Q2, not on the proposition.

Statistics: every decoder comparison uses three training seeds common to its arms; values in brackets are unadjusted paired 95% Student intervals over seeds, with repeated batches and masks averaged within a seed (Section 3.1, Appendix F.1). An interval that contains zero does not establish equivalence.

Reviewer_H3knOpenReview ↗

Reply to authors' rebuttal

Thank you for the detailed response and the additional experiments. I appreciate the effort put into addressing my questions.

Predictive gains

  • The new matched-training-time experiment addresses my concern, but the improvement is very small (~0.1% in validation bpb).
  • It is unclear whether this advantage persists at larger scales under matched training time or translates into better downstream performance.
  • The paper does not explain why RhoNorm improves prediction.

Robustness

  • In the late-layer 100× spike experiment, most of the improvement is also achieved with standard learnable output gains. The additional robustness benefit over standard output gains is limited to certain spike settings and reverses in others.
  • The paper does not explain why RhoNorm becomes more tolerant to spikes or why these artificial spikes matter in practice.

Novelty: As the authors acknowledge, the underlying operator is not new.

I maintain my score.

AuthorsOpenReview ↗

Response to Reviewer H3kn

Thank you for the follow-up. We would like to clarify the methodological contribution alongside the broader empirical results. RhoNorm investigates whether learning the relative channel contributions to a normalization radius improves Transformer training beyond output rescaling or spending the same parameter budget elsewhere. Our contribution combines an RMSNorm-initialized log-weight parameterization, an analysis linking predictive gradients and exact radius responses to finite activation spikes through weighted energy shares, and controlled experiments separating the effects of relative channel weighting, global scaling, output gains, and additional capacity. This identifies and tests a specific opportunity for Transformer training within the established family of weighted divisive normalization.

The approximately 0.1% improvement concerns one matched-training-time comparison at 1.38B parameters. The broader empirical results include:

  • OLMoE-1.3B: After 500B training tokens, RhoNorm achieves C4 loss 2.823, compared with the published 2.922 for RMSNorm and 2.900 for SeeDNorm. Against the published RMSNorm baseline, HellaSwag and PIQA improve by 2.8 and 1.3 percentage points, respectively.
  • ImageNet: The complete DeiT-Tiny configuration, including Q/K normalization, improves top-1 accuracy from 72.065% to 74.313%. Comparisons at the native normalization sites show gains of 0.465 points on DeiT-Base and 0.534 points on DeiT-Tiny.
  • nanochat: Our RhoNorm-based configuration achieves a five-seed mean CORE above the GPT-2 target after 88 training minutes on eight H100 GPUs, compared with 99 minutes for the fastest entry in the cited leaderboard snapshot—an 11.1% reduction.

These broader benchmarks demonstrate practical promise, while the paired ablations examine attribution. Predictive improvements occur at four decoder sizes; the d24 controls distinguish learning channel-energy weights from output rescaling and equal-parameter FFN additions, and the matched-time comparison accounts for the additional training-step cost.

The spike study contributes diagnostic evidence about trained-network behavior. Its interventions isolate a downstream radius-response contribution that accounts for most of the residual advantage over a vector gain in the tested late-layer d24 setting. We retain the setting-dependent findings and distinguish this diagnostic contribution from the predictive and benchmark evidence supporting the method’s utility.

Reviewer_w1XVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RhoNorm, a variant of RMSNorm that learns how much each channel’s energy contributes to the normalization denominator. It adds one zero-initialized parameter vector per normalization site, so the model starts with the same behavior as RMSNorm. The authors analyze the gradients and the response to activation spikes. In language-model experiments, RhoNorm consistently improves validation loss and outperforms output-gain controls. It also reduces the damage from the tested middle- and late-layer spikes. Additional experiments explore image classification, multimodal fine-tuning, and training efficiency, though these results are less tightly controlled.

Critical strengths
  • RhoNorm is a easy method to implement which adds only one vector per normalization site and starts with the same behavior as RMSNorm.

  • Validation loss improves across all four tested model sizes and all reported seed pairs.

  • Comparisons with scalar and vector output gains support the value of learning the denominator, rather than simply adding parameters.

Critical weaknesses
  • RhoNorm is closely related to earlier weighted divisive normalization, as the paper acknowledges. The exponential parameterization changes how the positive weights are learned, rather than introducing a new normalization family. Its main contribution is therefore the Transformer application and empirical evaluation. More analysis of the learned weights and their role in the observed gains would strengthen the contribution.

  • The spike tests use one fixed input batch, one mask seed, and one main perturbation setting: approximately 1% of coordinates amplified by 100×. These results show improved tolerance under the tested condition, but do not establish general robustness. Since the spike affects both the normalized branch and the residual bypass, the experiments also do not isolate the role of the learned metric. Tests across multiple batches, masks, and perturbation strengths would make the evidence more convincing.

  • Some vision results change the training recipe or normalization sites alongside the normalization method. For example, the DeiT-Base result uses stronger regularization without a matched LayerNorm baseline. Similarly, the CORE time-to-target comparison uses different complete training configurations, while the fixed-update systems experiment is approximately 5.1% slower with RhoNorm. These results are informative, but do not isolate normalization-specific gains in accuracy or training efficiency.

  • It would strengthen the paper to evaluate RhoNorm on larger models and test whether the gains remain consistent. Extending the evaluation to MoE architectures would also help establish its broader applicability.

Questions for authors

refer to my weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer w1XV (Part 1)

1. Novelty, and analysis of the learned weights.

We agree that RhoNorm is not a new normalization family; the abstract, introduction and Section 2 now say so, and Appendix A.3 gives the correspondence to weighted divisive normalization. On the learned weights, the revision adds the following.

  • They are strongly non-uniform. At the measured d24 pre-attention sites the SD of the effective log-weights is 0.844–1.198 at block 12 and 1.475–1.876 at block 18 across three seeds, and their mean is negative (−0.88 to −2.19), a learned global scale-up of the output (Table 32).
  • The clean gain comes mainly from relative weights, not from the global scale. Under Full replacement, a zero-mean metric (no free global scale) is better than scalar-only weights by 0.002004 bpb [0.000974, 0.003034] and is not distinguished from unconstrained RhoNorm (−0.000325 [−0.001374, +0.000723]) (Table 3).
  • Output gains recover part of the gain. At the 48 native block sites RhoNorm is better than an equal-parameter vector gain by 0.000360 bpb [0.000240, 0.000481] and than a scalar gain by 0.000499 [0.000147, 0.000850]; the two gains recover 69% and 58% of RhoNorm's improvement over RMSNorm there (Table 1).

What this does not show: we have no analysis that links individual channel weights to the clean-quality gain. The energy shares determine the radius response exactly (Equation 7), but under a uniform random mask their mean equals the mask fraction for any weights, so the downstream effect has to be measured; item 2 reports that measurement.

2. Spike tests: replication, and the role of the learned metric.

Replication. We agree that one batch, one mask and one setting do not establish general robustness, and the paper no longer claims it; the probe is described as a synthetic stress test (Section 3.5). The expanded d24 grid uses three training seeds, eight batches, five masks, blocks 6/12/18 and strengths 10/30/100 (5,760 evaluations including the 1× identity check). The mask fraction is still 1% throughout.

d24, block 18, 100× spikes (n=3; 8 batches × 5 masks per seed)Extra loss (nats/token)
RMSNorm12.853
Full vector gain (equal parameters)7.337
Full RhoNorm6.384
Vector gain − RhoNorm0.953 [0.749, 1.157]

Most of the tolerance relative to RMSNorm is shared by an equal-parameter output gain. RhoNorm's further advantage is 0.953 at block 18 and 1.007 [0.821, 1.194] at block 12 for 100× spikes; it is absent or reversed for weaker or shallower spikes (block 6, 100×: −0.392 [−0.589, −0.195]) (Table 30). Under Block replacement it reverses: vector gain is better than RhoNorm by 3.014 [2.625, 3.403], while RhoNorm remains better than scalar gain by 1.130 [0.527, 1.734] (Table 45).

Branch versus bypass. The reviewer is right that the spike enters both paths. Separating them shows that almost all of the damage enters through the bypass: for RhoNorm at block 18, 100×, branch-only injection costs 0.036 nats and bypass-only 6.391. The gain-minus-Rho contrast is 0.944 [0.755, 1.133] bypass-only and −0.014 [−0.029, 0.001] branch-only (Table 31, Table 39). The two models therefore do not differ at the injection-site normalizer.

Role of the learned metric. With all weights fixed, replacing RhoNorm's learned radius response by the uniform one at all native block sites from the injection block onward gives:

Full configuration / referenceReference − Rho gap, originalRho extra loss, CF − originalReference − Rho gap after CF
d24, block 18 / equal-parameter vector gain0.979 [0.781, 1.177]+0.851 [0.724, 0.977]0.128 [−0.027, 0.284]
d32, block 24 / RMSNorm8.724 [6.998, 10.450]+0.763 [0.621, 0.905]7.961 [6.100, 9.821]

Values are nats/token at 100×, six batches and four masks per seed (Table 6, Figure 3). The learned response accounts for most of the margin over vector gain at d24 and for less than a tenth of the margin over RMSNorm at d32; at half depth the effect is reversed at d24 (−0.273 [−0.297, −0.248]). A Full zero-mean metric keeps a 0.754 [0.546, 0.961] advantage over vector gain (Table 7). Not run: a Full scalar-gain stress control and separate interventions on Q/K, embedding and final sites. The interventions do not explain why training produces the tolerance.

AuthorsOpenReview ↗

Response to Reviewer w1XV (Part 2)

3a. Training efficiency.

The comparisons the reviewer describes did not isolate the normalizer; the new matched-time study does, for bpb.

Method, n=3UpdatesTraining-step time (min)Final bpb
RMSNorm5,67499.030.715935
RhoNorm5,37499.370.715258

RhoNorm minus RMSNorm is −0.000677 bpb [−0.001138, −0.000216] at about 99 minutes of training-step time on eight H100 GPUs, with graph replay and a complete schedule in both arms (Table 4, Appendix F.2). This is smaller than the 0.002323 fixed-update gain. The clock excludes graph capture, evaluation and logging; total job times are 104.43 and 104.13 minutes; the runs are not FLOP-matched; and full CORE on these checkpoints is unresolved (+0.0029 [−0.0379, +0.0437]).

The 5.1% overhead comes from a timing cohort whose arms use different execution modes; with graph replay in both arms the per-update overhead implied by the matched-time study is 6.0%.

The time-to-GPT-2 result remains a comparison of complete configurations, and Section 3.4 now says so: the 95% interval of the 88-minute mean includes the target, there is no RMSNorm run at 88 minutes, and in the matched-time study both arms exceed the target in mean at about 99 minutes (0.2586 and 0.2615). A normalizer-only time-to-target curve was not run.

3b. Vision recipe and normalization sites.

DeiT-Base now has the matched LayerNorm baseline, and DeiT-Tiny separates Q/K placement from the normalizer.

ConfigurationSitesnFinal top-1 (%)
Base / LayerNorm25best 3 of 581.642 ± 0.178
Base / RhoNorm25best 3 of 582.107 ± 0.136
Base / RhoNorm, all completed runs25581.951 ± 0.300
Tiny / LayerNorm25372.065 ± 0.138
Tiny / RhoNorm25372.599 ± 0.068
Tiny / LayerNorm + free Q/K49374.018 ± 0.068
Tiny / RhoNorm + free Q/K49373.898 ± 0.241
Tiny / RhoNorm + learned Q/K49374.313 ± 0.145

Base. Both methods use the 25 native sites, drop path 0.2 and mixup 1.0, with five full-length runs each. Two LayerNorm runs finished with collapsed accuracy. Under the best-three-of-five rule the difference is 0.465 points; this rule removes LayerNorm's two collapsed runs but RhoNorm's two lowest converged runs. Over all five completed RhoNorm runs the difference from LayerNorm's three converged runs is 0.309 points. Both are descriptive, and the runs do not establish a difference in training reliability.

Tiny. Adding parameter-free Q/K normalization raises LayerNorm by 1.953 points, which is most of the 2.248-point difference between the complete RhoNorm configuration and native LayerNorm. At the same 25 sites RhoNorm is 0.534 points above LayerNorm. With the same 49 sites, RhoNorm with learned Q/K weights is 0.295 points above LayerNorm + free Q/K, and RhoNorm with free Q/K is 0.120 points below it. These are three-seed configuration means without paired intervals (Table 8, Appendix F.6).

The abstract now reports gains of 0.3–0.5 points with matched recipes and sites instead of the earlier configuration-level figures. The per-size configuration study of the original submission is kept in Appendix D.1 and labelled as unmatched.

AuthorsOpenReview ↗

Response to Reviewer w1XV (Part 3)

4. Larger models and MoE architectures.

MoE: one run was added. Larger from-scratch models: none beyond 2.82B parameters.

We trained one OLMoE-1.3B model for 500B tokens with the architecture and recipe of Cai et al. (2026) and compare it with the baseline results published there.

OLMoE-1.3B, 500B tokensC4 lossPPLARC-CARC-EHellaSwagPIQA
RMSNorm, published2.92218.6332.362.255.272.6
DyT, published2.96819.4530.461.953.270.6
SeeDNorm, published2.90018.1234.565.456.873.1
RhoNorm, one run (ours)2.82316.8332.163.958.073.9

RhoNorm's C4 loss is 0.099 and 0.077 nats below the published RMSNorm and SeeDNorm values. Its downstream accuracy is the highest on HellaSwag and PIQA, between RMSNorm and SeeDNorm on ARC-Easy, and below both on ARC-Challenge (Table 2).

Limits. This is one seed. The baselines are published values that we did not retrain and that were scored by their authors' evaluation code. The run uses the zero-mean variant with its own metric learning rate, not the unconstrained Full configuration of the nanochat experiments (Appendix F.7). It indicates that the gain is not confined to nanochat decoders; it is not a paired effect size.

Scale. Paired from-scratch comparisons cover 0.29B–2.82B parameters, where the d32 reduction is 0.003730 bpb [0.003255, 0.004205]. The only larger model is the 7B Qwen2.5-VL fine-tuning study, which has one run per configuration and cannot separate the difference from seed variation. Whether the gain persists in larger from-scratch models is open, and the discussion lists this as a limitation.

Statistics: every decoder comparison uses three training seeds common to its arms; values in brackets are unadjusted paired 95% Student intervals over seeds, with repeated batches and masks averaged within a seed (Section 3.1, Appendix F.1). An interval that contains zero does not establish equivalence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

The revision makes three kinds of change. It narrows the claims, it adds the controlled comparisons the reviewers asked for, and it states plainly which requested experiments were not run.

Scope of the contribution. All three reviewers note that the operator is a restricted case of weighted divisive normalization. We agree. The abstract, introduction and Section 2 now say so directly (Appendix A.3 gives the correspondence). The contribution is the log-coordinate parameterization for Transformers, an analysis that ties the training gradient and the exact finite-spike response to one quantity, and a controlled evaluation.

New controlled evidence at d24 (1.38B parameters, three paired seeds).

  • Matched training-step time. With about 99 minutes of training-step time in both arms, RhoNorm's final bpb is lower by 0.000677 [0.000216, 0.001138]. This is smaller than the fixed-update gain of 0.002323 [0.002224, 0.002422]; in the matched budget RMSNorm receives 5.6% more updates (Table 4, Appendix F.2).
  • Equal parameters elsewhere. Two FFN branches that add exactly RhoNorm's 82,944 parameters do not improve on RMSNorm; RhoNorm is better than them by 0.002227 [0.001990, 0.002465] and 0.002254 [0.002181, 0.002327] bpb (Table 4, Table 5).
  • Spike tolerance against matched controls. At block 18 with 100× spikes, extra loss is 12.853 nats/token for RMSNorm, 7.337 for an equal-parameter Full vector gain and 6.384 for Full RhoNorm. Most of the tolerance relative to RMSNorm is therefore shared by an output gain. RhoNorm's further advantage, 0.953 [0.749, 1.157], holds for late, strong spikes under Full replacement, is absent or reversed for weak or shallow spikes, and reverses under Block replacement, where vector gain is better by 3.014 [2.625, 3.403] (Table 30, Table 45).
  • Fixed-checkpoint interventions. Replacing RhoNorm's learned radius response by the uniform one at downstream native block sites raises its extra loss by 0.851 [0.724, 0.977] at d24 and 0.763 [0.621, 0.905] at d32. This accounts for most of the 0.979-nat margin over vector gain at d24 and for less than a tenth of the 8.724-nat margin over RMSNorm at d32 (Table 6, Figure 3). We do not have a mechanistic explanation of why predictive training produces the tolerance, and the paper now says so.

Vision. DeiT-Base now uses the same sites and the same stronger recipe for LayerNorm and RhoNorm. DeiT-Tiny separates Q/K placement from learned weights: 1.953 of the 2.248-point configuration gain comes from adding Q/K normalization, which also helps LayerNorm (Table 8, Appendix F.6).

Statistics: every decoder comparison uses three training seeds common to its arms; values in brackets are unadjusted paired 95% Student intervals over seeds, with repeated batches and masks averaged within a seed (Section 3.1, Appendix F.1). An interval that contains zero does not establish equivalence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.