- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RhoNorm, a modification of RMSNorm that replaces uniform channel-energy aggregation with a learnable, positive diagonal metric. The method parameterizes channel weights exponentially, initializes them to recover standard RMSNorm, and introduces one learnable vector per normalization site.
The paper presents an analytical characterization of RhoNorm, including its scale-invariance properties, gradients with respect to the learned metric, and exact responses to finite multiplicative activation spikes. The analysis identifies weighted channel-energy shares as a common quantity governing metric gradients and normalization responses to perturbations.
Empirically, the authors evaluate RhoNorm on decoder-only language models ranging from 0.29B to 2.82B parameters. The main results demonstrate consistent improvements in validation bits per byte (bpb) compared with RMSNorm. Controlled ablations investigate scalar and vector output gains, relative metric weights, alternative positive parameterizations, and higher-capacity metric formulations.
The authors further report reduced prediction loss under artificially injected activation spikes, improvements on ImageNet classification and multimodal fine-tuning, and a practical time-to-GPT-2 benchmark using nanochat.
The central contribution is an empirically effective and parameter-efficient extension of RMSNorm that learns how individual channel energies contribute to the shared normalization denominator.
Critical strengths
- Consistent improvements across model scales, supported by meaningful controls that isolate denominator learning.
The strongest aspect of the paper is the empirical evidence that learning channel-energy weights can improve predictive performance beyond conventional RMSNorm.
In the principal 1.38B-parameter language-model experiment (Table 1), RhoNorm reduces validation bpb from 0.717272 to 0.714820, with improvements across all five paired seeds. The improvements also persist across the four evaluated model sizes, including the 2.82B-parameter model, where the mean reduction reaches approximately 0.00373 bpb.
Importantly, the paper does not rely solely on comparisons against parameter-free RMSNorm. Table 2 includes controls for scalar gains, vector gains, and zero-mean metric parameterizations. The Block-scope comparison shows that RhoNorm outperforms an equal-parameter vector-gain baseline, with a paired difference of approximately 0.00049 bpb and a confidence interval excluding zero. The zero-mean metric control further supports the claim that relative channel-energy weighting contributes beyond global scaling. These controls are consequential because they provide evidence that the observed improvements are not merely explained by adding a conventional learned output gain.
- A parameter-efficient formulation with useful analytical characterization and complementary stress-test evidence.
RhoNorm introduces a simple diagonal parameterization that adds approximately 0.006% parameters in the principal language-model setting. Table 3 shows that its observed mean validation quality is close to that of a substantially more expensive dense-metric formulation, although statistical equivalence is not established.
The analytical treatment is also reasonably clear. The derivations connect learned energy shares to parameter gradients and finite-spike normalization responses, helping explain what the additional degrees of freedom control.
The activation-spike experiments provide complementary evidence about the behavior of the resulting trained networks. For example, in the 2.82B-parameter model, perturbations at 75% depth produce approximately 15.0 nats of additional cross-entropy under RMSNorm, compared with 6.3 nats under RhoNorm.
Although these experiments do not establish general robustness or a complete causal mechanism, they demonstrate a potentially useful property beyond clean validation quality.
Overall, the combination of multi-scale improvements, parameter efficiency, targeted ablations, and analytical characterization makes the contribution more substantive than a straightforward architectural substitution.
Critical weaknesses
- The conceptual novelty is limited by existing divisive-normalization formulations, and the proposed explanation of spike tolerance lacks direct causal validation.
The mathematical operator underlying RhoNorm is closely related to established weighted divisive-normalization formulations. As the authors acknowledge in Section 4 and Appendix A.3, the proposed normalization corresponds to a restricted weighted L2 form within the generalized divisive normalization (GDN) family.
Consequently, the main mathematical identities, including the finite-spike radius response in Equation (5), follow directly from the weighted quadratic denominator. These results provide a useful characterization of the operator but do not, by themselves, establish a new normalization principle.
The more distinctive contribution is therefore the parameterization and empirical application of this operator to modern Transformers. However, the experiments do not fully establish why the learned metric improves resistance to activation spikes.
The stress tests compare separately trained RMSNorm and RhoNorm checkpoints. Differences in downstream loss could reflect the learned normalization weights, differences in the surrounding network parameters, or other consequences of the training trajectories. Furthermore, although Table 2 provides useful clean-quality controls, the corresponding vector-gain, scalar-gain, and zero-mean-metric configurations are not evaluated under the same spike perturbations. The authors explicitly acknowledge that their controls do not attribute spike tolerance to individual design choices.
The robustness analysis is also restricted to a single fixed evaluation batch and perturbation mask, with a primary spike magnitude of 100 and approximately 1% randomly selected tensor elements.
These limitations matter because improved spike tolerance is one of the paper's central empirical claims, while the proposed connection between learned energy shares and the observed robustness advantage remains largely indirect.
A more convincing analysis would distinguish the effect of the learned denominator from general changes in network representations and optimization.
- Several practical and cross-domain conclusions rely on incompletely matched comparisons, limiting their evidential strength.
The principal language-model improvements are reasonably well controlled, but important additional results involve differences beyond the normalization operator.
First, the ImageNet experiments use size-specific configurations selected using validation performance. In particular, the DeiT-Base result compares a more strongly regularized RhoNorm configuration against a standard-recipe LayerNorm baseline. Appendix D shows that under the original standard recipe, RhoNorm achieves lower validation accuracy than LayerNorm (81.45% versus 81.59%). The reported improvement to 82.03% follows changes to drop-path and mixup settings, without a corresponding stronger-regularization LayerNorm control.
Similarly, the DeiT-Tiny comparison introduces additional Q/K normalization sites, making it difficult to attribute the full reported improvement specifically to learned channel-energy weighting.
Second, the multimodal experiments on Qwen2.5-VL-7B use only one training run per configuration and nonstandard sampled evaluation subsets. Although the results are encouraging, they provide limited evidence of reproducible downstream improvements.
Third, the practical efficiency claim requires stronger control. Appendix E reports approximately 5.1% greater training-step time for RhoNorm in its systems comparison. However, the paper does not establish whether the clean-quality advantage persists when RMSNorm and RhoNorm are compared under equal wall-clock training budgets.
The time-to-GPT-2 result is also not a normalization-only comparison: the 88-minute RhoNorm result is compared with published configurations using different recipes. Moreover, its reported 95% interval for mean CORE, [0.2483, 0.2681], includes the GPT-2 reference score of 0.256525.
The authors appropriately disclose many of these limitations, but the headline results still risk conveying stronger evidence for cross-domain generalization and practical efficiency than the controlled experiments establish.
These issues are consequential because they affect whether RhoNorm should be regarded as a broadly preferable normalization replacement rather than a promising modification with demonstrated benefits under specific training configurations.
Questions for authors
- Attribution of the learned metric's contribution to spike tolerance (Critical Weakness 1).
Can the authors provide activation-spike evaluations for the matched output-gain and relative-metric controls from Table 2, ideally under multiple independently sampled batches, masks, and perturbation magnitudes?
In particular, does RhoNorm retain a meaningful robustness advantage over an equal-parameter vector-gain baseline, rather than only over parameter-free RMSNorm?
It would also be informative to examine whether the learned channel-energy shares quantitatively predict the observed differences in spike sensitivity.
This evidence could materially strengthen the paper's central claim that learning the denominator, rather than merely modifying the trained network, is responsible for improved perturbation tolerance.
- Compute-matched predictive quality and practical efficiency (Critical Weakness 2).
Does RhoNorm retain its validation-quality advantage over RMSNorm when both models are trained for the same wall-clock time, rather than the same number of optimization steps or tokens?
Relatedly, can the authors provide a time-to-GPT-2 comparison against an otherwise identical RMSNorm nanochat configuration, using the same evaluation protocol and comparable random seeds?
These results would clarify whether the reported efficiency benefit is attributable to RhoNorm itself and whether the additional per-step computational cost is justified by faster improvement in predictive quality.
- Normalization-only improvements on vision tasks (Critical Weakness 2).
Can the authors provide a matched-recipe DeiT-Base LayerNorm baseline using the stronger regularization settings selected for RhoNorm?
Additionally, for DeiT-Tiny and DeiT-Small, how much of the reported improvement remains when both baselines use identical normalization locations, particularly with respect to Q/K normalization?
These comparisons are important for determining whether the ImageNet gains are attributable to learned channel-energy weights rather than changes in normalization placement, regularization, or configuration selection.
2 comments
Response to Reviewer WCxr (Part 2)
Q2a. Does the quality advantage persist at equal wall-clock training time?
Yes at matched training-step time, by a smaller margin.
| Method, n=3 | Updates | Training-step time (min) | Final bpb |
|---|---|---|---|
| RMSNorm | 5,674 | 99.03 | 0.715935 |
| RhoNorm | 5,374 | 99.37 | 0.715258 |
RhoNorm minus RMSNorm is −0.000677 bpb [−0.001138, −0.000216], with all three pairs improving (Table 4, Table 26). Both arms use graph replay, eight H100 GPUs and a complete learning-rate schedule; pilot timing fixed the horizons. The fixed-update gain is 0.002323 bpb; in the matched budget RMSNorm receives 5.6% more updates. The clock sums training-step durations and excludes graph capture, evaluation and logging; total job times are 104.43 (RMSNorm) and 104.13 (RhoNorm) minutes. The runs are not FLOP-matched.
Q2b. Is there a time-to-GPT-2 comparison against an otherwise identical RMSNorm configuration?
No. That experiment was not run. What the available data show:
- Full 22-task CORE on the matched-time checkpoints, at about 99 minutes: 0.2586 (RMSNorm) and 0.2615 (RhoNorm), a difference of +0.0029 [−0.0379, +0.0437]. Both arms exceed the GPT-2 reference of 0.256525 in mean, and the difference is unresolved (Appendix F.2).
- The 88-minute result is a complete-configuration comparison with five seeds per horizon and no RMSNorm counterpart. As the reviewer notes, the 95% interval of its mean, [0.2483, 0.2681], includes the reference (Table 25).
Section 3.4 and the abstract now state that a time saving attributable to the normalizer is not established. The matched-time study supports a bpb advantage at equal training-step time; it does not support an efficiency claim in CORE.
Q3a. A matched-recipe DeiT-Base LayerNorm baseline.
Provided. RhoNorm remains ahead by 0.3–0.5 points, depending on how runs are selected.
| Configuration | Sites | n | Final top-1 (%) |
|---|---|---|---|
| Base / LayerNorm | 25 | 3 | 81.642 ± 0.178 |
| Base / RhoNorm | 25 | 3 | 82.107 ± 0.136 |
| Base / RhoNorm, all completed runs | 25 | 3 | 81.951 ± 0.300 |
| Tiny / LayerNorm | 25 | 3 | 72.065 ± 0.138 |
| Tiny / RhoNorm | 25 | 3 | 72.599 ± 0.068 |
| Tiny / LayerNorm + free Q/K | 49 | 3 | 74.018 ± 0.068 |
| Tiny / RhoNorm + free Q/K | 49 | 3 | 73.898 ± 0.241 |
| Tiny / RhoNorm + learned Q/K | 49 | 3 | 74.313 ± 0.145 |
Both Base methods use the 25 native sites, drop path 0.2 and mixup 1.0, with five full-length runs each.
Q3b. How much of the Tiny and Small improvement remains with identical normalization locations?
Tiny: 0.3–0.5 points of the 2.248. Small: not tested in the selected cohort.
- Tiny, same 25 native sites: RhoNorm is 0.534 points above LayerNorm.
- Tiny, same 49 sites: adding parameter-free Q/K normalization raises LayerNorm by 1.953 points. RhoNorm with learned Q/K weights is 0.295 points above that configuration, and RhoNorm with free Q/K is 0.120 points below it. These are three-seed configuration means without paired intervals (Appendix F.6).
- Small: no matched-site control was run in the selected timm/EMA cohort. The only same-site evidence is the DeiT-runner native-site diagnostic, +0.14 points in best-checkpoint accuracy (Table 20).
Multimodal and headline claims (W2). The multimodal study still has one run per configuration on sampled subsets; the text now states that the differences cannot be separated from seed variation (Section 3.7). The abstract no longer presents the DeiT, OLMoE and time-to-GPT-2 results as normalizer-only gains: it reports 0.3–0.5 ImageNet points with matched recipes and sites and labels the other two as a single run against published baselines and a complete-configuration comparison.
Statistics: every decoder comparison uses three training seeds common to its arms; values in brackets are unadjusted paired 95% Student intervals over seeds, with repeated batches and masks averaged within a seed (Section 3.1, Appendix F.1). An interval that contains zero does not establish equivalence.
Response to Reviewer WCxr (Part 1)
Conceptual novelty (W1). We agree. The abstract, introduction and Section 2 now state that the operator is a scale-invariant special case of weighted divisive normalization and that the identities follow from the weighted quadratic denominator (Appendix A.3). The contribution is the Transformer parameterization, the analysis and the controlled evaluation.
Q1. Spike evaluations for the matched controls, and attribution to the learned metric.
Does RhoNorm keep a robustness advantage over an equal-parameter vector gain? Under Full replacement and late, strong spikes, yes, but it is a small part of the gap to RMSNorm; under Block replacement, no.
The grid uses three seeds, eight batches, five masks, blocks 6/12/18 and strengths 10/30/100 (Table 30). An output gain removes 5.5 of the 6.5 nats between RMSNorm and RhoNorm. RhoNorm's further advantage is 0.953 at block 18 and 1.007 [0.821, 1.194] at block 12 for 100×, and is absent or reversed for weaker or shallower spikes.
Controls from the clean-quality table, evaluated under spikes:
Do differences in separately trained checkpoints reflect the learned denominator? To address this we intervene at fixed checkpoints: at all native block sites from the injection block onward, RhoNorm's perturbed-to-clean radius ratio is replaced by the uniform one, keeping each token's clean scale and all weights.
Values are nats/token at 100×, six batches and four masks per seed (Table 6, Figure 3, Appendix F.10). The learned response accounts for most of the margin over vector gain at d24 and for less than a tenth of the margin over RMSNorm at d32. At half depth the effect is reversed at d24 (−0.273 [−0.297, −0.248]) and unresolved at d32. The 0.979 and 0.953 d24 gaps are the same contrast on the six-batch and eight-batch evaluation sets.
Do the learned energy shares quantitatively predict spike sensitivity? No. The shares give the radius ratio exactly, sqrt(1 + (k² − 1) q_M) (Equation 7; predicted and measured ratios agree to four figures in Table 32). They do not give a calibrated predictor of downstream loss: under a uniform random mask the mean share equals the mask fraction for any weights, and the path-separated tests show that the model difference arises downstream of the injection site, not at it (Table 39). The intervention above is the direct test we ran. It does not explain why training produces the tolerance, and Q/K, embedding and final sites were not intervened on separately.