- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MultiNormStep, a family of coordinatewise optimizers. Each takes a steepest-descent step under an Orlicz-modular constraint with updates . Two weight profiles give two different maps: "log: h(x) = ln(1+x)" and "sigmoid: h(x) = x/(1+x)". SignSGD, PowerStep, and normalizedSGD are presented as special cases. Applying the same construction to the singular values gives a "matrix face" whose limiting case is Muon (not implemented).
The theory gives an convergence bound up to an neighborhood for momentum applied before the map with held fixed. The experiments have 3 parts:
- full parameter training on GPT 2 (124M/ 350M) and Qwen3 0.6B/ 4B. The maps end with about 0.01 nats gained over AdamW on validation and the log map must be clipped to stay stable at 4B.
- a Muon hybrid on Qwen3 0.6B/4B and a proprietary MoE, where Newton–Schulz Muon is held fixed on the 2D matrices and the maps replace AdamW on the embedding, LM head, norm and bias parameters, ending 0.001–0.03 nats below the AdamW branch on training loss
- ablations of λ's pooling granularity and EMA coefficient, and a measured saving of about one fp32 optimizer buffer relative to AdamW. Code for both codebases are released.
Critical strengths
-
The paper targets a real gap which appears in practice. In current Muon recipes, the non matrix parameters (embeddings, norms, bias, LM head) are left to AdamW. The paper asks directly what this branch should use, with a drop-in optimizer that keeps one buffer per parameter, and evaluates it across three codebases and model families up to Qwen3 4B and a production MoE, in both full-parameter and hybrid settings, with a per-method learning-rate grid on GPT 2. The memory accounting is exact and honest about its sharding limitation, and the diagnostics that expose the paper’s own confounds are published.
-
One robust design finding, made visible by the scale of the study. Whether the map is bounded decides stability at scale: at Qwen3-4B the unbounded log map diverges at every learning rate tried (2.5e−4, 3e−4, 5e−4) unless its update is clipped, while the bounded sigmoid map trains normally at the same rate. This does not appear at 0.6B or on GPT-2, so a smaller study would have missed it.
Critical weaknesses
- The theoretical framework and both "new" maps exist in uncited peer-reviewed literature! The update is dual-space preconditioning (https://epubs.siam.org/doi/10.1137/19M130858X). (https://arxiv.org/pdf/2502.08532) analyze exactly this iteration for separable and isotropic kernels. Their Table 1 contains both of this work's Young functions. Their example 1.6 applied the separable sigmoid kernel coordinate wise and notes it recovers Adam when beta_1 = beta_2 = 0. The (https://arxiv.org/pdf/2510.11312) work follows up with heavy ball momentum and a stochastic analysis. Row pooled second moments that remove Adam's v buffer exist in Adam-mini (https://arxiv.org/abs/2406.16793).
To be precise about the overlap, the final results are rederived with a different method, but the results stand the same. The convergence proof (Lemma 4, 5, Theorem 2, momentum on raw gradient) is the paper's own novel contribution. The experiments are new.
The “remarkably little attention” phrasing of the non-Muon branch omits the published Adam-free recipes for exactly these parameters, Scion (ICML 2025; https://arxiv.org/abs/2502.07529) and modular duality’s embedding dualization (ICML 2025; https://arxiv.org/pdf/2410.21265).
-
The reported benefits are too small relative to the paper’s own variability. The experimental gains seen are on a single seed and hence is a bit unclear on how to claim benefits with confidence. The Muon-hybrid validation margins over AdamW are 0.010/0.007 nats at 0.6B and 0.001/0.004 at 4B, where all five branches sit within 0.007 nats (Table 11); the paper’s own learning-rate sweep moves the log map by 0.02–0.04 nats between adjacent rates at 124M (Table 10). Model-X is training loss only, on a proprietary model and corpus. The 4B runs see $\approx$2.6 tokens per parameter, so “at scale” and “early in training” are confounded. If possible I would like the authors to perform the Qwen 0.6B and Qwen 4B run with 2 more seeds and at a larger training scale to allow for a confidence interval and substantiate the claims further.
-
The hybrid comparison confounds geometry with effective step size. All branches share one learning rate, at which the paper’s own Table 12 shows the maps’ per-coordinate update RMS on the vector branch is 3.4× (sigmoid), 5.7× (log) and 6.8× (signSGD) that of AdamW. There is no vector sweep of AdamW or RMS-matched control, and the 4B training loss ordering of the five branches tracks their update RMS. The two controls that Lemma 3 covers identically are not run. It would be very interesting and crucial to know how the identity link h(x) = x with the same row- and momentum (essentially Adam-mini with heavy-ball momentum), which isolates the nonlinearity, and coordinatewise clipping clip(m/, −1, 1), which isolates smooth saturation, operate.
Minor:
- the bound is vacuous once σ/G ≥ (1−γ)^{3/2} ≈ 0.032 at the used γ = 0.9
- I think App. F.2 should read λ ≈ RMS(g)/√(2γ), not RMS(g)/(√2·γ)? Also, the constant cannot be “absorbed into the learning-rate sweep” for a nonlinear h. This might have to be rephrased.
- eq. 8 is printed with the ascent sign
- Can the x axis for the training runs be reported in terms of tokens rather than steps? It would be more standardized and would be easier for readers!
- Can you report the validation curves and numbers in the main text and defer the training numbers to the appendix?
Questions for authors
- How many independent seeds underlie each entry of Tables 4, 9 and 11, and what is the standard deviation of final validation loss for AdamW and the sigmoid map at all model scales (GPT 2 and Qwen 3 runs)?
- What does the identity link h(x) = x (same row λ, momentum, β₂ and learning rate) and coordinatewise clipping clip(m/λ, −1, 1) look like? How does this compare with the two explored functions as a baseline? This would let us know how the nonlinearity plays a role in the optimization!
- Train vs validation at 4B. Why does the 4B hybrid training-loss margin of 0.030–0.033 nats (Table 4) collapse to 0.001–0.004 on validation (Table 11)?
- How does this paper position itself with that of https://arxiv.org/pdf/2502.08532, https://arxiv.org/pdf/2510.11312? What do you claim as new relative to these works?
- Why do runs diverge at lower learning rates than the selected one for Model-X (Table 8)? Can you report validation loss or perplexity?
Thank you for your review.
Prior work and novelty. [Revised, §2 and intro]
Oikonomidis et al. (ICML 2025, Table 1) give the same scalar log and rational-sigmoid maps and their generating functions; their experiments use radial variants on a toy objective and radial sigmoid on full-batch MNIST. Oikonomidis et al. (NeurIPS 2025) accumulate preconditioned gradients and analyze the base method without momentum, whereas we apply the link after accumulating raw stochastic gradients. We now credit both, together with Maddison et al. (dual-space preconditioning), Adam-mini, Scion, and the modular-duality framework, and we narrowed the "little attention" claim in the introduction to "comparatively little", noting these works touch the companion parameters. Our stated contribution is the unified Orlicz template for these links and their spectral counterparts, plus the LLM-pretraining and Muon-companion evaluation.
Confound between geometry and effective step size. [Revised, Tables 9 and 10; Open]
Two controls separate link shape from magnitude.
These are precisely the two controls you asked for. The identity link h(x)=x with the same row-λ and momentum (Adam-mini with heavy ball) isolates the nonlinearity, and the coordinatewise clip clip(m/λ,−1,1) isolates smooth saturation; both are now run in Table 10, and the companion-LR sweep of Table 9 supplies the AdamW vector sweep you noted was missing. They also answer the "ordering tracks update RMS" concern directly: the clip caps every coordinate at 1 yet the smooth maps still beat it (log 2.889, sigmoid 2.892 vs clip 2.895), so the ranking is set by link shape, not by raw step magnitude. A full RMS-matched sweep on the vector branch nonetheless remains Open, and Table 16 reports the per-coordinate update-RMS gaps (sigmoid 3.4×, log 5.7×, signSGD 6.8× AdamW) that motivate it.
Seeds, variability, horizon, and held-out evaluation. [Revised, Tables 5 and 6]
We ran the two extra seeds you asked for. Table 5 now reports three seeds (42, 1234, 2026) for the Qwen3 hybrids: validation loss 2.903±0.003 / 2.894±0.005 / 2.895±0.004 at 0.6B and 2.765±0.003 / 2.763±0.003 / 2.760±0.003 at 4B (AdamW / log / sigmoid). This answers the variability concern directly: both maps improve on AdamW in every paired seed at both scales, and at 0.6B the ≈0.008--0.009 nat margin exceeds the run-to-run std (0.003--0.005). On the 0.02--0.04 nat swing the log map shows between adjacent learning rates: that measures learning-rate sensitivity for all parameters, not the specific non-matrix branch we focused on. The remaining tables are single-seed point estimates; we state this and treat Table 5 as the reproducibility evidence.
On Model-X being training-loss only: Table 6 now adds held-out evaluation on Wikipedia and a Pile test split at both scales, where both links again improve on AdamW, so the Model-X result no longer rests on training loss alone.
On the 4B horizon confound: we agree "at scale" and "early in training" are entangled at about 2.6 tokens per parameter (vs 17.5 at 0.6B), and we narrowed the scale claim accordingly. The training-loss gains of 0.033 / 0.030 nats shrink to 0.001 / 0.004 on validation because the held-out gap needs a longer horizon to open, not because the effect vanishes; the training-loss margin in fact widens at 4B. Model-X's 55B / 147B-token runs supply the longer-horizon evidence, and a longer-budget Qwen 4B run remains Open due to limited resource.
Theory and formula corrections. [Revised]
Tokens on the x-axis and validation. [Revised]
Qwen3 and Model-X curves now use processed tokens (billions); GPT-2 stays in steps. Validation loss / PPL is now in the main text (Tables 5 and 6).
Model-X minor points. [Revised]
Table 12 now marks the untested low-rate Model-X entries as "na" (not searched) rather than "NaN" (diverged), with the caption updated.