acceptodds
Under review as a conference paper at ICLR 2027

Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping

Abstract

Matrix-valued optimizers such as Muon exploit the spectral structure of neural network updates through Newton–Schulz orthogonalization, but their near-flattening of the singular spectrum discards relative magnitude information across gradient modes. We introduce Soren (Spectral Orthogonal Reshaping), a matrix-valued optimizer that preserves the singular subspaces of the gradient while applying a bounded, monotone sigmoid transformation to its singular values. This smoothly compresses dominant modes without fully flattening the spectrum. We interpret Soren as a positive-definite preconditioned gradient method and establish convergence guarantees under relative smoothness and metric Polyak–\Lojasiewicz geometry. To avoid explicit singular value decomposition, we further develop a finite-depth Soft Newton–Schulz (SNS) polynomial realization of the sigmoid spectral map and characterize how its spectral approximation affects the induced convergence geometry. Experiments across LLM pre-training, supervised fine-tuning, and direct preference optimization demonstrate the effectiveness and robustness of Soren against established optimizers.

Primary area:
optimization
Keywords:
optimization, Muon, spectral reshaping, Newton-Schulz iteration, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hkgjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Soren, a matrix optimizer that applies a sigmoid map to gradient singular values. The map preserves singular vectors and compresses the spectrum without making all active singular values equal. The authors express the exact update as preconditioned gradient descent and prove a conditional contraction under metric relative smoothness and a metric PL assumption. They also propose Soft Newton-Schulz (SNS), a two-stream polynomial approximation that avoids an explicit SVD. Experiments compare Soren with AdamW, Muon, and HTMuon in LLM pre-training, supervised fine-tuning, and direct preference optimization.

Critical strengths
  • The sigmoid map is simple and gives a clear interpolation between retaining and flattening spectral magnitudes. The paper tests the method across pre-training, SFT, and DPO rather than on one small task.
  • The theoretical framing is coherent and internally consistent. The proofs in Appendices A.1-A.10 are correct under their stated assumptions as far as I checked. The paper also reports wall-clock cost and states that the CNN absolute-error experiment does not establish the relative-error assumption in Eq. 14.
Critical weaknesses
  • The convergence results do not cover the optimizer used in the experiments. Theorems 1 and 2 use a full gradient and the oracle step 1/β~t1/\tilde\beta_t or 1/β^t1/\hat\beta_t. Algorithm 2 instead uses Nesterov momentum, a fixed tuned rate, hybrid AdamW, and weight decay. The proof does not transfer to this update. Its uniform-rate assumptions also fail near a standard minimum. Since s(0)=1/2s(0)=1/2, β~t\tilde\beta_t diverges near zero even for L(x)=x2/2L(x)=x^2/2, and full row rank fails at stationarity.
  • The paper does not establish that finite-depth SNS approximates the sigmoid matrix map. Equation 14 is assumed rather than derived from Algorithm 1. Figure 3 tests a scalar, where normalization makes Q0≈1Q_0\approx1. Matrix coefficients can be much smaller because they are divided by ∥X∥F\lVert X\rVert_F. The unnormalized second stream is also unstable outside the tested range. A singular value of 10 gives about 27.43 instead of sigmoid⁡(10)≈1\operatorname{sigmoid}(10)\approx1. Figure 5 visibly disagrees with the claimed spectral match. Table 10 reports absolute errors and does not establish Eq. 14.
  • The closest theory and a natural baseline are missing. Theorem 4 in "Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence" (Nguyen et al.) uses the same induced-metric PL template, but is not cited. "Delving into Muon and Beyond: Deep Analysis and Extensions" (Qi et al.) already studies SVD-free UΣpV⊤U\Sigma^pV^\top compression. Comparing with these transforms is needed to isolate the benefit of sigmoid shaping.
Questions for authors
  • Can you prove a result for the actual stochastic momentum and hybrid AdamW algorithm with a fixed or computable step size? Such a result would materially improve my assessment of the theory.
  • Can you prove a finite-K matrix bound for Eq. 14 that accounts for Frobenius normalization, the raw singular-value scale, rank, and SNS depth? Without such a bound, Eq. 14 remains an assumption rather than a property of Algorithm 1. Please also explain the visible disagreement in Figure 5 and release the values and plotting code.
Reviewer_rFajOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Soren, an optimizer that preserves the gradient’s singular vectors while applying a sigmoid transformation to its singular values. The authors interpret this transformation as positive-definite preconditioning and establish convergence guarantees under a metric Polyak–Łojasiewicz condition. To avoid explicit singular value decomposition (SVD), they introduce Soft Newton–Schulz (SNS), an efficient approximation of the sigmoid transformation. The empirical evaluation spans C4 pre-training across four LLaMA model sizes, downstream evaluation of Qwen3-0.6B, supervised fine-tuning of Qwen2.5-3B, and direct preference optimization (DPO) of Mistral-7B. Comparisons with AdamW, Muon, and HTMuon demonstrate competitive or improved performance across these settings.

Critical strengths

Promising results across several distinct language-model training settings. The empirical evaluation spans C4 pre-training across four LLaMA model sizes, downstream evaluation of Qwen3-0.6B, supervised fine-tuning of Qwen2.5-3B, and direct preference optimization (DPO) of Mistral-7B. Across these settings, the proposed method consistently outperforms AdamW, Muon, and HTMuon.

Critical weaknesses

Figure 5 appears to show a substantially wider log-centered spectral range for SNS than for the exact sigmoid transformation, despite the authors' claim that the two closely match. Could the authors explain this discrepancy and report quantitative approximation errors on the actual gradient matrices used in Figure 5?

Questions for authors

Clarification of the implemented optimizer. What values of K, input normalization, and momentum coefficients were used in each experiment? Does the T stream receive X/4, as written in Algorithm 1, or the normalized input X^/4\widehat X/4? Were Muon and Soren matched in their use of Nesterov momentum, weight decay, update scaling, and the parameter subsets assigned to AdamW?

Reviewer_HsMvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develop a new Muon-type optimizer for neural network training. The authors used sigmoid function to reshape the singular values of the moment matrix so that approximately keeping the order of the singular values in the original update matrix. The convergence of this new optimizer is proved under certain smoothness conditions. Fo implementation, the authors used Soft Newton–Schulz polynomial to approximate sigmoid function. This new optimizer, Soren, has been tested in pre-training and post-training experiments and achieve good performances comparing with AdamW, Muon, and HTMuon.

Critical strengths

The paper is well-organized and the presentation is clear.

Critical weaknesses
  1. The theoretical part proves Soren convergence but did not show how Soren outperforms or has a better convergence rate comparing with Muon or AdamW. The theoretical evidence of why we should use Soren is weak.

  2. There are many variants and extensions of Muon, so only comparing with Muon, AdamW and HTMuon is fair in the experiments. Also, there are many related works worth mentioning or comparing empirically. For instance, [1] and [2] consider adaptive and scalable methods form Muon; [3] and [4] consider acceleration of Newton–Schulz iteration; [5], [6], [7], and [8] consider other singular value distribution shape for training.


[1] Si et al. 2025. AdaMuon: Adaptive Muon Optimizer

[2] Li et al. 2025. NorMuon: Making Muon more efficient and scalable

[3] Amsel et al. 2025. The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm

[4] Yang et al. 2026. PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Training

[5] Dong and Sawin. 2026. Muon$^p$: Muon with Fractional Spectral Powers

[6] Goyal and Ramkumar. 2026. Spectrally Targeted Muon.

[7] Pethick. 2026. When to use what Schatten-p norm in deep learning

[8] Shumaylov et al. 2026. Muon is Not That Special: Random or Inverted Spectra Work Just as Well

Questions for authors
  1. Why do we choose sigmoid function for singular value spectrum? How about some other increasing function? Is this choice contradicting with the claim in [8] above?

  2. How do the smallest relative-smoothness constant and PL constant in (7) and (8) change if we consider the pure Muon and HTMuon?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.