- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Soren, a matrix optimizer that applies a sigmoid map to gradient singular values. The map preserves singular vectors and compresses the spectrum without making all active singular values equal. The authors express the exact update as preconditioned gradient descent and prove a conditional contraction under metric relative smoothness and a metric PL assumption. They also propose Soft Newton-Schulz (SNS), a two-stream polynomial approximation that avoids an explicit SVD. Experiments compare Soren with AdamW, Muon, and HTMuon in LLM pre-training, supervised fine-tuning, and direct preference optimization.
Critical strengths
- The sigmoid map is simple and gives a clear interpolation between retaining and flattening spectral magnitudes. The paper tests the method across pre-training, SFT, and DPO rather than on one small task.
- The theoretical framing is coherent and internally consistent. The proofs in Appendices A.1-A.10 are correct under their stated assumptions as far as I checked. The paper also reports wall-clock cost and states that the CNN absolute-error experiment does not establish the relative-error assumption in Eq. 14.
Critical weaknesses
- The convergence results do not cover the optimizer used in the experiments. Theorems 1 and 2 use a full gradient and the oracle step or . Algorithm 2 instead uses Nesterov momentum, a fixed tuned rate, hybrid AdamW, and weight decay. The proof does not transfer to this update. Its uniform-rate assumptions also fail near a standard minimum. Since , diverges near zero even for , and full row rank fails at stationarity.
- The paper does not establish that finite-depth SNS approximates the sigmoid matrix map. Equation 14 is assumed rather than derived from Algorithm 1. Figure 3 tests a scalar, where normalization makes . Matrix coefficients can be much smaller because they are divided by . The unnormalized second stream is also unstable outside the tested range. A singular value of 10 gives about 27.43 instead of . Figure 5 visibly disagrees with the claimed spectral match. Table 10 reports absolute errors and does not establish Eq. 14.
- The closest theory and a natural baseline are missing. Theorem 4 in "Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence" (Nguyen et al.) uses the same induced-metric PL template, but is not cited. "Delving into Muon and Beyond: Deep Analysis and Extensions" (Qi et al.) already studies SVD-free compression. Comparing with these transforms is needed to isolate the benefit of sigmoid shaping.
Questions for authors
- Can you prove a result for the actual stochastic momentum and hybrid AdamW algorithm with a fixed or computable step size? Such a result would materially improve my assessment of the theory.
- Can you prove a finite-K matrix bound for Eq. 14 that accounts for Frobenius normalization, the raw singular-value scale, rank, and SNS depth? Without such a bound, Eq. 14 remains an assumption rather than a property of Algorithm 1. Please also explain the visible disagreement in Figure 5 and release the values and plotting code.