- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes PLUM, a Muon-based optimizer for native low-rank weight training where linear layer weights are represented as , where U is an orthonormal basis, and Y is unconstraned. The key observation is that matrix-sign updates have unit nonzero singular values, simplifying Grassmann geodesics and momentum transport into inexpensive matrix operations. The method additionally introduces a product-aware allocation of factor learning rates, provides convergence guarantees for finite Newton–Schulz, and report favorable perplexity and training efficiency.
Critical strengths
Elegant specialization of Grassmann geometry to matrix-sign optimization, avoiding repeated retractions. Finite-NS convergence analysis addresses approximation errors in the implemented algorithm. Relevant comparisons with LORO and Spectron across four pretraining scales, with promising efficiency results.
Critical weaknesses
-
Experimental reliability and statistical sufficiency: a. The C4 improvements over Spectron are consistently small (0.13–0.24 perplexity points), without reported multi-seed variance, confidence intervals, or significance analysis. b. GLUE uses a single seed, despite including small, high-variance benchmarks. c. The 1B dense AdamW baseline is also unusually weak (16.52 versus 15.56 in prior GaLore/LORO results under nominally comparable settings). Figure 2 provides a compute-matched dense Muon comparison only at 350M, without an equal-token counterpart. Stronger, consistently tuned dense baselines and repeated runs are needed.
-
Insufficient validation of finite-NS geometry. The method depends on approximate orthogonality, yet finite NS does not preserve it exactly. For an initially orthonormal basis (U) and horizontal approximate matrix-sign direction (Q), the one-step residual is Subsequent orthogonality errors also contaminate the horizontal projection and factor-step allocation. Theorem 1 controls accumulated drift only under restrictive, horizon-dependent learning-rate, momentum, and normalization conditions that differ substantially from the experiments (e.g. lr is 160 times larger than the what the theory). The authors should provide residual trajectories and sensitivity to NS iterations and periodic reorthogonalization.
-
Missing mechanism ablations and limited novelty isolation. The principal ablations investigate rank and learning-rate allocation. They do not isolate the proposed geometric contributions: a. Grassmann geodesic versus tangent projection with retraction. b. Parallel transport versus ordinary momentum projection. c. PLUM allocation versus Spectron-style spectral scaling. d. Exact (SVD based) matrix sign versus the finite-NS implementation.
These comparisons are important because fixed-rank Riemannian optimization (LORO, Riemannion), product-level stabilization (Spectron), and spectral steepest descent on low-rank manifolds (LoRA-Muon) are established prior art.
The geodesic and transport formulas themselves are standard geometric identities. I suppose that PLUM's main novelty is their computational simplification under matrix-sign updates, not the underlying geometry or low-rank Muon optimization. The current experiments do not establish whether this specific mechanism explains the observed gains.
- Reporting and reproducibility: Several GLUE averages are arithmetically inconsistent. For example, PLUM rank 8 averages to 84.54 rather than 85.94. Missing complete optimizer configurations, initialization details, seed variability should be supplied.
Questions for authors
Q1 Can the authors explain the discrepancies with previously published dense baselines and provide matched-token dense Muon comparisons?
Q2 Can the authors report what are the observed layerwise orthogonality errors during pretraining, and how sensitive are performance and stability to NS accuracy or periodic reorthogonalization?
Q3 How much of the improvement over Spectron comes specifically from Grassmann updates and momentum transport, rather than the factor-step allocation?
Q4 Did the authors consider the implications of the stale coordinates of Y given U_new is rotated to include new directions that Y could not have accounted for?
6 comments
Part 2
W2. Finite-NS geometry and orthogonality
We measured orthogonality over 100,000 updates of a 130M C4 comparison, five times the original 130M update budget. Both methods use the same data stream and LLaMA-2 tokenizer, rank 256, sequence length 256, and batches of 512 documents. The control applies Muon separately to the two unconstrained factors, both randomly initialized following the stronger initialization identified by Glentis et al. [1] and our matched-control experiments. It uses the best learning rate from the 130M grid . PLUM retains the original learning rate of . Both use linear warmup followed by cosine decay.
| Method | Validation loss at 100K updates |
|---|---|
| Vanilla Muon (both factors random) | 3.0636 |
| PLUM | 3.0180 |
At each checkpoint, we measure and report the mean and maximum across PLUM's constrained matrices.
| Updates | Mean orthogonality error | Maximum orthogonality error |
|---|---|---|
| 0 | 0.002443 | 0.002776 |
| 10,000 | 0.002452 | 0.002924 |
| 20,000 | 0.002460 | 0.002893 |
| 30,000 | 0.002469 | 0.002918 |
| 40,000 | 0.002473 | 0.002944 |
| 50,000 | 0.002476 | 0.002945 |
| 60,000 | 0.002481 | 0.002965 |
| 70,000 | 0.002483 | 0.002990 |
| 80,000 | 0.002483 | 0.002993 |
| 90,000 | 0.002484 | 0.002985 |
| 100,000 | 0.002482 | 0.002957 |
The mean error changes by only between the first and last checkpoints, and the maximum stays below 0.003. By 100K updates, the learning rate has reached its peak and begun cosine decay.
Low-precision parameter storage perturbs orthogonality through rounding after each update, as also examined in ProLand [4]. The largest measured residual, , gives . More directly, , so the basis introduces almost no norm distortion when mapping coefficient changes to weight changes. LORO associates poor factor conditioning with unstable low-rank training [5]. These measurements show that PLUM preserves a well-conditioned basis over the longer run with free geodesic and transport refinements of the projected updates.
The analysis separates newly introduced NS error from the effect of existing error on later updates. Let and . The orthogonality error obeys .
The terms carry the previous error, add the NS residual, and account for imperfect horizontality. At an orthonormal basis with a horizontal direction, only the second remains, giving . Thus, the NS residual enters the basis error at second order in the step size.
For subsequent steps, makes the feedback explicit. Lemma 1 jointly tracks and the normal momentum component . Under Theorem 1's conditions, on the event in its probability bound, the proof obtains , hence , where and bounds the NS orthogonality residual. Thus, the proof also controls the feedback through later projections and momenta. With the theorem's schedule , the accumulated finite-NS term is .
Corollary 1 relates these near-orthogonal iterates to constrained solutions. The analytical change , restores orthogonality while preserving . Under the stated assumptions and schedules, the average expected constrained stationarity measure is . This change of coordinates is used in the analysis and is not an additional training step.
Theorem 1 uses sufficient step-size conditions to control accumulated error and establish the convergence rates. Such conditions can be conservative under standard smoothness assumptions, as discussed for Muon by Riabinin et al. [6]. The longer-run measurements evaluate PLUM at the learning rate selected for pretraining and show that the basis remains well conditioned in this regime.
As an additional check, we compared PLUM with reorthogonalization every 2,000 steps at 60M, using the same seed and deterministic settings.
| Method | PPL | Final mean orthogonality error |
|---|---|---|
| PLUM | 32.76 | 0.0024 |
| PLUM with reorthogonalization every 2,000 steps | 32.85 | 0.0019 |
Part 3
Reorthogonalization lowers the mean error, with similar PPL in this paired run. ProLand likewise reports lower constraint residuals under retraction without higher task accuracy [4]. PLUM already maintains a well-conditioned basis, with slightly lower PPL here and no periodic factorization or additional stabilization hyperparameters. Its geometric simplification makes this control practical at every training step.
W3. Geometric updates and mechanism ablations
All 60M ablation comparisons below use the same seed, deterministic settings, and training budget.
(a,b) Geodesic retraction and parallel transport. In the exact matrix-sign setting, let with and . Since , the polar update simplifies to .
Equivalently, this is , the same Grassmann geodesic with angle . Since , its basis update differs from PLUM's by . Both formulas require only scalar coefficients and matrix additions. We evaluate the scalar polar formula by replacing Line 7 of Algorithm 1 with , retaining PLUM's momentum update.
For the generating momentum, with and ,
\qquad (I-U_+U_+^\top)M=\cos(\eta_U)\mathcal T_{\eta_U}(M).$$ Parallel transport preserves the momentum norm. In this exact-sign calculation, ordinary projection retains a $\cos(\eta_U)$ fraction of that norm. Correcting this loss requires a scalar division of the existing projection, with no additional matrix product. The relative difference is $1-\cos(\eta_U)=O(\eta_U^2)$. | Method | Validation loss | | --- | ---: | | PLUM | **3.4892** | | PLUM with naive momentum projection | 3.4956 | | PLUM with approximate polar | 3.4911 | At the small learning rates used here, the close validation losses are consistent with this local agreement. Edelman, Arias, and Smith [7, Sections 3.5–3.6] compare geodesic/transport algorithms with QR/projection alternatives and explain how higher-order differences can leave their local convergence behavior unchanged. The central observation is that matrix sign equalizes the singular values that govern the angular speeds of Grassmann motion. The matrix functions in the geodesic and transport maps then reduce to scalar coefficients. Geodesic plus parallel transport and scalar polar plus projection consequently have the same leading computational cost in this setting. The geodesic and transport refinements are therefore free, requiring no additional matrix decompositions, matrix products, or geometric hyperparameters. This turns operations that usually require costly decompositions into updates suited to GPU pretraining. Finite-NS deviations are addressed in W2. This simplification also applies to other orthogonality-constrained problems with horizontal matrix-sign directions. Their geometric differences grow with the step size, so the distinction can become more consequential in settings that permit larger steps. **(c) Allocation and spectral scaling.** Allocation has a local justification in the product update. At an orthonormal basis, with horizontal $Q_U$ and column-orthonormal sign directions, the two first-order contributions are orthogonal, so $$\lVert aQ_UY^\top+bUQ_Y^\top\rVert_F^2 =a^2\lVert Y\rVert_F^2+b^2\lVert U\rVert_F^2.$$ With $g_i=\langle M_i,Q_i\rangle_F$, maximizing momentum alignment under the stabilized product-step model gives $a:b=\frac{g_U}{\lVert Y\rVert_F^2+\epsilon}:\frac{g_Y}{\lVert U\rVert_F^2+\epsilon}$. PLUM normalizes the larger step to the scheduled learning rate, with zero steps for zero momentum. This is a locally motivated adaptive rule, with a heuristic choice of overall scale. Under finite NS, the norm expression becomes an approximation. Spectron instead applies a common scale based on factor spectral norms. Fig. 3 (Fig.4 after revise) compares PLUM allocation with equal factor steps. We additionally replaced allocation by the shared scale $(1+\lVert U\rVert_2+\lVert Y\rVert_2)^{-1}$ in the same PLUM parameterization, following Spectron's implementation. Learning rates are selected independently using the same grid. | Factor-step rule within PLUM | 60M validation loss | | --- | ---: | | Spectron-style shared spectral scale | 3.4913 | | PLUM allocation | **3.4892** |
Response to Reviewer 5qGX
Thank you for the detailed assessment and the suggestions for strengthening the evidence. We have added repeated runs, parameter-matched dense Muon comparisons, and mechanism ablations. We also corrected the omitted GLUE column and clarified the scope of the finite-NS analysis.
W1. Experimental reliability and dense baselines
(a) Variability of the pretraining gains. We evaluated 60M, 130M, and 350M with seeds {1,2,42}, using deterministic settings for each seed. Both methods share the seeds and data order, with their selected configurations fixed across runs. Entries are validation loss, reported as mean ± sample SD.
PLUM has lower loss in all three paired seeds at each model size. The mean paired reductions are 0.0150, 0.0206, and 0.0425 at 60M, 130M, and 350M, respectively.
(b) GLUE seed variability. We apologize for the typesetting omission: QNLI was evaluated, but its column was accidentally dropped when preparing the table. The complete six-task results over three different seeds are below.
(c) Dense AdamW and dense Muon comparisons. The 1B AdamW result of 16.52 PPL comes from our original training run under the reported settings. The shared 15.56 value in GaLore, SLTrain, and LORO traces back to GaLore. SLTrain explicitly inherits its baseline results, and LORO's Table 1 identifies GaLore and SLTrain as the baseline sources. These entries therefore share a reference value rather than provide separate reproductions.
Our independently trained baseline agrees closely with three recent reports:
The agreement with these independently reported results supports the accuracy of our original baseline.
At 350M, we added the equal-token dense Muon comparison and applied PLUM to a model with a matching stored parameter budget. All three configurations process 6.4B tokens under the same training and evaluation protocol.
To match the parameter budget, we retain 24 blocks and change the model width, FFN width, and rank from (1024,2736,256) to (1536,4096,384). Stored parameters include the factors, embeddings, output head, and normalization parameters. FLOPs count the model's forward and backward matrix multiplications at sequence length 256, with dense Muon normalized to 1×.
At stored parameter counts matched within 0.2%, PLUM reduces validation loss from 2.8249 to 2.7821 under the same token budget. The original PLUM uses about half the stored parameters. This adds equal-token and equal-parameter evidence to the equal-FLOP comparison in Fig. 2.