acceptodds
Under review as a conference paper at ICLR 2027

PLUM: Lightweight Low-Rank Updates for Parameter- and Memory-Efficient Training

Abstract

Training large models is constrained by memory for parameters, gradients, and optimizer states. Low-rank parameterization reduces these costs, but factor-wise optimization is not invariant to the non-unique factorization: equivalent representations of the same weight can induce different weight updates. We introduce PLUM (**P**arameter-efficient **L**ow-rank **U**pdates with **M**uon), a lightweight optimizer that represents each weight by an orthonormal subspace basis and a coefficient matrix, separating subspace motion from coefficient changes with a simple implementation. The unit nonzero singular values of Muon's msign update make the Grassmann geodesic and momentum transport particularly simple, reducing both to inexpensive matrix operations without matrix exponentials or repeated QR/polar retractions. A fixed Newton-Schulz approximation further yields a fully GEMM-based implementation with one momentum state per factor and no additional factor- or geometry-specific hyperparameters to tune. We establish an expected first-order stationarity rate and an bound on the expected maximum orthogonality error for fixed finite Newton-Schulz iterations under standard assumptions. Experiments on C4 pretraining and GLUE fine-tuning up to models with over 1B parameters show lower pretraining perplexity than prior low-rank methods together with a lightweight, GPU-friendly implementation.

Primary area:
optimization
Keywords:
Parameter Efficient Pretraining, Low-Rank Optimization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5qGXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PLUM, a Muon-based optimizer for native low-rank weight training where linear layer weights are represented as W=UY⊤W=UY^\top, where U is an orthonormal basis, and Y is unconstraned. The key observation is that matrix-sign updates have unit nonzero singular values, simplifying Grassmann geodesics and momentum transport into inexpensive matrix operations. The method additionally introduces a product-aware allocation of factor learning rates, provides convergence guarantees for finite Newton–Schulz, and report favorable perplexity and training efficiency.

Critical strengths

Elegant specialization of Grassmann geometry to matrix-sign optimization, avoiding repeated retractions. Finite-NS convergence analysis addresses approximation errors in the implemented algorithm. Relevant comparisons with LORO and Spectron across four pretraining scales, with promising efficiency results.

Critical weaknesses
  1. Experimental reliability and statistical sufficiency: a. The C4 improvements over Spectron are consistently small (0.13–0.24 perplexity points), without reported multi-seed variance, confidence intervals, or significance analysis. b. GLUE uses a single seed, despite including small, high-variance benchmarks. c. The 1B dense AdamW baseline is also unusually weak (16.52 versus 15.56 in prior GaLore/LORO results under nominally comparable settings). Figure 2 provides a compute-matched dense Muon comparison only at 350M, without an equal-token counterpart. Stronger, consistently tuned dense baselines and repeated runs are needed.

  2. Insufficient validation of finite-NS geometry. The method depends on approximate orthogonality, yet finite NS does not preserve it exactly. For an initially orthonormal basis (U) and horizontal approximate matrix-sign direction (Q), the one-step residual is E+=sin⁡2(ηU)(Q⊤Q−Ir).E^+=\sin^2(\eta_U)(Q^\top Q-I_r). Subsequent orthogonality errors also contaminate the horizontal projection and factor-step allocation. Theorem 1 controls accumulated drift only under restrictive, horizon-dependent learning-rate, momentum, and normalization conditions that differ substantially from the experiments (e.g. lr is 160 times larger than the what the theory). The authors should provide residual trajectories and sensitivity to NS iterations and periodic reorthogonalization.

  3. Missing mechanism ablations and limited novelty isolation. The principal ablations investigate rank and learning-rate allocation. They do not isolate the proposed geometric contributions: a. Grassmann geodesic versus tangent projection with retraction. b. Parallel transport versus ordinary momentum projection. c. PLUM allocation versus Spectron-style spectral scaling. d. Exact (SVD based) matrix sign versus the finite-NS implementation.

These comparisons are important because fixed-rank Riemannian optimization (LORO, Riemannion), product-level stabilization (Spectron), and spectral steepest descent on low-rank manifolds (LoRA-Muon) are established prior art.

The geodesic and transport formulas themselves are standard geometric identities. I suppose that PLUM's main novelty is their computational simplification under matrix-sign updates, not the underlying geometry or low-rank Muon optimization. The current experiments do not establish whether this specific mechanism explains the observed gains.

  1. Reporting and reproducibility: Several GLUE averages are arithmetically inconsistent. For example, PLUM rank 8 averages to 84.54 rather than 85.94. Missing complete optimizer configurations, initialization details, seed variability should be supplied.
Questions for authors

Q1 Can the authors explain the discrepancies with previously published dense baselines and provide matched-token dense Muon comparisons?

Q2 Can the authors report what are the observed layerwise orthogonality errors during pretraining, and how sensitive are performance and stability to NS accuracy or periodic reorthogonalization?

Q3 How much of the improvement over Spectron comes specifically from Grassmann updates and momentum transport, rather than the factor-step allocation?

Q4 Did the authors consider the implications of the stale coordinates of Y given U_new is rotated to include new directions that Y could not have accounted for?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5qGX

Thank you for the detailed assessment and the suggestions for strengthening the evidence. We have added repeated runs, parameter-matched dense Muon comparisons, and mechanism ablations. We also corrected the omitted GLUE column and clarified the scope of the finite-NS analysis.

W1. Experimental reliability and dense baselines

(a) Variability of the pretraining gains. We evaluated 60M, 130M, and 350M with seeds {1,2,42}\{1,2,42\}, using deterministic settings for each seed. Both methods share the seeds and data order, with their selected configurations fixed across runs. Entries are validation loss, reported as mean ± sample SD.

Method60M130M350M
Spectron3.5037 ± 0.00633.1883 ± 0.00692.9294 ± 0.0158
PLUM3.4887 ± 0.00053.1676 ± 0.00672.8869 ± 0.0112

PLUM has lower loss in all three paired seeds at each model size. The mean paired reductions are 0.0150, 0.0206, and 0.0425 at 60M, 130M, and 350M, respectively.

(b) GLUE seed variability. We apologize for the typesetting omission: QNLI was evaluated, but its column was accidentally dropped when preparing the table. The complete six-task results over three different seeds are below.

MethodRankCoLASTS-BMRPCRTESST-2QNLIAvg.
Full fine-tuning—62.35 ± 0.4090.82 ± 0.1791.35 ± 0.3679.30 ± 0.8394.69 ± 0.1892.60 ± 0.2585.18 ± 0.05
LoRA462.51 ± 1.4090.54 ± 0.3991.71 ± 1.1079.54 ± 0.9193.12 ± 0.2091.94 ± 0.2284.89 ± 0.43
GaLore461.87 ± 0.4290.85 ± 0.1092.41 ± 0.1880.39 ± 0.8394.69 ± 0.6392.23 ± 0.0485.41 ± 0.35
Spectron461.90 ± 0.2791.02 ± 0.0292.58 ± 0.6076.90 ± 0.0094.65 ± 0.3393.06 ± 0.1985.02 ± 0.22
PLUM463.04 ± 0.2890.85 ± 0.1192.94 ± 0.4279.54 ± 0.2194.65 ± 0.1893.29 ± 0.1385.72 ± 0.10
LoRA862.44 ± 0.6490.63 ± 0.5292.49 ± 0.7280.02 ± 0.9192.97 ± 0.0792.17 ± 0.1285.12 ± 0.18
GaLore860.72 ± 0.6490.83 ± 0.0292.56 ± 0.1080.39 ± 0.7593.96 ± 0.2692.21 ± 0.1085.11 ± 0.16
Spectron862.94 ± 0.4290.81 ± 0.0291.97 ± 0.0877.62 ± 0.3694.50 ± 0.1193.27 ± 0.3185.18 ± 0.10
PLUM862.98 ± 0.3691.15 ± 0.0393.13 ± 0.3779.18 ± 0.9194.69 ± 0.0793.66 ± 0.1885.80 ± 0.29

(c) Dense AdamW and dense Muon comparisons. The 1B AdamW result of 16.52 PPL comes from our original training run under the reported settings. The shared 15.56 value in GaLore, SLTrain, and LORO traces back to GaLore. SLTrain explicitly inherits its baseline results, and LORO's Table 1 identifies GaLore and SLTrain as the baseline sources. These entries therefore share a reference value rather than provide separate reproductions.

Our independently trained baseline agrees closely with three recent reports:

SourceDense optimizer1B PPL
Glentis et al. [1], Table 3Adam16.52
Joo et al. [2], Table 1Adam16.35
Shang et al. [3], Table 1AdamW, cosine schedule16.52
Our original runAdamW16.52

The agreement with these independently reported results supports the accuracy of our original baseline.

At 350M, we added the equal-token dense Muon comparison and applied PLUM to a model with a matching stored parameter budget. All three configurations process 6.4B tokens under the same training and evaluation protocol.

ConfigurationStored parametersValidation lossForward/backward FLOPs
Dense Muon367.97M2.82491.00×
PLUM (matched parameter budget)367.34M2.78210.97×
PLUM (original)185.22M2.89980.47×

To match the parameter budget, we retain 24 blocks and change the model width, FFN width, and rank from (1024,2736,256)(1024,2736,256) to (1536,4096,384)(1536,4096,384). Stored parameters include the factors, embeddings, output head, and normalization parameters. FLOPs count the model's forward and backward matrix multiplications at sequence length 256, with dense Muon normalized to 1×1\times.

At stored parameter counts matched within 0.2%, PLUM reduces validation loss from 2.8249 to 2.7821 under the same token budget. The original PLUM uses about half the stored parameters. This adds equal-token and equal-parameter evidence to the equal-FLOP comparison in Fig. 2.

W2. Finite-NS geometry and orthogonality

We measured orthogonality over 100,000 updates of a 130M C4 comparison, five times the original 130M update budget. Both methods use the same data stream and LLaMA-2 tokenizer, rank 256, sequence length 256, and batches of 512 documents. The control applies Muon separately to the two unconstrained factors, both randomly initialized following the stronger initialization identified by Glentis et al. [1] and our matched-control experiments. It uses the best learning rate from the 130M grid 10−3,2×10−3,4×10−3,6×10−3,8×10−3,10−2,2×10−2\\{10^{-3},2\times10^{-3},4\times10^{-3},6\times10^{-3},8\times10^{-3},10^{-2},2\times10^{-2}\\}. PLUM retains the original learning rate of 5×10−25\times10^{-2}. Both use linear warmup followed by cosine decay.

MethodValidation loss at 100K updates
Vanilla Muon (both factors random)3.0636
PLUM3.0180

At each checkpoint, we measure ∥Uℓ⊤Uℓ−Ir∥F\lVert U_\ell^\top U_\ell-I_r\rVert_F and report the mean and maximum across PLUM's constrained matrices.

UpdatesMean orthogonality errorMaximum orthogonality error
00.0024430.002776
10,0000.0024520.002924
20,0000.0024600.002893
30,0000.0024690.002918
40,0000.0024730.002944
50,0000.0024760.002945
60,0000.0024810.002965
70,0000.0024830.002990
80,0000.0024830.002993
90,0000.0024840.002985
100,0000.0024820.002957

The mean error changes by only 3.9×10−53.9\times10^{-5} between the first and last checkpoints, and the maximum stays below 0.003. By 100K updates, the learning rate has reached its peak and begun cosine decay.

Low-precision parameter storage perturbs orthogonality through rounding after each update, as also examined in ProLand [4]. The largest measured residual, δ=0.002993\delta=0.002993, gives κ(U)≤(1+δ)/(1−δ)<1.003\kappa(U)\leq\sqrt{(1+\delta)/(1-\delta)}<1.003. More directly, (1−δ)∥ΔY∥F2≤∥UΔY⊤∥F2≤(1+δ)∥ΔY∥F2(1-\delta)\lVert\Delta Y\rVert_F^2\leq\lVert U\Delta Y^\top\rVert_F^2\leq(1+\delta)\lVert\Delta Y\rVert_F^2, so the basis introduces almost no norm distortion when mapping coefficient changes to weight changes. LORO associates poor factor conditioning with unstable low-rank training [5]. These measurements show that PLUM preserves a well-conditioned basis over the longer run with free geodesic and transport refinements of the projected updates.

The analysis separates newly introduced NS error from the effect of existing error on later updates. Let Et=Ut⊤Ut−IrE_t=U_t^\top U_t-I_r and Qt=QU,tQ_t=Q_{U,t}. The orthogonality error obeys Et+1=cos⁡2(ηU,t)Et+sin⁡2(ηU,t)(Qt⊤Qt−Ir)−sin⁡(ηU,t)cos⁡(ηU,t)(Ut⊤Qt+Qt⊤Ut)E_{t+1}=\cos^2(\eta_{U,t})E_t+\sin^2(\eta_{U,t})(Q_t^\top Q_t-I_r)-\sin(\eta_{U,t})\cos(\eta_{U,t})(U_t^\top Q_t+Q_t^\top U_t).

The terms carry the previous error, add the NS residual, and account for imperfect horizontality. At an orthonormal basis with a horizontal direction, only the second remains, giving ∥Et+1∥F≤ηU,t2∥Qt⊤Qt−Ir∥F\lVert E_{t+1}\rVert_F\leq\eta_{U,t}^2\lVert Q_t^\top Q_t-I_r\rVert_F. Thus, the NS residual enters the basis error at second order in the step size.

For subsequent steps, Ut⊤(I−UtUt⊤)=−EtUt⊤U_t^\top(I-U_tU_t^\top)=-E_tU_t^\top makes the feedback explicit. Lemma 1 jointly tracks EtE_t and the normal momentum component Ut⊤MU,tU_t^\top M_{U,t}. Under Theorem 1's conditions, on the event in its probability bound, the proof obtains DT≤ΔbTη2+12DTD_T\leq\Delta_bT\eta^2+\tfrac12D_T, hence DT≤2ΔbTη2D_T\leq2\Delta_bT\eta^2, where DT=max⁡t≤T∥Et∥FD_T=\max_{t\leq T}\lVert E_t\rVert_F and Δb\Delta_b bounds the NS orthogonality residual. Thus, the proof also controls the feedback through later projections and momenta. With the theorem's schedule η=η0T−3/4\eta=\eta_0T^{-3/4}, the accumulated finite-NS term is O(T−1/2)O(T^{-1/2}).

Corollary 1 relates these near-orthogonal iterates to constrained solutions. The analytical change Uˉt=Ut(Ut⊤Ut)−1/2\bar U_t=U_t(U_t^\top U_t)^{-1/2}, Yˉt=Yt(Ut⊤Ut)1/2\bar Y_t=Y_t(U_t^\top U_t)^{1/2} restores orthogonality while preserving UˉtYˉt⊤=UtYt⊤\bar U_t\bar Y_t^\top=U_tY_t^\top. Under the stated assumptions and schedules, the average expected constrained stationarity measure is O(T−1/4)O(T^{-1/4}). This change of coordinates is used in the analysis and is not an additional training step.

Theorem 1 uses sufficient step-size conditions to control accumulated error and establish the convergence rates. Such conditions can be conservative under standard smoothness assumptions, as discussed for Muon by Riabinin et al. [6]. The longer-run measurements evaluate PLUM at the learning rate selected for pretraining and show that the basis remains well conditioned in this regime.

As an additional check, we compared PLUM with reorthogonalization every 2,000 steps at 60M, using the same seed and deterministic settings.

MethodPPLFinal mean orthogonality error
PLUM32.760.0024
PLUM with reorthogonalization every 2,000 steps32.850.0019

Reorthogonalization lowers the mean error, with similar PPL in this paired run. ProLand likewise reports lower constraint residuals under retraction without higher task accuracy [4]. PLUM already maintains a well-conditioned basis, with slightly lower PPL here and no periodic factorization or additional stabilization hyperparameters. Its geometric simplification makes this control practical at every training step.

W3. Geometric updates and mechanism ablations

All 60M ablation comparisons below use the same seed, deterministic settings, and training budget.

(a,b) Geodesic retraction and parallel transport. In the exact matrix-sign setting, let Q=msign⁡(M)Q=\operatorname{msign}(M) with U⊤U=Q⊤Q=IrU^\top U=Q^\top Q=I_r and U⊤Q=0U^\top Q=0. Since (U−ηUQ)⊤(U−ηUQ)=(1+ηU2)Ir(U-\eta_UQ)^\top(U-\eta_UQ)=(1+\eta_U^2)I_r, the polar update simplifies to polar⁡(U−ηUQ)=(U−ηUQ)[(1+ηU2)Ir]−1/2=(U−ηUQ)/1+ηU2\operatorname{polar}(U-\eta_UQ)=(U-\eta_UQ)[(1+\eta_U^2)I_r]^{-1/2}=(U-\eta_UQ)/\sqrt{1+\eta_U^2}.

Equivalently, this is cos⁡(arctan⁡ηU)U−sin⁡(arctan⁡ηU)Q\cos(\arctan\eta_U)U-\sin(\arctan\eta_U)Q, the same Grassmann geodesic with angle arctan⁡ηU\arctan\eta_U. Since arctan⁡ηU=ηU+O(ηU3)\arctan\eta_U=\eta_U+O(\eta_U^3), its basis update differs from PLUM's by O(ηU3)O(\eta_U^3). Both formulas require only scalar coefficients and matrix additions. We evaluate the scalar polar formula by replacing Line 7 of Algorithm 1 with U←(U−ηUQU)/1+ηU2U\leftarrow(U-\eta_UQ_U)/\sqrt{1+\eta_U^2}, retaining PLUM's momentum update.

For the generating momentum, with H=Q⊤MH=Q^\top M and U+=cos⁡(ηU)U−sin⁡(ηU)QU_+=\cos(\eta_U)U-\sin(\eta_U)Q,

\qquad (I-U_+U_+^\top)M=\cos(\eta_U)\mathcal T_{\eta_U}(M).$$ Parallel transport preserves the momentum norm. In this exact-sign calculation, ordinary projection retains a $\cos(\eta_U)$ fraction of that norm. Correcting this loss requires a scalar division of the existing projection, with no additional matrix product. The relative difference is $1-\cos(\eta_U)=O(\eta_U^2)$. | Method | Validation loss | | --- | ---: | | PLUM | **3.4892** | | PLUM with naive momentum projection | 3.4956 | | PLUM with approximate polar | 3.4911 | At the small learning rates used here, the close validation losses are consistent with this local agreement. Edelman, Arias, and Smith [7, Sections 3.5–3.6] compare geodesic/transport algorithms with QR/projection alternatives and explain how higher-order differences can leave their local convergence behavior unchanged. The central observation is that matrix sign equalizes the singular values that govern the angular speeds of Grassmann motion. The matrix functions in the geodesic and transport maps then reduce to scalar coefficients. Geodesic plus parallel transport and scalar polar plus projection consequently have the same leading computational cost in this setting. The geodesic and transport refinements are therefore free, requiring no additional matrix decompositions, matrix products, or geometric hyperparameters. This turns operations that usually require costly decompositions into updates suited to GPU pretraining. Finite-NS deviations are addressed in W2. This simplification also applies to other orthogonality-constrained problems with horizontal matrix-sign directions. Their geometric differences grow with the step size, so the distinction can become more consequential in settings that permit larger steps. **(c) Allocation and spectral scaling.** Allocation has a local justification in the product update. At an orthonormal basis, with horizontal $Q_U$ and column-orthonormal sign directions, the two first-order contributions are orthogonal, so $$\lVert aQ_UY^\top+bUQ_Y^\top\rVert_F^2 =a^2\lVert Y\rVert_F^2+b^2\lVert U\rVert_F^2.$$ With $g_i=\langle M_i,Q_i\rangle_F$, maximizing momentum alignment under the stabilized product-step model gives $a:b=\frac{g_U}{\lVert Y\rVert_F^2+\epsilon}:\frac{g_Y}{\lVert U\rVert_F^2+\epsilon}$. PLUM normalizes the larger step to the scheduled learning rate, with zero steps for zero momentum. This is a locally motivated adaptive rule, with a heuristic choice of overall scale. Under finite NS, the norm expression becomes an approximation. Spectron instead applies a common scale based on factor spectral norms. Fig. 3 (Fig.4 after revise) compares PLUM allocation with equal factor steps. We additionally replaced allocation by the shared scale $(1+\lVert U\rVert_2+\lVert Y\rVert_2)^{-1}$ in the same PLUM parameterization, following Spectron's implementation. Learning rates are selected independently using the same grid. | Factor-step rule within PLUM | 60M validation loss | | --- | ---: | | Spectron-style shared spectral scale | 3.4913 | | PLUM allocation | **3.4892** |
Reviewer_KkGsOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes PLUM, a method for low-rank pretraining of LLMs. They reparameterize each weight into an orthonormal subspace factor and a coefficient matrix. The subspace factor U stays on the Stiefel manifold, while the coefficient Y is a normal Euclidean matrix. This constraint breaks the non-orthogonal symmetry between the two low-rank factors. They build a hardware-efficient optimizer around this parameterization. They use Muon’s msign operator to simplify transport on the manifold. This removes the need for QR factorization (except at start). The update is made orthogonal to the current subspace, and Newton-Schulz keeps it orthonormal with just a scalar renormalization.

The method is validated on C4 pretraining and GLUE finetuning. They compare against recent low-rank baselines across several model sizes and show strong performance.

Critical strengths

The product-level control via parameterization is the core strength. They keep U on the Stiefel manifold and leave Y Euclidean. This solves the non-orthogonal symmetry problem because one factor is constrained. It is a clean fix to a real issue in per-factor low-rank updates.

The optimizer is genuinely hardware-efficient. I checked the provided code. The operations are only matrix multiplications. There are no eigen decompositions and no per step QR factorization. So the hardware-efficiency claim holds. The theoretical proof is also correct. The observation that Muon’s matrix-sign update forces all principal directions to rotate by the same angle is interesting and well-motivated.

The empirical results are positive. The method beats recent low-rank pretraining baselines across different model sizes.

Critical weaknesses

The writing is dense in places and this is the main weakness. The method and theory are sound once parsed, but the presentation requires substantial effort. Several early terms are introduced without clear definition (see questions below), which slows the reader in the intro before the method clarifies things.

This is a fixable, presentation-level concern rather than a substantive issue with the contribution.

Questions for authors
  • Around lines 048-049, what is meant by “ideal factor-wise Muon”? What does “ideal” refer to here?
  • What do you mean by “positive rescaling” in the same region?
  • What are Q_B and Q_A? Are these the same as ΔA and ΔB? Please define them explicitly.
  • What is c in that passage?
  • (Suggestion, not required) How does PLUM relate to a CoLA-like parameterization (Liu et al., 2025), where there is an activation between the factors? I am not sure how it connects to the current method, but it would be an interesting extra experiment.
  • (Minor) The Section 5 heading could be more descriptive, e.g., “Efficient Pretraining on C4.”

Liu, Ziyue, et al. “CoLA: Compute-efficient pre-training of LLMs via low-rank activation.” EMNLP 2025.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KkGs

Thank you for the encouraging review and for taking the time to point out the unclear passages. We appreciate both your support and the suggestion to connect PLUM with CoLA.

Several definitions should have appeared earlier. We will introduce the factors and Muon directions before the rescaling example and rewrite that passage to make the argument easier to follow.

“Ideal factor-wise Muon,” positive rescaling, and the notation. You are right that “ideal” was unclear. We meant the exact matrix-sign operation, which satisfies msign⁡(cM)=msign⁡(M)\operatorname{msign}(cM)=\operatorname{msign}(M) for c>0c>0. The word is unnecessary, and we will remove it.

Here QB=msign⁡(MB)Q_B=\operatorname{msign}(M_B) and QA=msign⁡(MA)Q_A=\operatorname{msign}(M_A) are the Muon directions, where MBM_B and MAM_A are the factor momenta. Thus, the increments in Eq. (1) are ΔB=−ηQB\Delta B=-\eta Q_B and ΔA=−ηQA\Delta A=-\eta Q_A. We will define these quantities explicitly at their first use.

“Positive rescaling” refers to replacing (B,A)(B,A) by (cB,c−1A)(cB,c^{-1}A), where c>0c>0 is an arbitrary scalar and the product BABA is unchanged. For correspondingly scaled factor gradients and momenta, the matrix-sign directions remain unchanged, but the first-order weight update becomes

\quad\longmapsto\quad -\eta(c^{-1}Q_BA+cBQ_A).$$ The example shows why unchanged Muon directions need not give the same weight update. The same argument applies when finite NS first divides its input by its norm, since this cancels any positive constant multiplying the input. We will revise those ambiguous expressions. **Connection to CoLA.** We found this suggestion interesting and thought through one possible approach. Unlike linear $BA$, a ReLU layer generally does not allow arbitrary rotations between its factors. It does allow a feature to be multiplied by a positive number and its output coefficient divided by the same number. Features and their output coefficients can also be reordered together. This suggests fixing each input feature's norm and using PLUM to learn the subspace containing these features. For a ReLU version of CoLA (Liu et al., 2025), $h(x)=B\operatorname{ReLU}(Ax)$, one possible approach is to write, for nonzero rows of $A$, $$A=DTU^\top,\qquad U^\top U=I_r,\qquad \lVert T_{i,:}\rVert_2=1,$$ where $D$ is a diagonal matrix with positive entries. Since $\operatorname{ReLU}(Dz)=D\operatorname{ReLU}(z)$, setting $V=BD$ gives $$h(x)=V\operatorname{ReLU}(TU^\top x).$$ Because $U^\top U=I_r$, each row of $TU^\top$ has unit norm. The original row norms of $A$ are absorbed into $V$, preserving the function and fixing the input-side norms that otherwise affect factor-wise updates. PLUM's geodesic formula could then update the input subspace $U$ without a matrix decomposition, while $T$ determines feature directions within it and $V$ weights their contributions to the output. Forming $TU^\top$ before the forward pass retains CoLA's two main matrix multiplications, with additional work to form and optimize the factors. For SiLU, multiplying the input by a constant cannot in general be reproduced by multiplying the output. The entries of $D$ would therefore remain inside the activation and require separate optimization. We think this is an interesting direction to explore. We will add CoLA to Related Work and discuss its connection to PLUM. **Section 5 heading.** We will adopt your suggestion and rename it “Efficient Pretraining on C4”. Thank you for pointing this out. Thank you again for your careful reading and encouraging assessment. We really appreciate your support and the helpful suggestions!
Reviewer_KkGsOpenReview ↗

thanks

Thank you , keeping my already positive assessment.

Reviewer_sp8qOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes PLUM to addresse native low-rank training by separating each weight into an orthonormal basis and a coefficient matrix. It exploits Muon’s update structure to simplify subspace updates and momentum transport, while adaptively balancing the two factor learning rates.

Critical strengths
  • The connection between Muon and low-rank geometry is technically appealing and enables an efficient implementation.
  • PLUM consistently improves perplexity over Spectron across the reported model scales, with a substantial speed advantage in the 350M experiment.
  • The paper provides convergence analysis accounting for finite Newton–Schulz iterations.
Critical weaknesses
  • Comparisons with AdamW-based methods such as SLTrain, GaLore, and LORO mix changes in parameterization and optimizer. Although Spectron and dense Muon provide useful controls, a matched vanilla factor-wise Muon baseline is needed to isolate PLUM’s contribution.

  • The training budgets are too short to establish convincing long-term trends; the largest reported budget is only 13.1B tokens. I would like to see at least one controlled comparison trained to 100B tokens or more, tracking validation loss and orthogonality error throughout training. This would clarify whether the gains persist and whether the approximate geometric updates remain stable.

Questions for authors
  • Please see the weakness part.

  • Beyond the reported RoBERTa/GLUE experiments, would PLUM’s benefits extend to larger-scale LoRA fine-tuning? Could it also be integrated with ReLoRA, particularly its periodic weight merging and factor reinitialization?

7 comments
AuthorsOpenReview ↗

Response to Reviewer sp8q

Thank you for the careful review and constructive suggestions. We have added matched Muon baselines, extended training measurements, and experiments on larger-model fine-tuning and ReLoRA.

W1. Matched vanilla factor-wise Muon

We agree that a matched vanilla factor-wise Muon baseline helps isolate PLUM's contribution. We evaluated this baseline at 60M and 130M, matching the architecture, rank, training tokens, data order, and optimizer for the remaining parameters. Each paired comparison uses the same seed and deterministic settings.

Independent factor updates can make native low-rank pretraining sensitive to initialization and factor scaling. LORO reports unstable training and poor results across several initializations and learning rates [1], while Spectron relates instability to uncontrolled changes in the represented weight [2]. Rescaling the factors as (cB,c−1A)(cB,c^{-1}A), c>0c>0, leaves W=BAW=BA unchanged, but factor-wise Muon can produce different first-order weight updates across these representations. Orthogonalizing each factor direction alone therefore does not control the induced product update. These difficulties motivated our original focus on methods with geometric or product-level stabilization.

PLUM makes the geodesic and transport refinements free relative to projected updates: the matrix-sign structure reduces them to scalar coefficients, with no additional matrix products or decompositions. This removes the repeated decomposition cost that can make conventional manifold updates expensive at every GPU pretraining step.

Glentis et al. [3] suggest that zero-factor initialization may also explain some earlier poor results with ordinary native low-rank pretraining. With both factors randomly initialized, they obtain much stronger plain low-rank results, although a gap to stronger methods remains. Following this observation, we tested both the standard LoRA initialization, with one random factor and one zero factor, and initialization with both factors random. We expanded vanilla Muon's learning-rate grid to 10−3,2×10−3,4×10−3,6×10−3,8×10−3,10−2,2×10−2\\{10^{-3},2\times10^{-3},4\times10^{-3},6\times10^{-3},8\times10^{-3},10^{-2},2\times10^{-2}\\}.

Method60M validation loss130M validation loss
Vanilla Muon (LoRA init.)5.45255.0060
Vanilla Muon (both factors random)3.51653.2011
PLUM3.48923.1751

Random initialization of both factors substantially reduces vanilla Muon's loss, while PLUM remains better at both sizes. This sensitivity concerns native pretraining, where the factors represent the weight itself. A zero-initialized LoRA adapter added to a pretrained weight has a different starting point. We agree that the matched Muon control makes PLUM's contribution clearer.

W2. Orthogonality stability over longer training

As a rough reference for a larger study, Chinchilla's compute-optimal scaling pairs 100B tokens with a model of about 5B parameters [4]. Within the resources available during rebuttal, we configured the 130M C4 runs for a 100B-token budget and report the results at 100,000 updates, five times the original 130M update budget.

PLUM and vanilla factor-wise Muon share the data stream and LLaMA-2 tokenizer, with rank 256, sequence length 256, and batches of 512 documents. Vanilla Muon uses both factors randomly initialized and the best learning rate from the 130M grid in W1. PLUM retains the original learning rate of 5×10−25\times10^{-2}. Both use linear warmup followed by cosine decay.

MethodValidation loss at 100K updates
Vanilla Muon (both factors random)3.0636
PLUM3.0180

For each PLUM basis, we measure ∥Uℓ⊤Uℓ−Ir∥F\lVert U_\ell^\top U_\ell-I_r\rVert_F and report the mean and maximum across constrained matrices.

UpdatesMean orthogonality errorMaximum orthogonality error
00.0024430.002776
10,0000.0024520.002924
20,0000.0024600.002893
30,0000.0024690.002918
40,0000.0024730.002944
50,0000.0024760.002945
60,0000.0024810.002965
70,0000.0024830.002990
80,0000.0024830.002993
90,0000.0024840.002985
100,0000.0024820.002957

The mean error changes by only 3.9×10−53.9\times10^{-5} between the first and last checkpoints, and the maximum remains below 0.003. By 100K updates, the learning rate has reached its peak and begun cosine decay. We are continuing the comparison over more training steps and plan to supplement these measurements later.

The small nonzero residual is consistent with rounding in low-precision parameter storage, an effect also examined in ProLand [5]. The largest measured residual, δ=0.002993\delta=0.002993, gives κ(U)≤(1+δ)/(1−δ)<1.003\kappa(U)\leq\sqrt{(1+\delta)/(1-\delta)}<1.003. Moreover, (1−δ)∥ΔY∥F2≤∥UΔY⊤∥F2≤(1+δ)∥ΔY∥F2(1-\delta)\lVert\Delta Y\rVert_F^2\leq\lVert U\Delta Y^\top\rVert_F^2\leq(1+\delta)\lVert\Delta Y\rVert_F^2, so the basis maps coefficient changes to weight changes with almost no distortion in norm. LORO also highlights the importance of factor conditioning for stable training [1]. Its experiments associate ill-conditioned weights with loss spikes, and its analysis explains how small factor gradients can coexist with a large gradient on the low-rank manifold.

PLUM maintains this basis conditioning with free geodesic and transport refinements of the projected updates. It introduces no additional stabilization hyperparameters, such as an exact-update interval, a penalty coefficient, or a spectral-norm power-iteration count.

Q1. Larger-scale LoRA fine-tuning and ReLoRA

We evaluated Llama 3.2 3B on commonsense reasoning with rank-32 adapters.

MethodBoolQPIQASIQAHellaSwagWinoGrandeARC-CARC-EOBQAAvg.
Spectron70.6784.4478.1089.4578.1469.8883.8077.0078.94
PLUM70.4684.2877.3391.3279.0171.5084.1379.4079.68

Both methods share the data, target modules, adapter scale, training budget, and learning-rate grid 5×10−5,5×10−4,5×10−3,5×10−2\\{5\times10^{-5},5\times10^{-4},5\times10^{-3},5\times10^{-2}\\}, using the same seed and deterministic settings. Spectron uses standard LoRA initialization, while PLUM uses random orthonormal UU and Y=0Y=0. Both preserve the pretrained model initially. Entries are percentage accuracies, and Avg. is their unweighted mean.

We also evaluated PLUM within ReLoRA at 60M and 130M, with plain PLUM as the control. Paired runs use the same seed and deterministic settings.

Method60M validation loss130M validation loss
PLUM3.48923.1751
ReLoRA + PLUM3.52763.2400

We attach adapters to all attention and MLP linear layers, using ranks 128/256 and scale α/r=1\alpha/r=1 at 60M/130M. Merging every 2,000/4,000 updates gives four restarts per run, with cosine cycles and 100/200 re-warmup steps. Each restart merges UY⊤UY^\top into WbaseW_{\mathrm{base}}, reinitializes UU orthonormally, and sets Y=0Y=0, preserving the represented weight. The factor momenta, step counter, and transport state are reset. Base linear weights are frozen between merges, while the remaining trainable parameters use AdamW.

Between restarts, PLUM applies unchanged. With G=∇WLG=\nabla_W\mathcal L, the restart gives ∇UL=GY=0\nabla_U\mathcal L=GY=0, while ∇YL=G⊤U\nabla_Y\mathcal L=G^\top U can be nonzero, so the coefficients begin learning before the basis moves.

The combination is compatible with merging and factor reinitialization, although it increases validation loss in these runs. Plain PLUM already trains the low-rank model directly, without choosing a merge interval or restart schedule. The suggested combination remains useful to consider when accumulating successive low-rank updates into a higher-rank weight is desired.

Thank you again for the helpful suggestions. We will take these comments into account in the revision.

References

[1] Mo et al. Parameter and Memory Efficient Pretraining via Low-rank Riemannian Optimization. ICLR, 2025.

[2] Janson et al. Stabilizing Native Low-Rank LLM Pretraining. ICML, 2026.

[3] Glentis et al. Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking. arXiv:2505.22922, 2025.

[4] Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS, 2022.

[5] Zhou et al. ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning. arXiv:2611.00236, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.