PLUM: Lightweight Low-Rank Updates for Parameter- and Memory-Efficient Training

Training large models is constrained by memory for parameters, gradients, and optimizer states. Low-rank parameterization reduces these costs, but factor-wise optimization is not invariant to the non-unique factorization: equivalent representations of the same weight can induce different weight updates. We introduce PLUM (**P**arameter-efficient **L**ow-rank **U**pdates with **M**uon), a lightweight optimizer that represents each weight by an orthonormal subspace basis and a coefficient matrix, separating subspace motion from coefficient changes with a simple implementation. The unit nonzero singular values of Muon's msign update make the Grassmann geodesic and momentum transport particularly simple, reducing both to inexpensive matrix operations without matrix exponentials or repeated QR/polar retractions. A fixed Newton-Schulz approximation further yields a fully GEMM-based implementation with one momentum state per factor and no additional factor- or geometry-specific hyperparameters to tune. We establish an $O(T^{-1/4})$ expected first-order stationarity rate and an $O(T^{-1/2})$ bound on the expected maximum orthogonality error for fixed finite Newton-Schulz iterations under standard assumptions. Experiments on C4 pretraining and GLUE fine-tuning up to models with over 1B parameters show lower pretraining perplexity than prior low-rank methods together with a lightweight, GPU-friendly implementation.