acceptodds
Under review as a conference paper at ICLR 2027

Approximate Muon with Low-Rank Adapters

Abstract

The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.

Primary area:
optimization
Keywords:
muon, lora, peft, optimization, finetuning, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sJdwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes sMuon, a low-rank optimizer that adapts Muon's full-layer update geometry to LoRA. Rather than orthogonalizing the two factor updates independently, sMuon projects a Muon-inspired update onto the feasible tangent space and exploits low-rank structure to perform the required computations through small matrices. The method also incorporates momentum transport and weight decay. The paper shows experiments results across 4 language models to show competitive performance, with promising efficiency and convergence results.

Critical strengths
  1. By reducing the main orthogonalization computations to a small core matrix, the proposed sMuon avoids materializing full-layer gradients and achieves computational complexity that scales favorably with the adapter rank.

  2. The ReLoRA experiments demonstrate promising optimization efficiency. sMuon achieves a validation loss comparable to Riemannion and outperforming AdamW, while substantially reducing optimizer overhead.

  3. The evaluation covers various models and tasks, including commonsense reasoning, knowledge benchmarks, code generation, and continued pretraining.

Critical weaknesses
  1. Some theoretical properties derived under exact arithmetic do not directly carry over to the implemented algorithm. In particular, the regularized pseudo-inverses used for numerical stability do not generally preserve exact reparameterization invariance, and the resulting approximate projections need not retain the idempotence assumed in the analysis. The full-rank assumptions also need more clarification in the presence of zero initialization or degenerate gradients. The paper should distinguish the properties of the idealized formulation from those of the implemented optimizer and discuss the effect of numerical regularization.

  2. The empirical advantages are not sufficiently consistent or statistically validated. Although the ReLoRA results are encouraging, the SFT experiments show substantial regressions on several model-task combinations, including code generation on Qwen and DeepSeek and ARC-C on Llama.

  3. The experiments do not fully isolate the contribution of the proposed full-layer update geometry. The proposed sMuon and its competitors differ in factor initialization, momentum handling, weight decay, and update scaling, making it difficult to attribute performance differences specifically to the projected Muon direction.

Questions for authors
  1. How sensitive is sMuon to numerical regularization and rank deficiency? To what extent do the reparameterization invariance and projection properties established in the analysis hold for the implemented optimizer?

  2. How stable are the reported improvements across random seeds and learning rates, particularly for the ReLoRA experiment and the SFT settings showing substantial regressions?

Reviewer_YfxKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose sMuon, an approximation of the Muon optimizer specifically designed for fine-tuning of LoRA modules via linearization and least-squares. Additionally, the authors provide an efficient implementation using only matmul operations. The proposed method is benchmarked on a variety of NLP tasks and datasets.

Critical strengths
  • The paper is well written, informative and clear to follow.
  • The paper introduces a novel Muon optimizer approximation specifically designed for LoRA fine-tuning.
Critical weaknesses
  • Although the authors state that results are model and eval dependent, the empirical results appear inconclusive when it comes to discussing performance on the benchmarked datasets and models, especially when considering Tables 1 and 2. There seems to be no clear overall better method.
  • The claim “Muon-based LoRA is strongest on the Muon-pretrained model.” is not sufficiently backed up by empirical evidence, as only 1 or 2 muon pretrained models are considered (Moonshot in Tables 1 and 2, and the transformer in Table 3?).
  • The related work discussion is overly short, and the additional information in the appendix provides mostly an overview, but does not limitations in sufficient detail. Likewise, it is not clear how the models and benchmarks in the experiments are chosen, and whether they are representative of some class of problems. Also for settings in experiments, e.g. initializations, the impact on results should be explained further (e.g. bf16 for weights and optimizer states, fp32 for step computation). More generally speaking, the main paper should offer more intuition on the reasoning and justification of the empirical evaluation and instead move the technical details to the appendix.
Questions for authors

Can you back up the main claim “Muon-based LoRA is strongest on the Muon-pretrained model.” by showing more Muon pre-trained models settings? And/or: can you characterise when sMuon is helpful, and when not?

Reviewer_qCdhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper derives sMuon, a Muon-style optimizer for LoRA adapters. Starting from spectral-norm-regularized steepest descent on ΔW = BA, the authors linearize the factor update and solve a least-squares problem to approximate the orthogonalized low-rank update. The resulting step has a closed form, is first-order invariant to the adapter parameterization, and admits a matmul-only implementation using only r×r operations. The paper adds split weight decay and momentum transport, then evaluates on SFT across four base models and on ReLoRA pretraining.

Critical strengths

1.Principled and efficient derivation. The closed-form low-rank Muon step and matmul-only implementation are the main contribution. This is consequential because it makes spectral optimization for LoRA practical at larger rank without Riemannian retraction/SVD overhead. 2.Good efficiency and ReLoRA evidence. The step-time comparison and ReLoRA result support the claim that sMuon is competitive with Riemannian at lower cost.

Critical weaknesses

1.First-order approximation lacks convergence guarantees. The method drops second-order terms and approximates the ideal full-rank Muon step. This matters because the regime where the approximation is accurate is not characterized theoretically. 2.Empirical gains are mixed and partly single-seed. sMuon is strong on the Muon-pretrained Moonlight model, but not uniformly best elsewhere; several baselines are self-implemented. This limits the strength of the “outperforms other Muon-LoRA methods” claim.

Questions for authors

1.Can you bound or empirically characterize the error introduced by the first-order linearization as a function of adapter rank and update scale? 2.Are the SFT results robust across multiple seeds, and are LoRA-Muon/Riemannian baselines tuned with equal effort? 3.How sensitive is sMuon to the AdamW-RMS learning-rate scaling heuristic and to the jitter/initialization choices?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.