acceptodds
Desk rejected at ICLR 2027

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

25 views

Abstract

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify spectral plasticity collapse: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose MuLoRA, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within the selected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly to the induced weight update. We establish a max–min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YCBXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MuLoRA for continual learning, focusing on the problem that LoRA updates may concentrate their energy in only a few singular modes. The method selects adaptation directions using current-task and historical feature statistics, then applies Muon-style orthogonalization to balance each update. The paper provides theoretical results on subspace selection and cumulative spectral behavior, and evaluates the method on five benchmarks across eight incremental settings.

Critical strengths
  1. The method addresses both the choice of adaptation directions and the use of the available low-rank capacity, rather than focusing only on reducing interference with previous tasks.

  2. Fixing the orthonormal basis gives an exact relationship between the singular values of factor updates and those of the corresponding weight updates, making the motivation for spectral balancing mathematically clear.

  3. The evaluation covers several benchmarks and incremental settings. The ablation results also show that both RRS and SSB contribute to the final performance.

Critical weaknesses

The experiments do not fully isolate the effect of spectral balancing. Muon changes the optimization dynamics as well as the update spectrum, but there is no norm-matched control to distinguish these effects. The mechanism comparison is also limited to one benchmark setting, making it difficult to determine how consistently the proposed components explain the gains across tasks.

Questions for authors
  1. Did you measure the cumulative anisotropy and orthogonalization-error conditions during training? If so, how often were the theoretical conditions satisfied across tasks and random seeds?

  2. Can you compare MuLoRA with a norm-matched optimization baseline to clarify whether the gains come from spectral balancing rather than differences in update magnitude?

  3. Why is the detailed mechanism comparison limited to 20-session ImageNet-A? Do the same trends hold on other benchmarks or incremental settings?

Reviewer_Qmk5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MuLoRA combines history-aware subspace selection (RRS) with Muon-based spectral balancing (SSB) to improve low-rank continual adaptation. Experiments across five benchmarks show competitive performance, supported by theoretical characterizations of subspace optimality and cumulative spectral balance.

Critical strengths
  1. Clear motivation for addressing spectral concentration beyond nominal LoRA rank.
  2. Consistent empirical improvements across multiple class-incremental settings, supported by component and mechanism ablations.
  3. Mathematically sound spectral preservation through a fixed row-orthonormal LoRA factor.
Critical weaknesses
  1. RRS builds on generalized eigenvalue-based subspace selection, while SSB adopts Muon's existing orthogonalization mechanism. Novelty relative to EBLoRA and other subspace-based approaches remains insufficiently distinguished.
  2. Theorem 4.2 and Corollary 4.3 require bounded cross-step anisotropy, without empirical verification of the assumed conditions. Neither result directly guarantees reduced forgetting or stability across tasks with changing adaptation subspaces.
  3. Improved spectral statistics correlate with higher accuracy, but existing ablations cannot isolate spectral balancing from changes in update magnitude and optimization dynamics. Norm-matched optimizer controls are needed to establish the claimed mechanism.
Questions for authors
  1. How frequently do actual optimization trajectories satisfy the anisotropy conditions in Theorem 4.2 and Corollary 4.3?
  2. Can norm-matched optimizer comparisons establish whether spectral balancing, rather than altered update scaling, drives the improvements?
  3. How much additional computation and memory does RRS require relative to competing methods?
Reviewer_nYqkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MuLoRA, which combines history-aware subspace selection with Muon-based spectral balancing for continual learning. A fixed orthonormal LoRA basis ensures that factor updates and weight updates share the same nonzero singular values. Experiments show competitive performance across five benchmarks.

Critical strengths
  1. The paper clearly distinguishes nominal adapter rank from effective spectral utilization, providing a useful perspective on continual adaptation.
  2. The fixed orthonormal parameterization offers an exact connection between factor-step and weight-step spectra.
  3. Experiments are extensive, with the highest mean performance in 15 of 16 reported metrics. Component ablations support combining subspace selection and spectral balancing.
Critical weaknesses
  1. The causal role of spectral balancing remains unclear. Higher effective rank accompanies better accuracy, but this does not establish that spectral concentration causes reduced plasticity. A norm-matched momentum baseline would help separate spectral reshaping from update-scale effects.

  2. The cumulative guarantees have limited practical validation. The bounds require controlled cross-step anisotropy, but the paper does not report whether these conditions hold during training. As acknowledged in the appendix, they apply within a task and do not directly guarantee reduced cross-task forgetting.

  3. Historical response is an imperfect proxy for task importance. Stored feature statistics become stale as the backbone changes, and unnormalized statistics depend on sample and token counts. Sensitivity to these choices and the regularization parameter ϵ\epsilon should be evaluated.

  4. Efficiency and reproducibility need stronger evidence. Historical covariance matrices and subspace construction introduce additional storage and computation. End-to-end runtime and peak memory are missing, while key settings such as polynomial coefficients, iteration count, and momentum configuration should be specified.

  5. Some improvements are small. For example, the final-accuracy gain over SplitLoRA on 20-session ImageNet-R is only 0.11 percentage points. Paired comparisons across more seeds would help establish the reliability of these gains.

Questions for authors
  1. How often do the cumulative spectral conditions hold in actual training?
  2. Does MuLoRA outperform a norm-matched momentum baseline?
  3. How sensitive is subspace selection to stale statistics, normalization, and ϵ\epsilon?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.