- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MuLoRA for continual learning, focusing on the problem that LoRA updates may concentrate their energy in only a few singular modes. The method selects adaptation directions using current-task and historical feature statistics, then applies Muon-style orthogonalization to balance each update. The paper provides theoretical results on subspace selection and cumulative spectral behavior, and evaluates the method on five benchmarks across eight incremental settings.
Critical strengths
-
The method addresses both the choice of adaptation directions and the use of the available low-rank capacity, rather than focusing only on reducing interference with previous tasks.
-
Fixing the orthonormal basis gives an exact relationship between the singular values of factor updates and those of the corresponding weight updates, making the motivation for spectral balancing mathematically clear.
-
The evaluation covers several benchmarks and incremental settings. The ablation results also show that both RRS and SSB contribute to the final performance.
Critical weaknesses
The experiments do not fully isolate the effect of spectral balancing. Muon changes the optimization dynamics as well as the update spectrum, but there is no norm-matched control to distinguish these effects. The mechanism comparison is also limited to one benchmark setting, making it difficult to determine how consistently the proposed components explain the gains across tasks.
Questions for authors
-
Did you measure the cumulative anisotropy and orthogonalization-error conditions during training? If so, how often were the theoretical conditions satisfied across tasks and random seeds?
-
Can you compare MuLoRA with a norm-matched optimization baseline to clarify whether the gains come from spectral balancing rather than differences in update magnitude?
-
Why is the detailed mechanism comparison limited to 20-session ImageNet-A? Do the same trends hold on other benchmarks or incremental settings?