McLoRA: Multiplicative Composition for LoRA-Based Continual Learning

Low-Rank Adaptation (LoRA) suffers from catastrophic forgetting in continual learning. Existing methods for addressing this problem still retain the standard additive composition, in which the LoRA update does not receive direct guidance from the backbone output. This paper proposes McLoRA (Multiplicative Composition LoRA), which replaces the standard additive composition with element-wise multiplicative composition between the low-rank branch and the frozen backbone output. This design makes the learning gradient of each output dimension in the low-rank branch explicitly correlated with the output magnitude of the corresponding backbone dimension, thereby encouraging updates to focus on dimensions that respond more strongly to the current task while suppressing updates on weakly activated dimensions. Empirically, McLoRA produces more concentrated updates with less cross-task interference. Experiments on multiple large language models and sequential learning benchmarks show that, under the same parameter budget and computational complexity as LoRA, McLoRA significantly alleviates catastrophic forgetting when a single shared LoRA module is trained sequentially, while maintaining comparable single-task performance.