- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
McLoRA replaces LoRA’s additive correction, Wx + BAx, with the elementwise product (Wx) ⊙ (1 + BAx), making the low-rank adapter gradients explicitly depend on the frozen backbone output. The authors argue that this directs updates toward dimensions that are more active for the current task and thereby reduces interference with earlier tasks. They provide a gradient derivation, activation interventions, and analyses of update concentration and cross-task overlap. Experiments across several backbones and sequential-learning streams generally show higher final performance and less forgetting than standard LoRA. The paper also studies a bounded variant, extensions to other LoRA methods, and single-task performance.
Critical strengths
-
Simple and well-specified method. McLoRA changes the composition rule without increasing LoRA’s trainable-parameter budget. The gradient derivation clearly identifies the effect of the backbone output on adapter updates.
-
Substantial empirical results. In Table 2, McLoRA improves final average performance in 27 of 28 backbone–sequence settings and reduces forgetting in 27 of 28. Longer task streams and settings without explicit task boundaries extend the evaluation.
-
Useful analyses beyond final scores. The activation interventions and update statistics help characterize the method’s behavior. Experiments with the bounded variant and other LoRA methods provide additional context.
Critical weaknesses
-
The central anti-forgetting motivation remains insufficiently justified. The paper provides evidence that dimensions with large backbone outputs influence current-task predictions, but does not establish why directing more updates to those dimensions should preserve previous tasks. Successive tasks may rely on the same strongly activated dimensions, in which case concentrating updates there could still create interference. The reported overlap statistics are supportive observations, but do not resolve when this update-allocation principle should work or fail.
-
The specific choice of multiplication is not adequately isolated. Appendix C derives the multiplicative form after imposing the desired Wx factor in the gradient; it does not independently justify that factor as a continual-learning principle. The motivation emphasizes activation magnitude, whereas the method uses the signed, unnormalized Wx. Multiplication also changes the forward function. Without controlled alternatives, it remains unclear which of these changes accounts for the performance gains.
-
Comparative and robustness evidence is incomplete. The main shared-adapter experiments do not provide matched comparisons with the specialized continual-learning methods discussed in the related work. The S1–S7 experiments use task-specific classification heads, although the longer shared-head streams partly address this limitation. The result tables also lack run-to-run variability, making close comparisons difficult to assess.
Questions for authors
-
How does McLoRA perform when successive tasks rely on substantially overlapping high-activation dimensions? Can the authors measure whether new-task updates affect dimensions important to earlier tasks and directly relate this to changes in old-task loss?
-
Under matched parameter counts and update scales, how do normalized or magnitude-based activation gates, shuffled activation signals, and activation-weighted additive updates compare with the proposed multiplication?
-
Can the authors report variability across random seeds and compare against specialized shared-adapter continual-learning methods under the same task stream and classification-head protocol?
6 comments
Response 2 (Weakness 2; Question 2): Design rationale and controlled alternatives
We thank the reviewer for this comment. We address the points one by one:
- Basis of the Wx factor. Figure 1 and Appendix A show that dimensions with larger backbone output magnitudes tend to exhibit higher sensitivity to the current task; we therefore want the backbone output to enter the gradient and guide the allocation of updates across dimensions. Appendix C proves that, among element-wise compositions, multiplication is the simplest mechanism that makes the gradient contain this factor. Why this factor benefits continual learning is explained by the first-order forgetting analysis in Response 1: it multiplies the interference between the new and old tasks in each dimension by the product of their backbone outputs, so the interference is suppressed as long as one side is weakly activated.
- Signed and unnormalized. Keeping the sign is required by multiplicative modulation itself: if were used instead, positive and negative responses would be scaled in opposite directions by the same BAx; in Table R12 below, the magnitude gate is also clearly worse than McLoRA. Per-token RMS normalization only changes the overall scale of the gate at each token, not the relative allocation across dimensions within the same token; in Table R12, the RMS-normalized gate is close to McLoRA.
Controlled alternatives: McLoRA can be written as . All variants use the same forward form and only replace the gate ; they have the same number of parameters as McLoRA, and the update scale is matched per layer. The experiments follow the protocol of Table 2 of the paper (S2, RoBERTa-large).
Table R12 | Controlled alternatives
| Method | Gate | Final Avg ↑ | Forgetting ↓ |
|---|---|---|---|
| LoRA | 1 | 73.1 | 22.4 |
| McLoRA | 81.8 | 11.8 | |
| Shuffled activation | with randomly permuted dimensions | 73.6 | 21.8 |
| Magnitude gate | 76.7 | 18.8 | |
| Static activation weighting | computed at the start of each task | 73.4 | 21.9 |
| RMS-normalized gate | with per-token RMS normalization | 81.6 | 12.1 |
Conclusion: (1) Shuffled activation is close to LoRA: when the distribution of gate values is kept and only the correspondence of dimensions is shuffled, the gain almost disappears, indicating that the gain relies on modulating each dimension by its own backbone output. (2) Static activation weighting is also close to LoRA: a fixed per-dimension weighting cannot replace an input-dependent gate. (3) The magnitude gate is clearly worse than McLoRA, indicating that the sign matters. (4) The RMS-normalized gate is close to McLoRA, indicating that the gain comes from the relative allocation across dimensions within the same token, rather than from the overall scale of the gate.
Revision: The revised version will add the first-order forgetting analysis of Response 1 to the main text as the continual-learning basis of this factor, and include the controlled alternatives in Table R12.
Response 3 (Weakness 3; Question 3): Specialized continual-learning baselines, classification heads, and multiple random seeds
We thank the reviewer for the suggestion. The LoRA-based continual-learning methods discussed in Section 2 of the paper do not match our NLP setting with a fixed budget and a single shared LoRA: O-LoRA, N-LoRA, AM-LoRA, and LB-CL introduce new LoRA modules or low-rank parameters for each task (these four have been compared under the multi-adapter protocol in Table 6 of the paper); BranchLoRA targets multimodal large models and adds a new router for each task; SD-LoRA, CL-LoRA, and C-LoRA target class-incremental learning in vision; Share stores a set of coefficients for each task. Therefore, we chose EWC and LwF, which can be applied directly to the same shared adapter without adding trainable parameters or replaying data, compared them under the same task stream and classification-head protocol as the main experiments, and report the mean ± standard deviation over 3 random seeds (S2, RoBERTa-large; λ and α selected on the validation set).
Table R13 | Specialized continual-learning baselines and multi-seed results
| Method | Final Avg ↑ | Forgetting ↓ |
|---|---|---|
| LoRA | 76.7 ± 3.6 | 18.2 ± 4.2 |
| McLoRA | 83.2 ± 1.7 | 9.3 ± 2.8 |
| LoRA + EWC | 84.1 ± 2.4 | 7.3 ± 2.2 |
| LoRA + LwF | 83.8 ± 2.9 | 7.7 ± 2.7 |
| McLoRA + EWC | 89.4 ± 1.4 | 1.1 ± 0.3 |
| McLoRA + LwF | 88.3 ± 2.1 | 2.3 ± 0.7 |
Conclusion: (1) The Final Avg of McLoRA is on average 6.5 points higher than that of LoRA, and its forgetting is on average 8.9 points lower, with smaller variation across seeds (standard deviation 1.7 vs. 3.6). (2) Used alone, McLoRA is comparable to LoRA + EWC and LoRA + LwF, without requiring an additional regularization term, Fisher estimation, or a teacher model; the two can be combined, and McLoRA + EWC performs best. (3) Regarding classification heads: S1–S7 follow the setting of related work and use task-specific classification heads; the long sequences LS1–LS4 and their boundary-free variants all use the shared language-modeling head, and McLoRA outperforms LoRA in all these settings (Table 3 of the paper).
Revision: The revised version will include the above multi-seed results and baseline comparisons.
Response 1 (Weakness 1; Question 1): The anti-forgetting mechanism, the overlapping case, and old-task loss
We thank the reviewer for this comment. We agree that concentrating updates on the high-activation dimensions of the current task does not by itself guarantee the protection of old tasks. We provide a conditional account of the mechanism and conduct experiments for both parts of the question.
Mechanism: (1) Figure 1 and Appendix A of the paper show that dimensions with larger backbone output magnitudes tend to exhibit higher sensitivity to the current task. (2) The high-activation dimensions of different tasks partially overlap but are not identical. (3) McLoRA modulates according to the backbone output. Consider one gradient-descent step on B with a new-task input x (learning rate η); the first-order change of the old-task loss on an old-task input x′ is:
where δ=∂L/∂h and u=Wx is the backbone output (same notation as Appendix C). Of the additional factors, ui(x) comes from the new-task training side, i.e., the learning signal is scaled by the backbone output of the new task; ui(x′) comes from the old-task evaluation side, i.e., the effect of the adapter change on the old-task output is scaled by the backbone output of the old task. (4) Therefore, as long as either the new or the old task is weakly activated in a dimension, the interference in that dimension is reduced accordingly; when both are strongly activated, it is not suppressed, and forgetting still occurs if the update directions conflict. (5) This predicts that the more the high-activation dimensions overlap, the smaller McLoRA's advantage. The experiments below test this prediction.
The overlapping case: On 24 "old task → new task" directions formed from 7 tasks (RoBERTa-large), we compute the overlap O between the new and old tasks from the top 20% dimensions in backbone output of each task on the frozen backbone (Jaccard, averaged over modules; chance level ≈ 0.11), and divide the directions into three groups by O:
Table R10 | Forgetting grouped by the overlap of high-activation dimensions (LoRA / McLoRA)
McLoRA reduces forgetting overall, but its advantage shrinks as the overlap increases, and the two are close in the highest-overlap group, consistent with the above analysis.
Effect on dimensions important to old tasks and on old-task loss: On the same training runs, we compute the first-order increase of the old-task loss, E[gold⊤Δhold], from the actual output change before and after training on the new task; the first-order prediction accounts for about 85% of the measured loss increase. We divide the dimensions into four groups according to whether they are highly activated for the new and the old task:
Table R11 | Dimension-wise decomposition of the first-order increase of the old-task loss
The reduction in interference occurs mainly in dimensions where at least one side is weakly activated; in dimensions highly activated for both tasks, it is reduced by only 11%, consistent with the above analysis. We also directly examine the dimensions important to the old task: selecting the old-task-important dimensions by the sensitivity of the old-task loss to each dimension's output (the same set for both methods), the increase in old-task loss caused by McLoRA on these dimensions is 26% smaller than that caused by LoRA; on the part of these dimensions not needed by the new task, it is 63% smaller.
Revision: The revised version will rephrase the mechanism in the above conditional form and include the above analyses.