- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on selecting a fixed subset of LoRA parameters for continual learning. It asks whether gradient attribution is worth its cost compared with random selection. Specifically, standard LoRA initialization gives A no gradient at the start. Gradient-based selection therefore picks only from B, while an unrestricted random mask draws from both A and B. This changes both the selection rule and the candidate pool. To isolate the selection rule, the authors use the same 10% budget within B for both methods. Experiments conducted on Qwen2.5-3B across five sequential tasks and seven seeds, and on Phi-2 with four seeds, show that random selection consistently yields lower loss-based backward transfer.
Critical strengths
- The paper provides a controlled comparison by restricting gradient-based and random selection to the same parameter pool and 10% update budget, allowing the selection strategies to be compared under matched conditions.
Critical weaknesses
1.The practical value of the study remains questionable. The paper fixes most LoRA parameters from initialization and uses the same mask across all tasks. While this may reduce forgetting, it may also limit the model's ability to learn new tasks, raising concerns about its usefulness in practical continual learning.
2.The evaluation does not adequately assess overall continual learning performance. Both loss-based and accuracy-based Backward Transfer (BWT) measure changes in previous-task performance, so a model that learns poorly may still exhibit low forgetting. The authors should consider reporting common metrics such as final average performance and average incremental performance [1] to better assess the balance between learning and retention.
[1]Towards Lifelong Learning of Large Language Models: A Survey. ACM Computing Surveys, 2025.
Questions for authors
-
What practical contribution does the proposed random masking strategy make to continual learning? Since the same randomly selected parameters remain fixed across all tasks, how does this design support learning new tasks while retaining previously acquired knowledge?
-
Could the authors compare random and gradient-based parameter selection starting from the same checkpoint after the first task? Applying the two selection strategies to subsequent tasks would help clarify whether random masking remains advantageous once the model has acquired task-specific knowledge.
-
The experimental results raise concerns about robustness. Table 19 shows that the advantage of Random-B is largely concentrated on the Medical task, suggesting that the findings may be sensitive to specific task characteristics.
-
Although Random-B consistently outperforms Grad-B in loss-based BWT, this advantage is not confirmed by the accuracy-based results. As shown in Figure 4, the mean Accuracy-BWT difference is only 0.94 percentage points, with a 95% confidence interval that includes zero. This raises concerns about whether the reported reduction in loss-based forgetting translates into meaningful improvements in task performance.
2 comments
Official Comment
Thank you for the response. My two main concerns remain unresolved.
- The authors explain why they select the mask at initialization. My concern is about the practical value of keeping the same mask fixed across all tasks. As the authors report, Random-B achieves higher loss than Grad-B immediately after learning each new task in all 35 paired comparisons. Reducing forgetting has limited practical value for continual learning if it comes at the cost of weaker learning on new tasks.
- As the authors acknowledge, lower loss-based forgetting does not imply better task performance. The response provides no additional results on final average performance or average incremental performance, so it remains difficult to assess the effectiveness of the method in continual learning. For these reasons, I will keep my original rating.
We thank the reviewer for raising the distinction between reduced forgetting and successful continual learning. We agree that the value of a static mask must be judged by both retention and acquisition, and that our evidence does not establish a generally superior continual-learning algorithm. We clarify the limits of the evidence.
Q1. Practical value and ability to learn new tasks.
Our contribution is a controlled negative result and evaluation protocol, not a new adaptive CL algorithm. With standard LoRA initialized at B = 0, initialization-time attribution assigns zero gradient to A. Unrestricted A+B random sampling therefore confounds the selection rule with the pool. By holding both the B-only pool and the 1,651,508-coordinate budget fixed, we test whether paying for gradient ranking improves retention relative to uniform selection. In this setting Random-B lowers loss-based BWT from 0.449 to 0.294 across seven paired Qwen2.5-3B seeds, while avoiding a separate 64-sample attribution pass.
We acknowledge the stability–plasticity cost of fixing one sparse mask across all tasks. Indeed, Grad-B obtains lower post-acquisition (diagonal) loss in all 35 seed × task comparisons (Appendix R, Table 20). Random-B therefore does not learn each new task better, and lower BWT alone would be insufficient evidence of practical superiority. However, the effect is not explained solely by a weaker starting point: averaging final held-out loss over the four prior tasks, Random-B is lower in six of seven seeds (Grad-B minus Random-B = +0.0813, paired p = 0.023; Appendix R, Table 21). This endpoint check does not establish superior task accuracy or the full plasticity–retention trade-off. We will emphasize that distinction.
Q2. Shared checkpoint after the first task.
We agree this is an important complementary experiment, but it answers a different question. Our mask is chosen before task 1 to isolate the zero-initialization confound. After task 1, B will generally be nonzero and ∂L/∂A need no longer vanish; the motivating confound is no longer guaranteed. A clean extension would first train an identical task-1 checkpoint, then construct equal-budget gradient and random masks from that checkpoint, train both on tasks 2–5 with identical data and seeds, and compare acquisition, final performance, and forgetting. We have not run this experiment, and do not claim that our initialization-time ordering extends to post-task adaptation. We will identify it explicitly as future work.
Q3. Medical-task concentration and robustness.
We agree: Medical contributes approximately 79.9% of the aggregate Qwen2.5-3B gap (Appendix Q, Table 19). The effect is not uniform across domains. Still, the per-task mean Random-B minus Grad-B BWT is negative on Code (−0.046), Medical (−0.496), and Math (−0.076); Legal is approximately neutral (−0.003). The density-matched Phi-2 replication preserves the aggregate direction in four of four seeds (mean difference −0.1152), although Medical still accounts for 67.2% of that gap. The medical concentration is a genuine limitation; the single-seed 7B probe also slightly favors gradient (0.441 vs. 0.447). We do not establish robustness to balanced task mixtures or multiple task orders; the existing reversed-order probe is only single-seed and shows that task order can substantially alter absolute BWT. We will further foreground this limitation.
Q4. Loss-based versus accuracy-based evidence.
We agree that lower loss-BWT does not demonstrate better task success. On Medical and Legal, accuracy-BWT differs by only +0.94 percentage points (Grad-B minus Random-B), with a 95% CI of [−2.91, +4.80] pp. This is inconclusive and fails our pre-specified 4-pp non-inferiority criterion. Nor have we reported the standard final average accuracy or average incremental accuracy requested by the reviewer. The stored task-by-checkpoint loss matrices permit analogous loss-based summaries, but these are not substitutes for task-success metrics. Reliable task-level evaluations for Code and Math, alongside final average and incremental accuracy, are required before making a deployment recommendation. We will revise the discussion to make this evidence gap explicit rather than implying that the loss result transfers to accuracy.
In summary, the supported claim is not that fixed random masking is the best practical CL strategy. It is that, for this particular static, initialization-time LoRA-B selection problem, gradient attribution does not outperform an equally sparse random baseline on loss-based retention, despite its calibration cost. Post-task checkpoint and broader performance evaluations remain important tests of external validity. We will reinforce the title’s qualifiers—static, within-B, and loss-based—throughout the abstract and discussion.