- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies differentially private continual learning with low-rank adaptation (LoRA). Through empirical analyses, the authors observe that DP exacerbates forgetting and worsens the stability–plasticity trade-off, while jointly perturbing both LoRA factors introduces a bilinear noise interaction. To address these issues, they propose DP-SPA, which constructs a stable subspace from previously privatized updates and a plastic subspace from pretrained weights within the stable subspace’s orthogonal complement. These bases remain fixed during each task, while the corresponding up-projection factors are trained using DP-SGD. Before merging each task update, the method recalibrates its stable-subspace component using historical and current update energies. The paper provides a privacy analysis and theoretical results concerning historical overlap and a conditional bound on old-task loss increase. Experiments across multiple continual learning settings and privacy budgets report improved average accuracy and reduced forgetting compared with the evaluated baselines.
Critical strengths
The analysis of bilinear noise interactions in Section 3 provides useful insight into the effects of jointly perturbing both LoRA factors under differential privacy.
Critical weaknesses
The paper requires substantial improvement in clarity, positioning, and exposition. In its current form, it reads more like a technical report describing a procedure than a research paper articulating a clear contribution and providing insight into its design. The novelty, scope of the theoretical results, and connections between the empirical observations and the proposed method are insufficiently explained. My specific concerns are as follows.
- Unclear positioning and novelty. The introduction does not clearly establish what is already known about differentially private LoRA in continual learning. Has this specific setting been studied previously, or are the authors introducing it? Which challenges remain unresolved by existing approaches, and what precisely does this work contribute beyond them? These questions should be addressed explicitly to make the novelty assessable.
- Insufficiently specified experimental setup in Section 3. Although the paper describes jointly updating both LoRA factors using DP-SGD, it does not clearly identify the continual learning baseline or explain how it relates to existing methods. In particular, it is unclear whether the experiments evaluate an established DP-LoRA continual learning approach or the authors’ own adaptation of standard DP-SGD to sequential LoRA training. This distinction, together with the relevant implementation details, is necessary to interpret the observations and assess their generality.
- Unclear scope of the privacy guarantee. Theorem 1 appears to state a general privacy composition result conditional on per-task guarantees with very limited novelty, and is not a specific theoretical result for the method in Section 3. The paper should explain how the clipping threshold , noise multiplier , sampling rate, and number of optimization steps are specified, and how the privacy accountant determines the resulting privacy parameters.
- Weak connection between the observations and the method. The challenges identified in Section 3 are supported primarily by empirical observations, alongside an algebraic characterization of the bilinear noise interaction. While this characterization provides a motivation for fixing one LoRA factor in 4.2, the motivations for the specific subspace construction in Section 4.1 and recalibration mechanism in Section 4.3 are much less developed. The authors should explain which observed failure mode each component addresses, why the proposed construction is appropriate, and what distinguishes it from existing methods. At present, Section 4 largely presents the procedure without sufficiently explaining the reasoning behind its design.
- Insufficient interpretation of the theoretical results. Section 5 presents theorems with no discussion of their intuition, explaination, and practical implications. The authors should discuss what each result establishes, what it does not establish, and how it supports the proposed design. Overall, the current manuscript needs substantial revision to make its contributions clear and assessable. Stronger positioning relative to prior work, more precise descriptions of the experimental and privacy setups, and a coherent explanation connecting the observations, method, and theory are necessary to meet the standard expected of a research paper.
Questions for authors
See weakness.