acceptodds
Withdrawn from ICLR 2027

Stable–Plastic Subspace Construction for Differentially Private Continual LoRA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning (CL) adapts pretrained models to a sequence of tasks while retaining knowledge acquired from earlier tasks. When task data are sensitive, differential privacy (DP) provides record-level protection, but maintaining both adaptation and retention under privacy constraints remains challenging. We study this problem in continual low-rank adaptation (LoRA) and observe that: 1) private training increases forgetting primarily during subsequent adaptation rather than initial task acquisition; 2) jointly perturbing both LoRA factors introduces a non-negligible bilinear noise interaction; and 3) DP aggravates the stability-plasticity dilemma, hindering both knowledge retention and adaptation to new tasks. Based on these observations, we propose DP-SPA, which constructs a stable subspace from previously privatized updates and a plastic subspace from pretrained weights within its orthogonal complement. The method fixes these bases and privately optimizes only the LoRA factor, eliminating the bilinear noise interaction within each step. Before merging each task update, it rescales stable-direction coefficients using historical and current update energies. We establish a privacy guarantee and show that recalibration does not increase historical overlap and tightens a conditional upper bound on old-task loss increase. Experiments across five datasets, eight settings, and multiple privacy budgets show that DP-SPA improves average accuracy and reduces forgetting compared with the evaluated baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, Continual Learning, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kjFSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies differentially private continual learning with low-rank adaptation (LoRA). Through empirical analyses, the authors observe that DP exacerbates forgetting and worsens the stability–plasticity trade-off, while jointly perturbing both LoRA factors introduces a bilinear noise interaction. To address these issues, they propose DP-SPA, which constructs a stable subspace from previously privatized updates and a plastic subspace from pretrained weights within the stable subspace’s orthogonal complement. These bases remain fixed during each task, while the corresponding up-projection factors are trained using DP-SGD. Before merging each task update, the method recalibrates its stable-subspace component using historical and current update energies. The paper provides a privacy analysis and theoretical results concerning historical overlap and a conditional bound on old-task loss increase. Experiments across multiple continual learning settings and privacy budgets report improved average accuracy and reduced forgetting compared with the evaluated baselines.

Critical strengths

The analysis of bilinear noise interactions in Section 3 provides useful insight into the effects of jointly perturbing both LoRA factors under differential privacy.

Critical weaknesses

The paper requires substantial improvement in clarity, positioning, and exposition. In its current form, it reads more like a technical report describing a procedure than a research paper articulating a clear contribution and providing insight into its design. The novelty, scope of the theoretical results, and connections between the empirical observations and the proposed method are insufficiently explained. My specific concerns are as follows.

  1. Unclear positioning and novelty. The introduction does not clearly establish what is already known about differentially private LoRA in continual learning. Has this specific setting been studied previously, or are the authors introducing it? Which challenges remain unresolved by existing approaches, and what precisely does this work contribute beyond them? These questions should be addressed explicitly to make the novelty assessable.
  2. Insufficiently specified experimental setup in Section 3. Although the paper describes jointly updating both LoRA factors using DP-SGD, it does not clearly identify the continual learning baseline or explain how it relates to existing methods. In particular, it is unclear whether the experiments evaluate an established DP-LoRA continual learning approach or the authors’ own adaptation of standard DP-SGD to sequential LoRA training. This distinction, together with the relevant implementation details, is necessary to interpret the observations and assess their generality.
  3. Unclear scope of the privacy guarantee. Theorem 1 appears to state a general privacy composition result conditional on per-task guarantees (ϵt,δt)(\epsilon_t,\delta_t) with very limited novelty, and is not a specific theoretical result for the method in Section 3. The paper should explain how the clipping threshold CC, noise multiplier σ\sigma, sampling rate, and number of optimization steps are specified, and how the privacy accountant determines the resulting privacy parameters.
  4. Weak connection between the observations and the method. The challenges identified in Section 3 are supported primarily by empirical observations, alongside an algebraic characterization of the bilinear noise interaction. While this characterization provides a motivation for fixing one LoRA factor in 4.2, the motivations for the specific subspace construction in Section 4.1 and recalibration mechanism in Section 4.3 are much less developed. The authors should explain which observed failure mode each component addresses, why the proposed construction is appropriate, and what distinguishes it from existing methods. At present, Section 4 largely presents the procedure without sufficiently explaining the reasoning behind its design.
  5. Insufficient interpretation of the theoretical results. Section 5 presents theorems with no discussion of their intuition, explaination, and practical implications. The authors should discuss what each result establishes, what it does not establish, and how it supports the proposed design. Overall, the current manuscript needs substantial revision to make its contributions clear and assessable. Stronger positioning relative to prior work, more precise descriptions of the experimental and privacy setups, and a coherent explanation connecting the observations, method, and theory are necessary to meet the standard expected of a research paper.
Questions for authors

See weakness.

Reviewer_Q9F6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies class-incremental LoRA under record-level differential privacy. The first task uses a basis initialized from public pretrained weights. From the second task onward, DP-SPA constructs a stable basis from privatized update history and a plastic basis from public pretrained weights. The bases remain fixed within each task while output-side coefficients are privately trained. Later-task updates undergo history-dependent rescaling before merging. The conditional analysis establishes privacy and geometric properties, including non-increase of an upper bound on old-task loss increase. Experiments cover five image datasets, eight continual-learning settings, and several privacy budgets.

Critical strengths
  1. The construction addresses a real constraint of private continual learning: adaptation directions cannot be selected freely from private task statistics. Reusing privatized history and public weights provides a coherent solution without an additional basis-selection privacy cost, under Theorem 1’s stated assumptions. The paper also correctly distinguishes the conditional loss bound from a guarantee that actual forgetting decreases.

  2. The improvement is not confined to the weaker joint-factor baseline. At 20 tasks on ImageNet-R, final accuracy rises from 64.54% for FA-LoRA to 68.14% for DP-SPA; on ImageNet-A it rises from 45.25% to 48.44% (Table 2). The branch ablations and the constant and norm-matched rescaling controls in Appendix E.2 make the empirical case more informative than a comparison with AB-LoRA alone.

Critical weaknesses

The experimental description leaves two aspects of privacy accounting unclear. Section 6.1 describes DP-SGD training, a cosine classifier trained alongside the LoRA coefficients, and selection of the clipping threshold from five values. However, Section 3.1 defines the private parameter vector using LoRA parameters, and Algorithm 1 explicitly describes private optimization of the output-side factor without mentioning the classifier. It is therefore not explicit whether the classifier is covered by that mechanism or a separately accounted private update. The data and privacy treatment used to select the clipping threshold are also unspecified. Appendix E.1 restates the requirement that all private-data-dependent operations be covered, but does not resolve these implementation details. This is a disclosure gap, not evidence that the implementation violates DP.

Questions for authors
  1. How was the trainable classifier handled in the privacy accounting? If trained on protected records, was it included in the LoRA clipping/noising mechanism or a separately accounted private update? Please identify the protected trainable parameters and specify the sampler and accountant inputs, including sampling rate, number of steps, and noise multiplier.

  2. Which data were used to select the clipping threshold? If selection used protected task data, how was its privacy cost included in the reported budget? This clarification could change the assessment of the claimed privacy–utility results.

Reviewer_TWeLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates differentially private continual LoRA and identifies three key challenges: DP exacerbates forgetting during subsequent adaptation, jointly privatizing AA and BB introduces a bilinear noise term, and DP further aggravates the stability–plasticity trade-off. To address these issues, the paper proposes DP-SPA, which constructs a stable basis from privatized historical information Ht−1H_{t-1} and a plastic basis from the public initialization W0W^0 within its orthogonal complement. It freezes AtA_t and privately optimizes only BtB_t using DP-SGD, followed by shrinking stable coefficients according to λjt=cjt/(hjt+cjt+τ)\lambda_j^t=c_j^t/(h_j^t+c_j^t+\tau). The paper further establishes a privacy guarantee (Theorem 1), a geometric characterization of the stable–plastic basis construction (Theorem 2), a proof that recalibration does not increase historical overlap (Theorem 3), and a conditional upper bound on the increase in old-task loss (Theorem 4).

Critical strengths

S1. A clear, quantified description of how DP degrades continual LoRA. The three observations: the temporal location of forgetting, the explicit bilinear noise interaction, and the stability–plasticity gap at matched plasticity provide an interpretable basis for method design.

S2. Correct privacy-first framing. Bases and recalibration use only privatized outputs and public weights as post-processing, and does not add privacy budget. The paper admits recalibration only lowers an upper bound, not actual old-task loss, which is restrained and credible.

Critical weaknesses

W1. Fixing A to remove ξ_B ξ_A is FFA-LoRA (Sun et al. 2024) moved to continual. History top-subspace is OGD/GPM applied to DP outputs. Therefore, the novelty of the concept is limited.

W2. Section 6.1 states a cosine classifier is trained alongside LoRA coefficients, yet Appendix E.1 accounts only for LoRA gradients (C=1, RDP-calibrated σ). Is the cosine classifier trained alongside LoRA not included in DP accounting?

Questions for authors

Is the cosine classifier also included in DP-SGD training? And is the cosine classifier covered by the per-task (ϵ,δ) budget? Can you provide a fair comparison of two external SOTA continuous LoRA methods adapted to DP?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.