acceptodds
Under review as a conference paper at ICLR 2027

Rank-Dependent DP-Noise Bounds for Frozen- LoRA

Abstract

We study differentially private SGD (DP-SGD) for frozen- LoRA, where a random factor is fixed and only is trained. Privacy accounting is dimension independent: the same noise multiplier, Poisson sampling rate, step count, adjacency convention, and accountant yield the same guarantee. Our main result is instead a utility statement for the restricted -objective: when gives , the reparameterization controls both gradient sensitivity and smoothness, and the DP-noise component of the stationarity bound scales with rather than . With inactive clipping, , and respective positive initial gaps , the ratio of the DP-noise terms is bounded by . With active clipping, method-specific bias terms remain and we do not claim an unconditional improvement in total error or practical utility. The guarantee controls , not unrestricted stationarity in -space. A Polyak–Lojasiewicz (PL) assumption is used only for excess-risk and optimal-rank results; we prove it for linear LoRA, while the informed- optimal-rank result is an oracle benchmark. We quantify, without a complete convergence theorem, the product-noise term arising when both factors are trained. A complete 1,800-run controlled synthetic matrix separates restricted error from projection error and demonstrates optimizer-dependent rank orderings; SST-2 experiments remain feasibility checks, not validation of nonlinear PL or the optimal-rank law. A completed 42-run matched-public-start BERT/SST-2 follow-up finds at most 0.153 percentage points mean accuracy difference between frozen and jointly trained factors across six private-stage settings, providing no clear practical advantage for either branch.

Primary area:
learning theory
Keywords:
differential privacy, LoRA, DP-SGD, low-rank adaptation, clipping bias, optimization bounds

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_b5RrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies DPSGD for frozen-AA LoRA, where the random factor AA is fixed and only BB is fine-tuned. The authors have a utility statement for the restricted-BB objective, with the privacy guarantees being unchanged compared to full fine-tuning of the adapter WW. They first show that under frozen-AA condition, the sensitivity and smoothness of loss w.r.t WW translates to those w.r.t BB. They then provide two convergence analysis of the frozen-AA setting: 1. under DPSGD and with loss smoothness and lipschitzness assumptions (with inactive and active clipping) 2. under DPSGD and with PL condition. They show that in all these scenarios, the DP-noise component of the stationarity bound (or excess risk bound under PL condition) scales with mrmr, rather than mnmn (for full fine-tuning). The convergence bounds guarantee stationarity in BB domain, not in WW domain. PL condition, proven for linear LoRA, is also used for excess-risk and optimal-rank results. The informed-AA optimal-rank results are only an oracle benchmark.

Critical strengths

This paper had no visible strength in my opinion. In fact, after reading the paper multiple times, it was hard for me to get a clear and useful message from the paper. I explain my concerns in the following.

Critical weaknesses

The paper overall had no clear message to me, and its writing was hard to follow.

  1. The convergence results driven by the authors seem trivial and have no new message to me. For instance, the convergence proof in Theorem 8 and Proposition 9 are trivial, as they prove convergence of DPSGD to a stationary point under smoothness and lipschitzness assumptions.

  2. The stationarity upper-bounds from the optimization results also show that DP noise component scales with mrmr, rather than mnmn (for full fine-tuning). This is again trivial, because the number of trainable parameters in frozen-AA setting is mrmr, as opposed to mnmn for full fine-tuning. Hence, DPSGD adds independent noise to fewer parameters.

  3. Further, as confirmed in line 118 by the authors, the dimensional advantage depends on κ\kappa. The dimensional advantage is not automatic if this quantity is large.

  4. Even the statement above about the DP noise component is useless, because a smaller "stationarity upper-bound" in BB domain does not necessarily guarantee a better practical utility, compared to full fine-tuning. The authors have also mentioned and confirmed this point multiple times, e.g. in lines 244 and 245.

  5. Finally, the excess risk results are conditional on the PL assumption, which is proven only for linear regression LoRA. The spectral decay assumption (Assumption 4) also may not hold. Therefore, the excess risk results and the following optimal-rank results are all conditional on PL and this assumption and are barely useful in practical settings. Finally, the experimental results are very limited with no clear explanations about the consider experimental settings.

Questions for authors

Overall, the paper was hard to follow. So I read it multiple times for a complete understanding of its main message. I ask the following questions about the remaining unclear parts:

  1. What is meant by this sentence in line 107? "The shorthand q2/σ2q^2/\sigma^2 scaling requires additional parameter restrictions and is not used here as an unconditional privacy certificate".

  2. In line 140, how is the conditional minibatch variance derived?

  3. What is meant by "rather than an unrestricted small-qq approximation" in line 210? Does it have the same meaning as in lines 232 and 233?

  4. What is meant by "NTK and local-quadratic arguments may motivate the assumption" in line 269?

2 comments

Thank you for listing the passages that were unclear. We should state the message more directly: freezing A does not automatically improve privacy or downstream utility. Our result is a conditional convergence analysis for the restricted objective, showing where the mrmr DP-noise term enters and which other terms can offset it.

The factor scale is essential. The bound depends on κeff=s∥A∥op\kappa_{\rm eff}=s\|A\|_{\rm op} and is not gauge invariant. In a new diagnostic, we held W=BAW=BA fixed under A′=cA,B′=B/cA'=cA,B'=B/c across five values of cc, three ranks, and 40 shared noise seeds. Frozen-A W-space noise changed substantially; a tangent-space construction using PRISM's official geometric components was invariant to numerical precision. We will state plainly that the theorem concerns the declared parameterization and does not establish superiority over full adaptation or PRISM.

The four passages have the following meanings.

  1. Line 107: ρ(α,q,σ)=O(q2/σ2)\rho(\alpha,q,\sigma)=O(q^2/\sigma^2) is only a small-sampling/high-noise shorthand under additional restrictions. We do not substitute it into the privacy claim. The reported (ε,δ)(\varepsilon,\delta) values come from the stated sampled-Gaussian RDP accountant with the declared adjacency convention.

  2. Line 140: conditional on the fixed dataset and A, let Ii∼Bernoulli(q)I_i\sim{\rm Bernoulli}(q) and use denominator b=qNb=qN. Independence gives E∥b−1∑i(Ii−q)gi∥F2=q(1−q)b−2∑i∥gi∥F2\mathbb E\|b^{-1}\sum_i(I_i-q)g_i\|_F^2=q(1-q)b^{-2}\sum_i\|g_i\|_F^2. Under inactive clipping, ∥gi∥F≤G\|g_i\|_F\le G, so this is at most (1−q)G2/b(1-q)G^2/b. We will include these two steps rather than state the result without derivation.

  3. Line 210: “rather than an unrestricted small-q approximation” refers to the same issue as the later warning: the exact accountant is used for privacy calibration, while a q2/σ2q^2/\sigma^2 scaling may be used only as intuition when its conditions hold.

  4. Line 269: NTK and local-quadratic arguments may suggest approximately regular local geometry, but they do not verify a PL inequality for Transformer fine-tuning. Our PL excess-risk and optimal-rank results are conditional. Moreover, Proposition 15's unconstrained quadratic has unbounded gradients and therefore does not instantiate all assumptions of Theorem 11 under unprojected Gaussian updates; we will correct that claim.

We will revise the introduction and the discussion around Theorem 8 to make this scope visible before presenting the conditional PL and rank results.

Reviewer_b5RrOpenReview ↗

Official comment

Thanks for the comments. After reading them carefully and reviewing my first comments, I have decided to keep my score.

Reviewer_a2BYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

LoRA-FA decomposes a d-by-n matrix weight matrix WW into W0+BAW_0 + BA, where BB is d-by-r and AA is r-by-n, then it freezes W0W_0 and AA, and only optimizes B with gradient descent on the data. DP-LoRA-FA replaces the gradient descent on BB with DP-SGD on BB, and provides differential privacy guarantee. This paper establishes convergence guarantees of DP-LoRA-FA, which scales with the dimension of BB, instead of that of the full gradient matrix WW, under different assumptions (smoothness, no effective clipping, or PL condition). Essentially, the proofs leverage the following argument: smoothness of the loss with respect to the full gradient matrix WW implies smoothness with respect to AA, given AA has (dimension independent) operator norm.

Critical strengths

This paper establishes a standard convergence guarantee of DP-LoRA-FA. This seems like a new result.

Critical weaknesses
  1. The theoretical results are standard, especially the sensitivity and smoothness transfer argument. Additionally, it is well-established that LoRA-FA can be viewed as a dimension reduction method on the gradient space ([1][2]), so an improved convergence guarantee for the DP version is not particularly surprising.

  2. Discussion of each result is very limited. At times, the paper reads like a list of results without discussing their implications, intuition, or relationships to one another. In addition, the paper does not situate its analysis within the existing literature. For example, [3] proved the privacy utility tradeoff of a very similar mechanism in the linear setting, but the paper does not mention their results.

  3. Experimental results are insufficient to complement the theoretical results. For example, the 95% confidence interval in table 2 shows no improvement.

[1] Hao et al. Flora: low rank adapters are secretly gradient compressors. ICML 2024. [2] Malekmohammadi and Farnadi. Low-Rank Adaptation secretly imitates differential private SGD. 2025 [3]Lev et al. The Gaussian Mixing Mechanism: Rényi Differential privacy via Gaussian Sketches. Neurips 2025

Questions for authors

A minor question: under the paragraph “restricted error and projection error”, should it be BTAB_T A instead of BATBA_T?

2 comments

Thank you for pointing out the missing connections to gradient compression, Gaussian sketches, and private LoRA mechanisms, and for flagging the decomposition and AI disclosure.

We agree that the ingredients are standard. The result we intend to claim is more specific than a trainable-parameter count: for centralized frozen-A DP-SGD, Theorem 8 tracks how the same fixed factor changes per-example sensitivity and smoothness, while keeping the DP-noise term separate from minibatch variance and active-clipping bias. The later results add restricted-versus-full approximation error and conditional PL excess risk. We will reorganize the presentation around this chain instead of presenting the results as separate observations.

We will add the cited work and state the differences directly. Flora interprets low-rank adapters as gradient compressors; Low-Rank Adaptation Secretly Imitates Differentially Private SGD studies the projection/noise connection; Gaussian Mixing analyzes a Gaussian-sketch mechanism in a linear setting. FFA-LoRA supplies the freeze-A parameterization in federated learning, whereas PRISM replaces factor-space DP-SGD with a gauge-invariant tangent-space mechanism. Our new 1,800-condition diagnostic checks this last distinction: rescaling equivalent factors changes frozen-A W-space noise substantially, while the PRISM tangent construction is invariant to numerical precision. This is a boundary check for our fixed-scale theorem, not evidence of an advantage over PRISM.

For the decomposition, you are correct: the iterate is BTAB_TA, not BATBA_T. The intended identity is

=\left[f(B_TA)-f_A^{\star}\right] +\left[f_A^{\star}-f_{\mathrm{full}}^{\star}\right].$$ The subscript $T$ indexes the optimized factor $B$ after $T$ updates, while $A$ is frozen. The occurrence of $f(BA_T)$ is a typographical error and will be corrected to $f(B_TA)$. The first bracket is restricted optimization error and the second is projection error. The numerical audit verifies the corrected identity to a maximum discrepancy of 1.78 × 10^-15. The manuscript's AI Use Statement already discloses assistance with literature review and polishing. We will repeat the same complete statement in the revision and OpenReview metadata.

Thank you for the response. After reading the author's responses to me and the other reviewers, I believe some of the most important concerns, i.e. novelty and readability, are not resolved. Therefore, I will maintain my score.

Reviewer_3UfpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies DP-SGD with a fixed random LoRA factor, derives an mrmr-dimensional noise term in a restricted-stationarity bound, and presents conditional PL results and synthetic/BERT experiments.

Critical strengths
  1. Theorem 8 clearly tracks sensitivity, smoothness and noise dimension and appears correct under its stated assumptions.

  2. The paper is explicit about clipping bias, approximation error and the limited gains in its neural experiments.

Critical weaknesses
  1. The method follows FFA-LoRA [1], LB≤∣A∣op2LWL_B \leq |A|_{\mathrm{op}}^2 L_W mainly sharpens its existing smoothness bound, and no experimental comparison shows what is gained over FFA.

  2. The identity E∣ΞBΞA∣F2=mnrσ4C4\mathbb{E}|\Xi_B\Xi_A|_F^2 = mnr\sigma^4 C^4 in Appendix I, equation (28), already appears in PRISM [2], but the paper neither discusses this work nor explains whether its guarantees go beyond it.

  3. PRISM shows that one-sided training leaves noise dependent on the frozen factor’s scale, while this paper bounds that scale without removing the dependence or showing an experimental advantage over PRISM.

  4. Proposition 15’s quadratic loss has unbounded gradients, so it fails Assumption 2 under Algorithm 1’s unprojected Gaussian updates and does not provide a valid example satisfying all the assumptions of Theorem 11.

  5. Table 2 shows no clear benefit from freezing A—the absolute mean difference is at most 0.153 percentage points—and its AdamW updates with 9.00–20.10% active clipping do not test Theorem 8’s SGD and inactive-clipping assumptions.

  6. In Figure 1’s teacher-rank-2 case, rank-2 SGD improves over full SGD in test objective (3.93238 versus 4.15274) but still performs worse than no update (3.92103), so this comparison does not demonstrate useful adaptation.

[1] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving LoRA in Privacy-preserving Federated Learning. ICLR 2024. [2] Shihao Wang and Xueru Zhang. PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA. ICML 2026.

Questions for authors
  1. Does the paper actually overcome the one-sided limitation identified by PRISM? Please include FFA-LoRA and PRISM as experimental baselines.

  2. Can Assumptions 1–3 hold simultaneously under Algorithm 1, as required by Theorem 11? How does Proposition 15’s quadratic objective satisfy Assumption 2’s uniform gradient bound when its gradients are unbounded?

1 comment

Thank you for pointing us to the overlap with FFA-LoRA and PRISM and for distinguishing the theorem from experiments that do not satisfy its assumptions.

We agree that freezing A is not new. Our contribution is the finite-time analysis of centralized frozen-A DP-SGD: it transfers both per-example sensitivity and smoothness through the fixed factor and keeps minibatch variance, DP noise, active-clipping bias, restricted optimization, and projection error separate. FFA-LoRA introduced the same nonzero-A/zero-B parameterization for federated learning, where the issue is the interaction between local client updates and server aggregation. In our centralized setting, an “FFA” arm would execute the same update as our frozen-A arm, so showing it as a second method would duplicate that arm. We will cite FFA-LoRA prominently, describe our implementation as centralized FFA-style DP-SGD, and make clear that Theorem 8—not the parameterization—is the added result.

Our method does not overcome PRISM's gauge-dependence objection. We tested this directly using the geometric components of the official implementation. In the paper's notation, we held W=BAW=BA fixed under A′=cA,B′=B/cA'=cA,B'=B/c for c∈{0.25,0.5,1,2,4}c\in\{0.25,0.5,1,2,4\}, ranks 2/4/8, and 40 shared Gaussian seeds. The three methods give 1,800 method/scale/seed conditions, paired through the shared seeds. For the PRISM tangent construction, the maximum relative W-space discrepancy from c=1c=1 was 1.35×10−71.35\times10^{-7} for the noise vector and 1.35×10−51.35\times10^{-5} for one clipped, noised, retracted tangent step. At the two extreme scales, the minimum noise-vector discrepancy was 0.75 for frozen-A Euclidean DP-SGD and 1.54 for ordinary joint-factor Euclidean DP-SGD. Thus a bound on ∥A∥op\|A\|_{\rm op} controls the scale in our theorem but does not make the update gauge invariant. We will add PRISM, attribute the identity in Appendix I, and report this diagnostic as a mechanism check rather than a full PRISM training reproduction.

You are also correct about Proposition 15. Its unconstrained quadratic loss is PL, but its per-example gradients are not uniformly bounded over the whole space. Algorithm 1 uses unprojected Gaussian updates, so a bounded-domain argument does not apply. Proposition 15 is therefore not an example satisfying all assumptions of Theorem 11. We will present it only as a linear PL calculation and remove the claim that it jointly verifies Assumptions 1--3. Theorem 11 remains conditional; neither this quadratic example nor the Transformer experiments instantiate all of its assumptions.

Finally, Table 2's AdamW runs with active clipping do not test Theorem 8's SGD/inactive-clipping case, and the teacher-rank-2 SGD result is worse than no update. We will keep both negative findings and describe them as limits of the empirical evidence. Theorem 8 remains the restricted-coordinate convergence result under its stated assumptions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.