- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies DPSGD for frozen- LoRA, where the random factor is fixed and only is fine-tuned. The authors have a utility statement for the restricted- objective, with the privacy guarantees being unchanged compared to full fine-tuning of the adapter . They first show that under frozen- condition, the sensitivity and smoothness of loss w.r.t translates to those w.r.t . They then provide two convergence analysis of the frozen- setting: 1. under DPSGD and with loss smoothness and lipschitzness assumptions (with inactive and active clipping) 2. under DPSGD and with PL condition. They show that in all these scenarios, the DP-noise component of the stationarity bound (or excess risk bound under PL condition) scales with , rather than (for full fine-tuning). The convergence bounds guarantee stationarity in domain, not in domain. PL condition, proven for linear LoRA, is also used for excess-risk and optimal-rank results. The informed- optimal-rank results are only an oracle benchmark.
Critical strengths
This paper had no visible strength in my opinion. In fact, after reading the paper multiple times, it was hard for me to get a clear and useful message from the paper. I explain my concerns in the following.
Critical weaknesses
The paper overall had no clear message to me, and its writing was hard to follow.
-
The convergence results driven by the authors seem trivial and have no new message to me. For instance, the convergence proof in Theorem 8 and Proposition 9 are trivial, as they prove convergence of DPSGD to a stationary point under smoothness and lipschitzness assumptions.
-
The stationarity upper-bounds from the optimization results also show that DP noise component scales with , rather than (for full fine-tuning). This is again trivial, because the number of trainable parameters in frozen- setting is , as opposed to for full fine-tuning. Hence, DPSGD adds independent noise to fewer parameters.
-
Further, as confirmed in line 118 by the authors, the dimensional advantage depends on . The dimensional advantage is not automatic if this quantity is large.
-
Even the statement above about the DP noise component is useless, because a smaller "stationarity upper-bound" in domain does not necessarily guarantee a better practical utility, compared to full fine-tuning. The authors have also mentioned and confirmed this point multiple times, e.g. in lines 244 and 245.
-
Finally, the excess risk results are conditional on the PL assumption, which is proven only for linear regression LoRA. The spectral decay assumption (Assumption 4) also may not hold. Therefore, the excess risk results and the following optimal-rank results are all conditional on PL and this assumption and are barely useful in practical settings. Finally, the experimental results are very limited with no clear explanations about the consider experimental settings.
Questions for authors
Overall, the paper was hard to follow. So I read it multiple times for a complete understanding of its main message. I ask the following questions about the remaining unclear parts:
-
What is meant by this sentence in line 107? "The shorthand scaling requires additional parameter restrictions and is not used here as an unconditional privacy certificate".
-
In line 140, how is the conditional minibatch variance derived?
-
What is meant by "rather than an unrestricted small- approximation" in line 210? Does it have the same meaning as in lines 232 and 233?
-
What is meant by "NTK and local-quadratic arguments may motivate the assumption" in line 269?
2 comments
Official comment
Thanks for the comments. After reading them carefully and reviewing my first comments, I have decided to keep my score.
Thank you for listing the passages that were unclear. We should state the message more directly: freezing A does not automatically improve privacy or downstream utility. Our result is a conditional convergence analysis for the restricted objective, showing where the mr DP-noise term enters and which other terms can offset it.
The factor scale is essential. The bound depends on κeff=s∥A∥op and is not gauge invariant. In a new diagnostic, we held W=BA fixed under A′=cA,B′=B/c across five values of c, three ranks, and 40 shared noise seeds. Frozen-A W-space noise changed substantially; a tangent-space construction using PRISM's official geometric components was invariant to numerical precision. We will state plainly that the theorem concerns the declared parameterization and does not establish superiority over full adaptation or PRISM.
The four passages have the following meanings.
Line 107: ρ(α,q,σ)=O(q2/σ2) is only a small-sampling/high-noise shorthand under additional restrictions. We do not substitute it into the privacy claim. The reported (ε,δ) values come from the stated sampled-Gaussian RDP accountant with the declared adjacency convention.
Line 140: conditional on the fixed dataset and A, let Ii∼Bernoulli(q) and use denominator b=qN. Independence gives E∥b−1∑i(Ii−q)gi∥F2=q(1−q)b−2∑i∥gi∥F2. Under inactive clipping, ∥gi∥F≤G, so this is at most (1−q)G2/b. We will include these two steps rather than state the result without derivation.
Line 210: “rather than an unrestricted small-q approximation” refers to the same issue as the later warning: the exact accountant is used for privacy calibration, while a q2/σ2 scaling may be used only as intuition when its conditions hold.
Line 269: NTK and local-quadratic arguments may suggest approximately regular local geometry, but they do not verify a PL inequality for Transformer fine-tuning. Our PL excess-risk and optimal-rank results are conditional. Moreover, Proposition 15's unconstrained quadratic has unbounded gradients and therefore does not instantiate all assumptions of Theorem 11 under unprojected Gaussian updates; we will correct that claim.
We will revise the introduction and the discussion around Theorem 8 to make this scope visible before presenting the conditional PL and rank results.