- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AS-LoRA for differentially private federated fine-tuning. At each communication round, the server selects whether to update the A or B factor independently for each LoRA layer, while sharing the selected mode across clients. Selection scores are computed from privatized local gradients, with a secant-based curvature estimate introduced late in training. The paper evaluates the method on language understanding, question answering, and image classification tasks, and provides analyses of reconstruction error, convergence, privacy, and flatness.
Critical strengths
-
The layer-wise, round-wise selection of LoRA factors is a useful extension of fixed alternating optimization. Sharing the selected mode across clients preserves aggregation consistency while allowing more flexible updates.
-
The experiments cover language understanding, question answering, and image classification under two privacy budgets. The gains on several language tasks are encouraging, and Table 8 suggests a small additional scoring cost under the evaluated configuration.
-
The main text is clear and well-organized.
Critical weaknesses
-
The reconstruction-error floor for alternating schedules is not proved. Appendix E bounds the number of matched updates by (T/(2N)), which still grows with (T). This can imply slower convergence, but does not establish a nonzero asymptotic error floor as claimed in Theorem
-
The score penalizes directional curvature , whereas Theorem 4 concerns . The proof does not establish the alignment conditions needed to relate their ordering. Differences in gradient norms can also cause selection of the higher-curvature block.
-
Algorithm 3 uses privatized-gradient secants, while Appendix D invokes a raw-minibatch Hessian without justifying the post-processing claim. Privacy composition across rounds also needs clarification, given the stated reinitialization of the privacy engine.
-
The proofs assume current scores and greedy selection, whereas the implementation uses estimated scores, stale scores for inactive factors, and stochastic selection. The effects of these differences are not quantified.
-
The experiments lack a layer-wise random selection baseline to isolate the value of score-driven selection. A first-order-only comparison would also clarify the contribution of curvature, which is used only in the final ten rounds.
Questions for authors
-
Does the argument for Theorem 2 establish a nonzero asymptotic error floor, or only a finite-horizon disadvantage? What prevents the error from vanishing as the number of matched updates grows with ?
-
What justifies the transition from lower directional curvature to a smaller maximum Hessian eigenvalue in Theorem 4? How does the guarantee account for unequal gradient norms between the two factors?
-
Algorithm 3 computes scores from privatized gradients, whereas Appendix D refers to a Hessian computed directly on the training minibatch. Which procedure is used in the private experiments? Please also clarify how the total privacy budget is composed across rounds despite reinitializing the privacy engine.
-
Which guarantees remain valid for stochastic selection based on estimated and potentially stale scores, and which apply only to the idealized greedy rule?
-
Are there existing results comparing against layer-wise random selection and first-order scoring throughout training? These comparisons would help isolate the contributions of score-driven selection and the late-stage curvature term.
7 comments
Response to Reviewer h453 (2/2)
[W.4 / Q.4] Theorems 3 and 4: Guarantees for stochastic selection with estimated and stale scores
Thank you for this question. When preparing the manuscript, we had derived Theorems 3 and 4 in two versions: one for greedy selection and one for the stochastic (temperature-scaled softmax) selection used in our algorithm. We presented the greedy version because we believed it highlighted the effect of the proposed score more clearly. Based on your comment, and to close the gap between the algorithm and the theory, we have replaced the greedy version with the stochastic selection of Eq. (4) of the revised manuscript.
The revised analysis also reflects the error due to staleness and estimation as in the algorithm, modeling it as , where is the estimated score used for selection (for an inactive factor, the one from its last active round) and is the score at the current iterate. Hence the revised Theorems 3 and 4 are valid for stochastic selection based on stale and estimated scores. However, we do not explicitly model the privatized gradients and the secant approximation beyond this error term. Tracking how the DP-SGD noise and the secant approximation error propagate and accumulate is highly complex, and we were concerned that it would distract from the main messages of Theorems 3 and 4.
Theorems 1 and 2 do not rely on the accuracy or freshness of the selection scores. And we revised Theorems 3 and 4 as described above. Thus, all four theorems in the revised manuscript apply to stochastic selection based on estimated and potentially stale scores under their respective assumptions.
[W.5 / Q.5] Ablations for score-driven selection and late-stage curvature
We thank the reviewer for this valuable suggestion.
First, to examine the contribution of score-driven selection beyond layer-wise selection, we compare random selection with Secant (). To compare learning speed rather than only the performance after a specific number of rounds, we plot the performance of both methods over training rounds and add this plot to Appendix J.5 (Figure 9) of the revised manuscript. The plot shows that score-driven mode selection is superior in final performance and also learns much faster.
Next, we measure the performance of three variants: first-order scoring throughout training (), the curvature score included from round 90 (Secant ()), and the curvature score used from the start (). Including the curvature score improves performance, and including it earlier gives a larger improvement. This confirms the contribution of also considering curvature. Also, we remark that the late-stage curvature term remains meaningful because it serves as a trade-off between accuracy and scoring cost.
[Table R1. Comparison of curvature estimators and activation schedules on MNLI with DP-SGD training ().]
| Method | DP guarantee | MNLI (m) | MNLI (mm) |
|---|---|---|---|
| FD () | No | ||
| HVP () | No | ||
| Secant () | Yes | ||
| Secant () | Yes | ||
| Secant () | Yes |
Table R1 extends the original Table 2 by adding Secant () and Secant (). It also appears as Table 2 in the revised manuscript.
Thank you for the detailed response and the additional experiments. After reading the revised manuscript and your replies, I have three remaining concerns.
-
Novelty. Several prior works have addressed aggregation error in federated LoRA through single-factor updates or alternating schedules. As I understand it, the main contribution of AS-LoRA is to decide which factor to update in each layer and communication round using scores estimated from privatized gradients. I see this as an incremental extension of existing single-factor update methods. The paper would benefit from a clearer account of what this adaptive rule adds beyond prior approaches.
-
Comparison with LA-LoRA. LA-LoRA is a relevant baseline because it also studies alternating factor updates in differentially private federated LoRA. It alternates between updating and at successive local steps and averages both factors at the server. AS-LoRA instead selects one factor per layer for the entire communication round. It would be helpful to include LA-LoRA alongside FFA-LoRA and RoLoRA. If feasible, please use the same privacy budget, LoRA rank, and number of local training steps.
-
Reliability of comparing scores from different rounds. For a given layer, the score for may have just been updated, while the score for may be several rounds old. If scores higher, it is unclear whether updating would yield a larger loss reduction at the current model state or whether the older score for no longer reflects that state. The comparison with random selection evaluates the overall strategy, but leaves this question open. Could the authors compare the score-based ranking with the observed loss reductions from updating and separately from the same checkpoint, and report how often the rankings agree as the older score becomes more stale?
Response to Reviewer h453 (1/2)
Thank you for this question. The argument establishes only a finite-horizon disadvantage. We made a logical jump from this finite-horizon disadvantage to a nonzero asymptotic floor. In particular, the number of matched rounds, T/(2N), grows linearly in T for fixed N, and these rounds alone give δT(n⋆)≤γn⋆T/(2N)δ0(n⋆)→0.
We have therefore narrowed the scope of Theorem 2 and Corollary 1 to FFA-LoRA, whose floor arises from the frozen subspace of A, and removed the floor claim for layer-tied alternation. We instead justify layer-wise selection by a score comparison. For dn=SA(n)−SB(n), the average score advantage of independent per-layer maximization over the best common mode is exactly
&\frac{1}{N}\sum_{n=1}^N\max_{a\in\lbrace A,B\rbrace} S_a^{(n)} -\max_{a\in\lbrace A,B\rbrace}\frac{1}{N}\sum_{n=1}^N S_a^{(n)} \\ &\qquad =\frac{1}{2N}\left(\sum_{n=1}^N|d_n|-\left|\sum_{n=1}^N d_n\right|\right)\ge 0, \end{aligned}$$ with equality if and only if all $d_n$ share the same sign. Since $\eta S_a^{(n)}$ is the second-order estimate of the one-step loss decrease from updating block $a$ of layer $n$ (ignoring cross-layer curvature), this indicates a larger estimated descent for AS-LoRA, without claiming that layer-tied alternation cannot eventually converge. In line with the narrowed scope, we have revised all corresponding statements in the manuscript, and added this score comparison, together with its derivation, immediately after Corollary 1. > **[W.2 / Q.2] Theorem 4: Directional curvature, maximum Hessian eigenvalues, and unequal gradient norms** Thank you for identifying these two gaps. We acknowledge that the original Theorem 4 contained a logical jump from lower directional curvature to a smaller maximum Hessian eigenvalue, and that its proof relied on the assumption that the gradient norms of the two factors are comparable, which is rough and may not hold in general. To address this, we add the assumption that the block with the smaller maximum curvature, $a^\dagger:=\arg\min_a\lambda_{\max}(H_a^{(n)})$, **is selected by AS-LoRA with probability at least 1/2**. Since this assumption is stated directly in terms of the maximum eigenvalue, it removes the jump from directional curvature. Under the stochastic selection of Eq. (4) of the revised manuscript with effective temperature $\tau^t(n)=T^t(\lvert\widehat S_A^t(n)\rvert+\lvert\widehat S_B^t(n)\rvert)$, it takes the analytic form $$\mathbb{E}\left[ \tanh\left( \frac{\widehat S^t_{a^\dagger}(n)-\widehat S^t_{a'}(n)} {2\tau^t(n)} \right) \right]\ge 0,$$ where $a'$ is the other block. This condition explicitly weighs the curvature against the gradient norm. For example, even when $\|g_{a'}\|^2>\|g_{a^\dagger}\|^2$, the assumption holds, i.e., the flatter block $a^\dagger$ is preferred, whenever $$\frac{\|g_{a'}\|^2}{\|g_{a^\dagger}\|^2} \le \frac{1-\eta\lambda_{a^\dagger}/2} {1-\eta\lambda_{a'}/2}$$ holds almost surely, where $\lambda_a=g_a^\top H_ag_a/\|g_a\|^2$. We acknowledge that this assumption does not always hold. However, we believe it is natural in the late-training regime, in which AS-LoRA activates the curvature term ($T^{\ast}=90$ in our experiments): as training converges, the gradient norms of the two factors tend to become comparable, so the score gap is governed by the curvature penalty. The revised Theorem 4 and its proof in Appendix H provide the details. > **[W.3 / Q.3] Privacy guarantees: Scoring from privatized gradients and composition across rounds** We thank the reviewer for this question. In our private experiments, we follow Algorithm 3 and **compute scores only from gradients clipped and privatized by DP-SGD**. The original presentation did not clearly distinguish the ideal score $S$ used in the analysis from the estimated score $\widehat{S}$ used for selection. We have clarified this distinction in Appendix C, including Remark 1, and revised Appendix D to base the privacy argument on the estimated scores. These scores are computed from privatized gradients using the secant estimator, without evaluating a Hessian on the training minibatch. Score computation and the subsequent randomized selection are therefore post-processing of DP-SGD outputs. Appendix I.4 incorrectly stated that the PrivacyEngine is reinitialized each round. Only the local model and optimizer are reinitialized. **Each client's PrivacyEngine is initialized once and reused across rounds**, preserving its accountant state for cumulative privacy accounting over the full training run. We have corrected Appendix I.4 and added Table 9, which reports each client's dataset size $N_k$, sampling rate $q_k=B/N_k$, and noise multiplier $\sigma_k$ for all datasets.