acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

Abstract

Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure, which is further amplified by DP noise and degrades both stability and accuracy. Existing remedies apply a single update mode uniformly across all layers and all communication rounds (or alternate them on a fixed schedule), ignoring both the structural asymmetry between the two LoRA factors and the round-wise dynamics of training. We propose AS-LoRA, an adaptive framework defined by three axes (i) layer-wise freedom, in which each layer independently selects its active component, (ii) round-wise adaptivity, in which the selection updates over communication rounds, and (iii) a curvature-aware score derived from a second-order approximation of the loss. Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost. Across GLUE, SQuAD, CIFAR-100, and Tiny-ImageNet under strict DP budgets and non-IID partitions, AS-LoRA improves over the federated LoRA baselines by up to pp on MNLI (m) and pp on SQuAD v2.0. Compared with SVD-based aggregation, it attains comparable accuracy with about additional client-side compute, two orders of magnitude lower server-side aggregation cost, and negligible communication cost. Code for the proposed method is available at https://anonymous.4open.science/r/as_lora-F75F/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated learning, LoRA, Differential privacy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_h453OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AS-LoRA for differentially private federated fine-tuning. At each communication round, the server selects whether to update the A or B factor independently for each LoRA layer, while sharing the selected mode across clients. Selection scores are computed from privatized local gradients, with a secant-based curvature estimate introduced late in training. The paper evaluates the method on language understanding, question answering, and image classification tasks, and provides analyses of reconstruction error, convergence, privacy, and flatness.

Critical strengths
  1. The layer-wise, round-wise selection of LoRA factors is a useful extension of fixed alternating optimization. Sharing the selected mode across clients preserves aggregation consistency while allowing more flexible updates.

  2. The experiments cover language understanding, question answering, and image classification under two privacy budgets. The gains on several language tasks are encouraging, and Table 8 suggests a small additional scoring cost under the evaluated configuration.

  3. The main text is clear and well-organized.

Critical weaknesses
  1. The reconstruction-error floor for alternating schedules is not proved. Appendix E bounds the number of matched updates by (T/(2N)), which still grows with (T). This can imply slower convergence, but does not establish a nonzero asymptotic error floor as claimed in Theorem

  2. The score penalizes directional curvature g⊤Hg/∥g∥2g^\top H g / \|g\|^2, whereas Theorem 4 concerns λmax⁡(H)\lambda_{\max}(H). The proof does not establish the alignment conditions needed to relate their ordering. Differences in gradient norms can also cause selection of the higher-curvature block.

  3. Algorithm 3 uses privatized-gradient secants, while Appendix D invokes a raw-minibatch Hessian without justifying the post-processing claim. Privacy composition across rounds also needs clarification, given the stated reinitialization of the privacy engine.

  4. The proofs assume current scores and greedy selection, whereas the implementation uses estimated scores, stale scores for inactive factors, and stochastic selection. The effects of these differences are not quantified.

  5. The experiments lack a layer-wise random selection baseline to isolate the value of score-driven selection. A first-order-only comparison would also clarify the contribution of curvature, which is used only in the final ten rounds.

Questions for authors
  1. Does the argument for Theorem 2 establish a nonzero asymptotic error floor, or only a finite-horizon disadvantage? What prevents the error from vanishing as the number of matched updates grows with TT?

  2. What justifies the transition from lower directional curvature to a smaller maximum Hessian eigenvalue in Theorem 4? How does the guarantee account for unequal gradient norms between the two factors?

  3. Algorithm 3 computes scores from privatized gradients, whereas Appendix D refers to a Hessian computed directly on the training minibatch. Which procedure is used in the private experiments? Please also clarify how the total privacy budget is composed across rounds despite reinitializing the privacy engine.

  4. Which guarantees remain valid for stochastic selection based on estimated and potentially stale scores, and which apply only to the idealized greedy rule?

  5. Are there existing results comparing against layer-wise random selection and first-order scoring throughout training? These comparisons would help isolate the contributions of score-driven selection and the late-stage curvature term.

7 comments
AuthorsOpenReview ↗

Response to Reviewer h453 (1/2)

[W.1 / Q.1] Theorem 2: Finite-horizon disadvantage and the asymptotic error floor

Thank you for this question. The argument establishes only a finite-horizon disadvantage. We made a logical jump from this finite-horizon disadvantage to a nonzero asymptotic floor. In particular, the number of matched rounds, T/(2N)T/(2N), grows linearly in TT for fixed NN, and these rounds alone give δT(n⋆)≤γn⋆T/(2N)δ0(n⋆)→0\delta^T(n^\star)\le\gamma_{n^\star}^{T/(2N)}\delta^0(n^\star)\to0.

We have therefore narrowed the scope of Theorem 2 and Corollary 1 to FFA-LoRA, whose floor arises from the frozen subspace of AA, and removed the floor claim for layer-tied alternation. We instead justify layer-wise selection by a score comparison. For dn=SA(n)−SB(n)d_n=S_A^{(n)}-S_B^{(n)}, the average score advantage of independent per-layer maximization over the best common mode is exactly

&\frac{1}{N}\sum_{n=1}^N\max_{a\in\lbrace A,B\rbrace} S_a^{(n)} -\max_{a\in\lbrace A,B\rbrace}\frac{1}{N}\sum_{n=1}^N S_a^{(n)} \\ &\qquad =\frac{1}{2N}\left(\sum_{n=1}^N|d_n|-\left|\sum_{n=1}^N d_n\right|\right)\ge 0, \end{aligned}$$ with equality if and only if all $d_n$ share the same sign. Since $\eta S_a^{(n)}$ is the second-order estimate of the one-step loss decrease from updating block $a$ of layer $n$ (ignoring cross-layer curvature), this indicates a larger estimated descent for AS-LoRA, without claiming that layer-tied alternation cannot eventually converge. In line with the narrowed scope, we have revised all corresponding statements in the manuscript, and added this score comparison, together with its derivation, immediately after Corollary 1. > **[W.2 / Q.2] Theorem 4: Directional curvature, maximum Hessian eigenvalues, and unequal gradient norms** Thank you for identifying these two gaps. We acknowledge that the original Theorem 4 contained a logical jump from lower directional curvature to a smaller maximum Hessian eigenvalue, and that its proof relied on the assumption that the gradient norms of the two factors are comparable, which is rough and may not hold in general. To address this, we add the assumption that the block with the smaller maximum curvature, $a^\dagger:=\arg\min_a\lambda_{\max}(H_a^{(n)})$, **is selected by AS-LoRA with probability at least 1/2**. Since this assumption is stated directly in terms of the maximum eigenvalue, it removes the jump from directional curvature. Under the stochastic selection of Eq. (4) of the revised manuscript with effective temperature $\tau^t(n)=T^t(\lvert\widehat S_A^t(n)\rvert+\lvert\widehat S_B^t(n)\rvert)$, it takes the analytic form $$\mathbb{E}\left[ \tanh\left( \frac{\widehat S^t_{a^\dagger}(n)-\widehat S^t_{a'}(n)} {2\tau^t(n)} \right) \right]\ge 0,$$ where $a'$ is the other block. This condition explicitly weighs the curvature against the gradient norm. For example, even when $\|g_{a'}\|^2>\|g_{a^\dagger}\|^2$, the assumption holds, i.e., the flatter block $a^\dagger$ is preferred, whenever $$\frac{\|g_{a'}\|^2}{\|g_{a^\dagger}\|^2} \le \frac{1-\eta\lambda_{a^\dagger}/2} {1-\eta\lambda_{a'}/2}$$ holds almost surely, where $\lambda_a=g_a^\top H_ag_a/\|g_a\|^2$. We acknowledge that this assumption does not always hold. However, we believe it is natural in the late-training regime, in which AS-LoRA activates the curvature term ($T^{\ast}=90$ in our experiments): as training converges, the gradient norms of the two factors tend to become comparable, so the score gap is governed by the curvature penalty. The revised Theorem 4 and its proof in Appendix H provide the details. > **[W.3 / Q.3] Privacy guarantees: Scoring from privatized gradients and composition across rounds** We thank the reviewer for this question. In our private experiments, we follow Algorithm 3 and **compute scores only from gradients clipped and privatized by DP-SGD**. The original presentation did not clearly distinguish the ideal score $S$ used in the analysis from the estimated score $\widehat{S}$ used for selection. We have clarified this distinction in Appendix C, including Remark 1, and revised Appendix D to base the privacy argument on the estimated scores. These scores are computed from privatized gradients using the secant estimator, without evaluating a Hessian on the training minibatch. Score computation and the subsequent randomized selection are therefore post-processing of DP-SGD outputs. Appendix I.4 incorrectly stated that the PrivacyEngine is reinitialized each round. Only the local model and optimizer are reinitialized. **Each client's PrivacyEngine is initialized once and reused across rounds**, preserving its accountant state for cumulative privacy accounting over the full training run. We have corrected Appendix I.4 and added Table 9, which reports each client's dataset size $N_k$, sampling rate $q_k=B/N_k$, and noise multiplier $\sigma_k$ for all datasets.
AuthorsOpenReview ↗

Response to Reviewer h453 (2/2)

[W.4 / Q.4] Theorems 3 and 4: Guarantees for stochastic selection with estimated and stale scores

Thank you for this question. When preparing the manuscript, we had derived Theorems 3 and 4 in two versions: one for greedy selection and one for the stochastic (temperature-scaled softmax) selection used in our algorithm. We presented the greedy version because we believed it highlighted the effect of the proposed score more clearly. Based on your comment, and to close the gap between the algorithm and the theory, we have replaced the greedy version with the stochastic selection of Eq. (4) of the revised manuscript.

The revised analysis also reflects the error due to staleness and estimation as in the algorithm, modeling it as et(n):=max⁡a∣S^at(n)−Sat(n)∣e^t(n):=\max_a\lvert\widehat S_a^t(n)-S_a^t(n)\rvert, where S^at(n)\widehat S_a^t(n) is the estimated score used for selection (for an inactive factor, the one from its last active round) and Sat(n)S_a^t(n) is the score at the current iterate. Hence the revised Theorems 3 and 4 are valid for stochastic selection based on stale and estimated scores. However, we do not explicitly model the privatized gradients and the secant approximation beyond this error term. Tracking how the DP-SGD noise and the secant approximation error propagate and accumulate is highly complex, and we were concerned that it would distract from the main messages of Theorems 3 and 4.

Theorems 1 and 2 do not rely on the accuracy or freshness of the selection scores. And we revised Theorems 3 and 4 as described above. Thus, all four theorems in the revised manuscript apply to stochastic selection based on estimated and potentially stale scores under their respective assumptions.

[W.5 / Q.5] Ablations for score-driven selection and late-stage curvature

We thank the reviewer for this valuable suggestion.

First, to examine the contribution of score-driven selection beyond layer-wise selection, we compare random selection with Secant (T∗=90T^*=90). To compare learning speed rather than only the performance after a specific number of rounds, we plot the performance of both methods over training rounds and add this plot to Appendix J.5 (Figure 9) of the revised manuscript. The plot shows that score-driven mode selection is superior in final performance and also learns much faster.

Next, we measure the performance of three variants: first-order scoring throughout training (T∗=100T^{\ast}=100), the curvature score included from round 90 (Secant (T∗=90T^{\ast}=90)), and the curvature score used from the start (T∗=0T^{\ast}=0). Including the curvature score improves performance, and including it earlier gives a larger improvement. This confirms the contribution of also considering curvature. Also, we remark that the late-stage curvature term remains meaningful because it serves as a trade-off between accuracy and scoring cost.

[Table R1. Comparison of curvature estimators and activation schedules on MNLI with DP-SGD training (ϵ=3\epsilon=3).]

MethodDP guaranteeMNLI (m)MNLI (mm)
FD (T∗=90T^{\ast}=90)No79.00±0.3079.00 \pm 0.3079.78±0.2579.78 \pm 0.25
HVP (T∗=90T^{\ast}=90)No79.36±0.2779.36 \pm 0.2780.62±0.1380.62 \pm 0.13
Secant (T∗=100T^{\ast}=100)Yes78.36±1.1078.36 \pm 1.1079.41±1.0279.41 \pm 1.02
Secant (T∗=90T^{\ast}=90)Yes78.83±0.2778.83 \pm 0.2779.97±0.5579.97 \pm 0.55
Secant (T∗=0T^{\ast}=0)Yes79.27±0.3179.27 \pm 0.3180.40±0.4180.40 \pm 0.41

Table R1 extends the original Table 2 by adding Secant (T∗=100T^{\ast}=100) and Secant (T∗=0T^{\ast}=0). It also appears as Table 2 in the revised manuscript.

Thank you for the detailed response and the additional experiments. After reading the revised manuscript and your replies, I have three remaining concerns.

  1. Novelty. Several prior works have addressed aggregation error in federated LoRA through single-factor updates or alternating schedules. As I understand it, the main contribution of AS-LoRA is to decide which factor to update in each layer and communication round using scores estimated from privatized gradients. I see this as an incremental extension of existing single-factor update methods. The paper would benefit from a clearer account of what this adaptive rule adds beyond prior approaches.

  2. Comparison with LA-LoRA. LA-LoRA is a relevant baseline because it also studies alternating factor updates in differentially private federated LoRA. It alternates between updating AA and BB at successive local steps and averages both factors at the server. AS-LoRA instead selects one factor per layer for the entire communication round. It would be helpful to include LA-LoRA alongside FFA-LoRA and RoLoRA. If feasible, please use the same privacy budget, LoRA rank, and number of local training steps.

  3. Reliability of comparing scores from different rounds. For a given layer, the score for AA may have just been updated, while the score for BB may be several rounds old. If AA scores higher, it is unclear whether updating AA would yield a larger loss reduction at the current model state or whether the older score for BB no longer reflects that state. The comparison with random selection evaluates the overall strategy, but leaves this question open. Could the authors compare the score-based ranking with the observed loss reductions from updating AA and BB separately from the same checkpoint, and report how often the rankings agree as the older score becomes more stale?

Reviewer_z88GOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AS-LoRA, a method for differentially private federated fine-tuning with LoRA. The motivation is that LoRA's multiplicative parameterization can amplify aggregation error and DP noise. AS-LoRA therefore selects which LoRA component is active per layer and per communication round, using a curvature-aware score from a second-order loss approximation. The paper provides privacy, reconstruction, convergence, and flatness analyses, and reports experiments across several privacy budgets.

Critical strengths
  1. A notable strength of the paper is its departure from fixed or globally synchronized LoRA update schedules. The layer-wise and round-wise selection is well motivated, and the curvature-aware score provides a plausible criterion for DP federated fine-tuning.
  2. The empirical evaluation spans multiple language and vision benchmarks and several DP budgets. Appendix J.4 further examines data heterogeneity and LoRA rank. The comparisons with FedLoRA, FFA-LoRA, RoLoRA, and FedSVD are informative, and the analyses of selection granularity, sharpness, computation, and communication add useful practical evidence.
Critical weaknesses

The main weakness is scale. Using six clients for language tasks and three for vision tasks, with full participation in every round, provides only a narrow test of an FL method. AS-LoRA aggregates client-level scores; with more clients and partial participation, these scores may become noisy or biased, and stale or absent clients may alter component selection. It therefore remains unclear whether the method is still effective when participation is uneven and local data are heterogeneous.

Questions for authors
  1. Additional experiments with a larger number of clients and partial client participation would strengthen the evaluation. In particular, it would be helpful to understand how aggregated score reliability, component selection, and final model quality change with participation rates and data heterogeneity.
  2. The rationale for activating the curvature term only during the last 10 of 100 communication rounds is not fully clear. An ablation comparing first-order-only selection, late-stage curvature-aware selection, and curvature-aware selection throughout training would help clarify the contribution of the curvature term. A sensitivity analysis over the activation round T* would further support this design choice.
  3. In Eqs. (2) and (3), component scores are estimated from privatized gradients, while inactive components retain their scores from the last active round. It would be useful to evaluate how DP noise and score staleness affect component selection. Experiments across different privacy budgets and inactive durations would help establish the robustness of the proposed mechanism.
3 comments
AuthorsOpenReview ↗

Response to Reviewer z88G (1/1)

[W.1 / Q.1] Evaluation scale: partial participation, and data heterogeneity

Thank you for this suggestion. We have added an experiment with a client participation rate of 0.5 on MNLI under ϵ=3\epsilon=3, keeping the total number of clients and the data partition fixed. As shown in Table R1, AS-LoRA outperforms the strongest baseline, FFA-LoRA, by 14.17 and 13.25 pp on MNLI (m) and MNLI (mm), respectively, with lower variability across runs. These results are included in Appendix J.7, Table 13 of the revised manuscript.

[Table R1. Performance comparison on MNLI under DP-SGD (ϵ=3\epsilon=3) with a client participation rate of 0.5.]

MethodMNLI (m)MNLI (mm)
FedLoRA57.90±16.1557.90 \pm 16.1559.26±16.5159.26 \pm 16.51
FFA-LoRA61.83±10.1561.83 \pm 10.1563.68±9.8163.68 \pm 9.81
RoLoRA58.37±11.2958.37 \pm 11.2959.53±11.3759.53 \pm 11.37
AS-LoRA76.00±3.46\mathbf{76.00 \pm 3.46}76.93±3.18\mathbf{76.93 \pm 3.18}

Separately, Appendix J.4, Figure 8(a) evaluates data heterogeneity using Dirichlet parameters α ∈ {0.1, 0.5, 1.0}. AS-LoRA outperforms the baselines across these settings, with larger gains under stronger heterogeneity.

[Q.2] Curvature ablation: Choosing when to activate the curvature term

Thank you for this suggestion. We use curvature only in the last 10 rounds to reduce the additional score-computation cost. To examine this choice, we have added first-order-only selection (T∗=100T^{\ast}=100) and curvature-aware selection throughout training (T∗=0T^{\ast}=0) to the comparison with our default setting (T∗=90T^{\ast}=90) in Table 2 of Section 5.

[Table R2. Comparison of curvature estimators and activation schedules on MNLI with DP-SGD training (ϵ=3\epsilon=3).]

MethodDP guaranteeMNLI (m)MNLI (mm)
FD (T∗=90T^{\ast}=90)No79.00±0.3079.00 \pm 0.3079.78±0.2579.78 \pm 0.25
HVP (T∗=90T^{\ast}=90)No79.36±0.2779.36 \pm 0.2780.62±0.1380.62 \pm 0.13
Secant (T∗=100T^{\ast}=100)Yes78.36±1.1078.36 \pm 1.1079.41±1.0279.41 \pm 1.02
Secant (T∗=90T^{\ast}=90)Yes78.83±0.2778.83 \pm 0.2779.97±0.5579.97 \pm 0.55
Secant (T∗=0T^{\ast}=0)Yes79.27±0.3179.27 \pm 0.3180.40±0.4180.40 \pm 0.41

Compared with first-order-only selection, T∗=90T^{\ast}=90 improves mean accuracy by 0.47 and 0.56 pp on MNLI (m) and MNLI (mm), respectively. Using curvature throughout training further improves accuracy by 0.44 and 0.43 pp over T∗=90T^{\ast}=90, but requires curvature-score computation in all 100 rounds rather than only 10. We therefore retain T∗=90T^{\ast}=90 as a compromise between accuracy and scoring cost. The revised Section 5 includes these results and explains the activation schedule.

[Q.3] Component selection with DP noise and stale scores

We thank the reviewer for this suggestion. Tables 3 and 4 show that AS-LoRA maintains competitive performance under a stricter privacy budget (ϵ=1\epsilon=1), indicating robustness to increased DP noise.

Regarding score staleness, Appendix J.2 compares our single-pass design with a two-pass variant that recomputes both component scores using the current global model. The two-pass variant improves MNLI-m and MNLI-mm by only 0.54 and 0.32 pp, respectively, at the cost of additional communication and computation. These results suggest a practical trade-off between score freshness and efficiency.

The revised Theorem 3 also accounts for estimation and staleness through the score error et(n):=max⁡a∣S^at(n)−Sat(n)∣e^t(n):=\max_a\lvert\widehat{S}_a^t(n)-S_a^t(n)\rvert, where S^at(n)\widehat{S}_a^t(n) is the score used for selection and Sat(n)S_a^t(n) is the score at the current iterate. For an inactive component, the former is retained from its last active round. This discrepancy enters the convergence bound explicitly. The analysis does not assume that retained scores remain accurate. The details are given in Section 4 and Appendix G.

The authors have addressed most of my concerns, although experiments with a larger number of clients are still missing. Overall, I am satisfied with the response and will keep my original rating.

We truly appreciate your constructive comments and engagement throughout the discussion. Your feedback has made our paper stronger. All the best:)

Reviewer_yCJiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies which LoRA factor to update during differentially private federated fine-tuning. AS-LoRA extends alternating single-factor updates by making a separate choice for each layer and communication round, shared by all clients. Clients estimate gradient magnitude and late-stage curvature from privatized gradients, and the server uses aggregated scores to select the next update modes. The method also uses random projection and a temperature schedule. Experiments cover language understanding, question answering, and image classification. The authors report improvements over fixed-factor and alternating baselines, with MNLI accuracy comparable to FedSVD and lower measured server aggregation cost.

Critical strengths
  1. Layer-wise selection extends RoLoRA's alternating framework by allowing different layers to update different factors while keeping the choice shared across clients. This provides scheduling flexibility while preserving exact aggregation of each layer's effective update.
  2. Computing scores from privatized gradients is a useful design for private adaptation. Algorithms 1–3 make adaptive selection possible without an additional query to private training data or an extra second-order backward pass.
  3. The reported MNLI comparison with FedSVD suggests a useful trade-off when server computation is constrained: AS-LoRA achieves comparable accuracy with much cheaper measured aggregation in the tested setup .
Critical weaknesses
  1. The T/(2N)T/(2N) upper bound on contracting rounds in Appendix E, Equations (33)–(34), does not establish a nonzero asymptotic error floor for layer-tied alternation.
  2. Theorem 4's proof overlooks that larger gradients can outweigh the curvature penalty. For L(a,b)=12(ab−1)2L(a,b)=\frac12(ab-1)^2 at (a,b)=(1,2)(a,b)=(1,2) with η=0.1\eta=0.1, Equation (1) gives SA=3.2>SB=0.95S_A=3.2>S_B=0.95 despite HA=4>HB=1H_A=4>H_B=1.
  3. Equation (3) does not fully debias the secant numerator. For an unclipped scalar quadratic with curvature hh and independent gradient noise of variance v2v^2, each corrected numerator term retains an extra η2hv2\eta^2 h v^2 in expectation because the next gradient depends on the previous noisy update.
  4. Appendix I.4 resets the privacy engine each round without explaining composition over all Tτ=1000T\tau=1000 local steps or reporting noise multipliers. Appendix D's RDP calculation also omits the order dependence, leaving the full-run privacy budgets unclear.
Questions for authors
  1. What prevents accumulated contracting updates from driving the layer-tied schedule's error to zero as T→∞T\to\infty in the construction used for Theorem 2?

  2. What additional assumptions or weaker conclusion would make Theorem 4 valid when gradient magnitude and curvature favor different blocks?

  3. How does Equation (3)'s correction account for the dependence between consecutive gradients induced by the DP-noisy parameter update?

  4. Can you provide the per-client privacy-accounting records, including sampling rates, noise multipliers, and composition over all 10001000 local steps despite round-wise engine reinitialization?

8 comments
AuthorsOpenReview ↗

Response to Reviewer yCJi (1/2)

[W.1 / Q.1] Theorem 2: Matched updates and the asymptotic error floor under layer-tied alternation

Thank you for identifying this gap. We agree that Eqs. (33)–(34) of the original submission do not establish a nonzero asymptotic floor for layer-tied alternation, and we acknowledge an error in the RoLoRA part of Theorem 2. We focused on the fact that the number of mismatched rounds for layer n⋆n^\star grows as Ω(T)\Omega(T), and overlooked that the number of contracting rounds, T/(2N)T/(2N), also grows linearly in TT.

We have narrowed the scope of Theorem 2 and Corollary 1 accordingly, and justify layer-wise selection instead by a score comparison. For dn=SA(n)−SB(n)d_n=S_A^{(n)}-S_B^{(n)}, the average score advantage of independent per-layer maximization over the best common mode is exactly

&\frac{1}{N}\sum_{n=1}^N\max_{a\in\{A,B\}} S_a^{(n)} -\max_{a\in\{A,B\}}\frac{1}{N}\sum_{n=1}^N S_a^{(n)} \\ &\qquad =\frac{1}{2N}\left(\sum_{n=1}^N|d_n|-\left|\sum_{n=1}^N d_n\right|\right)\ge 0, \end{aligned}$$ with equality if and only if all $d_n$ share the same sign. Since $\eta S_a^{(n)}$ is the second-order estimate of the one-step loss decrease from updating block $a$ of layer $n$ (ignoring cross-layer curvature), it indicates a larger estimated descent for AS-LoRA. In line with the narrowed scope, we have revised all corresponding statements in the manuscript. We have also added the above score advantage of independent per-layer maximization, together with its derivation, immediately after Corollary 1. > **[W.2 / Q.2] Theorem 4: When gradient magnitude outweighs the curvature penalty** Thank you for the counterexample. When preparing the manuscript, we had derived Theorems 3 and 4 in two versions: one for greedy selection and one for the stochastic (temperature-scaled softmax) selection used in our algorithm. We presented the greedy version because we believed it highlighted the effect of the proposed score more clearly. Based on the reviewers' comments, and to close the gap between the algorithm and the theory, the revised manuscript considers the stochastic selection of Eq. (4) of the revised manuscript for theory. In the original proof of Theorem 4, we handled the case where gradient magnitude and curvature favor different blocks by assuming that the gradient norms of the two blocks are comparable. This assumption can be regarded as rough and, as your example shows, does not hold in general. To make the proof rigorous, we instead assume that **the block with the smaller maximum curvature**, $a^\dagger:=\arg\min_a\lambda_{\max}(H_a^{(n)})$, **is selected by AS-LoRA with probability at least** $1/2$. Under the softmax rule with effective temperature $\tau^t(n)=T^t(|\widehat S_A^t(n)|+|\widehat S_B^t(n)|)$, the assumption can be written analytically as $$\mathbb{E}\left[ \tanh\left( \frac{\widehat S^t_{a^\dagger}(n)-\widehat S^t_{a'}(n)} {2\tau^t(n)} \right) \right]\ge 0,$$ where $a'$ is the other block. This condition explicitly weighs the curvature against the gradient norm. For example, even when $\|g_{a'}\|^2>\|g_{a^\dagger}\|^2$, the assumption holds, i.e., the flatter block $a^\dagger$ is preferred, whenever $$\frac{\|g_{a'}\|^2}{\|g_{a^\dagger}\|^2} \le \frac{1-\eta\lambda_{a^\dagger}/2} {1-\eta\lambda_{a'}/2}$$ holds almost surely, where $\lambda_a=g_a^\top H_ag_a/\|g_a\|^2$. In your example, $a^\dagger=B$ ($H_B=1<H_A=4$) and $S_B-S_A=0.95-3.2<0$, so the condition fails at every temperature and Theorem 4 makes no claim. We acknowledge that this assumption does not always hold. However, we believe it is natural in the late-training regime, in which AS-LoRA activates the curvature term ($T^*=90$ in our experiments): as training converges, the gradient norms of the two factors tend to become comparable, so the score gap is governed by the curvature penalty. The revised Theorem 4 and its proof in Appendix H provide the details.
AuthorsOpenReview ↗

Response to Reviewer yCJi (2/2)

[W.3 / Q.3] Equation (3): Dependence between consecutive gradients and residual secant bias

Thank you for this careful analysis. We agree that the corrected numerator retains η2hv2\eta^2hv^2 in the scalar example. If the target were the score of a noiseless (vanilla) SGD update computed from the population gradient and Hessian, as in Eq. (1), this term would indeed be a bias. However, since local training in our method is performed with DP-SGD, Eq. (3) aims to estimate the score of the actual DP-SGD update. For this target, the term is not a bias to be removed but a component that must be retained. We explain below how Eq. (3) accounts for the dependence between consecutive gradients from this perspective.

The error arises from applying the secant approximation to privatized gradients. Writing gˉi−1=ci−1+ξi−1\bar g_{i-1}=c_{i-1}+\xi_{i-1} with ci−1c_{i-1} the clipped batch gradient, si=−ηgˉi−1s_i=-\eta\bar g_{i-1}, and yi≈Hasi+ξi−ξi−1y_i\approx H_as_i+\xi_i-\xi_{i-1},

E⟨si,yi⟩=η2(⟨ci−1,Haci−1⟩+σξ2 tr(Ha))+ηFa.\mathbb{E}\langle s_i,y_i\rangle=\eta^2\big(\langle c_{i-1},H_ac_{i-1}\rangle+\sigma_\xi^2\,\mathrm{tr}(H_a)\big)+\eta F_a.

(i) The term ηFa\eta F_a arises because the same noise ξi−1\xi_{i-1} enters both sis_i and yiy_i. It is an artifact of the secant approximation that carries no curvature information, so Eq. (3) removes it exactly in expectation. Eq. (3) also removes the noise floor η2Fa\eta^2F_a from the denominator, so that the curvature term is normalized by the same signal energy ∥ci−1∥2\Vert c_{i-1}\Vert^2 as the first-order term Sg,aS_{g,a} in Eq. (2).

(ii) The term η2σξ2 tr(Ha)\eta^2\sigma_\xi^2\,\mathrm{tr}(H_a), which reduces to η2hv2\eta^2hv^2 in the scalar case, is the curvature response to the parameter displacement caused by the injected noise. This displacement actually occurs in the DP-SGD update, so Eq. (3) retains this term.

Writing cc for ci−1c_{i-1} and using ∇L≈c\nabla L\approx c, the score becomes

Sg,a(1−η2λ^a)≈∥c∥2−η2(⟨c,Hac⟩+σξ2 tr(Ha))≈−1η Eξ[L(W−ηgˉ)−L(W)],S_{g,a}\Big(1-\frac{\eta}{2}\hat\lambda_a\Big)\approx\Vert c\Vert^2-\frac{\eta}{2}\Big(\langle c,H_ac\rangle+\sigma_\xi^2\,\mathrm{tr}(H_a)\Big)\approx-\frac{1}{\eta}\,\mathbb{E}_\xi\big[L(W-\eta\bar g)-L(W)\big],

i.e., the expected one-step loss decrease of the actual DP-SGD update under the second-order approximation. Since the selected block is subsequently updated by DP-SGD, this is the appropriate criterion for block selection. In our theoretical analysis, the deviation of λ^a\hat\lambda_a from the noiseless Rayleigh quotient in Eq. (1) is absorbed into the score error et(n)e^t(n) of Theorem 3 (Remark 1).

In summary, Eq. (3) accounts for the dependence between consecutive gradients by removing the term introduced by the secant approximation while retaining the curvature cost of the actual DP-SGD update. We have clarified this target after Eq. (3) in the main text and added the full derivation in Appendix B.3 of the revised manuscript.

[W.4 / Q.4] Privacy accounting: Per-client calibration and composition across all 1,000 local steps

Thank you for raising this concern. Appendix I.4 of the original submission incorrectly stated that the PrivacyEngine is reinitialized at each communication round. In our implementation, only the local model and optimizer are reinitialized. Each client’s PrivacyEngine is initialized once and reused across rounds, without resetting its privacy accountant. Privacy loss is therefore accumulated over all 1,000 DP-SGD steps (100 rounds ×\times 10 local steps).

For each client, we calibrate the noise multiplier using the PRV accountant over these 1,000 steps with the target privacy budget (ϵ,δ)=(3,10−5)(\epsilon,\delta)=(3,10^{-5}). We have corrected Appendix I.4 and added Table 9, which reports the client-specific dataset sizes NkN_k, sampling rates qk=B/Nkq_k=B/N_k, and noise multipliers σk\sigma_k for all datasets.

Thanks for the revisions. The explanation of the secant estimator and the updated privacy accounting address my concerns on those points. I still have three questions about the revised theory:

  1. Theorem 4. In the final ten rounds, when the curvature term is active, how often across layers and rounds is the factor with the smaller maximum Hessian eigenvalue selected with probability at least 1/21/2?

  2. Corollary 1. The least-squares argument used in Appendix F gives the minimum reconstruction error over BB for a fixed AA. What bounds the extra reconstruction error at the actual iterate BTB_T in Eq. (41)?

  3. Theorem 3. The gain term in Eq. (11) tightens the bound only if Δ‾Ssoft>2eˉ\overline{\Delta}^{\mathrm{soft}}_S > 2\bar e. Do the experiments support this inequality when the scores used for selection are noisy and may be stale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.