acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Accelerating the Convergence of Federated LoRA

Abstract

Federated fine-tuning of large pretrained models often requires substantial communication and computation, making parameter-efficient methods such as LoRA attractive. A natural approach is FedAvg(LoRA), which combines FedAvg with LoRA. Despite its practical utility, the convergence theory of FedAvg(LoRA) remains limited: existing analyses typically assume bounded LoRA factors to ensure the smoothness of the factorized objective, an additional assumption not guaranteed by LoRA updates. More recent work on centralized LoRA shows that LoRA gradient descent achieves complexity without this assumption. This raises the question of whether FedAvg(LoRA) can similarly achieve communication complexity without this assumption. Unlike centralized LoRA, FedAvg(LoRA) performs multiple local steps that introduce client drift, and this drift can be amplified as the factor norms grow. We resolve this analytical challenge and establish an communication upper bound under full client participation. We then prove a lower bound for multiple local steps, even when the stepsize is chosen adaptively from the optimization history. To accelerate this communication complexity, we introduce FedStabLoRA, which uses persistent client corrections to reduce client drift and stabilize training via a quadratic penalty. FedStabLoRA achieves communication complexity without a bounded-factor assumption, and the rate also holds under partial participation. Across seven language and vision benchmarks, FedStabLoRA achieves the best performance among the compared methods. Its advantage persists under severe data heterogeneity, more local training, and partial participation.

Primary area:
optimization
Keywords:
LoRA, FL, Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HCTfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the convergence of federated LoRA without assuming uniformly bounded LoRA factors. It establishes an O(ϵ−6)O(\epsilon^{-6}) communication upper bound for multi-step FedAvg-LoRA and a matching lower bound within the specified update class. It then introduces FedStabLoRA, which combines persistent client corrections with a proximal penalty and a corresponding correction scale, and establishes an O(ϵ−4)O(\epsilon^{-4}) guarantee. Experiments cover language and vision tasks, with additional studies of heterogeneity, local training, and partial participation.

Critical strengths
  1. Removing the bounded-factor assumption addresses a meaningful gap in federated LoRA theory. The matching upper and lower bounds are the main contribution.

  2. The experiments cover both language and vision tasks and show consistent improvements under heterogeneous data and partial participation.

  3. The main text is clear and well-organized.

Critical weaknesses
  1. The algorithmic novelty is somewhat limited. The method combines SCAFFOLD-style correction with a FedProx-style penalty, both established ideas. The main new element is the correction scale tailored to the penalized local updates, while the more substantial contribution lies in the convergence analysis.

  2. The experiments support the method’s effectiveness, but provide limited direct evidence for factor-amplified client drift. The gains under stronger heterogeneity and more local updates are also consistent with ordinary client drift.

  3. The correction-scale ablation keeps the learning rate and penalty coefficient selected for the full method. This supports the benefit of the proposed scale at that configuration, but leaves its advantage over a separately tuned s=Kηs=K\eta variant unclear.

  4. In Theorem 5.2, the communication bound grows linearly with the number of clients at a fixed participation ratio. The paper does not explain whether this dependence is intrinsic to the method or an artifact of the analysis.

  5. The complexity results use the factor-gradient norm, which depends on the factor representation and can increase with factor norms even when the weight-space gradient is small. Appendix H acknowledges this dependence, but its role in the complexity gap remains unclear. This limits the practical interpretation of the lower bound.

Questions for authors

Please refer to the weaknesses above.

9 comments
AuthorsOpenReview ↗

Response to W1: Algorithmic Novelty of FedStabLoRA

We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.

Our paper studies the convergence of federated LoRA with multiple local updates and introduces FedStabLoRA to address client drift in this setting. We clarify our theoretical, algorithmic, and empirical contributions below.

1. Theoretical Contribution: Convergence Analysis for Federated LoRA

Existing convergence analyses of SCAFFOLD [1] consider standard federated optimization, where clients directly optimize model parameters and the local objectives are assumed to be smooth in those parameters. Federated LoRA has a different optimization structure: clients update two low-rank factors BB and AA, while the model update depends on their product BABA. Even when Fi(W)F_i(W) is smooth in weight space, the factorized objective Ji(B,A)=Fi(BA)J_i(B,A)=F_i(BA) is generally not globally smooth in (B,A)(B,A), as its local smoothness depends on the factor norms. With multiple local steps, this dependence can amplify the differences between client trajectories. Consequently, existing SCAFFOLD convergence guarantees do not directly apply to federated LoRA. For example, FSLoRA assumes that the factor iterates remain in a bounded region [2, Assumption 4.1].

Our theoretical analysis establishes convergence guarantees for federated LoRA without this bounded-factor assumption. First, we prove matching O(ϵ−6)O(\epsilon^{-6}) upper and Ω(ϵ−6)\Omega(\epsilon^{-6}) lower communication-complexity bounds for multi-step FedAvg(LoRA), where the lower bound also covers history-dependent learning rate. These results characterize the worst-case complexity caused by factor-amplified client drift. We then prove that our method achieves O(ϵ−4)O(\epsilon^{-4}) communication complexity under both full and partial participation. Our analysis controls the growth of LoRA factors together with the local-update and correction errors, without requiring a uniform bound on the factors. Under uniform partial participation, the corresponding complexity is O(N3M−2ϵ−4)O(N^3M^{-2}\epsilon^{-4}).

2. Algorithmic Contribution: Penalty-Aware Correction Update

FedStabLoRA uses client corrections similar to SCAFFOLD and a quadratic penalty similar to FedProx [3]. Combining these two components is non-trivial: the penalty changes the relationship between the corrected gradient and the accumulated local update, so SCAFFOLD's original correction scale KηK\eta is no longer appropriate. We derive a penalty-aware correction scale directly from the penalized local updates.

Specifically, each client returns its accumulated local update

Dit=Vi,Kt−VtD_i^t=V_{i,K}^t-V^t

and updates its correction state through

Yit+1=Yit−Dit/s.Y_i^{t+1}=Y_i^t-D_i^t/s.

The scale ss converts the accumulated local update into an estimate of the corrected gradient

git=∇Ji(Vt)−Cit.g_i^t=\nabla J_i(V^t)-C_i^t.

In SCAFFOLD Option II, s=Kηs=K\eta, since KK local steps give Dit=−KηgitD_i^t=-K\eta g_i^t when the corrected gradient is held fixed.

With the quadratic penalty, each local step additionally includes μ(Vi,kt−Vt)\mu(V_{i,k}^t-V^t), which pulls the local iterate toward the server iterate. To account for this term, let Δk=Vi,kt−Vt\Delta_k=V_{i,k}^t-V^t. Under the same fixed-gradient approximation, the local update satisfies

\qquad \Delta_0=0.$$ Applying this relation over $K$ steps gives $$D_i^t=-s_\mu g_i^t, \qquad s_\mu=\eta\sum_{k=0}^{K-1}(1-\eta\mu)^k.$$ Thus, the penalty changes the effective scale of the accumulated local update from $K\eta$ to $s_\mu$. Under our parameter conditions, $s_\mu<K\eta$ when $\mu>0$ and $K>1$. Retaining the original scale $K\eta$ would therefore underestimate the corrected gradient by a factor of $s_\mu/(K\eta)$. FedStabLoRA uses $s=s_\mu$, which preserves the gradient-estimation role of the correction update under the penalty. This relation is exact when the corrected gradient is fixed, and our analysis bounds the approximation error when gradients vary during local training. The proposed scale reduces to $K\eta$ when $\mu=0$ and requires no additional gradient evaluations. Under full participation, the $\mu=0$ special case is equivalent to SCAFFOLD Option II, as established in **Appendix G, Section G (Connection to SCAFFOLD Option II)**. **[1]** Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. [*SCAFFOLD: Stochastic Controlled Averaging for Federated Learning*](https://proceedings.mlr.press/v119/karimireddy20a.html). ICML, 2020. **[2]** Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G. Brinton. [*Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs*](https://arxiv.org/abs/2501.19389v4). ICML, 2026. **[3]** Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. [*Federated Optimization in Heterogeneous Networks*](https://arxiv.org/abs/1812.06127). MLSys, 2020.

3. Empirical Contribution: Improved Performance and Training Stability

Across seven language and vision benchmarks, FedStabLoRA achieves the highest mean accuracy among the compared methods, demonstrating its effectiveness in federated fine-tuning. FedStabLoRA also improves training stability at larger learning rates. As shown in Table 2, the correction-only variant diverges at η=0.3\eta=0.3, while FedStabLoRA remains stable. Furthermore, under the same learning rate and penalty coefficient, our penalty-aware correction scale improves MNLI-m accuracy from 82.99% to 85.07% compared with s=Kηs=K\eta. These experiments demonstrate the practical benefits of the quadratic penalty and the proposed correction update.

Additional controlled comparison of the correction scale. Our original ablation evaluates the contributions of gradient correction and quadratic penalty. In particular, the +Both (NaiveScale) and FedStabLoRA variants use the same learning rate (η=0.3\eta=0.3) and penalty coefficient (μ=0.03\mu=0.03), differing only in the correction scale. We further evaluate an independently tuned FedStab-NaiveScale variant to examine whether hyperparameter tuning can close this performance gap.

AblationVariantCorrectionPenaltyCorrection ScaleAccuracy
OriginalBaseNoNo--75.62 ± 1.63
Original+PenaltyNoYes--80.05 ± 0.34
Original+CorrectionYesNoKηK\eta81.25 ± 1.64
Original+Both (NaiveScale)YesYesKηK\eta82.99 ± 0.47
OriginalFedStabLoRAYesYesProposed85.07 ± 0.65
AdditionalFedStab-NaiveScale (Independently Tuned)YesYesKηK\eta83.22 ± 0.57

The proposed correction scale improves performance under matched hyperparameters. With the same learning rate (η=0.3\eta=0.3) and penalty coefficient (μ=0.03\mu=0.03), FedStabLoRA achieves 85.07%, outperforming the naive-scale variant (82.99%) by 2.08 percentage points. Since the two configurations differ only in the correction scale, this comparison isolates its contribution.

Independent tuning does not eliminate the performance gap. Tuning FedStab-NaiveScale independently improves its accuracy from 82.99% to 83.22%. However, it still trails the reported FedStabLoRA result by 1.85 percentage points, indicating that the observed advantage is not eliminated by hyperparameter tuning of the naive-scale variant.

Both correction and penalty contribute to the improvement. Adding the quadratic penalty improves the baseline from 75.62% to 80.05%. With the same learning rate and penalty coefficient, additionally applying gradient correction improves accuracy from 80.05% to 82.99%. Using the proposed correction scale further increases accuracy to 85.07%.

Overall, the controlled ablation isolates the benefit of the proposed correction scale, while the additional independently tuned comparison shows that the performance gap is not closed by tuning the naive-scale variant. These results support the effectiveness of our proposed penalty-aware correction scale.

Supporting material: Appendix I, Factor-Norm Growth and Factor-Amplified Client Drift, including Fig. 2, Fig. 3, and Table 18. Appendix J, Deterministic Synthetic Experiments, including Fig. 4, Fig. 5, Table 19, and Fig. 6.

AuthorsOpenReview ↗

Response to W2: Direct Evidence for Factor-Amplified Client Drift

We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.

We add Appendix I: Factor-Norm Growth and Factor-Amplified Client Drift, which directly examines the relationship between LoRA factorization scale and client-update drift.

The new appendix investigates two related questions about LoRA optimization in federated learning. First, how does the factor norm evolve as the local training budget increases, and does the observed factor-norm growth remain when the classification head is frozen? Second, does the scale of the LoRA factors affect how much local updates differ across clients?

1. Factor-Norm Growth across Different Local-Epoch Budgets

For LoRA layer ℓ\ell, let AℓtA_\ell^t and BℓtB_\ell^t denote the two factors at communication round tt. We measure their aggregate squared Frobenius norm RtR^t by

∑ℓ(∥Aℓt∥F2+∥Bℓt∥F2).\sum_\ell \left(\lVert A_\ell^t\rVert_F^2 + \lVert B_\ell^t\rVert_F^2\right).

We measure RtR^t for FedAvg, penalty only, correction only, and FedStabLoRA under E∈1,2,4E\in\\{1,2,4\\} local epochs per communication round.

Appendix I, Fig. 2 shows RtR^t over 50 communication rounds. Factor-norm growth becomes substantially faster for FedAvg and penalty only as EE increases. The correction-only and FedStabLoRA remain markedly lower and grow more slowly.

These results suggest that gradient correction plays the primary role in limiting factor-norm growth in our experiments, while the quadratic penalty provides an additional stabilizing effect when combined with correction.

2. Factor-Norm Growth with a Trainable or Frozen Classification Head

We next examine whether factor-norm growth depends on jointly training the task-specific classification head and the LoRA factors.

Appendix I, Fig. 3 compares joint training with a setting in which the classification head is frozen. The factor norm grows substantially for FedAvg and SCAFFOLD in both settings, whereas the FedStabLoRA trajectory remains much lower. Freezing the classification head does not remove this growth. Thus, joint updates to the dense head are not necessary for the factor-norm growth observed here.

3. Controlled Factorization-Rescaling Experiment

The factor-norm growth phenomenon shows how the factors evolve during training, but it does not by itself determine whether factorization scale affects or amplifies client drift after local optimization. To examine this, we conduct a controlled rescaling experiment.

Specifically, we rescale the LoRA factors at a fixed server checkpoint while preserving their product. For a LoRA layer with effective update Xt=BtAtX_t=B_tA_t, consider

Bt(c)=cBt,At(c)=1cAt,c∈0.5,1,2.B_t^{(c)}=cB_t, \qquad A_t^{(c)}=\frac{1}{c}A_t, \qquad c\in\\{0.5,1,2\\}.

The effective update is unchanged:

Bt(c)At(c)=BtAt=Xt.B_t^{(c)}A_t^{(c)}=B_tA_t=X_t.

Consequently, the model outputs and each client's loss at the initial point are unchanged; only the factor representation differs.

For each rescaled checkpoint, clients perform the same K=10K=10-step FedAvg local update. Let

ΔXi,K(c)=Xi,K(c)−Xt\Delta X_{i,K}^{(c)} = X_{i,K}^{(c)}-X_t

denote client ii's effective LoRA update after these local steps, and let

ΔX‾K(c)=∑i=1NpiΔXi,K(c)\overline{\Delta X}_{K}^{(c)} = \sum_{i=1}^{N}p_i\Delta X_{i,K}^{(c)}

be the aggregation-weighted mean update. We measure client drift by the weighted variance of client updates around their weighted mean:

DK(c)=∑i=1Npi∥ΔXi,K(c)−ΔX‾K(c)∥F2.D_K(c) = \sum_{i=1}^{N}p_i \left\lVert \Delta X_{i,K}^{(c)} - \overline{\Delta X}_{K}^{(c)} \right\rVert_F^2.

Equivalently,

DK(c)=12∑i=1N∑j=1Npipj∥ΔXi,K(c)−ΔXj,K(c)∥F2.D_K(c) = \frac{1}{2} \sum_{i=1}^{N}\sum_{j=1}^{N} p_ip_j \left\lVert \Delta X_{i,K}^{(c)} - \Delta X_{j,K}^{(c)} \right\rVert_F^2.

Thus, DK(c)D_K(c) measures client drift in effective weight space.

For each checkpoint, we fit a line to the three log-log pairs and denote its slope by

β^=slope⁡c∈0.5,1,2(log⁡Rt(c),,log⁡DK(c)).\widehat{\beta} = \operatorname{slope}_{c\in\\{0.5,1,2\\}} \left( \log R_t(c),\\, \log D_K(c) \right).

A slope near zero indicates little sensitivity of the measured client-update drift to factor norm. A slope near one corresponds to an approximately proportional relationship, and a slope greater than one indicates a superlinear relationship over these three points.

The MNLI experiment uses checkpoints from four methods: FedAvg, penalty only, correction only, and FedStabLoRA. For each source method, we use three local-epoch settings (E∈1,2,4E\in\\{1,2,4\\}) and 11 checkpoints at rounds 0,5,…,500,5,\ldots,50, giving 33 checkpoint-level slope estimates per method and 132 estimates in total. Every checkpoint is evaluated with the same K=10K=10 FedAvg local-update budget.

Reviewer_viVMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the convergence of FedAvg(LoRA) without assuming uniformly bounded LoRA factors. The authors first derive an O(ϵ−6)O(\epsilon^{-6}) upper bound on the communication complexity of FedAvg(LoRA) with multiple local steps and heterogeneous client objectives. They then establish a matching Ω(ϵ−6)\Omega(\epsilon^{-6}) worst-case lower bound, which holds even when the learning rate is selected adaptively based on the optimization history. To improve this convergence rate, the authors propose FedStabLoRA, which combines persistent client-specific gradient corrections with a quadratic penalty on local factor updates. The proposed method achieves an O(ϵ−4)O(\epsilon^{-4}) communication-complexity guarantee under full participation. The authors further extend the analysis to uniform partial participation, yielding an O(N3M−2ϵ−4)O(N^3M^{-2}\epsilon^{-4}) bound. The experimental evaluation spans seven language and vision benchmarks. FedStabLoRA achieves the highest mean accuracy among the compared methods on all seven tasks. Additional experiments examine client heterogeneity, local training budgets, partial participation, larger language models, component ablations, and communication overhead.

Critical strengths
  1. A substantial theoretical characterization of FedAvg(LoRA).

    The paper establishes both an upper bound and a matching worst-case lower bound for FedAvg(LoRA) with multiple local steps, with the latter even accounting for history-dependent learning-rate selection. This is a meaningful contribution: an upper bound alone would not show that the O(ϵ−6)O(\epsilon^{-6}) rate is unavoidable within the considered algorithm class. The analysis is detailed and tackles the lack of global smoothness in the factorized LoRA objective, providing a valuable foundation for understanding convergence under client heterogeneity.

  2. Consistent empirical improvements supported by component ablations.

    FedStabLoRA attains the highest mean accuracy on all seven evaluated benchmarks and performs favorably under stronger data heterogeneity, additional local training, and partial participation. The component ablation indicates that both the gradient correction and the quadratic penalty contribute to performance. Notably, the quadratic penalty stabilizes corrected LoRA updates at larger learning rates. These results support the proposed training procedure, though the scope of the comparisons remains a concern.

Critical weaknesses
  1. Insufficient explanation of correction and sampling errors in the main convergence analysis.

    Theorems 5.1 and 5.2 establish improved convergence rates using compact constants C2C_2 and C4C_4. Although Appendices D and E provide detailed proofs, the main text does not adequately explain how client-specific gradient corrections, local-update errors, and partial-participation sampling errors affect the convergence bounds.

    These mechanisms are central to FedStabLoRA, yet their effects are largely absorbed into constants in the main theoretical statements. A clearer presentation of the intermediate error bounds and their dependence on algorithmic parameters is needed to understand the source of the claimed convergence improvement.

  2. Limited theoretical justification and empirical isolation of the quadratic penalty.

    Theorems 5.1 and 5.2 establish the same O(ϵ−4)O(\epsilon^{-4}) communication complexity when the quadratic penalty is removed by setting μ=0\mu=0. Moreover, Appendix G establishes the equivalence of this special case to SCAFFOLD Option II under full participation. Therefore, the theoretical analysis does not establish a distinct asymptotic convergence benefit from the quadratic penalty.

    Although Table 2 suggests that the penalty improves stability, the ablation does not fully isolate its contribution from the correction scale and learning-rate selection. More controlled comparisons with independently tuned configurations are needed to establish the additional benefit of the proposed penalty-dependent correction mechanism.

  3. A substantial gap between the theoretical guarantees and experimental validation.

    The convergence analysis assumes exact local gradients and requires the effective step size to satisfy s=c1/Ts=c_1/\sqrt{T}, with c1≤1c_1\leq1. However, the experimental configurations do not follow this theoretically guaranteed parameter regime. For example, Appendix F, Table 9 reports s=33.11s=33.11 for MNLI-m, which substantially exceeds the theoretical upper limit for T=50T=50.

    Furthermore, the theoretical analysis considers deterministic local gradients, whereas the experiments use minibatch SGD. The paper does not establish convergence guarantees for these experimental settings.

    The experiments mainly report predictive accuracy under fixed communication budgets rather than factor-gradient stationarity or its dependence on ϵ\epsilon. Consequently, the reported results do not directly validate the claimed improvement from O(ϵ−6)O(\epsilon^{-6}) to O(ϵ−4)O(\epsilon^{-4}) communication complexity.

  4. Limited algorithmic novelty and insufficient comparisons with recent federated LoRA methods.

    FedStabLoRA combines SCAFFOLD-style gradient corrections with a quadratic penalty similar to proximal regularization. Since the same theoretical convergence rate holds without the penalty, the additional algorithmic contribution relies primarily on its empirical benefits.

    However, the main comparisons are dominated by established federated optimization methods such as FedDyn and SCAFFOLD. Although FedEx-LoRA and FFA-LoRA are included, their performance is substantially below the strongest baselines in Table 1. Other recent federated LoRA approaches discussed in Section 2, including FSLoRA, are not evaluated.

    As a result, the experiments establish improvements over the selected baselines but provide insufficient evidence of competitiveness against a broader range of recent federated LoRA methods, weakening the practical significance of the proposed algorithmic contribution.

Questions for authors
  1. Relationship between the theoretical analysis and experimental settings.

    Theorems 5.1 and 5.2 assume exact local gradients and use horizon-dependent effective step sizes, whereas the experiments use constant-step minibatch SGD. Can the authors clarify how the theoretical conditions relate to the implemented configurations? Can they provide convergence measurements based on factor-gradient norms, rather than only predictive accuracy, to support the claimed practical acceleration?

  2. Contribution of the quadratic penalty.

    The same O(ϵ−4)O(\epsilon^{-4}) theoretical guarantee holds when μ=0\mu=0, and Appendix G identifies this case with SCAFFOLD Option II under full participation. What additional convergence property of the penalized method is established by the analysis? Can the authors provide a more direct evaluation of the penalty's contribution to finite-round stability and performance, including comparisons under independently tuned hyperparameters?

  3. Competitiveness against more recent federated LoRA methods.

    Can the authors extend the main comparison to include additional federated LoRA approaches discussed in Section 2, such as FSLoRA, under comparable computational and communication budgets? Given the weaker performance of FedEx-LoRA and FFA-LoRA in Table 1, these comparisons are important for assessing the practical significance of the reported improvements.

  4. Interpretation of the full- and partial-participation guarantees.

    Can the authors explain more explicitly in the main text how the correction-tracking errors enter the Lyapunov analysis in Theorem 5.1 and how client sampling leads to the N3/2/MN^{3/2}/M factor in Theorem 5.2? This would clarify which effects are controlled by the analysis and the practical implications of the partial-participation bound.

9 comments
AuthorsOpenReview ↗

Response to W1: Correction Errors and Lyapunov Analysis

We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.

The key to our convergence analysis is to explicitly control local-update and correction errors and establish how they jointly affect convergence. The main steps underlying Theorem 5.1 are explained below.

1. Controlling Local-Update Errors

For client ii, let

git=∇Ji(Vt)−Citg_i^t=\nabla J_i(V^t)-C_i^t

denote its corrected gradient. If this gradient remained fixed during local training, the accumulated update would be Dit=−sgitD_i^t=-sg_i^t, where ss is the penalty-aware scale. Because the gradient changes across the KK local steps, the actual update differs from this idealized expression.

We define this deviation by ait:=git+Dit/sa_i^t:=g_i^t+D_i^t/s. Our local-update lemma (Appendix D, Section D.1: Local Update Approximation) proves that

Dit=−sgit+sait,∥ait∥F≤19∥git∥F,D_i^t=-sg_i^t+sa_i^t, \qquad \lVert a_i^t\rVert_F\le\frac19\lVert g_i^t\rVert_F,

under the stated smoothness, stepsize, and radius conditions. Thus, aita_i^t measures the inaccuracy of the accumulated local update, and its magnitude is bounded relative to the corrected gradient.

2. How Correction Updates Control Client Drift

The purpose of CitC_i^t is to estimate the difference between client ii's gradient and the global gradient. We measure the remaining mismatch by

Et=∑ipi∥git−∇J(Vt)∥F2=∑ipi∥Cit−[∇Ji(Vt)−∇J(Vt)]∥F2.\mathcal E_t =\sum_i p_i\lVert g_i^t-\nabla J(V^t)\rVert_F^2 =\sum_i p_i \lVert C_i^t-[\nabla J_i(V^t)-\nabla J(V^t)]\rVert_F^2.​

In other words, Et\mathcal E_t measures how accurately the corrections represent the client–global gradient differences. It is distinct from the local-update deviation aita_i^t.

Under full participation, the corrections satisfy

∑ipiCit=0.\sum_i p_iC_i^t=0.

Substituting the local-update decomposition (Dit=−sgit+saitD_i^t=-sg_i^t+sa_i^t) into the centered correction update gives

=\nabla J_i(V^t)-\nabla J(V^t) -(a_i^t-\bar a^t),$$ where $$\bar a^t=\sum_i p_i a_i^t.$$ **This equation is the key step:** after the correction update, the difference between $C_i^{t+1}$ and the desired client–global gradient difference at $V^t$ is exactly $a_i^t-\bar a^t$. The previous correction mismatch does not remain as an additional term. At the next server iterate, the remaining mismatch satisfies $$g_i^{t+1}-\nabla J(V^{t+1}) =(a_i^t-\bar a^t)\ +\nabla(J_i-J)(V^{t+1}) -\nabla(J_i-J)(V^t).$$ This equation shows the two sources of the next-round correction mismatch: the deviation of the previous local update, bounded in Step 1, and the change in the client–global gradient difference as the server iterate moves, bounded by local smoothness. ### 3. Joint Lyapunov Descent and Factor Control To control these errors jointly, we construct the Lyapunov function $$\mathcal L_t=\Delta_t+\frac{s}{2}\mathcal E_t, \qquad \mathcal Q_t=\sum_i p_i\lVert g_i^t\rVert_F^2.$$ Since the correction states satisfy $\sum_i p_iC_i^t=0$, we have $$\mathcal Q_t=\lVert \nabla J(V^t)\rVert_F^2+\mathcal E_t.$$ The key insight is that the two error bounds are coupled through the same average local-update error $\bar a^t$. Specifically, the objective descent bound contains a cross term involving $\langle\nabla J(V^t),\bar a^t\rangle$, while the correction-tracking bound provides a complementary negative quadratic term $-\lVert \bar a^t\rVert^2$. When combined through the Lyapunov function, these terms control the cross term. The remaining local-update errors are bounded relative to $\mathcal Q_t$, and the additional terms arising from gradient variation are absorbed under the stated stepsize and smoothness conditions. Consequently, we obtain $$\mathcal L_{t+1} \le \mathcal L_t-\frac{s}{4}\mathcal Q_t.$$ Importantly, this descent also enables us to verify the required local smoothness conditions through an induction that controls both accumulated local updates and the growth of LoRA factors. Summing the Lyapunov descent inequality with $s=\Theta(T^{-1/2})$ yields the stated $O(\epsilon^{-4})$ communication complexity. **Appendix reference:** Appendix D, *Proof of the Full-Participation Guarantee for FedStabLoRA*, particularly Section D.1, *Local Update Approximation*, and Section D.2, *Lyapunov Descent*.
AuthorsOpenReview ↗

Response to W2 and Q2: Contribution of the Quadratic Penalty

We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.

FedStabLoRA uses client corrections similar to SCAFFOLD [1] and a quadratic penalty similar to FedProx [2]. Combining these two components is non-trivial: the penalty changes the relationship between the corrected gradient and the accumulated local update, so SCAFFOLD's original correction scale KηK\eta is no longer appropriate. We derive a penalty-aware correction scale directly from the penalized local updates.

Specifically, each client returns its accumulated local update

Dit=Vi,Kt−VtD_i^t=V_{i,K}^t-V^t

and updates its correction state through

Yit+1=Yit−Dit/s.Y_i^{t+1}=Y_i^t-D_i^t/s.

The scale ss converts the accumulated local update into an estimate of the corrected gradient

git=∇Ji(Vt)−Cit.g_i^t=\nabla J_i(V^t)-C_i^t.

In SCAFFOLD Option II, s=Kηs=K\eta, since KK local steps give Dit=−KηgitD_i^t=-K\eta g_i^t when the corrected gradient is held fixed.

With the quadratic penalty, each local step additionally includes μ(Vi,kt−Vt)\mu(V_{i,k}^t-V^t), which pulls the local iterate toward the server iterate. To account for this term, let Δk=Vi,kt−Vt\Delta_k=V_{i,k}^t-V^t. Under the same fixed-gradient approximation, the local update satisfies

\qquad \Delta_0=0.$$ Applying this relation over $K$ steps gives $$D_i^t=-s_\mu g_i^t,\qquad s_\mu=\eta\sum_{k=0}^{K-1}(1-\eta\mu)^k.$$ Thus, the penalty changes the effective scale of the accumulated local update from $K\eta$ to $s_\mu$. Under our parameter conditions, $s_\mu<K\eta$ when $\mu>0$ and $K>1$. Retaining the original scale $K\eta$ would therefore underestimate the corrected gradient by a factor of $s_\mu/(K\eta)$. FedStabLoRA uses $s=s_\mu$, which preserves the gradient-estimation role of the correction update under the penalty. This relation is exact when the corrected gradient is fixed, and our analysis bounds the approximation error when gradients vary during local training. The proposed scale reduces to $K\eta$ when $\mu=0$ and requires no additional gradient evaluations. ### 1. Theoretical Role of the Penalty Theorems 5.1 and 5.2 establish the same $O(\epsilon^{-4})$ communication complexity when the quadratic penalty is removed by setting $\mu=0$. Moreover, **Appendix G, Section G: Connection to SCAFFOLD Option II**, establishes the equivalence of this special case to SCAFFOLD Option II under full participation. Therefore, our theoretical contribution is not a distinct asymptotic convergence exponent introduced by the quadratic penalty. ### 2. Controlled Ablation of the Penalty-Aware Correction Scale Our original ablation evaluates the contributions of gradient correction and quadratic penalty. In particular, the `+Both (NaiveScale)` and `FedStabLoRA` variants use the same learning rate ($\eta=0.3$) and penalty coefficient ($\mu=0.03$), differing only in the correction scale. This controlled comparison isolates the effect of the proposed scale. We further evaluate an independently tuned `FedStab-NaiveScale` variant to examine whether hyperparameter tuning can close this performance gap. |Ablation|Variant|Correction|Penalty|Correction Scale|Accuracy| |---|---|---|---|---|---| |Original|Base|No|No|--|75.62 ± 1.63| |Original|+Penalty|No|Yes|--|80.05 ± 0.34| |Original|+Correction|Yes|No|$K\eta$|81.25 ± 1.64| |Original|+Both (NaiveScale)|Yes|Yes|$K\eta$|82.99 ± 0.47| |Original|FedStabLoRA|Yes|Yes|Proposed|**85.07 ± 0.65**| |Additional|FedStab-NaiveScale (Independently Tuned)|Yes|Yes|$K\eta$|83.22 ± 0.57| **The results reveal three key findings.** 1. **The proposed correction scale improves performance under matched hyperparameters.** With the same learning rate ($\eta=0.3$) and penalty coefficient ($\mu=0.03$), FedStabLoRA achieves 85.07%, outperforming the naive-scale variant (82.99%) by 2.08 percentage points. Since the two configurations differ only in the correction scale, this comparison isolates its contribution. 2. **Independent tuning does not eliminate the performance gap.** Tuning FedStab-NaiveScale independently improves its accuracy from 82.99% to 83.22%. However, it still trails the reported FedStabLoRA result by 1.85 percentage points, indicating that the observed advantage is not eliminated by hyperparameter tuning of the naive-scale variant. **[1]** Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. [*SCAFFOLD: Stochastic Controlled Averaging for Federated Learning*](https://proceedings.mlr.press/v119/karimireddy20a.html). ICML, 2020. **[2]** Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. [*Federated Optimization in Heterogeneous Networks*](https://arxiv.org/abs/1812.06127). MLSys, 2020.
  1. Both correction and penalty contribute to the improvement. Adding the quadratic penalty improves the baseline from 75.62% to 80.05%. With the same learning rate and penalty coefficient, additionally applying gradient correction improves accuracy from 80.05% to 82.99%. Using the proposed correction scale further increases accuracy to 85.07%.

Overall, the controlled ablation isolates the benefit of the proposed correction scale, while the additional independently tuned comparison shows that the performance gap is not closed by tuning the naive-scale variant. These results support the effectiveness of our proposed penalty-aware correction scale.

Appendix reference: Appendix G, Connection to SCAFFOLD Option II. The empirical comparison is reported in the additional correction-scale ablation in the rebuttal.

Reviewer_kVucOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the convergence of federated LoRA fine-tuning without assuming uniformly bounded LoRA factors. To improve convergence, the paper proposes FedStabLoRA, which incorporates persistent client-specific corrections and quadratic regularization. The proposed method achieves an O(ϵ−4\epsilon^{-4}) communication complexity under full and partial client participation. Experiments on seven language and vision benchmarks demonstrate improvements over several federated learning baselines.

Critical strengths
  1. The paper addresses a meaningful theoretical challenge in federated LoRA optimization.

  2. The combination of upper and matching lower complexity bounds provides a potentially valuable characterization of multi-step FedAvg(LoRA).

  3. The experimental evaluation covers several language and vision tasks.

Critical weaknesses
  1. The convergence theorems are established for exact local gradients under specific step-size conditions. However, the experiments use stochastic mini-batch gradients, dropout, and substantially larger constant learning rates.

  2. The authors state that the task-specific classification head is trained densely and federated together with the LoRA factors. According to Appendix F.11, in RoBERTa-base, approximately 80% of the trainable parameters belong to the classification head. The authors should provide additional ablations that apply the proposed corrections separately to the LoRA factors and the classification head.

Questions for authors
  1. Can the authors clarify the algorithmic contribution of FedStabLoRA beyond combining SCAFFOLD-style correction and proximal regularization?

  2. How much of the observed improvement comes from applying the correction and penalty to LoRA factors versus the densely trained classification head?

5 comments
AuthorsOpenReview ↗

Response to W1: Gap between Theoretical Guarantees and Experimental Settings

We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.

The theoretical analysis considers exact local gradients and horizon-dependent step sizes, whereas the benchmark experiments use minibatch SGD, dropout, and constant learning rates. To provide direct numerical evidence under deterministic settings aligned with the theoretical analysis, we add Appendix J: Deterministic Synthetic Experiments.

Specifically, we use full-batch gradient descent, compute the exact gradient of each client's local objective at every step, and use full client participation. Dropout and minibatch sampling are disabled. This controlled setting retains heterogeneity across client objectives while removing stochastic-gradient and client-sampling noise. We evaluate two objectives: a Softmax-LoRA classification task with Dirichlet label skew (α=0.1\alpha=0.1) and a quadratic matrix-factorization task with heterogeneous client-specific target matrices. Both use ten clients and rank-three factorizations.

Effect of local steps. On Softmax-LoRA, we compare FedAvg with K∈1,2,5,20K\in\\{1,2,5,20\\} and FedStabLoRA with K=20K=20. As shown in Appendix J, Fig. 4, increasing the number of local steps leads to a higher final squared factor-gradient norm and larger factor norms for FedAvg. Specifically, the squared factor-gradient norm increases from 1.2×10−311.2\times10^{-31} at K=1K=1 to 2.91×10−12.91\times10^{-1} at K=20K=20, while the aggregate factor norm grows from 150150 to 3,5273{,}527. In contrast, FedStabLoRA at K=20K=20 maintains a squared factor-gradient norm of 1.9×10−301.9\times10^{-30} and an aggregate factor norm of 149149.

On quadratic matrix factorization, Appendix J, Fig. 5(a) shows that FedAvg reaches a near-zero squared factor-gradient norm for K=1K=1, while nonzero plateaus remain for K=2,5,20K=2,5,20. The corresponding final values are 1.1×10−61.1\times10^{-6}, 2.7×10−62.7\times10^{-6}, and 3.8×10−63.8\times10^{-6}. FedStabLoRA at K=20K=20 reaches approximately 2.2×10−262.2\times10^{-26}. For K=5K=5, the FedAvg gradient floor increases approximately quadratically with the per-round step size KηK\eta, with a fitted log-log slope of 1.92 (Appendix J, Fig. 5(b)).

Step-size sensitivity. We further fix K=5K=5 and vary the per-round step budget Kη∈4,2,1,0.5,0.25K\eta\in\\{4,2,1,0.5,0.25\\}. Appendix J, Table 19 reports the minimum and final squared factor-gradient norms, the final factor norm, and the first communication round at which each gradient threshold is reached. FedAvg's final squared factor-gradient norm ranges from 9.5×10−39.5\times10^{-3} to 1.71×10−21.71\times10^{-2} across the five step budgets. In contrast, FedStabLoRA maintains a final value of approximately 10−810^{-8} across all three settings. FedStabLoRA also reaches the 10−610^{-6} threshold in all three settings, whereas FedAvg does not reach this threshold in any of the five settings.

Horizon-dependent convergence trends. We additionally examine the average squared factor-gradient norm as the communication horizon TT increases, using step-size schedules motivated by our theoretical analysis. Appendix J, Fig. 6 reports the results for both synthetic tasks. For T≥1,000T\geq1{,}000, the fitted log-log slopes are −0.37-0.37 and −0.36-0.36 for FedAvg on the quadratic and Softmax-LoRA tasks, respectively, and approximately −0.50-0.50 for FedStabLoRA on both tasks. These finite-horizon slopes are close to the theoretical decay exponents of −1/3-1/3 for FedAvg and −1/2-1/2 for FedStabLoRA.

These deterministic experiments provide direct measurements of factor-gradient stationarity and numerical evidence consistent with the theoretical convergence rates. They isolate the effect of multiple local updates from stochastic-gradient noise and complement the predictive-accuracy evaluation in our seven benchmark experiments.

Supporting material: Appendix J, Deterministic Synthetic Experiments, including Fig. 4 (Softmax-LoRA), Fig. 5 (quadratic matrix factorization), Table 19 (step-size sensitivity), and Fig. 6 (horizon-dependent convergence).

AuthorsOpenReview ↗

Response to Q1: Algorithmic Contribution of FedStabLoRA Beyond Combining SCAFFOLD-Style Correction and Proximal Regularization

We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.

Our paper studies the convergence of federated LoRA with multiple local updates and introduces FedStabLoRA to address client drift in this setting. We clarify our theoretical, algorithmic, and empirical contributions below.

1. Theoretical Contribution: Convergence Analysis for Federated LoRA

Existing convergence analyses of SCAFFOLD [1] consider standard federated optimization, where clients directly optimize model parameters and the local objectives are assumed to be smooth in those parameters. Federated LoRA has a different optimization structure: clients update two low-rank factors BB and AA, while the model update depends on their product BABA. Even when Fi(W)F_i(W) is smooth in weight space, the factorized objective Ji(B,A)=Fi(BA)J_i(B,A)=F_i(BA) is generally not globally smooth in (B,A)(B,A), as its local smoothness depends on the factor norms. With multiple local steps, this dependence can amplify the differences between client trajectories. Consequently, existing SCAFFOLD convergence guarantees do not directly apply to federated LoRA. For example, FSLoRA assumes that the factor iterates remain in a bounded region [2, Assumption 4.1].

Our theoretical analysis establishes convergence guarantees for federated LoRA without this bounded-factor assumption. First, we prove matching O(ϵ−6)O(\epsilon^{-6}) upper and Ω(ϵ−6)\Omega(\epsilon^{-6}) lower communication-complexity bounds for multi-step FedAvg(LoRA), where the lower bound also covers history-dependent learning rate. These results characterize the worst-case complexity caused by factor-amplified client drift. We then prove that our method achieves O(ϵ−4)O(\epsilon^{-4}) communication complexity under both full and partial participation. Our analysis controls the growth of LoRA factors together with the local-update and correction errors, without requiring a uniform bound on the factors. Under uniform partial participation, the corresponding complexity is O(N3M−2ϵ−4)O(N^3M^{-2}\epsilon^{-4}).

2. Algorithmic Contribution: Penalty-Aware Correction Update

FedStabLoRA uses client corrections similar to SCAFFOLD and a quadratic penalty similar to FedProx [3]. Combining these two components is non-trivial: the penalty changes the relationship between the corrected gradient and the accumulated local update, so SCAFFOLD's original correction scale KηK\eta is no longer appropriate. We derive a penalty-aware correction scale directly from the penalized local updates.

Specifically, each client returns its accumulated local update

Dit=Vi,Kt−VtD_i^t=V_{i,K}^t-V^t

and updates its correction state through

Yit+1=Yit−Dit/s.Y_i^{t+1}=Y_i^t-D_i^t/s.

The scale ss converts the accumulated local update into an estimate of the corrected gradient

git=∇Ji(Vt)−Cit.g_i^t=\nabla J_i(V^t)-C_i^t.

In SCAFFOLD Option II, s=Kηs=K\eta, since KK local steps give Dit=−KηgitD_i^t=-K\eta g_i^t when the corrected gradient is held fixed.

With the quadratic penalty, each local step additionally includes μ(Vi,kt−Vt)\mu(V_{i,k}^t-V^t), which pulls the local iterate toward the server iterate. To account for this term, let Δk=Vi,kt−Vt\Delta_k=V_{i,k}^t-V^t. Under the same fixed-gradient approximation, the local update satisfies

\qquad \Delta_0=0.$$ Applying this relation over $K$ steps gives $$D_i^t=-s_\mu g_i^t, \qquad s_\mu=\eta\sum_{k=0}^{K-1}(1-\eta\mu)^k.$$ Thus, the penalty changes the effective scale of the accumulated local update from $K\eta$ to $s_\mu$. Under our parameter conditions, $s_\mu<K\eta$ when $\mu>0$ and $K>1$. Retaining the original scale $K\eta$ would therefore underestimate the corrected gradient by a factor of $s_\mu/(K\eta)$. FedStabLoRA uses $s=s_\mu$, which preserves the gradient-estimation role of the correction update under the penalty. This relation is exact when the corrected gradient is fixed, and our analysis bounds the approximation error when gradients vary during local training. The proposed scale reduces to $K\eta$ when $\mu=0$ and requires no additional gradient evaluations. Under full participation, the $\mu=0$ special case is equivalent to SCAFFOLD Option II, as established in **Appendix G, Section G (Connection to SCAFFOLD Option II)**. **[1]** Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. [*SCAFFOLD: Stochastic Controlled Averaging for Federated Learning*](https://proceedings.mlr.press/v119/karimireddy20a.html). ICML, 2020. **[2]** Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G. Brinton. [*Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs*](https://arxiv.org/abs/2501.19389v4). ICML, 2026. **[3]** Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. [*Federated Optimization in Heterogeneous Networks*](https://arxiv.org/abs/1812.06127). MLSys, 2020.

3. Empirical Contribution: Improved Performance and Training Stability

Across seven language and vision benchmarks, FedStabLoRA achieves the highest mean accuracy among the compared methods, demonstrating its effectiveness in federated fine-tuning. FedStabLoRA also improves training stability at larger learning rates. As shown in Table 2, the correction-only variant diverges at η=0.3\eta=0.3, while FedStabLoRA remains stable. Furthermore, under the same learning rate and penalty coefficient, our penalty-aware correction scale improves MNLI-m accuracy from 82.99% to 85.07% compared with s=Kηs=K\eta. These experiments demonstrate the practical benefits of the quadratic penalty and the proposed correction update.

Additional controlled comparison of the correction scale. Our original ablation evaluates the contributions of gradient correction and quadratic penalty. In particular, the +Both (NaiveScale) and FedStabLoRA variants use the same learning rate (η=0.3\eta=0.3) and penalty coefficient (μ=0.03\mu=0.03), differing only in the correction scale. We further evaluate an independently tuned FedStab-NaiveScale variant to examine whether hyperparameter tuning can close this performance gap.

AblationVariantCorrectionPenaltyCorrection ScaleAccuracy
OriginalBaseNoNo--75.62 ± 1.63
Original+PenaltyNoYes--80.05 ± 0.34
Original+CorrectionYesNoKηK\eta81.25 ± 1.64
Original+Both (NaiveScale)YesYesKηK\eta82.99 ± 0.47
OriginalFedStabLoRAYesYesProposed85.07 ± 0.65
AdditionalFedStab-NaiveScale (Independently Tuned)YesYesKηK\eta83.22 ± 0.57

The proposed correction scale improves performance under matched hyperparameters. With the same learning rate (η=0.3\eta=0.3) and penalty coefficient (μ=0.03\mu=0.03), FedStabLoRA achieves 85.07%, outperforming the naive-scale variant (82.99%) by 2.08 percentage points. Since the two configurations differ only in the correction scale, this comparison isolates its contribution.

Independent tuning does not eliminate the performance gap. Tuning FedStab-NaiveScale independently improves its accuracy from 82.99% to 83.22%. However, it still trails the reported FedStabLoRA result by 1.85 percentage points, indicating that the observed advantage is not eliminated by hyperparameter tuning of the naive-scale variant.

Both correction and penalty contribute to the improvement. Adding the quadratic penalty improves the baseline from 75.62% to 80.05%. With the same learning rate and penalty coefficient, additionally applying gradient correction improves accuracy from 80.05% to 82.99%. Using the proposed correction scale further increases accuracy to 85.07%.

Overall, the controlled ablation isolates the benefit of the proposed correction scale, while the additional independently tuned comparison shows that the performance gap is not closed by tuning the naive-scale variant. These results support the effectiveness of our proposed penalty-aware correction scale.

Supporting material: Appendix I, Factor-Norm Growth and Factor-Amplified Client Drift, including Fig. 2, Fig. 3, and Table 18. Appendix J, Deterministic Synthetic Experiments, including Fig. 4, Fig. 5, Table 19, and Fig. 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.