- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the convergence of federated LoRA without assuming uniformly bounded LoRA factors. It establishes an communication upper bound for multi-step FedAvg-LoRA and a matching lower bound within the specified update class. It then introduces FedStabLoRA, which combines persistent client corrections with a proximal penalty and a corresponding correction scale, and establishes an guarantee. Experiments cover language and vision tasks, with additional studies of heterogeneity, local training, and partial participation.
Critical strengths
-
Removing the bounded-factor assumption addresses a meaningful gap in federated LoRA theory. The matching upper and lower bounds are the main contribution.
-
The experiments cover both language and vision tasks and show consistent improvements under heterogeneous data and partial participation.
-
The main text is clear and well-organized.
Critical weaknesses
-
The algorithmic novelty is somewhat limited. The method combines SCAFFOLD-style correction with a FedProx-style penalty, both established ideas. The main new element is the correction scale tailored to the penalized local updates, while the more substantial contribution lies in the convergence analysis.
-
The experiments support the method’s effectiveness, but provide limited direct evidence for factor-amplified client drift. The gains under stronger heterogeneity and more local updates are also consistent with ordinary client drift.
-
The correction-scale ablation keeps the learning rate and penalty coefficient selected for the full method. This supports the benefit of the proposed scale at that configuration, but leaves its advantage over a separately tuned variant unclear.
-
In Theorem 5.2, the communication bound grows linearly with the number of clients at a fixed participation ratio. The paper does not explain whether this dependence is intrinsic to the method or an artifact of the analysis.
-
The complexity results use the factor-gradient norm, which depends on the factor representation and can increase with factor norms even when the weight-space gradient is small. Appendix H acknowledges this dependence, but its role in the complexity gap remains unclear. This limits the practical interpretation of the lower bound.
Questions for authors
Please refer to the weaknesses above.
9 comments
3. Empirical Contribution: Improved Performance and Training Stability
Across seven language and vision benchmarks, FedStabLoRA achieves the highest mean accuracy among the compared methods, demonstrating its effectiveness in federated fine-tuning. FedStabLoRA also improves training stability at larger learning rates. As shown in Table 2, the correction-only variant diverges at , while FedStabLoRA remains stable. Furthermore, under the same learning rate and penalty coefficient, our penalty-aware correction scale improves MNLI-m accuracy from 82.99% to 85.07% compared with . These experiments demonstrate the practical benefits of the quadratic penalty and the proposed correction update.
Additional controlled comparison of the correction scale. Our original ablation evaluates the contributions of gradient correction and quadratic penalty. In particular, the +Both (NaiveScale) and FedStabLoRA variants use the same learning rate () and penalty coefficient (), differing only in the correction scale. We further evaluate an independently tuned FedStab-NaiveScale variant to examine whether hyperparameter tuning can close this performance gap.
| Ablation | Variant | Correction | Penalty | Correction Scale | Accuracy |
|---|---|---|---|---|---|
| Original | Base | No | No | -- | 75.62 ± 1.63 |
| Original | +Penalty | No | Yes | -- | 80.05 ± 0.34 |
| Original | +Correction | Yes | No | 81.25 ± 1.64 | |
| Original | +Both (NaiveScale) | Yes | Yes | 82.99 ± 0.47 | |
| Original | FedStabLoRA | Yes | Yes | Proposed | 85.07 ± 0.65 |
| Additional | FedStab-NaiveScale (Independently Tuned) | Yes | Yes | 83.22 ± 0.57 |
The proposed correction scale improves performance under matched hyperparameters. With the same learning rate () and penalty coefficient (), FedStabLoRA achieves 85.07%, outperforming the naive-scale variant (82.99%) by 2.08 percentage points. Since the two configurations differ only in the correction scale, this comparison isolates its contribution.
Independent tuning does not eliminate the performance gap. Tuning FedStab-NaiveScale independently improves its accuracy from 82.99% to 83.22%. However, it still trails the reported FedStabLoRA result by 1.85 percentage points, indicating that the observed advantage is not eliminated by hyperparameter tuning of the naive-scale variant.
Both correction and penalty contribute to the improvement. Adding the quadratic penalty improves the baseline from 75.62% to 80.05%. With the same learning rate and penalty coefficient, additionally applying gradient correction improves accuracy from 80.05% to 82.99%. Using the proposed correction scale further increases accuracy to 85.07%.
Overall, the controlled ablation isolates the benefit of the proposed correction scale, while the additional independently tuned comparison shows that the performance gap is not closed by tuning the naive-scale variant. These results support the effectiveness of our proposed penalty-aware correction scale.
Supporting material: Appendix I, Factor-Norm Growth and Factor-Amplified Client Drift, including Fig. 2, Fig. 3, and Table 18. Appendix J, Deterministic Synthetic Experiments, including Fig. 4, Fig. 5, Table 19, and Fig. 6.
Response to W2: Direct Evidence for Factor-Amplified Client Drift
We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.
We add Appendix I: Factor-Norm Growth and Factor-Amplified Client Drift, which directly examines the relationship between LoRA factorization scale and client-update drift.
The new appendix investigates two related questions about LoRA optimization in federated learning. First, how does the factor norm evolve as the local training budget increases, and does the observed factor-norm growth remain when the classification head is frozen? Second, does the scale of the LoRA factors affect how much local updates differ across clients?
1. Factor-Norm Growth across Different Local-Epoch Budgets
For LoRA layer , let and denote the two factors at communication round . We measure their aggregate squared Frobenius norm by
We measure for FedAvg, penalty only, correction only, and FedStabLoRA under local epochs per communication round.
Appendix I, Fig. 2 shows over 50 communication rounds. Factor-norm growth becomes substantially faster for FedAvg and penalty only as increases. The correction-only and FedStabLoRA remain markedly lower and grow more slowly.
These results suggest that gradient correction plays the primary role in limiting factor-norm growth in our experiments, while the quadratic penalty provides an additional stabilizing effect when combined with correction.
2. Factor-Norm Growth with a Trainable or Frozen Classification Head
We next examine whether factor-norm growth depends on jointly training the task-specific classification head and the LoRA factors.
Appendix I, Fig. 3 compares joint training with a setting in which the classification head is frozen. The factor norm grows substantially for FedAvg and SCAFFOLD in both settings, whereas the FedStabLoRA trajectory remains much lower. Freezing the classification head does not remove this growth. Thus, joint updates to the dense head are not necessary for the factor-norm growth observed here.
3. Controlled Factorization-Rescaling Experiment
The factor-norm growth phenomenon shows how the factors evolve during training, but it does not by itself determine whether factorization scale affects or amplifies client drift after local optimization. To examine this, we conduct a controlled rescaling experiment.
Specifically, we rescale the LoRA factors at a fixed server checkpoint while preserving their product. For a LoRA layer with effective update , consider
The effective update is unchanged:
Consequently, the model outputs and each client's loss at the initial point are unchanged; only the factor representation differs.
For each rescaled checkpoint, clients perform the same -step FedAvg local update. Let
denote client 's effective LoRA update after these local steps, and let
be the aggregation-weighted mean update. We measure client drift by the weighted variance of client updates around their weighted mean:
Equivalently,
Thus, measures client drift in effective weight space.
For each checkpoint, we fit a line to the three log-log pairs and denote its slope by
A slope near zero indicates little sensitivity of the measured client-update drift to factor norm. A slope near one corresponds to an approximately proportional relationship, and a slope greater than one indicates a superlinear relationship over these three points.
The MNLI experiment uses checkpoints from four methods: FedAvg, penalty only, correction only, and FedStabLoRA. For each source method, we use three local-epoch settings () and 11 checkpoints at rounds , giving 33 checkpoint-level slope estimates per method and 132 estimates in total. Every checkpoint is evaluated with the same FedAvg local-update budget.
Response to W1: Algorithmic Novelty of FedStabLoRA
We appreciate the reviewer’s careful reading and thoughtful feedback, and welcome the opportunity to clarify this point.
Our paper studies the convergence of federated LoRA with multiple local updates and introduces FedStabLoRA to address client drift in this setting. We clarify our theoretical, algorithmic, and empirical contributions below.
1. Theoretical Contribution: Convergence Analysis for Federated LoRA
Existing convergence analyses of SCAFFOLD [1] consider standard federated optimization, where clients directly optimize model parameters and the local objectives are assumed to be smooth in those parameters. Federated LoRA has a different optimization structure: clients update two low-rank factors B and A, while the model update depends on their product BA. Even when Fi(W) is smooth in weight space, the factorized objective Ji(B,A)=Fi(BA) is generally not globally smooth in (B,A), as its local smoothness depends on the factor norms. With multiple local steps, this dependence can amplify the differences between client trajectories. Consequently, existing SCAFFOLD convergence guarantees do not directly apply to federated LoRA. For example, FSLoRA assumes that the factor iterates remain in a bounded region [2, Assumption 4.1].
Our theoretical analysis establishes convergence guarantees for federated LoRA without this bounded-factor assumption. First, we prove matching O(ϵ−6) upper and Ω(ϵ−6) lower communication-complexity bounds for multi-step FedAvg(LoRA), where the lower bound also covers history-dependent learning rate. These results characterize the worst-case complexity caused by factor-amplified client drift. We then prove that our method achieves O(ϵ−4) communication complexity under both full and partial participation. Our analysis controls the growth of LoRA factors together with the local-update and correction errors, without requiring a uniform bound on the factors. Under uniform partial participation, the corresponding complexity is O(N3M−2ϵ−4).
2. Algorithmic Contribution: Penalty-Aware Correction Update
FedStabLoRA uses client corrections similar to SCAFFOLD and a quadratic penalty similar to FedProx [3]. Combining these two components is non-trivial: the penalty changes the relationship between the corrected gradient and the accumulated local update, so SCAFFOLD's original correction scale Kη is no longer appropriate. We derive a penalty-aware correction scale directly from the penalized local updates.
Specifically, each client returns its accumulated local update
Dit=Vi,Kt−Vt
and updates its correction state through
Yit+1=Yit−Dit/s.
The scale s converts the accumulated local update into an estimate of the corrected gradient
git=∇Ji(Vt)−Cit.
In SCAFFOLD Option II, s=Kη, since K local steps give Dit=−Kηgit when the corrected gradient is held fixed.
With the quadratic penalty, each local step additionally includes μ(Vi,kt−Vt), which pulls the local iterate toward the server iterate. To account for this term, let Δk=Vi,kt−Vt. Under the same fixed-gradient approximation, the local update satisfies
\qquad \Delta_0=0.$$ Applying this relation over $K$ steps gives $$D_i^t=-s_\mu g_i^t, \qquad s_\mu=\eta\sum_{k=0}^{K-1}(1-\eta\mu)^k.$$ Thus, the penalty changes the effective scale of the accumulated local update from $K\eta$ to $s_\mu$. Under our parameter conditions, $s_\mu<K\eta$ when $\mu>0$ and $K>1$. Retaining the original scale $K\eta$ would therefore underestimate the corrected gradient by a factor of $s_\mu/(K\eta)$. FedStabLoRA uses $s=s_\mu$, which preserves the gradient-estimation role of the correction update under the penalty. This relation is exact when the corrected gradient is fixed, and our analysis bounds the approximation error when gradients vary during local training. The proposed scale reduces to $K\eta$ when $\mu=0$ and requires no additional gradient evaluations. Under full participation, the $\mu=0$ special case is equivalent to SCAFFOLD Option II, as established in **Appendix G, Section G (Connection to SCAFFOLD Option II)**. **[1]** Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. [*SCAFFOLD: Stochastic Controlled Averaging for Federated Learning*](https://proceedings.mlr.press/v119/karimireddy20a.html). ICML, 2020. **[2]** Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G. Brinton. [*Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs*](https://arxiv.org/abs/2501.19389v4). ICML, 2026. **[3]** Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. [*Federated Optimization in Heterogeneous Networks*](https://arxiv.org/abs/1812.06127). MLSys, 2020.