- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the convergence of federated LoRA under client data heterogeneity, considering full-batch local gradient descent with full client participation, simultaneous updates of both LoRA factors, and separate factor averaging at the server. The authors analyze how weight-space gradient heterogeneity affects factor-space gradients and local client drift, deriving local-stability conditions and a one-round descent bound that depends on factor norms, local steps, and learning rates. Based on this analysis, the paper establishes an stationarity rate with harmonic learning rates without assuming uniformly bounded factor norms, and further establishes an rate with a horizon-dependent constant learning rate under additional uniform trajectory bounds. The experiments examine learning-rate selection, client heterogeneity, and LoRA factor scaling, evaluating two norm-based adaptive learning-rate strategies on CIFAR-10 with a ResNet-18 backbone and on commonsense reasoning tasks with Qwen2.5-1.5B-Instruct.
Critical strengths
-
A detailed theoretical analysis of federated LoRA convergence.
The paper derives factor-dependent heterogeneity and local-drift bounds and establishes non-asymptotic convergence guarantees. In particular, the harmonic learning-rate analysis controls trajectory growth without assuming uniformly bounded LoRA factors. This provides a useful theoretical characterization of standard federated LoRA.
-
Empirical observations on learning-rate and factor-representation sensitivity.
The experiments examine how learning rates, client heterogeneity, and equivalent LoRA factor scalings affect optimization. The reported factor-norm and gradient-norm diagnostics provide useful observations about training stability.
Critical weaknesses
-
Limited methodological novelty and insufficient differentiation from existing theoretical studies.
The paper mainly analyzes the standard federated LoRA procedure based on simultaneous factor updates and server-side factor averaging, and does not introduce a substantially new aggregation or optimization algorithm. Moreover, Lemma 3.4 follows the existing centralized LoRA descent analysis, and Theorem 3.8 obtains the same rate as the centralized result discussed in the paper. Although the federated drift analysis is useful, its additional contribution relative to existing convergence studies is not sufficiently distinguished. The proposed adaptive learning-rate strategies are based on factor and gradient magnitudes but are not accompanied by specific convergence guarantees. Consequently, the paper lacks a clearly established methodological advance beyond its theoretical characterization and empirical observations.
-
Lack of competitive external baselines in the experimental evaluation.
The main experiments primarily compare different fixed learning rates with the two proposed adaptive learning-rate strategies under the same federated LoRA procedure, but the paper does not provide direct experimental comparisons with established federated optimization methods or recent federated LoRA approaches discussed in Section 2. As a result, the experiments mainly demonstrate differences between internally selected optimization configurations and do not establish whether the proposed learning-rate strategies are competitive with existing federated optimization methods, substantially limiting the practical significance of the reported experimental results.
-
A substantial gap between the theoretical convergence analysis and the experimental settings.
The convergence analysis assumes full-batch gradient descent and full client participation, whereas the experiments use minibatch SGD or AdamW and also consider partial client participation. Furthermore, Theorem 3.8 relies on a prescribed harmonic learning-rate schedule, while Theorem 3.10 requires additional uniform trajectory bounds to establish the rate. Neither theorem directly analyzes the norm-based adaptive learning-rate strategies evaluated in Section 4. Consequently, the theoretical guarantees do not directly establish the convergence behavior of the proposed adaptive strategies under the experimental settings, weakening the connection between the theoretical and empirical contributions.
-
Limited and inconsistent empirical improvements across experimental settings.
The reported benefits of adaptive learning rates are not consistent across the evaluated tasks. In Table 2, the fixed learning rate achieves accuracy for ResNet-18 under full participation, compared with and for the two adaptive strategies, and the differences under partial participation are also small. Similarly, Table 1 shows that the normalized adaptive strategy matches the fixed-learning-rate baseline at macro accuracy on the language-model benchmark, while the raw adaptive strategy achieves . Although improvements are observed in the logistic-regression experiments, the evidence does not demonstrate a consistent practical advantage across more complex models. The experimental evaluation also lacks an ablation that separately isolates the contributions of factor-norm and gradient-norm adaptation, so the effectiveness of the proposed learning-rate design remains insufficiently established.
Questions for authors
-
Novelty of the theoretical contribution.
Can the authors clarify the main theoretical advance over existing LoRA convergence analyses, particularly Mu and Klabjan (2025) [1] and the federated LoRA convergence studies discussed in Section 2? Which results establish a new convergence property that is not covered by previous work?
-
Competitiveness against existing optimization methods.
Can the authors provide comparisons against established federated optimization methods and recent federated LoRA approaches under matched training and communication budgets? Such comparisons are important for determining whether the proposed adaptive learning-rate strategies offer a practical advantage beyond tuning a fixed learning rate.
-
Relationship between theoretical guarantees and adaptive learning rates.
Theorems 3.8 and 3.10 analyze prescribed learning-rate schedules under full-batch and full-participation assumptions. How do these guarantees relate to the norm-based adaptive strategies evaluated using minibatch SGD, AdamW, and partial client participation? Can the authors provide theoretical or empirical evidence that more directly connects the adaptive strategies to the established convergence bounds?
-
Consistency and source of the empirical improvements.
The adaptive learning-rate strategies provide limited or inconsistent improvements on ResNet-18 and Qwen2.5-1.5B-Instruct. Can the authors clarify under which conditions the adaptive strategies offer a meaningful advantage over a well-tuned fixed learning rate? Can they also provide ablations separating factor-norm adaptation from gradient-norm adaptation to identify which component contributes to the observed improvements?
Reference
[1] Siqiao Mu and Diego Klabjan. On the convergence rate of LoRA gradient descent, 2025. URL https://arxiv.org/abs/2512.18248. Version 3, revised 11 May 2026.
4 comments
Response to Reviewer Lg7Z - Part 2
3. Relation between the theorems and the adaptive learning rates
We answer the two parts of this question in turn.
How they relate. The theorems give stepsize conditions under which the stated rates hold. By Eq. (10), the admissible stepsize shrinks as the factor norm and the client-gradient norm grow. The adaptive rules use exactly this dependence: they adjust the learning rate according to these two quantities. The conditions are sufficient, not necessary, so a learning rate that does not satisfy them may still converge (with some unknown convergence rate); our analysis simply gives no guarantee for it. The submission is explicit about this scope (abstract, Section 3.4, conclusion) and does not claim a guarantee for the adaptive rules.
More direct evidence from theory. The revised appendix extends the analysis to partial participation ( of clients per round). Proposition B.1 keeps the rate with probability at least without trajectory assumptions (proof sketch; the full proof will be in the final version). Corollary B.2 gives in expectation under Assumption 3.9. Both use full-batch local gradients.
More direct evidence from experiments. We ran the adaptive rules closer to the setting of the theory: full-batch local gradients and full participation, with the model, data split and heterogeneity level of the main ResNet-18 experiment. Both adaptive rules reach a lower loss and a higher accuracy than the fixed rate.
| Method | Train loss | Test acc |
|---|---|---|
| Fixed | 0.7757 | 81.08% |
| Adaptive raw | 0.7422 | 81.54% |
| Adaptive normalized | 0.7322 | 81.56% |
This run uses 10 rounds instead of 200 because full gradients are expensive, hence the lower accuracy than in Table 2. Data augmentation is off, and the rules take their reference values at initialization.
4. Consistency and source of the improvements
The paper does not claim that the adaptive rules improve every setting. Section 4.2 itself reports that the fixed rate is best for ResNet-18 under full participation and that the benefits vary across settings. The numbers quoted in the review are therefore consistent with what the paper states, not in conflict with it.
When the rules help. They help when the quantities in Eq. (10) change a lot during training. In logistic regression grows more than tenfold (Figure 1(c)), and accuracy improves by 1.3 points for full participation and 0.8–0.9 points for partial. For ResNet-18 and Qwen2.5-1.5B it changes by about 3% (Figures 2(c) and 3(b)), so a tuned fixed rate stays suitable and the adaptive rules match it.
Additional models. We added Qwen3-0.6B and Qwen2.5-0.5B, with the same dataset, optimizer, rank and participation settings. (Partial participation runs for 50 rounds and full participation for 20, so the two regimes are not directly comparable.)
Qwen3-0.6B:
| Participation | LR | Loss ↓ | Macro ↑ | Worst-task ↑ |
|---|---|---|---|---|
| partial | fixed | 0.2561 | 69.7% | 52.0% |
| partial | raw | 0.2423 | 69.7% | 51.5% |
| partial | normalized | 0.2374 | 70.9% | 55.5% |
| full | fixed | 0.3269 | 69.4% | 50.0% |
| full | raw | 0.3260 | 68.9% | 49.5% |
| full | normalized | 0.3244 | 69.9% | 51.0% |
Qwen2.5-0.5B:
| Participation | LR | Loss ↓ | Macro ↑ | Worst-task ↑ |
|---|---|---|---|---|
| partial | fixed | 0.3850 | 63.5% | 46.5% |
| partial | raw | 0.3888 | 65.0% | 48.5% |
| partial | normalized | 0.3854 | 64.5% | 47.0% |
| full | fixed | 0.3819 | 63.9% | 46.5% |
| full | raw | 0.3809 | 64.0% | 46.5% |
| full | normalized | 0.3811 | 64.0% | 47.5% |
The normalized rule improves macro accuracy by 0.5–1.2 points on Qwen3-0.6B (worst-task accuracy by 1.0–3.5 points) and by 0.1–1.0 points on Qwen2.5-0.5B.
Ablation on adaptive learning rate. As requested, we separated the two statistics on Qwen2.5-1.5B (setting of Table 1):
| Method | Training loss | Macro accuracy (%) | Worst-task accuracy (%) |
|---|---|---|---|
| Fixed LR | 0.2058 | 82.0 | 76.5 |
| Adaptive raw (both) | 0.2024 | 81.8 | 76.5 |
| Adaptive normalized (both) | 0.2001 | 82.0 | 77.0 |
| Gradient-only | 0.1966 | 81.8 | 77.0 |
| Factor-only | 0.2057 | 81.9 | 77.0 |
The gradient-only rule gives the lowest loss. The factor-only rule equals the fixed rate (0.2057 vs. 0.2058), as expected when the factor norm changes by only 3%. Accuracy differences among all variants are within 0.5 points. The rule uses both statistics because both appear in Eq. (10).
[1] Li et al. Federated Optimization in Heterogeneous Networks. MLSys 2020.
[2] Sun et al. Improving LoRA in Privacy-preserving Federated Learning. ICLR 2024.
[3] Wang et al. FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. NeurIPS 2024.
Thank you to the authors for their detailed response and revisions.
Based on your comments, I have re-read the paper by Mu and Klabjan (2025). I appreciate the clearer distinction made in the revised version between prior research and the novel theoretical contributions of this paper, which has helped me better assess the paper’s originality. Consequently, I tend to take a more positive view of my main concern about the Theoretical Novelty.
Thank you for taking the time to provide some additional comparisons of the model’s applications. These extra experiments, ablation analyses, and comparisons with existing benchmark models addressed some of my concerns regarding the practical applicability, effectiveness and competitiveness of the proposed method.
Given these improvements, I will raise the score. I will also discuss the revised manuscript with the other reviewers and AC to make a further decision.
Thank you for reconsidering your assessment. We are glad that the revised version makes the distinction between prior work and our new results clearer, and that the additional experiments address some of your concerns.
We will keep these changes in the final version, together with the full proof of Proposition B.1 and the full comparison in our additional experiments. We are happy to answer any further questions during the author-reviewer discussion period.
Response to Reviewer Lg7Z - Part 1
We thank Reviewer Lg7Z for the review and for recognizing our theoretical analysis. We respond to each concern below.
1. Contribution and novelty
We respectfully disagree that the absence of a new algorithm should be considered as a weakness of a paper. This paper is a theoretical study of standard federated LoRA. Understanding how an existing, widely used algorithm behaves is a research goal in its own right. Our question is how the LoRA factorization changes the effect of client heterogeneity on convergence, and also, two factorizations of the same model, with identical weight-space client gradients, have different factor-space heterogeneity.
What is inherited and what is new. The stacked reformulation and Lemma 3.4 come from Mu & Klabjan (2025), and the submission attributes them explicitly. Using a centralized lemma here is natural: in any federated algorithm, local training on one client is a centralized procedure that involves no other client. This lemma alone does not handle heterogeneous clients, local steps or factor averaging. The following results do:
Same order, different result. We see this point differently from the review: sharing the O(1/logT) order with the centralized result does not indicate limited novelty. Heterogeneous clients, K local steps and factor averaging typically introduce a heterogeneity residual. Showing that the order is preserved, with heterogeneity only changing the constant (1+σg)/log(T+1), is the result itself, not a shortcoming. The paragraph after Theorem 3.8 already makes this point.
Relation to federated LoRA analyses. The Introduction and Section 2 state the differences: FSLoRA assumes gradient dissimilarity directly in factor space, FedRot-LoRA isolates an aggregation-error term, and Chen et al. (2025) assume bounded factors and gradients. To our knowledge, no existing analysis covers standard factor averaging under a weight-space heterogeneity assumption without bounded factors. If the reviewer has a specific result in mind, we would be glad to compare with it.
Adaptive learning rates. We do not present the adaptive rules as a new method, so we do not think the lack of a separate guarantee for them weakens the theoretical contribution. They apply the stepsize condition, which says that the admissible stepsize shrinks as the factor norm and the client-gradient norm grow (see point 3).
2. External baselines
Our fixed-rate baseline is standard FedAvg-LoRA with separate factor averaging. Comparing with other federated methods therefore means comparing federated LoRA itself with alternative algorithms, which falls outside our central focus: understanding federated LoRA under client heterogeneity and testing the learning-rate guidance from this analysis. Our experiments fix the aggregation rule and change only the learning-rate rule; changing both would confound the comparison. We therefore do not agree that the absence of cross-algorithm baselines limits the value of these experiments.
Nevertheless, to address your request, we added FedProx[1], FFA-LoRA[2], FLoRA[3], and linear and cosine decay, with the same data partition, rank, communication rounds and local steps as the main experiments (per-round communication differs by design for FFA-LoRA and FLoRA).
Qwen2.5-1.5B, full participation:
The normalized rule is at least as good as FFA-LoRA and FLoRA on all three metrics, and matches the tuned fixed rate in macro accuracy with a lower training loss. Linear and cosine decay are 1.5–1.7 points lower than the fixed rate.
CIFAR-10 with ResNet-18, full participation, αDir=0.1: