acceptodds
Under review as a conference paper at ICLR 2027

Federated LoRA under Client Heterogeneity: Factor Representation and Convergence

Abstract

Low-rank adaptation (LoRA) offers a parameter-efficient approach to fine-tuning foundation models. In federated learning, LoRA can reduce local training and communication costs by restricting updates to the low-rank factors. However, the role of the low-rank parameterization in heterogeneous local training and convergence is not yet fully understood. In this paper, we develop a non-asymptotic analysis of full-batch federated LoRA with full client participation, simultaneous updates of both factors, and factor averaging at the server. We relate weight-space client heterogeneity to local drift in the LoRA factors, deriving a one-round descent bound that tracks factor norms, local steps, and stepsize. From this bound, we establish convergence rates for the minimum squared factor-gradient norm over communication rounds. With harmonic stepsizes, we obtain an rate by controlling trajectory growth, without assuming uniformly bounded factor norms. Under additional uniform trajectory bounds, a stepsize chosen according to and held constant within each run yields an rate. Experiments illustrate how client heterogeneity and learning-rate selection affect training dynamics, suggesting that factor and gradient magnitudes can inform local stepsize selection.

Primary area:
optimization
Keywords:
LoRA, Federated Learning, Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Lg7ZOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the convergence of federated LoRA under client data heterogeneity, considering full-batch local gradient descent with full client participation, simultaneous updates of both LoRA factors, and separate factor averaging at the server. The authors analyze how weight-space gradient heterogeneity affects factor-space gradients and local client drift, deriving local-stability conditions and a one-round descent bound that depends on factor norms, local steps, and learning rates. Based on this analysis, the paper establishes an O(1/log⁡T)O(1/\log T) stationarity rate with harmonic learning rates without assuming uniformly bounded factor norms, and further establishes an O(T−2/3)O(T^{-2/3}) rate with a horizon-dependent constant learning rate under additional uniform trajectory bounds. The experiments examine learning-rate selection, client heterogeneity, and LoRA factor scaling, evaluating two norm-based adaptive learning-rate strategies on CIFAR-10 with a ResNet-18 backbone and on commonsense reasoning tasks with Qwen2.5-1.5B-Instruct.

Critical strengths
  1. A detailed theoretical analysis of federated LoRA convergence.

    The paper derives factor-dependent heterogeneity and local-drift bounds and establishes non-asymptotic convergence guarantees. In particular, the harmonic learning-rate analysis controls trajectory growth without assuming uniformly bounded LoRA factors. This provides a useful theoretical characterization of standard federated LoRA.

  2. Empirical observations on learning-rate and factor-representation sensitivity.

    The experiments examine how learning rates, client heterogeneity, and equivalent LoRA factor scalings affect optimization. The reported factor-norm and gradient-norm diagnostics provide useful observations about training stability.

Critical weaknesses
  1. Limited methodological novelty and insufficient differentiation from existing theoretical studies.

    The paper mainly analyzes the standard federated LoRA procedure based on simultaneous factor updates and server-side factor averaging, and does not introduce a substantially new aggregation or optimization algorithm. Moreover, Lemma 3.4 follows the existing centralized LoRA descent analysis, and Theorem 3.8 obtains the same O(1/log⁡T)O(1/\log T) rate as the centralized result discussed in the paper. Although the federated drift analysis is useful, its additional contribution relative to existing convergence studies is not sufficiently distinguished. The proposed adaptive learning-rate strategies are based on factor and gradient magnitudes but are not accompanied by specific convergence guarantees. Consequently, the paper lacks a clearly established methodological advance beyond its theoretical characterization and empirical observations.

  2. Lack of competitive external baselines in the experimental evaluation.

    The main experiments primarily compare different fixed learning rates with the two proposed adaptive learning-rate strategies under the same federated LoRA procedure, but the paper does not provide direct experimental comparisons with established federated optimization methods or recent federated LoRA approaches discussed in Section 2. As a result, the experiments mainly demonstrate differences between internally selected optimization configurations and do not establish whether the proposed learning-rate strategies are competitive with existing federated optimization methods, substantially limiting the practical significance of the reported experimental results.

  3. A substantial gap between the theoretical convergence analysis and the experimental settings.

    The convergence analysis assumes full-batch gradient descent and full client participation, whereas the experiments use minibatch SGD or AdamW and also consider partial client participation. Furthermore, Theorem 3.8 relies on a prescribed harmonic learning-rate schedule, while Theorem 3.10 requires additional uniform trajectory bounds to establish the O(T−2/3)O(T^{-2/3}) rate. Neither theorem directly analyzes the norm-based adaptive learning-rate strategies evaluated in Section 4. Consequently, the theoretical guarantees do not directly establish the convergence behavior of the proposed adaptive strategies under the experimental settings, weakening the connection between the theoretical and empirical contributions.

  4. Limited and inconsistent empirical improvements across experimental settings.

    The reported benefits of adaptive learning rates are not consistent across the evaluated tasks. In Table 2, the fixed learning rate achieves 91.1691.16\\% accuracy for ResNet-18 under full participation, compared with 90.9490.94\\% and 90.9990.99\\% for the two adaptive strategies, and the differences under partial participation are also small. Similarly, Table 1 shows that the normalized adaptive strategy matches the fixed-learning-rate baseline at 82.082.0\\% macro accuracy on the language-model benchmark, while the raw adaptive strategy achieves 81.881.8\\%. Although improvements are observed in the logistic-regression experiments, the evidence does not demonstrate a consistent practical advantage across more complex models. The experimental evaluation also lacks an ablation that separately isolates the contributions of factor-norm and gradient-norm adaptation, so the effectiveness of the proposed learning-rate design remains insufficiently established.

Questions for authors
  1. Novelty of the theoretical contribution.

    Can the authors clarify the main theoretical advance over existing LoRA convergence analyses, particularly Mu and Klabjan (2025) [1] and the federated LoRA convergence studies discussed in Section 2? Which results establish a new convergence property that is not covered by previous work?

  2. Competitiveness against existing optimization methods.

    Can the authors provide comparisons against established federated optimization methods and recent federated LoRA approaches under matched training and communication budgets? Such comparisons are important for determining whether the proposed adaptive learning-rate strategies offer a practical advantage beyond tuning a fixed learning rate.

  3. Relationship between theoretical guarantees and adaptive learning rates.

    Theorems 3.8 and 3.10 analyze prescribed learning-rate schedules under full-batch and full-participation assumptions. How do these guarantees relate to the norm-based adaptive strategies evaluated using minibatch SGD, AdamW, and partial client participation? Can the authors provide theoretical or empirical evidence that more directly connects the adaptive strategies to the established convergence bounds?

  4. Consistency and source of the empirical improvements.

    The adaptive learning-rate strategies provide limited or inconsistent improvements on ResNet-18 and Qwen2.5-1.5B-Instruct. Can the authors clarify under which conditions the adaptive strategies offer a meaningful advantage over a well-tuned fixed learning rate? Can they also provide ablations separating factor-norm adaptation from gradient-norm adaptation to identify which component contributes to the observed improvements?

Reference

[1] Siqiao Mu and Diego Klabjan. On the convergence rate of LoRA gradient descent, 2025. URL https://arxiv.org/abs/2512.18248. Version 3, revised 11 May 2026.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Lg7Z - Part 1

We thank Reviewer Lg7Z for the review and for recognizing our theoretical analysis. We respond to each concern below.

1. Contribution and novelty

We respectfully disagree that the absence of a new algorithm should be considered as a weakness of a paper. This paper is a theoretical study of standard federated LoRA. Understanding how an existing, widely used algorithm behaves is a research goal in its own right. Our question is how the LoRA factorization changes the effect of client heterogeneity on convergence, and also, two factorizations of the same model, with identical weight-space client gradients, have different factor-space heterogeneity.

What is inherited and what is new. The stacked reformulation and Lemma 3.4 come from Mu & Klabjan (2025), and the submission attributes them explicitly. Using a centralized lemma here is natural: in any federated algorithm, local training on one client is a centralized procedure that involves no other client. This lemma alone does not handle heterogeneous clients, local steps or factor averaging. The following results do:

  • Lemma 3.5. Weight-space heterogeneity σg2\sigma_g^2 becomes at most 4∣V∣F2σg24\\|V\\|_F^2\sigma_g^2 in factor space.
  • Lemmas A.3 and 3.6. Local stability and drift over KK local steps, without a globally Lipschitz factor gradient.
  • Lemma 3.7. One-round descent for local updates with factor averaging.
  • Theorem 3.8. Joint control of factor growth and the accumulated heterogeneity residual, without bounded factors.

Same order, different result. We see this point differently from the review: sharing the O(1/log⁡T)O(1/\log T) order with the centralized result does not indicate limited novelty. Heterogeneous clients, KK local steps and factor averaging typically introduce a heterogeneity residual. Showing that the order is preserved, with heterogeneity only changing the constant (1+σg)/log⁡(T+1)(1+\sigma_g)/\log(T+1), is the result itself, not a shortcoming. The paragraph after Theorem 3.8 already makes this point.

Relation to federated LoRA analyses. The Introduction and Section 2 state the differences: FSLoRA assumes gradient dissimilarity directly in factor space, FedRot-LoRA isolates an aggregation-error term, and Chen et al. (2025) assume bounded factors and gradients. To our knowledge, no existing analysis covers standard factor averaging under a weight-space heterogeneity assumption without bounded factors. If the reviewer has a specific result in mind, we would be glad to compare with it.

Adaptive learning rates. We do not present the adaptive rules as a new method, so we do not think the lack of a separate guarantee for them weakens the theoretical contribution. They apply the stepsize condition, which says that the admissible stepsize shrinks as the factor norm and the client-gradient norm grow (see point 3).

2. External baselines

Our fixed-rate baseline is standard FedAvg-LoRA with separate factor averaging. Comparing with other federated methods therefore means comparing federated LoRA itself with alternative algorithms, which falls outside our central focus: understanding federated LoRA under client heterogeneity and testing the learning-rate guidance from this analysis. Our experiments fix the aggregation rule and change only the learning-rate rule; changing both would confound the comparison. We therefore do not agree that the absence of cross-algorithm baselines limits the value of these experiments.

Nevertheless, to address your request, we added FedProx[1], FFA-LoRA[2], FLoRA[3], and linear and cosine decay, with the same data partition, rank, communication rounds and local steps as the main experiments (per-round communication differs by design for FFA-LoRA and FLoRA).

Qwen2.5-1.5B, full participation:

MethodFinal training lossMacro accuracy (%)Worst-task accuracy (%)
FedAvg-LoRA (fixed LR)0.205882.076.5
FedAvg-LoRA (adaptive raw)0.202481.876.5
FedAvg-LoRA (adaptive normalized)0.200182.077.0
FedProx (fixed LR)0.213181.275.0
FFA-LoRA (fixed LR)0.206381.475.5
FLoRA (fixed LR)0.203581.876.0
FedAvg-LoRA (linear decay)0.222980.373.5
FedAvg-LoRA (cosine decay)0.222980.573.5

The normalized rule is at least as good as FFA-LoRA and FLoRA on all three metrics, and matches the tuned fixed rate in macro accuracy with a lower training loss. Linear and cosine decay are 1.5–1.7 points lower than the fixed rate.

CIFAR-10 with ResNet-18, full participation, αDir=0.1\alpha_{\mathrm{Dir}}=0.1:

MethodTest accuracy (%)
FedAvg-LoRA (fixed LR)91.16 ± 0.28
FedAvg-LoRA (adaptive raw)90.94 ± 0.35
FedAvg-LoRA (adaptive normalized)90.99 ± 0.40
FedProx (fixed LR)90.86 ± 0.28
FFA-LoRA (fixed LR)90.12 ± 0.36
FLoRA (fixed LR)89.98 ± 0.13
FedAvg-LoRA (linear decay)90.38 ± 0.41
FedAvg-LoRA (cosine decay)90.51 ± 0.39
AuthorsOpenReview ↗

Response to Reviewer Lg7Z - Part 2

3. Relation between the theorems and the adaptive learning rates

We answer the two parts of this question in turn.

How they relate. The theorems give stepsize conditions under which the stated rates hold. By Eq. (10), the admissible stepsize shrinks as the factor norm and the client-gradient norm grow. The adaptive rules use exactly this dependence: they adjust the learning rate according to these two quantities. The conditions are sufficient, not necessary, so a learning rate that does not satisfy them may still converge (with some unknown convergence rate); our analysis simply gives no guarantee for it. The submission is explicit about this scope (abstract, Section 3.4, conclusion) and does not claim a guarantee for the adaptive rules.

More direct evidence from theory. The revised appendix extends the analysis to partial participation (mm of NN clients per round). Proposition B.1 keeps the O(1/log⁡T)O(1/\log T) rate with probability at least 1−δ1-\delta without trajectory assumptions (proof sketch; the full proof will be in the final version). Corollary B.2 gives O(T−2/3+(N−m)/(m(N−1)T))O\big(T^{-2/3}+\sqrt{(N-m)/(m(N-1)T)}\big) in expectation under Assumption 3.9. Both use full-batch local gradients.

More direct evidence from experiments. We ran the adaptive rules closer to the setting of the theory: full-batch local gradients and full participation, with the model, data split and heterogeneity level of the main ResNet-18 experiment. Both adaptive rules reach a lower loss and a higher accuracy than the fixed rate.

MethodTrain lossTest acc
Fixed0.775781.08%
Adaptive raw0.742281.54%
Adaptive normalized0.732281.56%

This run uses 10 rounds instead of 200 because full gradients are expensive, hence the lower accuracy than in Table 2. Data augmentation is off, and the rules take their reference values at initialization.

4. Consistency and source of the improvements

The paper does not claim that the adaptive rules improve every setting. Section 4.2 itself reports that the fixed rate is best for ResNet-18 under full participation and that the benefits vary across settings. The numbers quoted in the review are therefore consistent with what the paper states, not in conflict with it.

When the rules help. They help when the quantities in Eq. (10) change a lot during training. In logistic regression ∣Vt∣F\\|V_t\\|_F grows more than tenfold (Figure 1(c)), and accuracy improves by 1.3 points for full participation and 0.8–0.9 points for partial. For ResNet-18 and Qwen2.5-1.5B it changes by about 3% (Figures 2(c) and 3(b)), so a tuned fixed rate stays suitable and the adaptive rules match it.

Additional models. We added Qwen3-0.6B and Qwen2.5-0.5B, with the same dataset, optimizer, rank and participation settings. (Partial participation runs for 50 rounds and full participation for 20, so the two regimes are not directly comparable.)

Qwen3-0.6B:

ParticipationLRLoss ↓Macro ↑Worst-task ↑
partialfixed0.256169.7%52.0%
partialraw0.242369.7%51.5%
partialnormalized0.237470.9%55.5%
fullfixed0.326969.4%50.0%
fullraw0.326068.9%49.5%
fullnormalized0.324469.9%51.0%

Qwen2.5-0.5B:

ParticipationLRLoss ↓Macro ↑Worst-task ↑
partialfixed0.385063.5%46.5%
partialraw0.388865.0%48.5%
partialnormalized0.385464.5%47.0%
fullfixed0.381963.9%46.5%
fullraw0.380964.0%46.5%
fullnormalized0.381164.0%47.5%

The normalized rule improves macro accuracy by 0.5–1.2 points on Qwen3-0.6B (worst-task accuracy by 1.0–3.5 points) and by 0.1–1.0 points on Qwen2.5-0.5B.

Ablation on adaptive learning rate. As requested, we separated the two statistics on Qwen2.5-1.5B (setting of Table 1):

MethodTraining lossMacro accuracy (%)Worst-task accuracy (%)
Fixed LR0.205882.076.5
Adaptive raw (both)0.202481.876.5
Adaptive normalized (both)0.200182.077.0
Gradient-only0.196681.877.0
Factor-only0.205781.977.0

The gradient-only rule gives the lowest loss. The factor-only rule equals the fixed rate (0.2057 vs. 0.2058), as expected when the factor norm changes by only 3%. Accuracy differences among all variants are within 0.5 points. The rule uses both statistics because both appear in Eq. (10).

[1] Li et al. Federated Optimization in Heterogeneous Networks. MLSys 2020.

[2] Sun et al. Improving LoRA in Privacy-preserving Federated Learning. ICLR 2024.

[3] Wang et al. FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. NeurIPS 2024.

Thank you to the authors for their detailed response and revisions.

Based on your comments, I have re-read the paper by Mu and Klabjan (2025). I appreciate the clearer distinction made in the revised version between prior research and the novel theoretical contributions of this paper, which has helped me better assess the paper’s originality. Consequently, I tend to take a more positive view of my main concern about the Theoretical Novelty.

Thank you for taking the time to provide some additional comparisons of the model’s applications. These extra experiments, ablation analyses, and comparisons with existing benchmark models addressed some of my concerns regarding the practical applicability, effectiveness and competitiveness of the proposed method.

Given these improvements, I will raise the score. I will also discuss the revised manuscript with the other reviewers and AC to make a further decision.

Thank you for reconsidering your assessment. We are glad that the revised version makes the distinction between prior work and our new results clearer, and that the additional experiments address some of your concerns.

We will keep these changes in the final version, together with the full proof of Proposition B.1 and the full comparison in our additional experiments. We are happy to answer any further questions during the author-reviewer discussion period.

Reviewer_kwdkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to provide a mathematical understanding of low-rank adaptation (LoRA) technique in federated learning with heterogeneous clients. The core novelty of the paper lies in its theoretical results.

Critical strengths

The analysis of the paper seem to be the core novelty of the paper. To much knowledge such results do not exist in the literature of federated LoRA and can inspire future work.

Critical weaknesses
  1. Parameter rr in LoRA (i.e., its rank) should have a direct impact on all the obtained results as it tunes the precision of fine-tuning. Nevertheless, the majority of the main results in the text do not explicitly show such effect on the convergence.

  2. Heterogeneous clients (which is the scenario of interest in this paper) in FL will have different computation capability and thus can choose different LoRA ranks (supplemented by padding during aggregation or sketching matrices). It is not clear how such heterogeneity is considered in the analysis. Specifically, the data heterogeneity is captured by Assumption 3.3, but other forms of heterogeneity, which are unique to federated LoRA setting, are not explicit in the analysis.

  3. It is not clear how the theory is connected to experiments. Specifically, the simulation results seem to be just some observations made from careful investigation of existing federated LoRA methods. For example, it is not clear what is the new observation which is of interest to the community.

Questions for authors
  1. Can you please clarify the effect of rank of LoRA on the analysis and results?

  2. Can you please clarify the effect of client computation heterogeneity on the analysis?

  3. Can you please clarify the new observations made in the experiments?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kwdk - Part 1

We thank Reviewer kwdk for the review and for recognizing that our analysis is new to the federated LoRA literature and can inspire future work. We respond to each concern below.

1. Effect of the rank rr

  • The rank does not appear as a separate symbol because our bounds are written in terms of the factor norm ∣Vt∣F\\|V_t\\|_F, and the rank is contained in this norm.
  • An explicit dependence on rr follows from ∥Vt∥2=∥At∥2+∥Bt∥2≤r(∥At∥op2+∥Bt∥op2)\Vert V_t\Vert^2=\Vert A_t\Vert^2+\Vert B_t\Vert^2 \le r(\Vert A_t\Vert_{\mathrm{op}}^2+\Vert B_t\Vert_{\mathrm{op}}^2), where ∣⋅∣op\\|\cdot\\|_{op} is the spectral norm. We state our results with the Frobenius norm because it is tighter.
  • Our theorems are about reaching a stationary point at a fixed rank. They do not describe how the rank changes the approximation quality. To show this effect empirically, we ran a rank ablation on Qwen2.5-1.5B with r∈{4,16,32}r\in\{4,16,32\}, keeping αLoRA/r\alpha_{\mathrm{LoRA}}/r fixed and all other settings as in the paper.
RankTraining lossMacro accuracy (%)Worst-task accuracy (%)
40.229780.375.0
16 (original)0.205882.076.5
320.190582.776.5

A larger rank gives a lower training loss and a higher macro accuracy. Worst-task accuracy does not improve beyond r=16r=16. We have added this discussion to the revision.

2. Heterogeneous ranks and computation

Thank you for pointing to this direction. In our paper, “client heterogeneity” means data heterogeneity: Assumption 3.3 bounds the difference between client gradients at the same weight-space point, and all clients use the same rank. We will state this scope explicitly in the abstract and the introduction.

Our analysis does not cover heterogeneous ranks. Two steps of the proof would have to change. First, the proof uses that all clients start a round from the same factor VtV_t; with different ranks each client receives a truncated or projected version. Second, the server update is a plain average of the factors; padding makes averaging possible but changes this update. Both points depend on the specific protocol. We leave this extension to future work and will add it to the discussion.

To explore this setting empirically, we ran Qwen2.5-1.5B with heterogeneous client ranks: of the 10 clients, 3 use rank 4, 4 use rank 16, and 3 use rank 32. Aggregation uses zero-padding, and all other settings follow Table 1 in the paper.

MethodFinal training lossMacro accuracy (%)Worst-task accuracy (%)
Fixed LR0.233580.574.5
Adaptive raw0.230581.075.5
Adaptive normalized0.230781.075.5

Both adaptive rules improve on the fixed rate by 0.5 points in macro accuracy and 1.0 point in worst-task accuracy.

AuthorsOpenReview ↗

Response to Reviewer kwdk - Part 2

3. Connection between theory and experiments, and new observations

Thank you for the suggestion of highlighting the new observations. Each experiment is tied to a part of the analysis. The main observations are the following.

  • The same model trains differently under different factorizations. By Eq. (14), rescaling (B,A)→(sB,A/s)(B,A)\to(sB,A/s) leaves BABA and all weight-space client gradients unchanged but changes the factor-space heterogeneity. Figure 4 confirms this: with ηℓ=0.1\eta_\ell=0.1 the final training loss changes substantially with ss. So in federated LoRA the factor scale and the learning rate have to be chosen together.
  • The factor norm grows during training, and this limits the stepsize. By Eq. (10), the admissible stepsize shrinks as the factor norm and the client-gradient norm grow. Figures 1–3 show that the factor norm grows in every setting. A learning rate that adapts to these two quantities improves accuracy by about 1.3 points in logistic regression, where they change most, and matches a tuned fixed rate where they change little.
  • The factor norm behaves differently in two regimes, which matches our two theorems. When the classifier is trained from scratch (logistic regression), ∣Vt∣F\\|V_t\\|_F grows more than tenfold (Figure 1(c)). When a pretrained model is fine-tuned (ResNet-18, Qwen2.5-1.5B), it changes by about 3%.
  • Client sampling amplifies the cost of heterogeneity. When moving from i.i.d. data to αDir=0.1\alpha_{\mathrm{Dir}}=0.1, accuracy drops more sharply under partial participation than under full participation. This observation is consistent with Corollary B.2 in the revised appendix, where the client-sampling term is proportional to σg2\sigma_g^2.
  • The right direction of learning-rate adaptation depends on the setting. During the rebuttal, we added two LLM experiments (Qwen3-0.6B and Qwen2.5-0.5B) and examined the learning-rate multiplier mtm_t of the normalized rule (ηt=η0max⁡(mt,mmin⁡)\eta_t=\eta_0 \max (m_t, m_{\min})). The rule lowers the rate on Qwen2.5-0.5B (final mt=0.63m_t=0.63) and raises it on Qwen3-0.6B (final mt=1.50m_t=1.50), which a schedule that only decays cannot do.
ExperimentRulemtm_tAccuracy vs. fixed rate (points)
CIFAR-10, full gradientraw1.02+0.46
CIFAR-10, full gradientnormalized1.11+0.48
Qwen3-0.6B, partialraw5.900.0
Qwen3-0.6B, partialnormalized1.50+1.2
Qwen2.5-0.5B, partialraw0.68+1.5
Qwen2.5-0.5B, partialnormalized0.63+1.0

We will reorganize Section 4 around these points.

Dear Reviewer kwdk,

We would like to gently remind you that our rebuttal clarifies the effect of the LoRA rank with a rank ablation, adds an experiment with heterogeneous client ranks, and summarizes the new observations from our experiments. Please let us know if you have any further questions, and we are happy to discuss further.

Best, Authors

Reviewer_o4Z4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the convergence of federated LoRA under heterogeneous client data distributions. It analyzes how weight-space heterogeneity affects factor-space gradients and local drift under simultaneous factor updates and factor averaging. Non-asymptotic convergence rates are established for harmonic and horizon-dependent constant stepsizes. Experiments on CIFAR-10 and Qwen2.5-1.5B investigate the effects of client heterogeneity, factor scaling, and adaptive learning rates.

Critical strengths
  1. The paper addresses an interesting theoretical problem concerning the convergence of federated LoRA under client heterogeneity.

  2. The analysis connects weight-space heterogeneity with factor-space local drift and provides corresponding convergence guarantees.

  3. The paper is generally well organized, and the experimental evaluation covers different model architectures and heterogeneous data settings.

Critical weaknesses
  1. The main concern is the theoretical novelty. The analysis builds heavily upon the existing LoRA convergence framework of Mu and Klabjan (2025/2026), including the factor reformulation and descent lemma. The essential theoretical advances beyond extending this framework to federated local updates could be better highlighted.

  2. The theoretical setting is relatively restrictive, requiring full-batch gradient descent and full client participation. Furthermore, the faster convergence rate in Theorem 3.10 relies on the additional uniform trajectory bounds of Assumption 3.9, whose validity for practical deep neural networks is unclear.

  3. There is a gap between the theoretical analysis and the proposed adaptive learning-rate strategies. The convergence theorems require specific stepsize schedules, whereas the experiments use heuristic norm-based adaptation, stochastic gradients, and partial client participation. The theoretical guarantees therefore do not directly support the evaluated strategies.

  4. The effectiveness of adaptive learning rates is limited. In Table 2, the improvements on ResNet-18 are marginal or negative, while Table 1 shows almost identical performance on Qwen2.5-1.5B. Comparisons with other established learning-rate adaptation strategies are also missing.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer o4Z4 - Part 1

We thank Reviewer o4Z4 for the careful review and for recognizing that the paper addresses an interesting theoretical problem and connects weight-space heterogeneity with factor-space local drift. We respond to each concern below.

1. Theoretical novelty relative to Mu & Klabjan

From Mu & Klabjan we take two things, and we credit them for both: the stacked reformulation V=[BT,A]TV=[B^T,A]^T and the descent lemma for a single objective (Lemma 3.4). Everything after Lemma 3.4 is new and specific to the federated setting:

  • Lemma 3.5. Weight-space heterogeneity σg2\sigma_g^2 becomes at most 4∣V∣F2σg24\\|V\\|_F^2\sigma_g^2 in factor space.
  • Lemmas A.3 and 3.6. The factor objective has no globally Lipschitz gradient, so we prove local stability and bound the drift over KK local steps with a factor-dependent constant.
  • Lemma 3.7. A one-round descent bound for local updates with factor averaging, with a single heterogeneity residual.
  • Theorem 3.8. The proof controls the growth of the factor norm and the accumulated residual together, without assuming bounded factors. The O(1/log⁡T)O(1/ \log T) order is the same as in the centralized case, but the result is different: it holds with heterogeneous clients, KK local steps and factor averaging, and heterogeneity only changes the constant, $(1+ \sigma_g)/ \log(T+1)$, without a non-vanishing residual.

We will state this split between inherited and new results explicitly in Section 3.4.

2. Restrictive setting and Assumption 3.9

Full-batch gradients and full participation. We chose this setting to separate data heterogeneity from sampling noise. The revised appendix now extends the analysis to partial participation, where m of the N clients are sampled in each round:

  • Proposition B.1 (no trajectory assumption). With harmonic stepsizes, the O(1/log⁡T)O(1/ \log T) rate of Theorem 3.8 is kept with probability at least 1−δ1-\delta; the stepsize constant then depends on δ\delta.
  • Corollary B.2 (under Assumption 3.9). With a horizon-dependent constant stepsize, min⁡t<TE∣∇J(Vt)∣2=O(T−2/3+(N−m)/(m(N−1)T))\min_{t<T}\mathbb{E}\\|\nabla J(V_t)\\|^2=O\big(T^{-2/3}+\sqrt{(N-m)/(m(N-1)T)}\big). The second term comes from client sampling and vanishes when m=Nm=N, which recovers Theorem 3.10.

We added a proof sketch for the partial participation result in the appendix B; the full proof will be in the final version.

Assumption 3.9. The assumption reduces to a bound on the factor norm: by LL-smoothness, ∣Vt∣F2≤BV\\|V_t\\|_F^2\le B_V implies Gt≤G0+L(BV+R02)G_t\le G_0+L(B_V+R_0^2). Bounded-factor conditions are also used in earlier analyses of federated and decentralized LoRA (Guo et al., 2025; Ghiasvand et al., 2025), and they are consistent with our experiments: the factor norm changes by only about 3% over training for both ResNet-18 (Figure 2(c)) and Qwen2.5-1.5B (Figure 3(b)).

3. Gap between the theory and the adaptive learning rates

The adaptive rules are heuristics built on the stepsize condition behind them (Lemma 3.7), so the admissible stepsize shrinks as the factor norm and the client-gradient norm grow. The adaptive rules track these two quantities and lower the learning rate when they grow (Section 4.1). Note that Figures 1–3 and Table 1 already use full participation.

To test the rules in the setting of the theory, we added an experiment with full-batch local gradients and full participation, using the model and data split and heterogeneity level of the main ResNet-18 experiment. The adaptive rules use the values at initialization as reference. Full gradients are time-consuming, so we ran 10 rounds instead of 200 during the rebuttal period. We will add the full experiment in the revision.

MethodTrain lossTest acc
Fixed0.775781.08%
Adaptive raw0.742281.54%
Adaptive normalized0.732281.56%

Both adaptive rules reach a lower training loss and a higher test accuracy than the fixed rate.

AuthorsOpenReview ↗

Response to Reviewer o4Z4 - Part 2

4. Effectiveness of the adaptive learning rates

Thank you for this observation. The adaptive rules are meant to test the stepsize guidance from our analysis. This guidance consists of sufficient conditions for the stated convergence rates, not necessary ones: a learning rate that does not satisfy them, such as a well-tuned fixed rate, does not necessarily diverge. Our analysis therefore does not imply that the adaptive rules must outperform a tuned fixed rate in every setting. It suggests that adaptation matters most when the quantities in these conditions change substantially during training. Below we (i) explain the small gains on ResNet-18 and Qwen2.5-1.5B, (ii) report two additional models where the gains are relatively significant, and (iii) add the requested comparison with other learning-rate schedules.

  • (i) Why the gains on ResNet-18 and Qwen2.5-1.5B are small. By Eq. (10), the admissible stepsize depends on the factor norm and the client-gradient norm, and the adaptive rules follow these two quantities. In logistic regression, the factor norm grows more than tenfold during training, and the adaptive rules improve accuracy by 1.3 points under full participation and 0.8–0.9 points under partial participation. When fine-tuning ResNet-18 and Qwen2.5-1.5B, the factor norm changes by only about 3% (Figures 2 and 3. A fixed rate tuned at the start then remains suitable for the whole run, and the adaptive rules perform on par with it. The following ablation on Qwen2.5-1.5B confirms this: a rule that uses only the factor norm behaves like the fixed rate.
MethodTraining lossMacro accuracy (%)Worst-task accuracy (%)
Fixed LR0.205882.076.5
Adaptive raw (both)0.202481.876.5
Adaptive normalized (both)0.200182.077.0
Gradient-only0.196681.877.0
Factor-only0.205781.977.0

The learning-rate multiplier mtm_t (ηt=η0max⁡(mt,mmin⁡)\eta_t=\eta_0 \max (m_t, m_{\min}), so mt=1m_t=1 is the fixed rate) shows the same pattern in the experiments added during the rebuttal (the full-gradient CIFAR-10 run of point 3 and the two models in (ii) below): the gain is small where mtm_t stays close to 1.

ExperimentRulemtm_tAccuracy vs. fixed rate (points)
CIFAR-10, full gradientraw1.02+0.46
CIFAR-10, full gradientnormalized1.11+0.48
Qwen3-0.6B, partialraw5.900.0
Qwen3-0.6B, partialnormalized1.50+1.2
Qwen2.5-0.5B, partialraw0.68+1.5
Qwen2.5-0.5B, partialnormalized0.63+1.0
  • (ii) Additional models. We repeated the LLM experiment on Qwen3-0.6B and Qwen2.5-0.5B, with the same dataset, optimizer, LoRA rank and participation settings. (As in Section 4.1, partial participation runs for 50 rounds and full participation for 20 rounds. )

Qwen3-0.6B:

ParticipationLRLoss ↓Macro ↑Worst-task ↑
partialfixed0.256169.7%52.0%
partialraw0.242369.7%51.5%
partialnormalized0.237470.9%55.5%
fullfixed0.326969.4%50.0%
fullraw0.326068.9%49.5%
fullnormalized0.324469.9%51.0%

Qwen2.5-0.5B:

ParticipationLRLoss ↓Macro ↑Worst-task ↑
partialfixed0.385063.5%46.5%
partialraw0.388865.0%48.5%
partialnormalized0.385464.5%47.0%
fullfixed0.381963.9%46.5%
fullraw0.380964.0%46.5%
fullnormalized0.381164.0%47.5%

On Qwen3-0.6B, the normalized rule is the best on all three metrics in both regimes. On Qwen2.5-0.5B, the adaptive rules improve macro accuracy by 1.0–1.5 points under partial participation and are on par with the fixed rate under full participation.

  • (iii) Comparison with other learning-rate schedules. As requested, we compared with linear and cosine decay on Qwen2.5-1.5B in the setting of Table 1.
MethodFinal training lossMacro accuracy (%)Worst-task accuracy (%)
Fixed LR0.205882.076.5
Adaptive raw0.202481.876.5
Adaptive normalized0.200182.077.0
Linear decay0.222980.373.5
Cosine decay0.222980.573.5

Both decay schedules are below the fixed rate, by 1.5–1.7 points in macro accuracy and 3.0 points in worst-task accuracy, while the adaptive rules stay on par with it.

In the revision, we will state clearly that the benefit of the adaptive rules depends on the setting, and we will add these results.

Dear Reviewer o4Z4,

We would like to gently remind you that our rebuttal clarifies the theoretical novelty, extends the analysis to partial participation, and adds comparisons with other learning-rate schedules and additional models. Please let us know if you have any further questions, and we are happy to discuss further.

Best, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their time and constructive feedback. We have replied to each reviewer individually and uploaded a revised paper, with all changes marked in blue. Below is a brief summary of what is new.

New theoretical results

  • An extension of the convergence analysis to partial client participation (Appendix B).
  • A discussion of our new results for the federated setting (Lemmas 3.5–3.7 and Theorem 3.8), and of the role of the LoRA rank (Section 3.4).

New experiments

  • Comparisons with FedProx, FFA-LoRA and FLoRA, and with linear and cosine learning-rate decay, on Qwen2.5-1.5B and on CIFAR-10 with ResNet-18.
  • An ablation that separates the factor-norm and gradient-norm parts of the adaptive rule.
  • Two additional language models (Qwen3-0.6B and Qwen2.5-0.5B).
  • An experiment with full-gradient local updates, closer to the setting of our theorems.
  • A rank ablation and an experiment with heterogeneous client ranks.

Clarifications

  • "Client heterogeneity" in our paper means data heterogeneity (abstract and introduction).
  • Our stepsize conditions are sufficient, not necessary, so the adaptive rules are not expected to outperform a tuned fixed rate in every setting. They help most when the factor norm and the gradient norm change substantially during training (Section 4.3).

We hope these additions address the main concerns, and we are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.