acceptodds
Under review as a conference paper at ICLR 2027

Asymmetric EMA for Federated LoRA: Feed Back A, Not B

Abstract

Federated LoRA can suffer from majority‑class collapse in the high‑heterogeneity regime (Dirichlet ), and whether a run collapses often depends on the random seed. The two LoRA factors multiply, so per‑round errors in either factor compound from one round to the next. Applying a time‑axis exponential moving average (EMA) over rounds is a natural remedy, but the choice of which factor to feed back is not obvious: feeding a factor back keeps only a fraction of its per‑round displacement, while not feeding it back lets the gap between the EMA and the training iterate accumulate across rounds. We show that both effects are exact server‑side identities that hold for any LoRA rank, local optimizer, and number of local steps, and that this common cost acts asymmetrically. Drift is most damaging for the direction factor , whereas the displacement squeeze is most damaging for the scale factor . We propose Stabilized A-Feedback (), which feeds only the direction factor back through the EMA, keeps the scale factor on the path, and adds one line of code with no new hyperparameters. Under a rank‑1, single‑step SGD model we quantify four consequences: a residual error that does not vanish as the round budget grows when ‑feedback is absent, standard SGD‑style convergence of , an optimization‑bias penalty when is fed back that cannot be recovered by retuning the learning rate within the stability constraint, and a heterogeneity crossover below which wins. Across four federated LoRA methods and six tasks (four GLUE and two MedMNIST; 24 five‑seed cells), outperforms both and EMA in 22 cells and reduces the MNLI seed standard deviation from up to to at most ; ablations that hold the deployed model fixed separate training‑time ‑feedback from deployment‑time averaging. We also report where the method stops: at 355M outperforms on all three seeds but its ordering against EMA is seed‑dependent, and stacking with FedSA‑LoRA is structurally compatible but not shown to be additive.

Primary area:
optimization
Keywords:
federated learning, LoRA, PEFT, EMA, data heterogeneity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aPEJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author argues that existing federated LoRA methods do not adequately address cross-round adapter drift under severe data heterogeneity, which can cause majority-class collapse, and that feeding EMA-smoothed values back into both LoRA factors can hinder learning in B. The author proposes Stabilized A-Feedback (SAF), which feeds the EMA-smoothed A back into training, keeps B on the FedAvg update path, and uses EMA-averaged factors for deployment. The paper provides theoretical analysis under a simplified model and experiments across six classification tasks, reporting improved accuracy and stability over FedAvg and full EMA feedback in most evaluated settings.

Critical strengths
  • The proposed method is simple to implement and adds no extra communication or client computation.
  • The proposed method improves accuracy and stability in most tested settings with severe data heterogeneity.
  • The authors clearly discuss the method’s limitations and report negative results.
Critical weaknesses
  • The empirical results do not clearly support the necessity of the proposed asymmetric feedback rule. The paper argues that A requires EMA feedback to prevent drift, whereas B should avoid feedback to prevent displacement squeeze. However, Table 2 shows that EMA-B, which violates both stated conditions, achieves nearly the same mean accuracy as SAF on FedIT (77.8% vs. 78.7%) and FedEx-LoRA (78.1% vs. 79.0%), and a higher mean accuracy on HetLoRA (73.6% vs. 72.1%). EMA-EVAL also achieves a slightly higher mean accuracy on LoRA-A² (76.5% vs. 75.8%). These differences should not be interpreted as statistically significant without further analysis, but they raise an important question: why is A-only feedback necessary if B-only feedback can achieve comparable performance? The authors acknowledge these results in Section 6.2 and Appendix C.11, but the explanation remains incomplete. Additional controlled experiments and direct drift measurements comparing SAF with EMA-B would help establish whether the proposed mechanism explains the observed gains, and under which conditions it applies.

  • The theoretical analysis seems too simplified to fully explain why SAF works in practice. The quantitative bounds analyze A and B separately under rank-1, single-step SGD, without explicitly tracking their coupled updates. This matters because A’s gradient depends on B, and initializing B to zero makes A’s initial task-loss gradient zero. The theory also provides no guarantee for the deployed EMA model, even though averaging B and the classification head improves accuracy by a median of 12.2 percentage points over SAF’s training-path model. The authors acknowledge these limitations, but a substantial gap remains between the theory and the reported gains.

Questions for authors
  • Why does EMA-B achieve nearly the same accuracy as SAF on FedIT/FedEx-LoRA and a higher mean accuracy on HetLoRA, despite violating both proposed feedback conditions?
  • Does SAF still outperform full A/B feedback when neither method feeds back the classification head? Could the authors provide this controlled comparison across the 24 settings in Table 1?
  • What explains SAF’s collapse on seed 43 and its lower average accuracy than EMA on RoBERTa-large? Can the authors verify this comparison under the same software environment and clarify whether the findings extend to decoder-only LLMs and generation tasks?
  • Please also refer to the weakness part.
Reviewer_tE6dOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-round EMA for federated LoRA under high data heterogeneity. The key observation is that EMA feedback affects the two LoRA factors differently: A benefits from feedback because it reduces cross-round drift, while feeding back B suppresses its per-round displacement. Based on this observation, the paper proposes SAF, which feeds back the EMA of A while keeping B on the FedAvg trajectory. The paper provides theoretical analysis under a simplified setting and evaluates SAF across multiple federated LoRA methods and classification tasks.

Critical strengths
  1. SAF introduces a clean asymmetric EMA strategy with essentially no additional communication or computational overhead.

  2. SAF improves over both FedAvg and standard EMA in most evaluated settings, while substantially reducing seed-dependent collapse. The feedback ablations help separate the effects of A-feedback and deployment-time averaging.

Critical weaknesses
  1. The convergence analysis assumes rank-1 LoRA and single-step SGD, whereas the main experiments use rank-8 LoRA with multiple local AdamW steps. Therefore, the theory mainly provides intuition for the observed behavior rather than directly explaining the practical training regime.

  2. The paper presents cross-round feedback and A-side stabilization as relatively unexplored design dimensions, but several closely related recent works are not sufficiently discussed. For example, LoRA-FAIR (ICCV 2025) explicitly studies client-side initialization lag across communication rounds and modifies the server output used for subsequent client initialization, while FedRot-LoRA (ICML 2026) studies factor/subspace misalignment in federated LoRA and introduces alignment before aggregation. These works are not identical to SAF, but they are closely related to its motivations of cross-round initialization and A/direction stability under heterogeneity. A clearer comparison is needed to establish precisely what is novel about SAF relative to existing initialization- and alignment-based approaches.

  3. The rank/local-step experiments, heterogeneity sweep, and several other robustness studies rely largely on single-seed results, while the 355M experiment shows seed-dependent behavior against EMA. More multi-seed evidence would strengthen the claimed generality.

Questions for authors
  1. How does SAF differ conceptually from prior federated LoRA methods that explicitly address client initialization across communication rounds?
  2. Can the authors provide stronger theoretical justification for the practical rank-r, multi-step AdamW setting, beyond the current rank-1 single-step SGD analysis?
  3. Can the authors provide multi-seed results for the heterogeneity sweep and rank/local-step robustness experiments?
Reviewer_8qqZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stabilized A-Feedback (SAF) for federated LoRA under severe label heterogeneity. SAF feeds the EMA of factor A back into training while keeping B on the FedAvg trajectory, and uses EMA-averaged parameters for deployment. The authors motivate this asymmetry through the different optimization roles of A and B, supported by feedback identities and simplified SGD analyses. Across four federated LoRA methods and six classification tasks, SAF outperforms FedAvg and full-feedback EMA in 22 of 24 five-seed settings.

Critical strengths

The method is simple, communication-efficient, and easy to integrate into existing federated LoRA frameworks. The factor-wise treatment of cross-round EMA feedback offers an interesting perspective on training stability. The experiments cover multiple methods and tasks, with substantial accuracy improvements and reduced seed variability in several highly heterogeneous settings. The authors also provide useful ablations and acknowledge cases where the method is less effective.

Critical weaknesses

The incremental benefit of A-feedback over EMA-EVAL is not consistently demonstrated. Although the MNLI ablation supports A-feedback, the five-seed results in the supplementary materials show that SAF outperforms EMA-EVAL in only 16 of 24 settings. On PathMNIST, EMA-EVAL achieves comparable or higher mean accuracy across all four methods. This suggests that the advantage of training-time A-feedback is more task-dependent than the main comparison indicates.

The justification for choosing A-only feedback remains incomplete. Proposition 1 establishes symmetric feedback identities for A and B, while the proposed direction/scale interpretation does not fully explain why A-feedback should be preferred under their coupled optimization dynamics. EMA-B also achieves competitive results on MNLI.

Finally, the proof of Proposition 3 appears incomplete. Appendix A.4 replaces an uncentered gap second moment with its centered variance despite a nonzero stationary mean, and the descent argument does not explicitly account for the EMA lag term.

Questions for authors
  1. The supplementary EMA-EVAL results suggest that training-time A-feedback provides task-dependent benefits. Could the authors clarify when A-feedback is preferable to maintaining EMA weights solely for evaluation, particularly given the PathMNIST results?

  2. What further evidence supports choosing A-only rather than B-only feedback? Could the competitive EMA-B results and the symmetric feedback identities be reconciled with the proposed A/B asymmetry?

  3. Could the authors isolate B-feedback and deployment-time B averaging from the effects of classification-head feedback and averaging?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.