acceptodds
Under review as a conference paper at ICLR 2027

Alternating LoRA for Decentralized Fine-Tuning: A Misalignment–Staleness Trade-Off

Abstract

Low-rank adaptation (LoRA) offers a communication-efficient approach to decentralized federated fine-tuning, but mixing independently trained LoRA factors can create discrepancies in their effective updates. Alternating factor updates mitigate a related aggregation problem in centralized federated learning, where server synchronization maintains a shared frozen factor; neighbor-only mixing does not provide this guarantee. We propose TAD-LoRA, which alternates local updates of the two factors in fixed-length phases while jointly mixing both factors at each communication round. Joint mixing reduces frozen-factor disagreement with the same per-round communication payload as standard decentralized LoRA. Under an idealized one-step stochastic-gradient model, we derive a finite-horizon bound for the loss of an averaged effective adapter. The bound separates a phase-averaged factor-product discrepancy that decreases with the switching interval from a delayed block-coverage term that increases with it; the minimizer of a loose upper-bound surrogate depends on network mixing. On four GLUE tasks, a matched-\(T=1\) ablation shows that joint mixing generally improves over active-factor-only mixing, while inter-client similarity measurements support improved factor alignment. Fixed-interval sweeps reveal that favorable switching regimes depend on network connectivity, with the largest best-observed gains over decentralized baselines under sparse communication. Additional experiments examine the method across multiple data-heterogeneity levels, communication topologies, and client counts.

Primary area:
optimization
Keywords:
decentralized federated learning, low-rank adaptation, parameter-efficient fine-tuning, decentralized optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EAk3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends RoLoRA (alternating trainable LoRA factors) from centralized federated learning to decentralized federated learning. In the decentralized setting, the major problem is that the clients may enter the same phase but with different versions of the frozen factors, which invalidates the alternating optimization idea. To solve this problem, authors do the aggregation (mixing) for both factors (line 238) to make both factors updated. Authors conduct experiments on GLUE datasets, TAD-LoRA achieves slight performance gain in certain settings.

Critical strengths

The decentralized federated LoRA is a relatively underexplored topic. The structure and writing of the draft is acceptable and good for readers to understand. Ablation results seem to be rich enough.

Critical weaknesses

I admire authors time and commitment on this work, while I feel this draft is not ready to be published as a full paper. The major "novelty" seems to be the equation of line 238 only, while this mixing rule itself is also standard for prior decentralized FL work. Other parts of the paper are mainly interpretation and explanation on existing works (RoLoRA).

  1. Related work section is very limited. For section 2.1, authors may consider provide more background on decentralized FL, as this is not a default setting, .e.g, mention the mixing matrix W_t earlier, explain how standard decentralized FL conducts aggregation. For section 2.2, I want to remind authors that some recent work starts to do the alignment for factors B or A to reduce the aggregation noise (to name a few, FedRot-LoRA, FedPA-LoRA), the current draft fully ignored this direction.
  2. The paper feels like a decentralized version of prior RoLoRA. The equation of line 238 feels like a standard extension instead of a solid contribution. Also, if you mix B and A together, how can you ensure the alternating optimization idea still validates?
  3. Limited baselines for experiments. Please consider include more federated LoRA and standard decentralized FL methods.
Questions for authors

I have no further questions. My major concern is that the paper's contribution/novelty is too limited for a full paper.

1 comment

Thank you for raising the questions about novelty, alternating optimization, and baseline coverage.

1. Contribution and validity of alternating optimization. We agree that the neighbor averaging in Eq. 238 is standard. Our contribution is to identify and analyze a condition that changes when alternating LoRA moves from server-based aggregation to peer-to-peer mixing. In RoLoRA, the server synchronizes the factor that will be frozen in the next phase. With a shared frozen factor, averaging the active factors also averages the corresponding adapter products. In a peer-to-peer network, clients can enter the same phase with different versions of the frozen factor, so this identity can fail. Active-only mixing leaves that disagreement unchanged throughout the phase.

In TAD-LoRA, during an AA-phase, each client holds BB fixed throughout its local optimizer steps and updates only AA. At communication, each client averages AA and BB separately using the same mixing matrix. Thus, the local optimization still alternates between factors. Under the stated mixing assumptions, communication contracts disagreement in the locally frozen factor. Our analysis characterizes the trade-off between frozen-factor disagreement and delayed updates to each factor as the switching interval changes.

2. Controlled comparison of active-only and joint mixing. On severely non-IID MNLI with RoBERTa-Large, p=0.02p=0.02, and T=15T=15, we ran a fresh paired comparison with identical seeds, learning rates, communication graphs, mixing weights, and local computation. Active-only mixing achieved 55.37±4.8155.37 \pm 4.81% accuracy, and joint mixing achieved 63.93±4.7963.93 \pm 4.79%. Joint mixing improved accuracy in all three seeds, with a paired gain of 8.57±1.708.57 \pm 1.70 percentage points. This comparison fixes contact opportunities. Joint mixing uses twice the adapter payload per contact of active-only mixing and the same adapter payload as standard decentralized LoRA.

3. Larger-model results. We evaluated Qwen2.5-3B on MATH with 10 clients, a Dirichlet partition (α=0.3\alpha=0.3), and 150 rounds. The table compares standard decentralized LoRA with different alternating schedules at a common learning rate of 5×10−55\times10^{-5}. Each client was evaluated on 200 examples from the official MATH test split. Accuracy (%) is reported as mean ±\pm standard deviation across three training seeds, except where marked.

Methodp=0.02p=0.02p=0.1p=0.1p=0.5p=0.5
Standard decentralized LoRA22.03±0.3522.03 \pm 0.3527.58±0.5027.58 \pm 0.5028.42±1.7128.42 \pm 1.71
Active-only alternation (T=1T=1)24.25±0.9524.25 \pm 0.9529.90±0.8029.90 \pm 0.80 (2 seeds)32.37±1.0532.37 \pm 1.05
TAD-LoRA (T=1T=1)26.63±1.0526.63 \pm 1.0530.20±0.7030.20 \pm 0.7031.97±0.7331.97 \pm 0.73
TAD-LoRA (T=10T=10)——32.05±0.3232.05 \pm 0.32
TAD-LoRA (T=30T=30)30.00±0.6830.00 \pm 0.6831.97±0.8731.97 \pm 0.8732.63±1.0532.63 \pm 1.05
TAD-LoRA (T=75T=75)32.87±0.2832.87 \pm 0.28——

Across the reported MATH schedules, the best observed TAD-LoRA configuration at each communication rate improved over standard decentralized LoRA by 10.84, 4.39, and 4.21 percentage points at p=0.02p=0.02, 0.10.1, and 0.50.5, respectively. The largest gain was at p=0.02p=0.02, where TAD-LoRA with T=75T=75 achieved 32.87% versus 22.03% for standard decentralized LoRA. This schedule switched once over 150 rounds. In the fixed-T=1T=1 ablation at the same communication rate, joint mixing achieved 26.63% versus 24.25% for active-only mixing.

4. Decentralized FL background and related work. We will introduce the time-varying mixing matrix WtW_t and standard neighbor averaging earlier in Sec. 2.1. In Sec. 2.2, we will discuss FedRot-LoRA's rotational alignment and FedPA-LoRA's product alignment and explain how they relate to the frozen-factor disagreement we study in peer-to-peer alternating optimization.

Reviewer_haCXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies decentralized federated learning (DFL) with Low-Rank Adaptation (LoRA). The authors address the factor-product discrepancy problem, where averaging factor matrices independently across clients causes product-level interference causing model performance degradation.

To address this issue, they propose a framework named TAD-LoRA. The proposed TAD-LoRA integrates alternating updates with joint mixing. In the framework, clients alternate between training factor in synchronized phases of length. Then, at each communication step, clients mix both factors (the active one and the frozen one) with neighbors. This is intended to resolve phase-state mismatch and block-wise drift of the frozen parameters.

In the theoretical analyses, convergence of the methods is explored based on an idealized one-step SGD model to formalize a trade-off between factor misalignment and update staleness. In the experimental analyses, the proposed TAD-LoRA is examined on four GLUE tasks using RoBERTa-Large.

Critical strengths

The paper provides a clear analysis of why alternating optimization fails in decentralized topologies. To this end, they elucidate how independent and uncoordinated client trajectories cause block-wise drift of the frozen factor.

Critical weaknesses

The core proposed joint mixing method is an incremental correction. In decentralized federated learning, it is standard to achieve consensus on the entire model state. In alternating LoRA, the model state consists of both factors AA and BB. Restricting consensus-mixing to only the active factor while letting the frozen factor drift is a flawed baseline design. Correcting this baseline by mixing the frozen factor as well is a straightforward engineering choice rather than a deep or novel algorithmic breakthrough.

In the theoretical study, the convergence analysis relies on a one-step, unbiased stochastic gradient (SGD) model. However, the experiments use multi-step local AdamW. This gap is mathematically significant. More precisely, in decentgralized federated learning, the behaviour of momentum-based adaptive optimizers (e.g. AdamW) under local steps is fundamentally different from one-step SGD. Moreover, nulti-step local training causes severe client drift. The theoretical bounds and the derived surrogate minimizer do not model these dynamics.

Therefore, the mathematical results provide limited qualitative analogy without formal guarantees for the actual algorithm implemented. In the experimental analyses, the main results are generated by retrospectively choosing the best switching interval for each task and network connectivity. While this is useful for exploring the trade-off, it presents an overly optimistic view of the algorithm's performance. In practice, the paper offers no test-free, practical heuristic to determine TT on a new dataset. A practitioner must perform an expensive, brute-force grid search over TT to achieve the reported gains. The theoretical minimizer is too detached from the multi-step AdamW dynamics to provide usable quantitative guidance.

Additionally, the misalignment-staleness trade-off is framed as a novel phenomenon specific to alternating LoRA. However, this is simply the standard, well-documented consensus-drift trade-off in decentralized optimization. In decentralized federated learning, communicating less frequently allows more local progress but leads to model divergence across nodes. Communicating more frequently maintains consensus. However, it is limited by the network topology and delays local convergence. Translating this classic trade-off into the language of LoRA blocks does not constitute a new theoretical insight.

Questions for authors

In the experimental analyses, K=20K = 20 local steps were used per communication rounds. If KK is increased (i.e. accelerating client drift), how does the optimal switching interval behave? Does a higher local step count require a larger or smaller switching interval to prevent factor-product divergence?

Since both A and B are lightweight adapter matrices, joint mixing does not significantly increase communication costs compared to the active-only baseline. Are there any specific settings (e.g., highly dense topologies or non-IID data) where active-only mixing actually outperforms joint mixing?

Also, Tsur(ρ)T_{sur}(\rho) depends on the network contraction rate ρ\rho. Could you estimate the spectral gap of the communication graph at runtime and use it to adaptively scale TT during training, rather than relying on retrospective grid searches?

1 comment

We thank the reviewer for the questions about joint mixing, local optimization, and switching-interval selection.

Joint mixing and when active-only is competitive. We agree that mixing the full adapter state is a standard consensus operation. To measure the effect of mixing the locally frozen factor, we ran a paired ablation on severe non-IID MNLI at p=0.02p=0.02 with the previously selected T=15T=15. Within each seed, we matched the learning rate, local training schedule, and communication graphs.

RoBERTa-Large settingppTTActive-onlyJoint mixing
MNLI, severe non-IID0.021555.37 ± 4.8163.93 ± 4.79
SST-2, Figure 50.5194.5194.43

Accuracy is reported in percent over three seeds, with standard deviations where shown. On sparse MNLI, the paired gain was 8.57 ± 1.70 percentage points, with joint mixing ahead in all three seeds. At fixed contact opportunities, joint mixing sent 3.00 MiB of adapter parameters per client per contact, compared with 1.50 MiB for active-only. In the dense SST-2 setting, active-only was ahead by 0.08 points on average, a difference driven by one seed. This illustrates a setting where active-only can be competitive at half the adapter payload per contact.

Relation to the consensus–drift trade-off. Varying TT changes when each LoRA factor is locally trainable while holding the communication schedule fixed. The LoRA-specific quantity is the discrepancy between the product of the mixed factors and the mixture of the local products. Under the one-step SGD assumptions, the bound contains a factor-product discrepancy term that decreases with TT and a delayed block-coverage term that increases with TT. We will narrow our contribution statement to this trade-off.

Effect of more local steps. The new MNLI ablation uses K=20K=20 local AdamW steps per round. Increasing KK changes both the number of optimizer steps between communication rounds and the duration of each phase in optimizer steps (KTKT). The direction in which the preferred TT shifts remains open pending a matched sweep over KK and TT.

Scope of the theory and runtime interval selection. The convergence result and surrogate minimizer apply to the one-step SGD model. The multi-step AdamW results are empirical, and the reported best-TT scores come from retrospective sweeps. One possible online approach is to estimate disagreement contraction from a recent window of mixing matrices and use the estimate to adjust TT at phase boundaries. The mapping from estimated contraction to TT could be calibrated on a held-out split of the training data. This approach remains to be evaluated under multi-step AdamW.

Comparison with standard LoRA on Qwen2.5-3B/MATH. A single shared interval of T=1T=1 improves over standard LoRA by 4.60, 2.62, and 3.55 percentage points at p=0.02p=0.02, 0.10.1, and 0.50.5, respectively. The same comparison at T=30T=30 yields gains of 7.97, 4.39, and 4.21 points. At p=0.02p=0.02, the best tested schedule with a factor switch (T=75T=75) reaches 32.87 ± 0.28%, compared with 22.03 ± 0.35% for standard LoRA, a gain of 10.84 percentage points. A shared interval therefore yields gains across the three tested communication regimes, while the sweep shows that the choice of TT can substantially affect the size of the gain.

Reviewer_vVikOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies alternating LoRA (train only A or only B in phases of T rounds, as in RoLoRA) in decentralized FL. It argues that if only the active factor is gossiped, clients disagree on the frozen factor, and proposes TAD-LoRA, which gossips both factors every round. Theory (one-step SGD, doubly-stochastic mixing) gives a finite-horizon bound with a factor-product discrepancy term decreasing in T and a coverage term growing as T², whose surrogate minimizer grows as mixing weakens.

Critical strengths

Paper is easy to follow.

Critical weaknesses
  1. The gains over decentralized LoRA depend on choosing T on the test set. TAD-LoRA's numbers in Tables 1-3 and Fig. 2 are the best of six T values (each with an lr grid), picked by mean test accuracy. Each baseline gets one configuration. The resulting average gains are small for p = 0.5…0.01, with 3 seeds and MNLI stds up to 0.04. The only full per-T grid is for MNLI (Fig. 3). There, T=1 is at or below LoRA in 5 of 6 columns, and averaging over T at p=0.2 gives -0.5 pts. The p=0.02 headline also includes LoRA SST-2 = 0.8668 ± 0.0811, which looks like a collapsed seed (LoRA gets 0.9182 at the sparser p=0.01). Using 0.9186 for that cell cuts TAD's margin from 2.3 to about 1.0 pt. Finally, Table 2 and Fig. 3 disagree on MNLI. TAD-LoRA is +0.0138, +0.0319 and +0.0171 at p=0.2, 0.05 and 0.01 in Table 2, but the best heatmap cells are +0.004, +0.023 and +0.023. So I cannot tell which runs the tables report.

  2. The comparisons and the theory don't support the "sparse communication" story as stated. (a) TAD-LoRA sends both factors every round. RoLoRA-DFL as implemented sends only the active factor, and FFA-LoRA only needs B. At equal p, TAD therefore uses twice their bytes, and this confounds Fig. 5. At equal expected bytes (TAD at p vs. baseline at 2p), the picture is mixed: TAD at p=0.1 scores 0.8418 vs. RoLoRA at p=0.2 with 0.8511. (b) Lemma A.9 with m=10 implies the surrogate T_sur should grow about 60-fold from p=0.5 to p=0.01. The observed median best T goes 1->9->4 and reverses at p=0.01 (Table 4). (c) The mixing weights used in the experiments are never given, although A.7 says they are needed to tell whether A.2 applies. Plain neighbour averaging on ER graphs is not doubly stochastic. (d) At p ≤ 0.02, a client is isolated in 83–91% of rounds. Without a local-only reference, gains in this regime could come from alternation acting on local training rather than from mixing.

Other issues

  • section_3 labels Σ w_i² B_iA_i as the "desired average update". It should be Σ w_i B_iA_i, which App. eq. 5 gets right.
  • section_3 says frozen-block discrepancies "accumulate" under active-only mixing. They stay constant, and page_12 correctly says they "do not contract".
  • "Topology-aware": nothing in the method adapts to topology.
  • App. B.2 calls Fig. 6 the non-monotonicity "predicted by our theory", but Fig. 6 peaks at the grid edge T=15. Its caption also says the y-axis is gain while the plot shows accuracy.
  • B.5.1 calls the ring an "extreme stress test", yet ring LoRA MNLI (0.8185) beats ER p=0.5 (0.8132).
  • Tables 5-6 report no std and don't say how T was chosen. Fig. 4 omits FFA-LoRA.
  • Table 2: LoRA average at p=0.05 recomputes to 0.8160, not 0.8173.
  • Positioning: Dec-LoRA (Ghiasvand et al., REALM workshop at ACL 2025) already studies decentralized LoRA with gossip of both factors and a convergence analysis. It is essentially the "LoRA" baseline here and should be cited.
Questions for authors
  1. What mixing matrix did the experiments use (Metropolis, lazy Laplacian, uniform neighbour average)? Is it doubly stochastic?
  2. Can you compare against RoLoRA and FFA-LoRA at equal communicated bytes, and add a local-only run at p ≤ 0.02?
3 comments
AuthorsOpenReview ↗

Response to vVik (1/2) - Selection, communication budget and local-only control

Thank you for identifying the Figure 3 inconsistency and suggesting the communication-budget and local-only comparisons.

1. Hyperparameter selection and variability. All methods use the same grid of five learning rates. For each method and seed, we select the best-performing learning rate and then average across seeds. TAD-LoRA additionally selects the best of six candidate intervals TT by mean test accuracy. This retrospective search gives TAD-LoRA additional tuning flexibility. The main tables report the best observed fixed intervals. Selecting TT independently of the reported evaluation remains open.

The T=1T=1 comparison and the −0.5-0.5-point average across intervals at p=0.2p=0.2 refer to the earlier sweep plotted in Figure 3. They illustrate the sensitivity to TT. We also agree that the variability across three seeds warrants caution. At p=0.02p=0.02, substituting 91.86% for the high-variance SST-2 LoRA entry as a sensitivity check reduces the four-task TAD-LoRA advantage over standard LoRA to 1.05 percentage points. Excluding SST-2 from both methods gives 1.14 points over the other three tasks.

Figure 3 and Tables 2–3. We updated Table 2 after rerunning MNLI, while Figure 3 retained results from the earlier sweep. The updated TAD-LoRA minus DFL LoRA differences are +1.38, +3.19, and +1.71 points at p=0.2p=0.2, 0.050.05, and 0.010.01, respectively. We have regenerated the heatmap and will include it in a later revision. The corrected four-task LoRA average at p=0.05p=0.05 is 0.8160, and the corresponding weak-regime average in Table 3 is approximately 0.7871 from the displayed values.

2(a). Matched expected adapter bytes. TAD-LoRA and standard DFL LoRA both send AA and BB per contact. RoLoRA-DFL and FFA-LoRA each send one factor in our setup. The factors have equal size, so comparing TAD-LoRA at pp with the single-factor baselines at 2p2p matches expected adapter bytes over the same number of rounds. The four-task averages from Table 2 are:

TAD ppBaseline ppTAD-LoRARoLoRA-DFLFFA-LoRA
0.010.0277.76%77.17%78.17%
0.050.1083.42%82.60%82.56%
0.100.2084.18%85.11%83.80%

Each method leads in one of the three comparisons. Realized traffic can vary across seeds.

2(d). Local-only control. We ran severe non-IID MNLI with zero contacts (p=0p=0), retaining the corresponding main-table training settings and using T=15T=15 for local alternation. Across three seeds, standard local LoRA achieved 35.25 ± 0.52%, and local alternating LoRA achieved 35.12 ± 0.86%. The paired difference was −0.14 ± 0.43 points, compared with +1.97 ± 3.08 points in the updated main-table comparison at p=0.02p=0.02. The mean gain observed with communication was absent in the local-only control, and the gain at p=0.02p=0.02 varied across seeds.

AuthorsOpenReview ↗

Response to vVik (2/2) - Mixing matrix, theory and manuscript corrections

2(b–c). Mixing matrix and scope of the surrogate. We use sequential pairwise gossip averaging. Within each round we process the sampled edges in order. At each contact, both endpoints replace each communicated factor with the arithmetic mean of their two pre-contact copies. All methods use the same contact order and averaging rule for a given trace. Each contact operator is doubly stochastic, and the round-level operator is their ordered product, which is also doubly stochastic. Under independent ER sampling with 0<p<10<p<1, each single-edge round has positive probability. Together with nonexpansiveness of the pairwise operators, this gives strict expected contraction on the centered subspace for a communicated factor.

Lemma A.9 analyzes a simultaneous lazy-Laplacian update. The approximately 60-fold change applies to the continuous surrogate under that example's mapping from pp to ρ\rho. The empirical intervals maximize final accuracy over a finite grid. The median increases from 1 to 9 as pp drops from 0.5 to 0.02, then falls to 4 at p=0.01p=0.01. We interpret this as a qualitative tendency over part of the range. We also agree that “topology-aware” overstates the implementation. We select TT offline for each evaluated communication setting and hold it fixed during training.

Figure 5 and the paired mixing control. Figure 5 holds contact opportunities fixed and tests the effect of also exchanging the frozen factor, which doubles the adapter payload. Some archived Figure 5 cells use different or incomplete seed sets. To obtain a strictly paired comparison, we reran both mixing variants on severe non-IID MNLI at p=0.02p=0.02 and T=15T=15, matching contact traces, learning rates, and local computation. Joint mixing scored 63.93 ± 4.79% versus 55.37 ± 4.81% for active-only mixing, a paired gain of 8.57 ± 1.70 points over three seeds. The bidirectional adapter payload was 6 MiB versus 3 MiB per contact. This result measures the benefit at fixed contact opportunities.

Other corrections and clarifications.

  • §3. The target average is ∑iwiBiAi\sum_i w_iB_iA_i. The term ∑iwi2BiAi\sum_i w_i^2B_iA_i is the diagonal part of the product of the averaged factors. Under active-only mixing, pre-existing differences in the frozen factor persist within the phase.
  • Figure 6 and the ring experiment. Figure 6 plots accuracy, with the highest observed point at the grid boundary T=15T=15. We will describe it as an interval sensitivity plot and correct its caption. We will describe the ring as a structured topology comparison, consistent with the reported LoRA MNLI accuracies of 81.85% on the ring and 81.32% on ER at p=0.5p=0.5.
  • Tables 5–6 and Figure 4. Tables 5–6 currently report point estimates. We agree that they should state how TT was selected and report across-seed variability. Figure 4 omits FFA-LoRA. Its AA factor is fixed, and the trajectory of its trainable BB remains a relevant comparison.
  • Dec-LoRA. We will cite it as directly relevant prior work for our standard DFL LoRA baseline. It uses gossip on both factors and analyzes convergence. Dec-LoRA

Thank you for the detailed rebuttal. I raised my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.