- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper extends RoLoRA (alternating trainable LoRA factors) from centralized federated learning to decentralized federated learning. In the decentralized setting, the major problem is that the clients may enter the same phase but with different versions of the frozen factors, which invalidates the alternating optimization idea. To solve this problem, authors do the aggregation (mixing) for both factors (line 238) to make both factors updated. Authors conduct experiments on GLUE datasets, TAD-LoRA achieves slight performance gain in certain settings.
Critical strengths
The decentralized federated LoRA is a relatively underexplored topic. The structure and writing of the draft is acceptable and good for readers to understand. Ablation results seem to be rich enough.
Critical weaknesses
I admire authors time and commitment on this work, while I feel this draft is not ready to be published as a full paper. The major "novelty" seems to be the equation of line 238 only, while this mixing rule itself is also standard for prior decentralized FL work. Other parts of the paper are mainly interpretation and explanation on existing works (RoLoRA).
- Related work section is very limited. For section 2.1, authors may consider provide more background on decentralized FL, as this is not a default setting, .e.g, mention the mixing matrix W_t earlier, explain how standard decentralized FL conducts aggregation. For section 2.2, I want to remind authors that some recent work starts to do the alignment for factors B or A to reduce the aggregation noise (to name a few, FedRot-LoRA, FedPA-LoRA), the current draft fully ignored this direction.
- The paper feels like a decentralized version of prior RoLoRA. The equation of line 238 feels like a standard extension instead of a solid contribution. Also, if you mix B and A together, how can you ensure the alternating optimization idea still validates?
- Limited baselines for experiments. Please consider include more federated LoRA and standard decentralized FL methods.
Questions for authors
I have no further questions. My major concern is that the paper's contribution/novelty is too limited for a full paper.
Thank you for raising the questions about novelty, alternating optimization, and baseline coverage.
1. Contribution and validity of alternating optimization. We agree that the neighbor averaging in Eq. 238 is standard. Our contribution is to identify and analyze a condition that changes when alternating LoRA moves from server-based aggregation to peer-to-peer mixing. In RoLoRA, the server synchronizes the factor that will be frozen in the next phase. With a shared frozen factor, averaging the active factors also averages the corresponding adapter products. In a peer-to-peer network, clients can enter the same phase with different versions of the frozen factor, so this identity can fail. Active-only mixing leaves that disagreement unchanged throughout the phase.
In TAD-LoRA, during an A-phase, each client holds B fixed throughout its local optimizer steps and updates only A. At communication, each client averages A and B separately using the same mixing matrix. Thus, the local optimization still alternates between factors. Under the stated mixing assumptions, communication contracts disagreement in the locally frozen factor. Our analysis characterizes the trade-off between frozen-factor disagreement and delayed updates to each factor as the switching interval changes.
2. Controlled comparison of active-only and joint mixing. On severely non-IID MNLI with RoBERTa-Large, p=0.02, and T=15, we ran a fresh paired comparison with identical seeds, learning rates, communication graphs, mixing weights, and local computation. Active-only mixing achieved 55.37±4.81% accuracy, and joint mixing achieved 63.93±4.79%. Joint mixing improved accuracy in all three seeds, with a paired gain of 8.57±1.70 percentage points. This comparison fixes contact opportunities. Joint mixing uses twice the adapter payload per contact of active-only mixing and the same adapter payload as standard decentralized LoRA.
3. Larger-model results. We evaluated Qwen2.5-3B on MATH with 10 clients, a Dirichlet partition (α=0.3), and 150 rounds. The table compares standard decentralized LoRA with different alternating schedules at a common learning rate of 5×10−5. Each client was evaluated on 200 examples from the official MATH test split. Accuracy (%) is reported as mean ± standard deviation across three training seeds, except where marked.
Across the reported MATH schedules, the best observed TAD-LoRA configuration at each communication rate improved over standard decentralized LoRA by 10.84, 4.39, and 4.21 percentage points at p=0.02, 0.1, and 0.5, respectively. The largest gain was at p=0.02, where TAD-LoRA with T=75 achieved 32.87% versus 22.03% for standard decentralized LoRA. This schedule switched once over 150 rounds. In the fixed-T=1 ablation at the same communication rate, joint mixing achieved 26.63% versus 24.25% for active-only mixing.
4. Decentralized FL background and related work. We will introduce the time-varying mixing matrix Wt and standard neighbor averaging earlier in Sec. 2.1. In Sec. 2.2, we will discuss FedRot-LoRA's rotational alignment and FedPA-LoRA's product alignment and explain how they relate to the frozen-factor disagreement we study in peer-to-peer alternating optimization.