- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates federated LoRA fine-tuning. The proposed method, CFLoRA, introduces a shared complementary channel masking strategy: for each latent LoRA channel, during each communication round, either the corresponding column of B or the corresponding row of A is updated, but not both simultaneously. Since these two updates are complementary, the bilinear cross-terms disappear, enabling factor-level aggregation to precisely reproduce the product updates induced by constrained local endpoints. This framework is further extended to heterogeneous client ranking. Experiments also demonstrate its effectiveness.
Critical strengths
The method is operationally simple. CFLoRA does not require repeated high-dimensional SVDs or server-side reconstruction. This method eliminates LoRA aggregation bias through the use of complementary masks.
Critical weaknesses
1.The method offers limited innovation, and the complementary channel selection mechanism lacks a sufficient design rationale. The complementary channel update mechanism proposed in this paper is essentially similar to existing single-factor freezing or alternating factor update methods; the main difference lies in refining the factor-level alternating updates to the LoRA channel level. This update constraint may limit the model’s optimization flexibility and affect training efficiency under a limited number of communication rounds. This paper employs a random mechanism based on the Bernoulli distribution to determine which channels are updated. The authors also fail to explain why this simple random allocation strategy is reasonable or effective. 2.The approach to handling heterogeneous-rank clients is relatively straightforward, and there is a significant gap between the relevant theoretical guarantees and the actual optimization objectives. To address the heterogeneous rank constraints across clients, this paper primarily enables different clients to train low-rank submodels of varying scales by randomly selecting different numbers of channels from the server-side LoRA parameters. However, this design is essentially a random parameter subset selection strategy, and it lacks an in-depth theoretical explanation for why specific channels should be selected for different clients, or how such selection effectively promotes global model optimization.
Questions for authors
- The experiments suggest that (p=0.9) achieves relatively strong performance. However, this setting heavily favors updates to factor (B), potentially limiting the adaptation of factor (A), especially for low-rank clients. Could the authors provide further justification and analysis for this choice?
- According to Eq. (16), varying (p) also changes the effective factor-specific learning rates through (1/p) and (1/(1-p)), even when the base learning rate remains fixed. How does this affect the experimental conclusions? A controlled ablation that isolates the effect of (p) would be helpful.
- What Dirichlet concentration parameter is used for non-IID data partitioning? The authors should provide additional experiments with different values to evaluate the robustness of CFLoRA under varying degrees of data heterogeneity.
5 comments
Response (2/2): Answers to Questions & Convergence Guarantees
Q1. Justification of and Adaptation of Factor for Low-Rank Clients
We justify from both empirical evidence and theoretical analysis:
i) does not starve factor : Although updates on 10% of active channels per round, inverse-probability scaling () preserves unbiased global gradients (Lemma A.1) and the same expected cumulative step size. Meanwhile, uniform channel sampling () ensures low-rank clients can still update all rows. Empirically, if were starved at , accuracy would collapse toward (frozen , as in FFA-LoRA); instead, those 10% updates yield substantial gains over across both homogeneous and heterogeneous settings: +4.39% / +2.55% on RoBERTa homo/hetero (Tables 3–4) and +4.54% / +3.64% on LLaMA-3.2-3B homo/hetero (Tables D.5–D.6). This aligns with prior LoRA studies showing that while requires heavier task-specific updates, lightweight co-adaptation of is sufficient yet critical (Hayou et al., 2024; Chen et al., 2025).
ii) Theoretical justification for asymmetric : As noted in our response to Reviewer p6wj, Corollary 4 favors only because Eq. (35) in Appendix A.5 bounds the exact factor-decomposed variance via a factor-agnostic worst-case bound with . Retaining separate gradient second-moment bounds for and in Eq. (35) yields the tighter factor-aware variance , minimized at
Consistent with LoRA+ (Hayou et al., 2024), since zero-initialized directly drives output-space adaptation whereas randomly initialized acts as an input projection, often exhibits larger gradient energy during fine-tuning (), shifting into the regime. We have added this discussion in Section 4.4, Appendix A.7, and Limitations.
Q2. Isolating the Effect of from Factor-Specific Learning Rates
As shown in Lemma A.1 (Eq. 33), the scaling and in Eq. (16) serves as an inverse-probability correction. Without this scaling, varying would shrink expected step sizes to and ; Eq. (16) therefore ensures the expected effective step size per local step remains for both and across all .
Symmetric learning rate already reported at : We would like to point out that factor-specific learning-rate scaling is already isolated at , where eliminates any relative learning-rate asymmetry between and . Even at , CFLoRA achieves 78.70% (RoBERTa homo, Table 3), 78.74% (RoBERTa hetero, Table 4), 73.40% (LLaMA homo, Table D.5), and 74.32% (LLaMA hetero, Table D.6), consistently outperforming every baseline across all four settings, while varying further improves accuracy within a stable ~1% range (Appendix D.3).
Q3. Dirichlet Concentration Parameter and Non-IID Robustness
Thank you for catching this, and we apologize for unintentionally omitting the Dirichlet parameter in the original manuscript. Following the standard federated settings in FSLoRA and AS-LoRA, our main experiments use (severe non-IID), which we have now explicitly stated in Section 5.
Furthermore, we added a sensitivity study across in Section 5.3 (Tables 5–6) (also shown in Tables R1–R2 in our response to Reviewer 2VDX). Together with our main results, CFLoRA consistently achieves the highest average accuracy across all regimes, with its margin over baselines widening under severe heterogeneity (: +3.04% homo / +2.37% hetero).
Remark on Theoretical Guarantees vs. Optimization Objectives
Regarding our convergence guarantees, we believe we provide non-trivial theoretical results demonstrating the effectiveness of CFLoRA. Under homogeneous ranks (), Corollary 4 (Eq. 21) guarantees convergence directly on the exact LoRA objective with zero non-vanishing error floor. Under heterogeneous ranks (), we added Corollary A.2 in Appendix A.7 (Eqs. 49–50) (see our response to Reviewer p6wj) proving that CFLoRA converges at rate to an neighborhood of .
Thank you again for your time and review of our submission. We would like to check whether our clarifications, theoretical extensions, and additional experiments have addressed your concerns, and whether you would kindly consider re-evaluating our work in light of the rebuttal. If any concerns remain, we would be more than happy to address them before the discussion phase closes.
Response (1/2): Methodological Innovation & Channel Sampling Rationale
Thank you for your time and comments. We respond to the weaknesses and questions point by point.
Methodological Innovation & Theoretical Rationale
We respectfully clarify that CFLoRA differs fundamentally from existing factor-freezing or alternating methods, and argue that its simplicity is a design strength rather than a limitation.
i) CFLoRA exploits a matrix-multiplication insight overlooked by prior work: Aggregation error in federated LoRA stems from the bilinear cross-term ΔBiΔAk from client i and k. We observe that, by basic matrix multiplication rule (ΔBiΔAk)x,y=∑j=1R(ΔBi)x,j(ΔAk)j,y, freezing either column j of B or row j of A in each channel j eliminates cross-terms without freezing entire B or A. Existing exact methods, e.g., single-factor freezing (FFA-LoRA) or round/layer alternation (RoLoRA, AS-LoRA), have not exploited this, preventing A and B from co-adaptation within the same layer in a round. We revised and highlighted the intuition in Section 4.1. To the best of our knowledge, this is the first work that simultaneously updates both factors within a round with exact factor-wise aggregation and distribution in federated LoRA.
ii) Theoretical and empirical necessity of intra-round co-adaptation: Unlike factor-freezing baselines that freeze A or B for K steps, CFLoRA can update them simultaneously. One of our contributions is to show its necessity. Theoretically, simultaneous updates with p∈(0,1) guarantee O(1/T) convergence to the full LoRA objective with zero non-vanishing error floor (Theorem 3, Corollary 4), whereas single-factor freezing (p=1) restricts optimization and diverges as p→1. Empirically, under identical rounds and matched 50% parameter/uplink budgets, CFLoRA outperforms FFA-LoRA, RoLoRA, and AS-LoRA by +4.17% to +4.40% on GLUE homogeneous (Table 1), +7.18% to +12.24% on GLUE heterogeneous (Table 2), and +2.01% to +4.66% on LLaMA-3.2-3B (Table D.3), while p=1.0 suffers clear accuracy drops (Section 5.2).
iii) Lightweight Bernoulli channel allocation is principled: First, the R inner LoRA dimensions are permutation-invariant (BPP⊤A=BA), so no channel j is a priori privileged. Second, sampling zjt∼Bernoulli(p) independently of Xt and local data ensures E[Zt]=pIR and E[IR−Zt]=(1−p)IR. With scaling in Eq. (16), masked aggregated updates remain unbiased estimators of full-matrix global gradients (E[GBt]=∇BΦ(Xt), E[GAt]=∇AΦ(Xt); Lemma A.1), eliminating the selection bias while communicating only a shared random seed. Conversely, deterministic or norm-based heuristics couple Zt with the model state (E[Zt∣SBt]=pIR), breaking unbiasedness and adding overhead.
Rationale for Uniform Channel Sampling (Eit) in Heterogeneous-Rank Settings
Under heterogeneous ranks, we uniformly sample ri LoRA channels for two key reasons:
i) Zero selection overhead under memory budget: Under CFLoRA, client i only downloads and trains compact rank-ri factors B~it=BtEit∈Rdout×ri and A~it=(Eit)⊤At∈Rri×din (Section 4.2, Eqs. 9–11). By contrast, data-driven selection (e.g., ranking all R channels by gradient norm on Di) would often require downloading the full rank-R adapter and running a full rank-R forward/backward pass before training, incurring extra overhead and exceeding the memory budget.
ii) Unbiased parameter updates: Channel selection such as norm-based methods couples Hit with Xt, violating unbiasedness. Conversely, uniformly sampling Hit=Eit(Eit)⊤∼Hi yields E[Hit]=RriIR, ensuring every client's Di updates all R global channels uniformly and unbiasedly (EH∼Hi[∇fiH(Xt)]=∇ϕi(Xt); Lemma A.1, Eq. 32). Empirically, despite not relying on energy-based SVD extraction or heuristic channel selection, our simple uniform sampling, paired with exact aggregation, consistently outperforms principal-subspace SVD method FlexLoRA (+3.54% on GLUE / +3.77% on LLaMA-3.2-3B).