acceptodds
Under review as a conference paper at ICLR 2027

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Abstract

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present *CFLoRA*, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates *CFLoRA* achieves convergence rate of the *original* LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that *CFLoRA* achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Primary area:
optimization
Keywords:
Federated learning, Low-rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9TuWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates federated LoRA fine-tuning. The proposed method, CFLoRA, introduces a shared complementary channel masking strategy: for each latent LoRA channel, during each communication round, either the corresponding column of B or the corresponding row of A is updated, but not both simultaneously. Since these two updates are complementary, the bilinear cross-terms disappear, enabling factor-level aggregation to precisely reproduce the product updates induced by constrained local endpoints. This framework is further extended to heterogeneous client ranking. Experiments also demonstrate its effectiveness.

Critical strengths

The method is operationally simple. CFLoRA does not require repeated high-dimensional SVDs or server-side reconstruction. This method eliminates LoRA aggregation bias through the use of complementary masks.

Critical weaknesses

1.The method offers limited innovation, and the complementary channel selection mechanism lacks a sufficient design rationale. The complementary channel update mechanism proposed in this paper is essentially similar to existing single-factor freezing or alternating factor update methods; the main difference lies in refining the factor-level alternating updates to the LoRA channel level. This update constraint may limit the model’s optimization flexibility and affect training efficiency under a limited number of communication rounds. This paper employs a random mechanism based on the Bernoulli distribution to determine which channels are updated. The authors also fail to explain why this simple random allocation strategy is reasonable or effective. 2.The approach to handling heterogeneous-rank clients is relatively straightforward, and there is a significant gap between the relevant theoretical guarantees and the actual optimization objectives. To address the heterogeneous rank constraints across clients, this paper primarily enables different clients to train low-rank submodels of varying scales by randomly selecting different numbers of channels from the server-side LoRA parameters. However, this design is essentially a random parameter subset selection strategy, and it lacks an in-depth theoretical explanation for why specific channels should be selected for different clients, or how such selection effectively promotes global model optimization.

Questions for authors
  1. The experiments suggest that (p=0.9) achieves relatively strong performance. However, this setting heavily favors updates to factor (B), potentially limiting the adaptation of factor (A), especially for low-rank clients. Could the authors provide further justification and analysis for this choice?
  2. According to Eq. (16), varying (p) also changes the effective factor-specific learning rates through (1/p) and (1/(1-p)), even when the base learning rate remains fixed. How does this affect the experimental conclusions? A controlled ablation that isolates the effect of (p) would be helpful.
  3. What Dirichlet concentration parameter is used for non-IID data partitioning? The authors should provide additional experiments with different values to evaluate the robustness of CFLoRA under varying degrees of data heterogeneity.
5 comments
AuthorsOpenReview ↗

Response (1/2): Methodological Innovation & Channel Sampling Rationale

Thank you for your time and comments. We respond to the weaknesses and questions point by point.

Methodological Innovation & Theoretical Rationale

We respectfully clarify that CFLoRA differs fundamentally from existing factor-freezing or alternating methods, and argue that its simplicity is a design strength rather than a limitation.

i) CFLoRA exploits a matrix-multiplication insight overlooked by prior work: Aggregation error in federated LoRA stems from the bilinear cross-term ΔBiΔAk\Delta\mathbf{B}_i \Delta\mathbf{A}_k from client ii and kk. We observe that, by basic matrix multiplication rule (ΔBiΔAk)x,y=∑j=1R(ΔBi)x,j(ΔAk)j,y(\Delta\mathbf{B}_i\Delta\mathbf{A}_k)_{x,y} = \sum_{j=1}^{R} (\Delta\mathbf{B}_i)_{x,j}(\Delta\mathbf{A}_k)_{j,y}, freezing either column jj of B\mathbf{B} or row jj of A\mathbf{A} in each channel jj eliminates cross-terms without freezing entire B\mathbf{B} or A\mathbf{A}. Existing exact methods, e.g., single-factor freezing (FFA-LoRA) or round/layer alternation (RoLoRA, AS-LoRA), have not exploited this, preventing A\mathbf{A} and B\mathbf{B} from co-adaptation within the same layer in a round. We revised and highlighted the intuition in Section 4.1. To the best of our knowledge, this is the first work that simultaneously updates both factors within a round with exact factor-wise aggregation and distribution in federated LoRA.

ii) Theoretical and empirical necessity of intra-round co-adaptation: Unlike factor-freezing baselines that freeze A\mathbf{A} or B\mathbf{B} for KK steps, CFLoRA can update them simultaneously. One of our contributions is to show its necessity. Theoretically, simultaneous updates with p∈(0,1)p \in (0,1) guarantee O(1/T)\mathcal{O}(1/\sqrt{T}) convergence to the full LoRA objective with zero non-vanishing error floor (Theorem 3, Corollary 4), whereas single-factor freezing (p=1p=1) restricts optimization and diverges as p→1p \to 1. Empirically, under identical rounds and matched 50% parameter/uplink budgets, CFLoRA outperforms FFA-LoRA, RoLoRA, and AS-LoRA by +4.17% to +4.40% on GLUE homogeneous (Table 1), +7.18% to +12.24% on GLUE heterogeneous (Table 2), and +2.01% to +4.66% on LLaMA-3.2-3B (Table D.3), while p=1.0p=1.0 suffers clear accuracy drops (Section 5.2).

iii) Lightweight Bernoulli channel allocation is principled: First, the RR inner LoRA dimensions are permutation-invariant (BPP⊤A=BA\mathbf{B}\mathbf{P}\mathbf{P}^\top\mathbf{A} = \mathbf{B}\mathbf{A}), so no channel jj is a priori privileged. Second, sampling zjt∼Bernoulli(p)z_j^t \sim \mathrm{Bernoulli}(p) independently of Xt\mathbf{X}^t and local data ensures E[Zt]=pIR\mathbb{E}[\mathbf{Z}^t] = p\mathbf{I}_R and E[IR−Zt]=(1−p)IR\mathbb{E}[\mathbf{I}_R - \mathbf{Z}^t] = (1-p)\mathbf{I}_R. With scaling in Eq. (16), masked aggregated updates remain unbiased estimators of full-matrix global gradients (E[GBt]=∇BΦ(Xt)\mathbb{E}[\mathbf{G}_B^t] = \nabla_{\mathbf{B}}\Phi(\mathbf{X}^t), E[GAt]=∇AΦ(Xt)\mathbb{E}[\mathbf{G}_A^t] = \nabla_{\mathbf{A}}\Phi(\mathbf{X}^t); Lemma A.1), eliminating the selection bias while communicating only a shared random seed. Conversely, deterministic or norm-based heuristics couple Zt\mathbf{Z}^t with the model state (E[Zt∣SBt]≠pIR\mathbb{E}[\mathbf{Z}^t \mid \mathbf{S}_B^t] \neq p\mathbf{I}_R), breaking unbiasedness and adding overhead.


Rationale for Uniform Channel Sampling (Eit\mathbf{E}_i^t) in Heterogeneous-Rank Settings

Under heterogeneous ranks, we uniformly sample rir_i LoRA channels for two key reasons:

i) Zero selection overhead under memory budget: Under CFLoRA, client ii only downloads and trains compact rank-rir_i factors B~it=BtEit∈Rdout×ri\tilde{\mathbf{B}}_i^t = \mathbf{B}^t\mathbf{E}_i^t \in \mathbb{R}^{d_{\mathrm{out}}\times r_i} and A~it=(Eit)⊤At∈Rri×din\tilde{\mathbf{A}}_i^t = (\mathbf{E}_i^t)^\top\mathbf{A}^t \in \mathbb{R}^{r_i \times d_{\mathrm{in}}} (Section 4.2, Eqs. 9–11). By contrast, data-driven selection (e.g., ranking all RR channels by gradient norm on Di\mathcal{D}_i) would often require downloading the full rank-RR adapter and running a full rank-RR forward/backward pass before training, incurring extra overhead and exceeding the memory budget.

ii) Unbiased parameter updates: Channel selection such as norm-based methods couples Hit\mathbf{H}_i^t with Xt\mathbf{X}^t, violating unbiasedness. Conversely, uniformly sampling Hit=Eit(Eit)⊤∼Hi\mathbf{H}_i^t = \mathbf{E}_i^t(\mathbf{E}_i^t)^\top \sim \mathcal{H}_i yields E[Hit]=riRIR\mathbb{E}[\mathbf{H}_i^t] = \frac{r_i}{R}\mathbf{I}_R, ensuring every client's Di\mathcal{D}_i updates all RR global channels uniformly and unbiasedly (EH∼Hi[∇fiH(Xt)]=∇ϕi(Xt)\mathbb{E}_{\mathbf{H}\sim\mathcal{H}_i}[\nabla f_i^{\mathbf{H}}(\mathbf{X}^t)] = \nabla \phi_i(\mathbf{X}^t); Lemma A.1, Eq. 32). Empirically, despite not relying on energy-based SVD extraction or heuristic channel selection, our simple uniform sampling, paired with exact aggregation, consistently outperforms principal-subspace SVD method FlexLoRA (+3.54% on GLUE / +3.77% on LLaMA-3.2-3B).

AuthorsOpenReview ↗

Response (2/2): Answers to Questions & Convergence Guarantees

Q1. Justification of p=0.9p=0.9 and Adaptation of Factor A\mathbf{A} for Low-Rank Clients

We justify p=0.9p=0.9 from both empirical evidence and theoretical analysis:

i) p=0.9p=0.9 does not starve factor A\mathbf{A}: Although 1−p=0.11-p = 0.1 updates A\mathbf{A} on 10% of active channels per round, inverse-probability scaling (11−p\frac{1}{1-p}) preserves unbiased global gradients (Lemma A.1) and the same expected cumulative step size. Meanwhile, uniform channel sampling (E[Hit]=riRIR\mathbb{E}[\mathbf{H}_i^t] = \frac{r_i}{R}\mathbf{I}_R) ensures low-rank clients can still update all RR rows. Empirically, if A\mathbf{A} were starved at p=0.9p=0.9, accuracy would collapse toward p=1.0p=1.0 (frozen A\mathbf{A}, as in FFA-LoRA); instead, those 10% A\mathbf{A} updates yield substantial gains over p=1.0p=1.0 across both homogeneous and heterogeneous settings: +4.39% / +2.55% on RoBERTa homo/hetero (Tables 3–4) and +4.54% / +3.64% on LLaMA-3.2-3B homo/hetero (Tables D.5–D.6). This aligns with prior LoRA studies showing that while B\mathbf{B} requires heavier task-specific updates, lightweight co-adaptation of A\mathbf{A} is sufficient yet critical (Hayou et al., 2024; Chen et al., 2025).

ii) Theoretical justification for asymmetric p>0.5p > 0.5: As noted in our response to Reviewer p6wj, Corollary 4 favors p=0.5p=0.5 only because Eq. (35) in Appendix A.5 bounds the exact factor-decomposed variance E∥Rt(u)∥F2=∥uB∥F2p+∥uA∥F21−p\mathbb{E}\\\|\mathcal{R}_t(\mathbf{u})\\\|_F^2 = \frac{\\\|\mathbf{u}_B\\\|_F^2}{p} + \frac{\\\|\mathbf{u}_A\\\|_F^2}{1-p} via a factor-agnostic worst-case bound κ∥u∥F2\kappa\\\|\mathbf{u}\\\|_F^2 with κ=max⁡1/p,1/(1−p)\kappa = \max\\{1/p, 1/(1-p)\\}. Retaining separate gradient second-moment bounds (MB,MA)(M_B, M_A) for B\mathbf{B} and A\mathbf{A} in Eq. (35) yields the tighter factor-aware variance MBp+MA1−p\frac{M_B}{p} + \frac{M_A}{1-p}, minimized at p⋆=MBMB+MA.p^\star = \frac{\sqrt{M_B}}{\sqrt{M_B}+\sqrt{M_A}}.

Consistent with LoRA+ (Hayou et al., 2024), since zero-initialized B\mathbf{B} directly drives output-space adaptation whereas randomly initialized A\mathbf{A} acts as an input projection, B\mathbf{B} often exhibits larger gradient energy during fine-tuning (MB>MAM_B > M_A), shifting p⋆>0.5p^\star > 0.5 into the [0.7,0.9][0.7, 0.9] regime. We have added this discussion in Section 4.4, Appendix A.7, and Limitations.


Q2. Isolating the Effect of pp from Factor-Specific Learning Rates

As shown in Lemma A.1 (Eq. 33), the scaling ηB=η/p\eta_B = \eta/p and ηA=η/(1−p)\eta_A = \eta/(1-p) in Eq. (16) serves as an inverse-probability correction. Without this scaling, varying pp would shrink expected step sizes to pηp\eta and (1−p)η(1-p)\eta; Eq. (16) therefore ensures the expected effective step size per local step remains η\eta for both B\mathbf{B} and A\mathbf{A} across all p∈(0,1)p \in (0,1).

Symmetric learning rate already reported at p=0.5p=0.5: We would like to point out that factor-specific learning-rate scaling is already isolated at p=0.5p=0.5, where ηB=ηA=2η\eta_B = \eta_A = 2\eta eliminates any relative learning-rate asymmetry between B\mathbf{B} and A\mathbf{A}. Even at p=0.5p=0.5, CFLoRA achieves 78.70% (RoBERTa homo, Table 3), 78.74% (RoBERTa hetero, Table 4), 73.40% (LLaMA homo, Table D.5), and 74.32% (LLaMA hetero, Table D.6), consistently outperforming every baseline across all four settings, while varying p∈[0.5,0.9]p \in [0.5, 0.9] further improves accuracy within a stable ~1% range (Appendix D.3).


Q3. Dirichlet Concentration Parameter and Non-IID Robustness

Thank you for catching this, and we apologize for unintentionally omitting the Dirichlet parameter α\alpha in the original manuscript. Following the standard federated settings in FSLoRA and AS-LoRA, our main experiments use α=0.1\alpha = 0.1 (severe non-IID), which we have now explicitly stated in Section 5.

Furthermore, we added a sensitivity study across α∈0.5,1.0\alpha \in \\{0.5, 1.0\\} in Section 5.3 (Tables 5–6) (also shown in Tables R1–R2 in our response to Reviewer 2VDX). Together with our α=0.1\alpha = 0.1 main results, CFLoRA consistently achieves the highest average accuracy across all regimes, with its margin over baselines widening under severe heterogeneity (α=0.1\alpha = 0.1: +3.04% homo / +2.37% hetero).


Remark on Theoretical Guarantees vs. Optimization Objectives

Regarding our convergence guarantees, we believe we provide non-trivial theoretical results demonstrating the effectiveness of CFLoRA. Under homogeneous ranks (ri=Rr_i = R), Corollary 4 (Eq. 21) guarantees O(1/T)\mathcal{O}(1/\sqrt{T}) convergence directly on the exact LoRA objective with zero non-vanishing error floor. Under heterogeneous ranks (ri<Rr_i < R), we added Corollary A.2 in Appendix A.7 (Eqs. 49–50) (see our response to Reviewer p6wj) proving that CFLoRA converges at rate O(1/T)\mathcal{O}(1/\sqrt{T}) to an O(ρˉ2)\mathcal{O}(\bar{\rho}^2) neighborhood of F(X)F(\mathbf{X}).

Thank you again for your time and review of our submission. We would like to check whether our clarifications, theoretical extensions, and additional experiments have addressed your concerns, and whether you would kindly consider re-evaluating our work in light of the rebuttal. If any concerns remain, we would be more than happy to address them before the discussion phase closes.

Reviewer_p6wjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
Paper Summary
  1. CFLoRA addresses the mismatch between averaging LoRA factors separately and averaging their products through a shared complementary channel mask.
  2. Each round updates either a column of B or its corresponding row of A, ensuring exact aggregation without rank expansion; client-specific channel selection accommodates heterogeneous ranks.
  3. The paper establishes convergence guarantees and reports improved accuracy and communication efficiency on RoBERTa/GLUE and LLaMA-3.2-3B commonsense reasoning benchmarks.
  4. I feel this algorithm share a similar spirit of FSLoRA.
Critical strengths
Critical Strengths
  1. Simple and mathematically clear design. Complementary masks eliminate bilinear update interactions, providing an exact aggregation identity with minimal changes to standard federated LoRA.
  2. Useful support for resource heterogeneity. The method accommodates different client ranks while preserving aggregation exactness and reducing uplink communication through selective updates.
  3. Consistent empirical improvements. Experiments cover homogeneous and heterogeneous ranks, multiple baselines, and four random seeds. CFLoRA improves average accuracy on both model families despite training fewer parameters per round.
Critical weaknesses
Critical Weaknesses
  1. The benefit of exact aggregation is insufficiently isolated. Complementary masking changes both the optimization trajectory and aggregation behavior. Controlled comparisons with matched update budgets and factor-specific learning rates, together with measurements of aggregation error, would better establish what drives the gains.
  2. The theoretical guarantees have limited explanatory scope. Under heterogeneous ranks, convergence concerns a masked-submodel surrogate rather than the full-adapter objective. Moreover, the bound favors p=0.5, whereas experiments favor p=0.9, leaving the practical advantage of asymmetric channel allocation unexplained.
  3. Experimental coverage and reproducibility need strengthening. Evaluation is restricted to two model sizes and classification or commonsense benchmarks. Broader generative tasks, sensitivity to local steps and data heterogeneity, and clearer reporting of baseline tuning, optimizers, and partition parameters would strengthen the conclusions.
Questions for authors

Please refer to the weakness.

4 comments
AuthorsOpenReview ↗

Response (1/2): Isolating Exact Aggregation, Analysis of p & Surrogate Gap

Thank you so much for your positive evaluation! We are glad you recognize that our design is simple and mathematically clear.

Isolating the Effect of Exact Aggregation

We provide complementary evidence on the gain of exact aggregation:

i) Isolation of scaling effects (p=0.5p=0.5): The inverse-probability scaling factors 1/p1/p and 1/(1−p)1/(1-p) are essential to ensure unbiased expected updates across all channels. However, when p=0.5p=0.5, the scaling factors for B\mathbf{B} and A\mathbf{A} in Eq. (16) are identical, eliminating learning-rate asymmetry between B\mathbf{B} and A\mathbf{A}. Under such a setting isolating scaling, CFLoRA achieves 78.70% on RoBERTa/GLUE homogeneous (Table 3) and 78.74% on heterogeneous (Table 4), outperforming unmasked FedIT (76.69% homo/67.20% hetero), alternating RoLoRA (75.52% homo/67.33% hetero), and rank-stacking FLoRA (76.01% homo/75.45% hetero).

ii) Comparison against exact baselines with matched update budgets: FFA-LoRA, RoLoRA, and AS-LoRA also update 50% of the LoRA parameters per round with zero aggregation error, yet lag CFLoRA by 4.17%–4.40% on GLUE homogeneous (Table 1) and 2.01%–4.66% on LLaMA-3.2-3B homogeneous (Table D.3). This demonstrates our gain over entire-factor freezing or round/layer alternation under the matched update budgets.

iii) Direct measurement of bilinear aggregation error: To quantify bilinear interference and empirically validate Propositions 1 and 2, we track the relative server-side aggregation error across communication rounds, defined as the normalized Frobenius discrepancy between the reconstructed global update and the exact weighted average of participating clients' local updates. The empirical analysis is provided in Appendix D.6, confirming that while unmasked baselines incur substantial non-zero bilinear error throughout training, CFLoRA maintains exact zero aggregation error at every communication round.


Theoretical Explanation for Asymmetric p=0.9p = 0.9

Thank you for this valid question. Beyond our empirical discussion in Appendix D.3 already in the original manuscript, we can further explain the theoretical preference for p=0.5p = 0.5 and the empirical superiority of p>0.5p > 0.5 from the same exact factor-decomposed variance in Eq. (35) of Appendix A.5, under two levels of bounding tightness:

  1. Theorem 3 and Corollary 4 yield p=0.5p = 0.5, as they are factor-agnostic: In our manuscript, Eq. (35) upper-bounds the two factor terms using a worst-case bound κ∥u∥F2=max⁡1/p,1/(1−p)(∥uB∥F2+∥uA∥F2)\kappa \\\|\mathbf{u}\\\|_F^2 = \max\\{1/p, 1/(1-p)\\}(\\\|\mathbf{u}_B\\\|_F^2 + \\\|\mathbf{u}_A\\\|_F^2). Since this relaxation does not distinguish the gradient magnitudes of B\mathbf{B} and A\mathbf{A}, minimizing it yields p=0.5p = 0.5.

  2. Optimal p>0.5p > 0.5 (e.g., p=0.9p = 0.9) can be explained from a tighter bound: Instead of merging the two terms via max⁡1/p,1/(1−p)\max\\{1/p, 1/(1-p)\\}, retaining separate gradient second-moment bounds MBM_B and MAM_A for factors B\mathbf{B} and A\mathbf{A} in Eq. (35) yields the factor-aware variance MBp+MA1−p\frac{M_B}{p} + \frac{M_A}{1-p}, which is minimized at p⋆=MBMB+MA.p^\star = \frac{\sqrt{M_B}}{\sqrt{M_B} + \sqrt{M_A}}.

In LoRA fine-tuning, B\mathbf{B} and A\mathbf{A} play asymmetric roles: because B\mathbf{B} is zero-initialized and drives output-space adaptation while A\mathbf{A} acts as a random Gaussian input projection (Hayou et al., 2024), B\mathbf{B} often exhibits larger gradient energy (MB>MAM_B > M_A), implying p⋆>0.5p^\star > 0.5, matching our experiment results. We have added this interpretation to Section 4.4, Appendix A.7, and Limitations of the manuscript.


Bridging the Masked-Submodel Surrogate Φ(X)\Phi(\mathbf{X}) and Full-Adapter Objective F(X)F(\mathbf{X})

We agree that specifying the gap between the surrogate Φ(X)\Phi(\mathbf{X}) and the full-adapter objective F(X)F(\mathbf{X}) under heterogeneous ranks (ri<Rr_i < R) makes our theoretical guarantee more complete. We have added Corollary A.2 in Appendix A.7 to characterize this relationship under smoothness and boundedness conditions:

1T∑t=0T−1E∣∇F(Xt)∣F2≤2T∑t=0T−1E∣∇Φ(Xt)∣F2+2CC2(∑i=1Nwi(1−riR))2.\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\\|\nabla F(\mathbf{X}^t)\\|_F^2 \le \frac{2}{T}\sum_{t=0}^{T-1}\mathbb{E}\\|\nabla \Phi(\mathbf{X}^t)\\|_F^2 + 2C_{\mathcal{C}}^2 \left(\sum_{i=1}^{N} w_i \left(1 - \frac{r_i}{R}\right)\right)^2.

Thus, CFLoRA converges at rate O(T−1/2)\mathcal{O}(T^{-1/2}) to an O(ρˉ2)\mathcal{O}(\bar{\rho}^2) neighborhood of a stationary point of F(X)F(\mathbf{X}), with ρˉ:=∑i=1Nwi(1−ri/R)\bar{\rho} := \sum_{i=1}^{N} w_i (1 - r_i/R). Note that this O(ρˉ2)\mathcal{O}(\bar{\rho}^2) bias is a consequence of rank truncation under heterogeneous ranks. We reiterate that in homogeneous-rank cases, our bound guarantees exact O(T−1/2)\mathcal{O}(T^{-1/2}) convergence to a stationary point of the full-LoRA objective F(X)F(\mathbf{X}) with zero non-vanishing error floor (Corollary 4).

AuthorsOpenReview ↗

Response (2/2): Additional Experiments, Ablations & Details

Additional Experiments, Ablations, and Details

Beyond evaluating RoBERTa on GLUE and LLaMA-3.2-3B-Instruct on commonsense reasoning in the main paper, we followed your suggestions to evaluate an additional model family (Qwen2.5-1.5B) in Table R3 below. To further test the scalability of CFLoRA on larger models and generative instruction-tuning benchmarks, we fine-tune LLaMA-7B on the Wizard, Dolly-15k, and Alpaca datasets and report MMLU accuracy in Table R4, where CFLoRA consistently achieves the best average performance. We have added these additional experiments in Appendix D.5.

We apologize for unintentionally omitting the Dirichlet parameter α\alpha. Following standard federated LoRA in FSLoRA and AS-LoRA, our default non-IID partition uses Dirichlet α=0.1\alpha = 0.1, confirming CFLoRA's effectiveness under severe heterogeneity. We have explicitly stated this in Section 5 (Implementation Details) and Table D.2 of the revised manuscript. Additionally, we conducted a sensitivity study across α∈{0.5,1.0}\alpha \in \\\{0.5, 1.0\\\} in Section 5.3 (see Tables R1–R2 in our response to Reviewer 2VDX), demonstrating that CFLoRA consistently achieves the highest average accuracy under moderate and milder non-IID regimes as well. Finally, we expanded Appendix D.1 to fully document our experimental protocol: non-IID splits use Dirichlet α=0.1\alpha = 0.1 across N=25N=25 clients (q=0.4q=0.4 default; varied over q∈[0.2,1.0]q \in [0.2, 1.0] in Tables D.7–D.9), and local training uses the AdamW optimizer.

Table R3. Testing accuracy for fine-tuning Qwen2.5-1.5B on commonsense reasoning tasks under homogeneous client ranks.

MethodARC-cARC-eBoolQOBQAPIQASIQAAvg.
FedIT67.56 ± 0.4386.39 ± 0.1063.27 ± 0.3664.00 ± 0.8764.28 ± 1.3054.15 ± 0.8366.61
FFA-LoRA68.39 ± 0.4884.26 ± 0.4862.17 ± 0.0167.78 ± 0.6563.88 ± 0.8655.84 ± 0.8767.05
RoLoRA68.12 ± 0.1288.83 ± 0.3665.31 ± 0.2868.83 ± 0.1465.61 ± 1.1656.27 ± 0.7468.83
AS-LoRA70.84 ± 0.7786.64 ± 0.5562.19 ± 0.1566.30 ± 0.4263.35 ± 0.4157.84 ± 0.8767.86
FLoRA71.06 ± 0.4184.23 ± 0.3864.45 ± 0.5167.37 ± 0.5262.25 ± 0.8254.34 ± 0.4767.28
CFLoRA72.87 ± 0.5088.28 ± 0.0865.08 ± 0.1673.60 ± 0.5366.39 ± 1.6460.49 ± 0.5271.12

Table R4. Testing accuracy for fine-tuning LLaMA-7B on the Wizard, Dolly-15k, and Alpaca datasets under homogeneous client ranks.

MethodWizardDolly-15kAlpacaAvg.
FedIT26.61 ± 0.3027.01 ± 0.4227.59 ± 0.2527.07
FFA-LoRA27.16 ± 0.4428.05 ± 0.1326.17 ± 0.0827.13
RoLoRA28.62 ± 0.4130.14 ± 0.2626.33 ± 0.2128.36
AS-LoRA25.90 ± 0.5029.72 ± 0.3428.41 ± 0.1628.01
CFLoRA30.07 ± 0.2933.66 ± 0.1528.26 ± 0.2430.66

Thank you for your response. I will keep my score.

Thank you very much for your follow-up and for maintaining your positive evaluation!

To further address your suggestion regarding sensitivity to local steps, we have also added experiments with varying K×TK \times T (Appendix D.8), where CFLoRA consistently achieves the highest accuracy across configurations.

We truly appreciate your time and constructive feedback!

Reviewer_2VDXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents CFLoRA, which selects complementary channels for LoRA B and A factors for update. CFLoRA only updates one factor per channel and freezes the channel on the other side, ensuring the exact aggregation for federated LoRA finetuning. CFLoRA is able to generalize to heterogeneous rank setting, and the experiment shows empirical evidence that CFLoRA is able to outperform SOTA methods.

Critical strengths
  1. The design of CFLoRA not only restores the exact aggregation for federated learning setup, but also is able to be extended to heterogeneous rank setting. The authors also show effectiveness of CFLoRA for both homo and hetero rank setting.
  2. The paper compares CFLoRA with a comprehensive range of baselines and conducts reasonable ablations, which solidates the effectiveness of CFLoRA.
Critical weaknesses
  1. The paper did not mention the Dirichlet α\alpha for their FL data heterogeneity. The author should report the Dirichlet α\alpha, and if the heterogeneity is not strong enough, it is highly recommended to include an extreme non-IID setup (e.g., α=0.1\alpha = 0.1) in the experiment.
  2. I think it is better for author to explicitly discuss the threat model in this setting, including which information is assumed to be private and which is not. For example, in heterogeneous rank setting, from how many channels are selected to be updated, the server will be able to guess the computation resources of the device and infer the device type. This may or may not leak privacy information, so it will be better to have relevant discussion.
Questions for authors

For now, it seems the server is randomly selecting channels for clients to update. Would this design be further enhanced by making it client-dependent? Maybe depending on client's task type or data, client self-determine which channel is more suitable for training?

4 comments
AuthorsOpenReview ↗

Response (1/2): Dirichlet Heterogeneity Clarifications & Sensitivity Study

Thank you so much for your positive evaluation and for recognizing the effectiveness of our proposed CFLoRA framework!

Dirichlet Parameter α\alpha Clarifications

We apologize for unintentionally omitting the Dirichlet parameter α\alpha and agree that it must be explicitly reported. Following standard federated LoRA settings in FSLoRA and AS-LoRA, our default non-IID partition uses α=0.1\alpha = 0.1. Thus, our experiments already evaluate CFLoRA in the non-IID regime you highlighted, confirming its effectiveness under severe heterogeneity. We have explicitly specified α=0.1\alpha = 0.1 in Section 5 (Implementation Details) and Table D.2 in the revised manuscript.

Importantly, as shown in Section 3.2 (Eqs. 3–4), the bilinear aggregation error of standard factor-wise averaging increases with inter-client update divergence ∥ΔBi−ΔB‾∥\\\|\Delta\mathbf{B}_i - \overline{\Delta\mathbf{B}}\\\| and ∥ΔAi−ΔA‾∥\\\|\Delta\mathbf{A}_i - \overline{\Delta\mathbf{A}}\\\|. Under high heterogeneity (α=0.1\alpha = 0.1), unmasked methods like FedIT suffer severe accuracy degradation due to large update divergence, whereas CFLoRA eliminates cross-client bilinear error regardless of data heterogeneity.

To further evaluate CFLoRA across varying degrees of heterogeneity, we added evaluation across α∈{0.5,1.0}\alpha \in \\\{0.5, 1.0\\\} in Section 5.3 of the main paper (see Tables R1–R2 below), showing that CFLoRA consistently achieves the highest average accuracy under moderate and less non-IID settings. Note that CFLoRA's average accuracy margin over the strongest baseline generally widens as heterogeneity increases: +1.36% (α=1.0\alpha = 1.0), +1.22% (α=0.5\alpha = 0.5), and +3.04% (α=0.1\alpha = 0.1), revealing that CFLoRA performs more robustly than others under data heterogeneity.

Table R1. Accuracy comparison under different data heterogeneity settings using RoBERTa on the GLUE benchmark under homogeneous client ranks.

Data DirMethodQNLIMRPCCoLAMNLI-MMNLI-MMQQPAvg.
Dir(0.5)FedIT90.43 ± 0.9480.49 ± 0.6378.53 ± 0.8686.24 ± 0.0785.81 ± 0.2880.21 ± 1.2083.62
FFA-LoRA88.38 ± 1.4580.98 ± 0.1279.30 ± 1.6383.88 ± 0.2784.10 ± 0.4280.73 ± 1.3182.90
RoLoRA87.83 ± 0.6881.97 ± 0.3780.37 ± 0.9684.59 ± 0.3984.69 ± 0.4781.11 ± 2.0583.43
AS-LoRA87.98 ± 0.7582.39 ± 0.9480.03 ± 1.6285.77 ± 0.2485.89 ± 0.3781.54 ± 1.4183.93
FLoRA89.33 ± 0.7579.97 ± 0.5979.41 ± 1.4683.82 ± 0.3284.12 ± 0.2780.41 ± 2.0482.84
CFLoRA90.19 ± 0.8884.49 ± 0.3481.58 ± 1.2686.12 ± 0.1585.60 ± 0.2082.91 ± 1.2185.15
Dir(1.0)FedIT91.98 ± 0.3282.97 ± 0.6282.19 ± 0.1287.61 ± 0.0787.40 ± 0.0383.39 ± 1.6685.92
FFA-LoRA90.93 ± 0.6283.09 ± 0.2981.85 ± 0.5885.65 ± 0.0886.03 ± 0.0883.71 ± 1.7285.21
RoLoRA91.37 ± 0.3283.38 ± 0.9283.29 ± 0.3786.16 ± 0.0686.29 ± 0.1284.56 ± 1.5985.84
AS-LoRA91.38 ± 0.3084.42 ± 0.9082.40 ± 0.3986.15 ± 0.0586.31 ± 0.0384.33 ± 1.5385.83
FLoRA90.77 ± 0.1583.33 ± 0.4981.92 ± 0.4385.68 ± 0.1385.97 ± 0.0883.64 ± 1.7285.22
CFLoRA92.76 ± 0.4686.57 ± 0.6984.26 ± 0.2787.48 ± 0.0787.31 ± 0.0685.32 ± 1.3787.28

Table R2. Accuracy comparison under different data heterogeneity settings using RoBERTa on the GLUE benchmark under heterogeneous client ranks.

Data DirMethodQNLIMRPCCoLAMNLI-MMNLI-MMQQPAvg.
Dir(0.5)FedIT88.81 ± 0.8582.23 ± 1.1680.73 ± 0.9986.16 ± 0.7386.05 ± 1.4878.87 ± 1.3183.81
FLoRA89.10 ± 0.2879.58 ± 0.4079.17 ± 1.2284.49 ± 0.2584.48 ± 0.1879.23 ± 1.1582.68
FlexLoRA90.51 ± 0.4681.98 ± 1.1980.44 ± 0.3985.82 ± 0.2385.47 ± 0.3880.04 ± 1.2184.04
FSLoRA91.00 ± 1.2881.56 ± 1.4779.81 ± 1.4185.03 ± 0.2484.86 ± 0.1279.63 ± 0.9783.65
CFLoRA92.39 ± 0.4884.52 ± 0.3381.25 ± 0.6585.56 ± 0.1585.40 ± 0.0683.56 ± 0.3885.45
Dir(1.0)FedIT91.78 ± 0.2580.61 ± 1.6081.54 ± 1.6288.08 ± 0.2787.77 ± 0.0683.26 ± 1.0485.51
FLoRA90.98 ± 0.5283.35 ± 1.0682.96 ± 0.2986.70 ± 0.0886.89 ± 0.0581.96 ± 1.7685.47
FlexLoRA91.54 ± 0.1985.13 ± 0.7582.20 ± 0.1087.85 ± 0.3287.96 ± 0.1884.47 ± 1.4486.53
FSLoRA92.63 ± 0.4984.73 ± 0.8282.64 ± 0.4087.25 ± 0.1787.21 ± 0.1684.22 ± 0.3986.45
CFLoRA93.46 ± 0.1487.17 ± 0.4683.92 ± 0.4187.98 ± 0.1487.66 ± 0.0186.32 ± 0.5687.75
AuthorsOpenReview ↗

Response (2/2): Threat Model, Privacy & Channel Selection

Threat Model and Device-Capacity Privacy in Heterogeneous-Rank Settings

Thank you for raising this valid point. In this paper, we primarily focus on eliminating aggregation error and improving training efficiency in federated LoRA under standard FL privacy assumptions. Following existing heterogeneous-rank FL frameworks (FlexLoRA, FLoRA, FSLoRA), we assume an honest-but-curious server where raw client datasets Di\mathcal{D}_i are private, while the rank budget rir_i is a system parameter used to prevent out-of-memory (OOM) failures and reduce communication-computing costs on edge devices. Consequently, in heterogeneous settings, we fully agree that the server can infer clients' computing and/or communication capabilities, since each client uploads parameters of varying sizes for aggregation.

While hiding device capabilities is beyond our main scope, CFLoRA readily supports two mitigations:

1. Coarse-grained rank tiering (kk-Anonymity): Rather than exposing fine-grained memory footprints, CFLoRA maps devices into a small set of discrete canonical rank tiers (ri∈4,8,16,32r_i \in \\{4, 8, 16, 32\\} in Appendix D.1), ensuring that many diverse physical devices fall into the same tier.

2. Compatibility with linear secure aggregation (SecAgg): To completely hide rir_i and the active channel indices Eit\mathbf{E}_i^t from the server altogether, clients can locally sample Eit\mathbf{E}_i^t and zero-pad their compact updates into the global rank-RR space (ΔB~it(Eit)⊤∈Rdout×R\Delta\tilde{\mathbf{B}}_i^t(\mathbf{E}_i^t)^\top \in \mathbb{R}^{d_{\mathrm{out}}\times R} and EitΔA~it∈RR×din\mathbf{E}_i^t\Delta\tilde{\mathbf{A}}_i^t \in \mathbb{R}^{R\times d_{\mathrm{in}}}) before applying cryptographic SecAgg masks under standard linear secure aggregation protocols [R1, R2]. Because CFLoRA’s server update in Eq. (13) is linear and uses a shared Zt\mathbf{Z}^t, Proposition 2 guarantees exact aggregation over the SecAgg sum without exposing individual client ranks rir_i or channel supports Eit\mathbf{E}_i^t (trading off uplink compression for rank privacy).

We have added this Threat Model and Privacy Discussion in Appendix C, clarifying how device-capability privacy leakage can be mitigated.

[R1] Bonawitz et al., Practical Secure Aggregation for Privacy-Preserving Machine Learning, CCS 2017.
[R2] Bell et al., Secure Single-Server Aggregation with (Poly)Logarithmic Overhead, CCS 2020.


Can Channel Selection Be Client-Dependent?

Thanks for this insightful question. We clarify that while the complementary factor mask Zt\mathbf{Z}^t must be shared across all clients, the client rank-support matrix Eit\mathbf{E}_i^t can be client-dependent in principle. However, we adopt uniform random sampling because data-dependent selection introduces optimization bias and potentially violates client memory budgets:

  1. The complementary factor mask Zt\mathbf{Z}^t must be shared: By Proposition 2 (Appendix A.2, Eq. 24), the cross-client bilinear error term between clients ii and kk is ∑i,kwiwkq2ΔB~it(Eit)⊤Zt(IR−Zt)EktΔA~kt.\sum_{i,k} \frac{w_i w_k}{q^2} \Delta\widetilde{\mathbf{B}}_i^t (\mathbf{E}_i^t)^\top \mathbf{Z}^t (\mathbf{I}_R - \mathbf{Z}^t) \mathbf{E}_k^t \Delta\widetilde{\mathbf{A}}_k^t. If clients independently choose which factor to update on global channel jj (Zit≠Zkt\mathbf{Z}_i^t \neq \mathbf{Z}_k^t), then Zit(IR−Zkt)≠0\mathbf{Z}_i^t(\mathbf{I}_R - \mathbf{Z}_k^t) \neq \mathbf{0}, re-introducing cross-client bilinear error.

  2. The rank activation mask Eit\mathbf{E}_i^t can be client-dependent, but data-driven selection incurs bias and memory overhead: In Eq. (24), as long as Zt\mathbf{Z}^t is shared, (Eit)⊤Zt(IR−Zt)Ekt=0(\mathbf{E}_i^t)^\top \mathbf{Z}^t (\mathbf{I}_R - \mathbf{Z}^t) \mathbf{E}_k^t = \mathbf{0} holds for any client-specific selection matrices Eit\mathbf{E}_i^t and Ekt\mathbf{E}_k^t, meaning clients can self-select their rir_i active channels based on local data or task importance while preserving error-free aggregation. However, data-dependent greedy selection (e.g., choosing the top-rir_i channels by gradient norm), while eliminating aggregation errors, introduces two drawbacks: (i) it breaks the unbiasedness condition E[∇fiHit(Xt)]=∇ϕi(Xt)\mathbb{E}[\nabla f_i^{\mathbf{H}_i^t}(\mathbf{X}^t)] = \nabla \phi_i(\mathbf{X}^t) in Lemma A.1, introducing a non-vanishing selection-bias floor and starving unselected channels; and (ii) evaluating channel importance often requires computing gradients across all RR channels first, which violates the client's rank-rir_i memory budget. Therefore, we consider uniform random sampling of Eit\mathbf{E}_i^t (E[Hit]=riRIR\mathbb{E}[\mathbf{H}_i^t] = \frac{r_i}{R}\mathbf{I}_R) as a simple and effective design choice: it respects client memory limits with no extra pre-computation overhead while guaranteeing unbiased global coverage and convergence.

We have added this discussion to Appendix C of the revised manuscript.

The authors' response has addressed my concerns. I appreciate authors' effort in running additional experiments and providing further analysis. I would like to maintain my positive score.

Thank you so much for your follow-up and for maintaining your positive evaluation. We are glad that our responses addressed all your concerns! We truly appreciate your time and your constructive comments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank you for your time, constructive feedback, and insightful suggestions, which have significantly strengthened our work. Based on your comments, we have uploaded a revised manuscript (with major revisions highlighted in blue):

  • Major Updates and Improvements: We (i) explicitly reported our default non-IID setting (α=0.1\alpha = 0.1) and added sensitivity experiments across α∈0.5,1.0\alpha \in \\{0.5, 1.0\\} (Section 5.3, Table D.2); (ii) derived a factor-aware variance bound (p⋆=MBMB+MA>0.5p^\star = \frac{\sqrt{M_B}}{\sqrt{M_B} + \sqrt{M_A}} > 0.5) for asymmetric channel allocation and added Corollary A.2 bounding the gap between the surrogate Φ(X)\Phi(\mathbf{X}) and full-adapter objective F(X)F(\mathbf{X}) under heterogeneous ranks (Section 4.4, Appendix A.7); (iii) evaluated server-side aggregation error across rounds (Appendix D.6), expanded experiments to Qwen2.5-1.5B-Instruct and LLaMA-7B instruction tuning (Appendix D.5), provided ablations against importance-based channel sampling and partitioning (Appendix D.7), and added ablations on the number of local steps (KK) (Appendix D.8); and (iv) clarified the co-adaptation intuition and threat model (Section 4.1, Appendix C).

We are encouraged that two reviewers have maintained their positive scores (3). In addition, to ensure a clear shared understanding of our core design across the discussion thread, we would like to highlight two clarifications about our novelty:

  1. Core Novelty over Prior Factor-Freezing and Alternating Methods: CFLoRA addresses a fundamental limitation of prior exact methods (FFA-LoRA, RoLoRA, AS-LoRA), all of which freeze an entire factor A\mathbf{A} or B\mathbf{B} across local steps and thus prevent intra-round co-adaptation. By exploiting complementary channel partitioning (not explored by any prior work), CFLoRA is the first framework to enable simultaneous intra-round co-adaptation of B\mathbf{B} and A\mathbf{A} while guaranteeing exact factor-wise aggregation and rank-rir_i communications. Both our theory (O(1/T)\mathcal{O}(1/\sqrt{T}) exact convergence vs. divergence as p→1p \to 1) and experiments (+2.01 to +12.24 accuracy points across GLUE and LLaMA-3.2-3B) confirm that intra-round co-adaptation is essential.

  2. Rationale for Lightweight Bernoulli and Uniform Channel Allocation: Our core algorithmic contribution lies in the complementary factor-partitioning principle that eliminates cross-client bilinear error (ΔBiΔAk\Delta \mathbf{B}_i \Delta \mathbf{A}_k). Channel sampling (Zt,Eit\mathbf{Z}^t, \mathbf{E}_i^t) is not intended as our primary algorithmic novelty; instead, we deliberately keep it state-independent and lightweight so that CFLoRA's gains stem cleanly from exact co-adaptation without extra system overhead. Moreover, independent Bernoulli⁡(p)\operatorname{Bernoulli}(p) and uniform sampling are not only simple but also effective choices in PEFT because they: (i) preserve unbiased global gradients (Lemma A.1) without selection bias over permutation-invariant LoRA channels, (ii) incur zero pre-computation overhead while respecting client rank-rir_i memory budgets with only O(1)\mathcal{O}(1)-bit seed sharing, and (iii) are empirically comparable to importance-based heuristics (Tables R5–R6; Appendix D.7).

Thank you again for your valuable time, constructive engagement, and thoughtful consideration of our submission!

Best wishes,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.