acceptodds
Under review as a conference paper at ICLR 2027

FedCoDi: Directional Coherence for Personalized Federated Low-Rank Adaptation

Abstract

Federated low-rank adaptation enables efficient fine-tuning of pretrained language models on decentralized data. Under heterogeneous data, however, client updates can point in conflicting directions and partially cancel when averaged. We characterize this directional cancellation through pairwise directional coherence and observe it in client adapters during federated training. Mitigating cancellation must also preserve client-specific adaptations, rather than simply enforce uniform directions. To address this tension, we propose FedCoDi, a personalized federated fine-tuning framework that coordinates how clients incorporate global directions and form subsequent local updates. Building on a magnitude–direction decomposition, FedCoDi retains trainable weight magnitudes locally and shares both low-rank factors. A row-wise coherence gate blends broadcast and previous local factors, retaining more local state where their directions disagree. During local training, a complementary regularizer penalizes angular disagreement between the local low-rank product and the unblended global product, which serves as a shared reference across clients. Both mechanisms operate on clients, while the server retains standard factor-wise averaging. Experiments with RoBERTa-large and Llama-3-8B on four GLUE tasks, GSM8K, and Dolly-15K show consistent gains over the evaluated federated LoRA baselines, achieving substantial improvements over the strongest compared baseline in each setting.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Personalization, LLMs, LoRA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BMC2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies directional cancellation in personalized federated low-rank adaptation under heterogeneous data. The authors first use cosine similarity between clients’ low-rank products to characterize the loss in norm caused by averaging their updates, distinguishing it from the aggregation bias that arises when LoRA factors are averaged separately. Building on DoRA’s magnitude–direction decomposition, FedCoDi keeps magnitude parameters local to each client. Its DDCA component mixes global factors with historical local factors based on row-wise directional similarity, while DCR constrains the direction of each local low-rank product relative to the unmixed global product. The server continues to use standard factor averaging. Experiments cover four GLUE tasks with RoBERTa-large, as well as GSM8K and Dolly-15K with Llama-3-8B. The paper reports consistent improvements over the compared baselines and supports the method’s effectiveness through component ablations and an analysis of the communication rounds needed to reach a target accuracy.

Critical strengths

In Table 1, FedCoDi achieves the highest mean accuracy in all 12 GLUE task–client configurations. With 3, 10, and 50 clients, its mean accuracy exceeds that of the strongest baseline at each scale by 0.70, 1.09, and 1.19 percentage points, respectively; these settings include partial client participation. The improvement on GSM8K is 3.18 percentage points, suggesting that the method merits further investigation.

Critical weaknesses

1、FedCoDi learns client-specific adaptation directions during local training, but the server still averages A and B separately. The resulting global product B^gA^g generally differs from the weighted average of the client products and may therefore be affected by both directional cancellation and factor-wise aggregation mismatch. Although DDCA mitigates interference from global parameters by retaining historical local factors, DCR subsequently encourages the local product to align with this global reference. The paper does not sufficiently clarify under what conditions these two mechanisms preserve beneficial personalized directions and when they may instead weaken them.

2、DDCA uses the cosine similarity between global and local factors to determine their mixing ratio. This implicitly assumes that greater directional alignment favors the global parameters, while disagreement favors retaining local parameters. Similarity alone, however, cannot establish which candidate is better for the local task. The guarantee in Proposition 2 relies on an additional “similarity–quality” assumption, which the appendix acknowledges has not been directly validated. The ablation in Table 3 shows that adding DDCA improves performance, but without a comparison against a tuned fixed mixing weight, it remains unclear whether the gain comes from retaining historical local parameters or from the proposed directional gating mechanism. This weakens the case for the necessity of that design.

Questions for authors

1、Section 5.1 specifies B^0=0, so the global product cached in the first round is W^g=B^0A^0=0. The cosine distance in Equation (14), however, is defined only for nonzero products. How does the implementation handle this case? Also, in the 50-client partial-participation setting, how are the historical local factors required by Algorithm 1 initialized when a client is selected for the first time at t>0?

2、Could you report the inter-client product similarity and cancellation ratio during training for the baseline, DDCA only, DCR only, and the full method, and relate these measurements to the accuracy curves?

3、Are the selected baselines and comparisons sufficient to establish the proposed method’s advantage? The GLUE experiments include methods such as LoRA-FAIR, FedEx-LoRA, and RoLoRA, whereas the GSM8K and Dolly-15K experiments compare only FedIT, FFA-LoRA, and FedSA-LoRA. Please explain why the two sets of experiments use different baselines. Could you also report results for the former methods on Llama-3-8B to determine whether FedCoDi’s performance advantage still holds?

8 comments
AuthorsOpenReview ↗

Rebuttal Part 1 by Authors

Thank you for your careful assessment of the global reference, personalization, and experimental comparisons. We clarify the interaction between DDCA and DCR, explain first participation, and report the fixed-mixing, training-dynamics, and expanded baseline results below.

Response to W1: Shared reference quality and personalized adaptation

DCR uses the unblended broadcast product as a common directional reference. The product BgAgB^gA^g can be affected by both directional cancellation and factor-aggregation mismatch. Its role is to provide a shared reference for clients participating in the same round. Using the DDCA-blended product would instead anchor each client to its own personalized initialization.

DDCA and DCR act at different stages and levels. DDCA blends the broadcast and retained local factors row-wise before local training, preserving more local state where directional compatibility is low. DCR then applies a soft angular penalty to the evolving local product during task-driven optimization. Both factors remain trainable, and the magnitude vector remains local. This design combines a personalized starting point with continued guidance from the shared reference.

The two mechanisms can complement each other when the shared reference contains directions useful to the local task and the regularization influence is balanced with task optimization. When beneficial local directions conflict strongly with the shared reference, or the regularization influence becomes too strong, DCR can suppress useful local adaptation. Their interaction therefore depends on task–reference compatibility and the relative influence of the task and regularization gradients.

Table 3 supports their combined benefit in the evaluated setting. With the same local-magnitude parameterization, the baseline achieves 88.9%, DCR alone 89.6%, DDCA alone 89.9%, and the full method 90.2%. Adding DCR to DDCA improves accuracy by 0.3 percentage points, while adding DDCA to DCR improves it by 0.6 points. We have clarified the role of the broadcast reference and the conditions governing its interaction with personalization in Appendix A.7, and revised Section 4.2 to describe product-level guidance alongside task-driven local optimization.

Response to W2a: Coherence as a directional compatibility signal

Thank you for identifying this distinction. We agree that cosine similarity alone cannot determine which candidate has lower local-task loss. DDCA uses coherence as a directional compatibility signal between the broadcast factors and the retained local state. Low coherence favors preserving more of the locally trained state, while high coherence permits a larger contribution from the shared factors. This is a transfer heuristic rather than a calibrated estimator of candidate quality.

Response to W2b: Fixed mixing and shuffled-gate controls

We compared DDCA with a validation-tuned fixed mixing weight on MNLI with three clients, Dirichlet α = 0.5, and 1,000 communication rounds. We evaluated global mixing coefficients {0, 0.25, 0.50, 0.75, 1}, retaining DCR, local magnitudes, and the remaining ablation settings. Validation performance selects α = 0.25. This fixed-mixing baseline achieves 88.70% mean test accuracy over three independent runs, compared with 90.20% for FedCoDi, a gain of 1.50 percentage points.

The coefficients 0, 0.50, and 0.75 yield mean accuracies of 88.86%, 87.81%, and 88.37%, respectively. The endpoint α = 1 corresponds to the DCR-only variant in Table 3 (89.60%). FedCoDi therefore also exceeds the highest reported result across the entire fixed-coefficient grid by 0.60 percentage points. Because fixed mixing already retains historical local factors, and DCR and local magnitudes are held constant, this comparison supports the additional contribution of directional gating in the evaluated setting. We have added the selection procedure and comparison to Section 5.3.

Shuffled-gate control. We additionally permute DDCA gate values across rows within each client and adaptation layer, separately for A and B. This preserves the gate-value distribution within each factor while disrupting its correspondence with the rows that produced those weights. DCR, local magnitudes, and the remaining training settings are unchanged. Using the same MNLI data splits, 1,000-round budget, and model-selection procedure as the full method, this control achieves 87.39% mean test accuracy over three runs, compared with 90.20% for the original gates, a difference of 2.81 percentage points. Together with the fixed-mixing comparison, this supports the practical contribution of matching coherence-derived mixing weights to their corresponding rows. We have added this control to Section 5.3.

AuthorsOpenReview ↗

Rebuttal Part 2 by Authors

Response to Q1: Zero reference and first participation

For DCR, the stabilized cosine follows F.cosine_similarity:

{\max(\\|x\\|_2,\epsilon)\max(\\|y\\|_2,\epsilon)}, \qquad \epsilon=10^{-8}.$$ The broadcast reference is fixed and detached during local optimization. When $B^0A^0=0$, the cosine is zero for every local product, so $\mathcal L_{\mathrm{DCR}}=1$ and its gradient with respect to the local factors is zero. The objective in that round is therefore $(1+\lambda)\mathcal L_{\mathrm{task}}$, with zero directional gradient from DCR. Section 4.2 now states this case explicitly. DDCA is applied when a client has **cached local factors**. A client participating for the first time initializes from the current broadcast, whether it first appears in round 1 or a later round. Algorithm 1 now checks for cached state and uses the current broadcast in the initialization branch. This covers first participation in the 50-client setting. ### **Response to Q2: Product coherence, cancellation, and accuracy** We added a 300-round MNLI diagnostic with three clients for the four requested variants. Mean best validation accuracy is computed over three independent runs. | Variant | κ: rounds 1 / 10 / 300 | ρ: rounds 1 / 10 / 300 | Mean best val. accuracy (%) | |---|---|---|---:| | Factor-averaging baseline | 0.641 / 0.876 / 0.986 | 0.095 / 0.680 / 0.985 | 86.2 | | DDCA only | 0.655 / 0.592 / 0.990 | 0.131 / 0.077 / 0.985 | 86.8 | | DCR only | 0.655 / 0.976 / 0.986 | 0.131 / 0.958 / 0.983 | 86.7 | | FedCoDi | 0.620 / 0.976 / 0.996 | 0.013 / 0.957 / 0.987 | **87.2** | DCR-only and FedCoDi show stronger early coherence than the baseline. DDCA-only improves mean best validation accuracy while retaining lower early coherence, and FedCoDi attains the highest accuracy. The reported κ–accuracy Spearman correlations across 30 evaluation checkpoints are 0.407 for DDCA-only and 0.776 for DCR-only. These describe co-evolution along the training trajectory; interpretation must account for the common training trend and temporal dependence. We have added the summary table and analysis to Appendix A.1 and linked them from Section 5.3. **We also provide continuous baseline-versus-FedCoDi trajectories under the server's factor-aggregation rule** (Appendix A.1, Figure 5). This comparison uses MNLI with three clients, Dirichlet α = 0.5, and 300 communication rounds; curves are means over three runs. The remaining settings follow the multi-client diagnostics, including rank 8, 50 local steps, and sample-count weighting. Here $\kappa_{\mathrm{agg}}$ measures the norm of the product of separately aggregated factors relative to the weighted sum of client-product norms. FedCoDi approaches a ratio of one earlier, while the baseline shows larger early norm deficits; the gap narrows later. This figure supplements the four-variant product-average $\kappa$ summaries above with a direct comparison under the server aggregation rule. ### **Response to Q3: Additional Llama-3-8B baselines** The original LLM comparison included the baseline experiments completed by the submission deadline. Time constraints prevented us from completing the remaining comparisons before submission. We extended both Llama-3-8B experiments to include LoRA-FAIR, FedEx-LoRA, and RoLoRA, using the same training, model-selection, and client-test procedures as Table 2. The updated comparison uses the same baseline set as GLUE and reports means over three independent runs. | Method | GSM8K accuracy (%) | Dolly-15K ROUGE-L F1 | |---|---:|---:| | FedIT | 46.10 | 0.3105 | | FFA-LoRA | 46.93 | 0.2906 | | FedSA-LoRA | 46.47 | 0.3047 | | LoRA-FAIR | 45.33 | 0.3119 | | FedEx-LoRA | 46.32 | 0.3027 | | RoLoRA | 46.09 | 0.3082 | | FedCoDi | **50.11** | **0.3162** | FedCoDi remains the highest-scoring method in both experiments. It exceeds the strongest GSM8K baseline, FFA-LoRA, by **3.18 percentage points** and the strongest Dolly-15K baseline, LoRA-FAIR, by **0.0043 ROUGE-L F1**. We have updated Table 2 and the discussion in Section 5.2.
AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Dear Reviewer BMC2,

Thank you for your careful review. We would appreciate it if you could review our response and revised manuscript before the discussion closes. Please let us know if any points need further clarification.

Best regards, The Authors

Reviewer_L4qQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents FedCoDi a method for improving federated LoRA training by addressing the problem of conflicting local updates that cancel when averaged. The method hinges on two key factors. First, the method compares the current local LoRA factors to the globally averaged updates and initializes per-client LoRA factors based on the similarity to the global update. Large amounts of dissimilarity means the client retains most of its own local LoRA factor. Second, the method applies a loss term that penalizes dissimilarity to the global update. This way subsequent optimization maintains client updates that are more similar to the global average. The combination of these two factors improves optimization of federated LoRA relative to prior baselines including FedIT, FFA-LoRA, and FedSA-LoRA which are well-studied in prior literature.

Critical strengths
  1. Strong Motivation - Section 3 is well written and provides a theoretical grounding for the problem being studied. It includes analysis for averaging the LoRA B and A factors separately as well as the average of the product of B and A. Figure 1 demonstrates the problem empirically providing further justification for the method
  2. Empirical Results - The method compares against six related baselines and demonstrates consistent improvements a variety of tasks including instruction following (Dolly15K) and math (GSM8K). These results are interesting and show the broad effectiveness of the method.
Critical weaknesses
  1. Potential Evaluation Bias - FedCoDi introduces client-local parameters, specifically the magnitude vector, that is also trainable (Figure 1 says it is fixed locally but the text explicitly mentions it is "trainable and local"). Having personalized parameters provides the method with an unfair advantage especially considering that prior baselines are largely not personalized. Table 4 also shows that the method adds roughly 40K parameters, which while marginal, could swing evaluation in favor of FedCoDi. In a similar vein, FFA-LoRA and RoLoRA use and communicate significantly fewer parameters. A fair comparison would double their rank so trainable and communicated parameters are the same.
  2. Diminishing Returns at Large Communication Rounds - Figure 1 shows that the effect of cancellation is really only significant in the first 50 or so rounds. However, GLUE training uses 1000 rounds, and it is likely the cancellation effects do not actually persist in these long-horizon training regimes.
  3. Missing Ablation - The paper claims that reducing the cancellation effects is responsible for the improvement in performance. While this may be true, there is no ablation demonstrating the impact of the method on the cancellation effects. Similarly, there is no ablation showing that the cancellation effects correlate with downstream performance, something mentioned in the Appendix. To prove this method achieves the intended downstream effect, these ablations are necessary in my opinion.
  4. Consequence of Separate Averaging - While the motivation section includes theory for averaging the product of B and A together as well as averaging B and A separately, figure 1 only uses the average of the product of B and A together to show cancellation effects. A similar plot that uses separate averaging would be important since this is the actual setting that is used in the rest of the paper.
Questions for authors
  1. In Figure 2, should it say "Lower Coherence -> Less Global"?
  2. Is there a reason to only use 3 clients in your Figure 1 analysis? I believe it would be a lot more compelling to show the cancellation artifacts still exist with a large number of clients since with only 3 heterogeneous clients, randomness can play a larger role in the aggregation of the LoRA parameters.
  3. Is the magnitude vector trainable as the text mentions or fixed as the figure says?
  4. "The resulting product is also distinct from a client’s personalized product after DDCA blending" - From a clarity perspective, would it be better to mention this only after you've included the formulation for what DDCA blending is?
6 comments
AuthorsOpenReview ↗

Rebuttal Part 1 by Authors

Thank you for the detailed suggestions. We have added component-wise training diagnostics and clarified the presentation of personalized products.

Response to W1: Personalization, parameter capacity, and rank-expanded comparisons

Local magnitude as a personalization design. FedCoDi retains a trainable magnitude vector locally so that each client can adapt weight scales to its own data while sharing both low-rank directional factors. This is an explicit part of our personalized federated learning design. Client-local trainable parameters also appear in our baseline set: FedSA-LoRA shares A and retains B locally for client-specific adaptation. All methods are evaluated on each client's own test split using the same reporting procedure.

Contributions of magnitude retention, DDCA, and DCR. Table 3 separates these contributions on MNLI. With DDCA and DCR disabled, sharing the magnitude vector gives 88.3% accuracy, while retaining it locally gives 88.9%, a gain of 0.6 percentage points. Starting from the local-magnitude variant, adding DCR gives 89.6% (+0.7 points), adding DDCA gives 89.9% (+1.0 point), and combining both gives 90.2% (+1.3 points). The full method also exceeds DDCA alone by 0.3 points and DCR alone by 0.6 points.

All five variants use the same magnitude–direction parameterization. The shared- and local-magnitude variants differ in whether the magnitude vector is aggregated; both train that vector. DDCA and DCR operate on the existing factors without introducing additional trainable parameters. These comparisons therefore show the benefit of local magnitude retention at fixed parameter capacity and the further gains from DDCA and DCR with the same local magnitude design. Together, the results support personalization through local magnitudes and complementary improvements from the two directional mechanisms.

Rank-expanded FFA-LoRA and RoLoRA comparisons.

Following your suggestion, we doubled the rank of FFA-LoRA and RoLoRA from 8 to 16 and compared them with FedCoDi at rank 8 on the four GLUE tasks with 3, 10, and 50 clients. The rank-16 runs follow Table 1: RoBERTa-large, Dirichlet α = 0.5, 1,000 communication rounds, and the same hyperparameter-selection and client-test procedures. Only the baseline rank changes. Entries report mean accuracy ± standard deviation over three independent runs.

At rank 16, FFA-LoRA and RoLoRA each have 1.83M trainable parameters and upload 0.78M parameters per client per round. FedCoDi at rank 8 has 1.88M trainable parameters and uploads 0.78M parameters per client per round (Table 4). These settings have the same reported per-round upload count and comparable trainable parameter counts. FedCoDi's remaining 0.05M trainable-parameter difference is considered alongside the fixed-capacity ablations in Table 3 discussed above.

ClientsMethodRankMNLIQNLIQQPSST-2Avg.
3FFA-LoRA1687.72 ± 0.1492.97 ± 0.2186.73 ± 0.1795.18 ± 0.1190.65
3RoLoRA1688.78 ± 0.1192.53 ± 0.2787.31 ± 0.0994.96 ± 0.1390.90
3FedCoDi890.21 ± 0.1493.30 ± 0.3487.79 ± 0.1895.41 ± 0.1791.68
10FFA-LoRA1686.84 ± 0.2591.37 ± 0.5386.62 ± 0.1995.33 ± 0.1690.04
10RoLoRA1687.44 ± 0.2391.37 ± 0.1686.38 ± 0.2696.53 ± 0.0790.43
10FedCoDi887.97 ± 0.2392.75 ± 0.6188.19 ± 0.1597.12 ± 0.2491.51
50FFA-LoRA1686.55 ± 0.2889.76 ± 0.3582.48 ± 1.2194.52 ± 0.4288.33
50RoLoRA1687.11 ± 0.1989.48 ± 0.4484.77 ± 0.2494.87 ± 0.2489.06
50FedCoDi888.60 ± 0.2691.72 ± 0.2786.48 ± 0.4196.21 ± 0.3490.75

FedCoDi achieves higher mean accuracy than both rank-16 baselines in all 12 task–client configurations. Its four-task average exceeds the stronger rank-16 baseline by 0.78, 1.08, and 1.69 percentage points with 3, 10, and 50 clients, respectively; these differences use the displayed two-decimal averages. We have added the rank-expanded comparison to Appendix C and summarized it in Section 5.2.

AuthorsOpenReview ↗

Rebuttal Part 2 by Authors

Response to W2: Early cancellation and performance under a long training budget

We agree that cancellation is most pronounced during early training in the reported diagnostics. We use 1,000 communication rounds to give all methods sufficient training to converge and compare their performance under a common, long training budget.

The new baseline-versus-FedCoDi trajectory in Appendix A.1 (Figure 5) further shows that the difference in the factor-aggregation ratio κagg\kappa_{\mathrm{agg}} is most pronounced early and narrows in later rounds. This comparison uses MNLI with three clients, Dirichlet α = 0.5, and 300 communication rounds; curves are means over three runs. Its sample-count weights and separately aggregated factor product match the server's aggregation rule.

Table 3 shows that both directional mechanisms improve accuracy under this 1,000-round protocol. With local magnitudes retained in every variant, the baseline achieves 88.9% on MNLI; adding DCR raises accuracy to 89.6%, adding DDCA to 89.9%, and combining them to 90.2%. Thus, DCR and DDCA provide gains of 0.7 and 1.0 percentage points individually, and 1.3 points together, under the full training budget.

Table 4 also shows that FedCoDi reaches the common target accuracy in fewer communication rounds. On MNLI, it takes 68 rounds, compared with 108–187 for the baselines; on QQP, it takes 132 rounds, compared with 168–195. The target for each task is 95% of the strongest baseline accuracy in Table 1. These results establish faster attainment of the target, alongside the accuracy improvements under long training in Table 3.

We therefore scope the cancellation-based motivation to the early stage where the effect is most pronounced. DDCA provides selective global–local factor mixing, and DCR provides directional guidance during local training. The evidence supports their benefits in training efficiency and accuracy under a long budget; it does not isolate early cancellation reduction as the sole cause of the final accuracy gains.

We have clarified the training-budget rationale in Section 5.1, scoped the cancellation motivation to early training in Section 3.2, and linked the long-budget ablation and rounds-to-target evidence in Sections 5.3–5.4.

Response to W3: Cancellation, component ablations, and downstream accuracy

We conducted a separate 300-round diagnostic on MNLI with three clients, tracking inter-client product coherence ρ\rho, the product-average cancellation ratio κ\kappa, and validation accuracy. The four variants use the same round budget. Best validation accuracy is selected within each run and then averaged across three independent runs.

Variantκ: rounds 1 / 10 / 300ρ: rounds 1 / 10 / 300Mean best val. accuracy (%)
Factor-averaging baseline0.641 / 0.876 / 0.9860.095 / 0.680 / 0.98586.2
DDCA only0.655 / 0.592 / 0.9900.131 / 0.077 / 0.98586.8
DCR only0.655 / 0.976 / 0.9860.131 / 0.958 / 0.98386.7
FedCoDi0.620 / 0.976 / 0.9960.013 / 0.957 / 0.98787.2

At round 10, DCR-only and FedCoDi reach κ = 0.976, compared with 0.876 for the baseline. DDCA-only retains a lower early κ while achieving higher mean best validation accuracy than the baseline. The full method has the highest mean best validation accuracy among the four variants. These results support complementary roles: DCR promotes early directional coherence, while DDCA's benefit is compatible with retaining directional differences.

The reported Spearman correlations between κ and validation accuracy across 30 evaluation checkpoints are 0.407 for DDCA-only and 0.776 for DCR-only. We interpret these as descriptive associations along training trajectories, where the common training trend and temporal dependence affect inference. Appendix A.1 reports the diagnostic summaries, with the main findings summarized in Section 5.3, separately from the 1,000-round test-accuracy ablation.

We also provide continuous baseline-versus-FedCoDi trajectories under the server's factor-aggregation rule (Appendix A.1, Figure 5). This comparison uses MNLI with three clients, Dirichlet α = 0.5, and 300 communication rounds; curves are means over three runs. The remaining settings follow the multi-client diagnostics, including rank 8, 50 local steps, and sample-count weighting. Here κagg\kappa_{\mathrm{agg}} measures the norm of the product of separately aggregated factors relative to the weighted sum of client-product norms. FedCoDi approaches a ratio of one earlier, while the baseline shows larger early norm deficits; the gap narrows later. This figure supplements the four-variant product-average κ\kappa summaries above with a direct comparison under the server aggregation rule.

AuthorsOpenReview ↗

Rebuttal Part 3 by Authors

Response to W4: Measuring the product of separately aggregated factors

We added the requested diagnostic using separately averaged A and B, followed by their product. For the active clients in each round, its row-wise ratio is

=\frac{\\| (\sum_i w_i B_i)_{j,:}(\sum_i w_i A_i)\\|_2} {\sum_i w_i\\|(B_iA_i)_{j,:}\\|_2}.$$ All six new panels use **sample-count weights**, $w_i=n_i/\sum_{k\in S_t}n_k$, where $n_i$ is client $i$'s training sample count and $S_t$ is the active client set. These are the same weights used by the server to aggregate A and B separately in these experiments. The numerator therefore measures the product of the factors actually aggregated by the server. The added figures cover 10 clients with full participation and a 50-client population with 5 participants per round, over 200 communication rounds under Dirichlet α ∈ {0.5, 1.0}. The factor-averaging baseline has DDCA and DCR disabled. The first-20-round mean aggregate norm deficits are **30.4% / 26.6%** for 10 clients and **15.0% / 14.8%** for the 50-client population, at α = 0.5 / 1.0. Thus, early norm deficits remain visible when the numerator uses the product of separately aggregated factors. We added both three-panel figures and the metric definition to Appendix A.1 and reference them from Section 3.2. The original Figure 1 retains the product-average ratio analyzed by Theorem 1. The new figures measure the factor-average product, which also includes aggregation mismatch. The captions identify the black curve as the **equal-weight, equal-norm product-average reference** from Theorem 1(c). The measured curves use the sample-weighted server rule defined above. The percentages quantify norm deficits relative to the weighted sum of client-product norms. ### **Response to Q1: Direction of the coherence gate** Yes. We corrected Figure 2 to read **“Lower coherence → Less global.”** This matches the DDCA gate: lower global–local similarity assigns a smaller weight to the global factors and retains more of the cached local factors. ### **Response to Q2: Cancellation diagnostics with larger client populations** The original three-client Figure 1 serves as a small illustrative configuration. We have extended the diagnostic to **10 clients with full participation** and a **50-client population with 5 participants per round**, using the separately aggregated factor product described above. Both settings exhibit early-round norm deficits. The additional figures show coherence versus the aggregation ratio, its 200-round trajectory, and the first-20-round summary. For the partially participating setting, the theorem reference uses the **5 active clients**, and the captions state both the population size and participation count. The two new figures are in Appendix A.1, with a pointer in Section 3.2. All six panels are embedded from vector PDFs so that the plots remain sharp when enlarged. ### **Response to Q3: Is the magnitude vector trainable?** **The magnitude vector is trainable and remains local.** Its gradient comes through the task loss, scaled by the multiplicative DCR factor: $$\nabla_{M^{(i)}}\mathcal L_{\mathrm{total}} =(1+\lambda\mathcal L_{\mathrm{DCR}}) \nabla_{M^{(i)}}\mathcal L_{\mathrm{task}}.$$ Each client retains its magnitude vector and uploads only the low-rank factors. We have made this update rule explicit in Section 4.2. Figure 2 now labels the magnitude as **“Trainable”** and **“Not transmitted,”** consistent with Section 4.2 and Algorithm 1. The revised caption also specifies that DCR uses the unblended broadcast product, cached before DDCA and detached during local optimization. ### **Response to Q4: Introducing the personalized product after DDCA** We moved the distinction between the broadcast product and the personalized blended product to Section 4.1, immediately after the DDCA mixing equations. Section 3.3 now introduces the product of the averaged factors before analyzing its difference from the average of client products.
Reviewer_3StwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on a key challenge in federated LoRA fine-tuning of language models: parameter updates from different clients have divergent directions and may counteract each other during aggregation. To tackle this issue, the authors propose FedCoDi, where each client leverages Decoupled Directional Coherence Alignment (DDCA) to determine the mixing ratio of local and global parameters based on directional similarity after receiving global parameters. In local training, Directional Coherence Regularization (DCR) is applied to penalize directional discrepancies. The authors compare FedCoDi with the federated LoRA baseline on four GLUE tasks using RoBERTa-large and Llama-3-8B. Experiments yield promising results, and ablation studies verify the effectiveness of the two core modules.

Critical strengths

1. To address the issues of directional inconsistency and aggregation cancellation among clients in federated LoRA, the paper proposes a method that mixes global and local parameters based on directional similarity and applies a penalty for directional divergence during local training. The motivation behind the method is clear, offering a valuable approach to achieve the balance between global and local information. 2. The experiments cover different models, tasks, and client scales and demonstrate performance advantages over baselines. Ablation studies also confirm the contributions of DDCA and DCR and support the method's effectiveness.

Critical weaknesses
  1. The approach is too direct. Both DDCA and DCR are straightforward methods to deal with the inconsistency of directions. There is no sufficient demonstration of what methodological advancement they actually offer.
  2. Existing ablation studies cannot confirm whether the effects comes from parameter mixing itself or DDCA directional similarity gating. Without a comparison with a fixed-ratio mixing method, it’s hard to confirm whether the performance gains stem from dynamic selection. As this mechanism is the core of the method, the lack of such evidence weakens the paper's credibility.
  3. The subjects of the experimental evaluation and the rules for aggregating results are unclear. It is not specified whether Table 1 and Table 2 use each client's trained parameters or server’s shared parameters for testing. And it cannot be verified whether baselines use the same evaluation method.
  4. DDCA needs to aggregate two low‑rank factors. However, Proposition 2 only verifies the effectiveness of one factor with the other fixed. For the practical scenario where both factors are updated, existing theory cannot guarantee the quality of the factor product.
  5. The core assumption of DDCA lacks direct experimental proof: the consistency of direction only measures the similiarity of the parameters and cannot determine whether the global or local factor better fits the corresponding client.
Questions for authors

Under the same settings, does DDCA outperform a fixed-ratio mix of global and local parameters? If this comparison has not been conducted, could the authors clarify whether the gains come from DDCA or from mixing global and local parameters in general? For Tables 1 and 2, do the results use server’s shared parameters or each client's trained parameters ? If the second are used, how are the results combined? Do all baselines use the same evaluation method? What does “±” in Table 1 represent? How many independent runs were conducted for each experiment?

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1 by Authors

Thank you for your careful review. We address the methodological contribution, fixed-mixing comparison, evaluation protocol, two-factor analysis, and interpretation of directional coherence below.

Response to W1: Methodological contribution

Thank you for asking us to clarify the methodological contribution. FedCoDi develops a client-side design for coordinating global transfer and subsequent local optimization in personalized federated low-rank adaptation. Our analysis distinguishes directional cancellation from factor-aggregation mismatch: cancellation can occur even when client update products are averaged exactly. This motivates examining how clients incorporate shared directions and form their subsequent updates.

FedCoDi addresses these decisions at different stages and levels. At initialization, DDCA uses row-wise coherence to blend both broadcast factors with retained local factors. During local training, DCR applies a product-level directional penalty relative to the fixed, unblended broadcast product. The local magnitude vector remains personalized. Together, these choices provide selective global transfer followed by shared directional guidance, while retaining standard factor-wise server aggregation.

The controlled comparisons support the contribution of these design choices. With DCR and local magnitudes held fixed, DDCA achieves 90.20% accuracy, compared with 88.70% for the validation-selected fixed-mixing baseline. This comparison evaluates directional gating while retaining historical local factors in both settings. A shuffled-gate control, which preserves gate values but permutes their row assignments within each client and layer separately for A and B, achieves 87.39% under the same protocol, compared with 90.20% for the original gates. The 2.81-percentage-point difference further supports the contribution of matching mixing weights to their corresponding rows. Table 3 further shows that, starting from the same local-magnitude variant at 88.9%, adding DCR gives 89.6%, adding DDCA gives 89.9%, and combining them gives 90.2%. These results support the practical value of selective factor mixing and continued product-level directional guidance within the proposed personalized training framework.

Response to W2 / Q1: Fixed mixing and shuffled-gate controls

We added a controlled comparison that replaces DDCA with a fixed global mixing coefficient for both factors. All variants retain DCR and local magnitudes and use the same MNLI ablation setting: three clients, Dirichlet α = 0.5, and 1,000 communication rounds. Results are means over three independent runs.

Global mixing coefficientAccuracy (%)
088.86
0.2588.70
0.5087.81
0.7588.37
DDCA (FedCoDi)90.20

Validation performance selects α = 0.25 from the evaluated fixed mixing coefficients. Its mean test accuracy over three independent runs is 88.70%, compared with 90.20% for FedCoDi, a gain of 1.50 percentage points.

The endpoint α = 1 is equivalent to the DCR-only variant already reported in Table 3 (89.6%), so we refer to that ablation rather than duplicate the row here. Including this endpoint, DDCA improves accuracy by 0.60 percentage points over the highest reported fixed-coefficient result. Because DCR and local magnitudes are held constant, this comparison supports the contribution of directional gating within the full method. At coefficient 0, the initialization retains the local factors and DCR still uses the broadcast reference. We have added this comparison to Section 5.3.

Shuffled-gate control. We additionally permute DDCA gate values across rows within each client and adaptation layer, separately for A and B. This preserves the gate-value distribution within each factor while disrupting its correspondence with the rows that produced those weights. DCR, local magnitudes, and the remaining training settings are unchanged. Using the same MNLI data splits, 1,000-round budget, and model-selection procedure as the full method, this control achieves 87.39% mean test accuracy over three runs, compared with 90.20% for the original gates, a difference of 2.81 percentage points. Together with the fixed-mixing comparison, this supports the practical contribution of matching coherence-derived mixing weights to their corresponding rows. We have added this control to Section 5.3.

Response to W3 / Q2: Evaluation protocol and client aggregation

Tables 1 and 2 evaluate each client's locally trained model on its own test split. Within each run, we take the unweighted arithmetic mean of the client scores. This procedure is the same for FedCoDi and all baselines. The score is accuracy for GLUE and GSM8K, and ROUGE-L F1 for Dolly-15K.

AuthorsOpenReview ↗

Rebuttal Part 2 by Authors

Response to Q3: Meaning of ± and independent runs

The reported results in Tables 1–3 are means over three independent runs. The “±” values in Table 1 are standard deviations across those runs; the Avg. column averages the four task accuracies. We have added the evaluation and reporting procedure to Section 5.1.

Response to W4: Simultaneous blending of both factors

We have added a two-factor extension to the DDCA analysis in Appendix A.5. Let

EB=(I−DB)(Bl−Bg),EA=(I−DA)(Al−Ag),E_B=(I-D_B)(B^l-B^g),\qquad E_A=(I-D_A)(A^l-A^g),

where DBD_B and DAD_A are diagonal matrices containing the row gates. The blended product satisfies the exact identity

=E_BA^g+B^gE_A+E_BE_A.$$ For a fixed compatible reference pair $(B^{\ast},A^{\ast})$, applying Proposition 2(a)'s triangle-inequality argument to each factor gives bounds $\\|B_{\mathrm{al}}-B^{\ast}\\|_F\le U_B$ and $\\|A_{\mathrm{al}}-A^{\ast}\\|_F\le U_A$, and hence $$\\|B_{\mathrm{al}}A_{\mathrm{al}}-B^{\ast}A^{\ast}\\|_F \le U_B\\|A^{\ast}\\|_F+\\|B^{\ast}\\|_F U_A+U_BU_A.$$ This extension explicitly accounts for both factor errors and their interaction. The reference pair is fixed consistently for the product bound; the separately defined conditional row minimizers in Proposition 2 need not form a joint local minimizer. With both local–global deviations of order $\varepsilon$, the cross term has norm $O(\varepsilon^2)$. The sharp-gating result concerns selection between the two candidates in each row under the stated coherence–quality assumptions. The revised appendix states the resulting product-error bound and its conditions directly. ### **Response to W5: Directional compatibility and local-task quality** Thank you for identifying this distinction. We agree that cosine similarity alone cannot determine which candidate has lower local-task loss. DDCA uses coherence as a directional compatibility signal between the broadcast factors and the retained local state. Low coherence favors preserving more of the locally trained state, while high coherence permits a larger contribution from the shared factors. This is a transfer heuristic rather than a calibrated estimator of candidate quality. The selection guarantees in Proposition 2 are conditional on the additional coherence–quality assumption. The fixed-mixing and shuffled-gate comparisons reported above evaluate the practical benefit of the gate in the tested setting. Original DDCA achieves **90.20%** mean test accuracy, compared with **87.39%** after permuting gate assignments while retaining their values and the remaining training settings. This evaluates the gate through downstream task performance; the proposition retains its stated assumption on reference-parameter errors.
AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Dear Reviewer 3Stw,

Thank you for your time and thoughtful feedback. As the discussion deadline approaches, we would appreciate it if you could take a look at our response and revised manuscript. Please let us know if you have any remaining questions or concerns; we would be happy to clarify.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.