- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies directional cancellation in personalized federated low-rank adaptation under heterogeneous data. The authors first use cosine similarity between clients’ low-rank products to characterize the loss in norm caused by averaging their updates, distinguishing it from the aggregation bias that arises when LoRA factors are averaged separately. Building on DoRA’s magnitude–direction decomposition, FedCoDi keeps magnitude parameters local to each client. Its DDCA component mixes global factors with historical local factors based on row-wise directional similarity, while DCR constrains the direction of each local low-rank product relative to the unmixed global product. The server continues to use standard factor averaging. Experiments cover four GLUE tasks with RoBERTa-large, as well as GSM8K and Dolly-15K with Llama-3-8B. The paper reports consistent improvements over the compared baselines and supports the method’s effectiveness through component ablations and an analysis of the communication rounds needed to reach a target accuracy.
Critical strengths
In Table 1, FedCoDi achieves the highest mean accuracy in all 12 GLUE task–client configurations. With 3, 10, and 50 clients, its mean accuracy exceeds that of the strongest baseline at each scale by 0.70, 1.09, and 1.19 percentage points, respectively; these settings include partial client participation. The improvement on GSM8K is 3.18 percentage points, suggesting that the method merits further investigation.
Critical weaknesses
1、FedCoDi learns client-specific adaptation directions during local training, but the server still averages A and B separately. The resulting global product B^gA^g generally differs from the weighted average of the client products and may therefore be affected by both directional cancellation and factor-wise aggregation mismatch. Although DDCA mitigates interference from global parameters by retaining historical local factors, DCR subsequently encourages the local product to align with this global reference. The paper does not sufficiently clarify under what conditions these two mechanisms preserve beneficial personalized directions and when they may instead weaken them.
2、DDCA uses the cosine similarity between global and local factors to determine their mixing ratio. This implicitly assumes that greater directional alignment favors the global parameters, while disagreement favors retaining local parameters. Similarity alone, however, cannot establish which candidate is better for the local task. The guarantee in Proposition 2 relies on an additional “similarity–quality” assumption, which the appendix acknowledges has not been directly validated. The ablation in Table 3 shows that adding DDCA improves performance, but without a comparison against a tuned fixed mixing weight, it remains unclear whether the gain comes from retaining historical local parameters or from the proposed directional gating mechanism. This weakens the case for the necessity of that design.
Questions for authors
1、Section 5.1 specifies B^0=0, so the global product cached in the first round is W^g=B^0A^0=0. The cosine distance in Equation (14), however, is defined only for nonzero products. How does the implementation handle this case? Also, in the 50-client partial-participation setting, how are the historical local factors required by Algorithm 1 initialized when a client is selected for the first time at t>0?
2、Could you report the inter-client product similarity and cancellation ratio during training for the baseline, DDCA only, DCR only, and the full method, and relate these measurements to the accuracy curves?
3、Are the selected baselines and comparisons sufficient to establish the proposed method’s advantage? The GLUE experiments include methods such as LoRA-FAIR, FedEx-LoRA, and RoLoRA, whereas the GSM8K and Dolly-15K experiments compare only FedIT, FFA-LoRA, and FedSA-LoRA. Please explain why the two sets of experiments use different baselines. Could you also report results for the former methods on Llama-3-8B to determine whether FedCoDi’s performance advantage still holds?
8 comments
Rebuttal Part 2 by Authors
Response to Q1: Zero reference and first participation
For DCR, the stabilized cosine follows F.cosine_similarity:
Follow-up on our response and revised manuscript
Dear Reviewer BMC2,
Thank you for your careful review. We would appreciate it if you could review our response and revised manuscript before the discussion closes. Please let us know if any points need further clarification.
Best regards, The Authors
Rebuttal Part 1 by Authors
Thank you for your careful assessment of the global reference, personalization, and experimental comparisons. We clarify the interaction between DDCA and DCR, explain first participation, and report the fixed-mixing, training-dynamics, and expanded baseline results below.
Response to W1: Shared reference quality and personalized adaptation
DCR uses the unblended broadcast product as a common directional reference. The product BgAg can be affected by both directional cancellation and factor-aggregation mismatch. Its role is to provide a shared reference for clients participating in the same round. Using the DDCA-blended product would instead anchor each client to its own personalized initialization.
DDCA and DCR act at different stages and levels. DDCA blends the broadcast and retained local factors row-wise before local training, preserving more local state where directional compatibility is low. DCR then applies a soft angular penalty to the evolving local product during task-driven optimization. Both factors remain trainable, and the magnitude vector remains local. This design combines a personalized starting point with continued guidance from the shared reference.
The two mechanisms can complement each other when the shared reference contains directions useful to the local task and the regularization influence is balanced with task optimization. When beneficial local directions conflict strongly with the shared reference, or the regularization influence becomes too strong, DCR can suppress useful local adaptation. Their interaction therefore depends on task–reference compatibility and the relative influence of the task and regularization gradients.
Table 3 supports their combined benefit in the evaluated setting. With the same local-magnitude parameterization, the baseline achieves 88.9%, DCR alone 89.6%, DDCA alone 89.9%, and the full method 90.2%. Adding DCR to DDCA improves accuracy by 0.3 percentage points, while adding DDCA to DCR improves it by 0.6 points. We have clarified the role of the broadcast reference and the conditions governing its interaction with personalization in Appendix A.7, and revised Section 4.2 to describe product-level guidance alongside task-driven local optimization.
Response to W2a: Coherence as a directional compatibility signal
Thank you for identifying this distinction. We agree that cosine similarity alone cannot determine which candidate has lower local-task loss. DDCA uses coherence as a directional compatibility signal between the broadcast factors and the retained local state. Low coherence favors preserving more of the locally trained state, while high coherence permits a larger contribution from the shared factors. This is a transfer heuristic rather than a calibrated estimator of candidate quality.
Response to W2b: Fixed mixing and shuffled-gate controls
We compared DDCA with a validation-tuned fixed mixing weight on MNLI with three clients, Dirichlet α = 0.5, and 1,000 communication rounds. We evaluated global mixing coefficients {0, 0.25, 0.50, 0.75, 1}, retaining DCR, local magnitudes, and the remaining ablation settings. Validation performance selects α = 0.25. This fixed-mixing baseline achieves 88.70% mean test accuracy over three independent runs, compared with 90.20% for FedCoDi, a gain of 1.50 percentage points.
The coefficients 0, 0.50, and 0.75 yield mean accuracies of 88.86%, 87.81%, and 88.37%, respectively. The endpoint α = 1 corresponds to the DCR-only variant in Table 3 (89.60%). FedCoDi therefore also exceeds the highest reported result across the entire fixed-coefficient grid by 0.60 percentage points. Because fixed mixing already retains historical local factors, and DCR and local magnitudes are held constant, this comparison supports the additional contribution of directional gating in the evaluated setting. We have added the selection procedure and comparison to Section 5.3.
Shuffled-gate control. We additionally permute DDCA gate values across rows within each client and adaptation layer, separately for A and B. This preserves the gate-value distribution within each factor while disrupting its correspondence with the rows that produced those weights. DCR, local magnitudes, and the remaining training settings are unchanged. Using the same MNLI data splits, 1,000-round budget, and model-selection procedure as the full method, this control achieves 87.39% mean test accuracy over three runs, compared with 90.20% for the original gates, a difference of 2.81 percentage points. Together with the fixed-mixing comparison, this supports the practical contribution of matching coherence-derived mixing weights to their corresponding rows. We have added this control to Section 5.3.