acceptodds
Withdrawn from ICLR 2027

FedRDAC: Representation–Decision Adaptation and Global–Local Coordination for Personalized Federated LoRA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated low-rank adaptation (LoRA) enables parameter-efficient adaptation of pretrained language models without sharing raw client data. However, statistically heterogeneous client data make it difficult to preserve transferable knowledge across clients while supporting client-specific adaptation. Representative personalized federated LoRA methods mainly organize personalization along the representation pathway, leaving decision-level adaptation and the coordination of shared and local components insufficiently unified. We refer to this joint design issue as the personalization coverage–coordination gap. To address it, we propose FedRDAC, a representation–decision adaptation and global–local coordination framework for personalized federated LoRA. Its representation–decision dual adaptation mechanism extends client-specific adaptation across representation and task-decision levels, while its adaptive global–local coordination mechanism controls local contributions and supports module-specific optimization during client training. We evaluate FedRDAC on sentiment classification and semantic equivalence tasks under heterogeneous client partitions using average client accuracy, worst-client accuracy, and cross-client variability. FedRDAC achieves the best average and worst-client results among the evaluated baselines on both tasks. Heterogeneity, hyperparameter sensitivity, and ablation analyses further support the robustness and component-level contributions of the proposed design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
personalized federated learning, federated LoRA, parameter-efficient fine-tuning, non-IID data, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_RdrxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors identify a personalization coverage–coordination gap in representative personalized federated LoRA designs: personalization remains centered on representation adaptation, while decision-level adaptation and the coordination of shared and local components are not jointly organized. To solve this, they propose FedRDAC, integrating RDDA to extend client-specific adaptation across representation and decision levels and AGLC to adaptively coordinate shared and personalized components during local training.

Critical strengths
  1. This paper is well written and easy to follow.
  2. The motivation of this work is clear, i.e., identifying a personalization coverage–coordination gap in representative personalized federated LoRA designs and proposing a new method to solve it.
Critical weaknesses
  1. The “coverage–coordination gap” is mainly a description, not yet a tested rule. This work explains the gap, but it does not provide experimental results that predict when a local task head or adaptive coordination will help. Experiments where a local head is not needed, or comparisons with a parameter-matched local-head-only method, would better support the claimed reason for the gains.
  2. The experiment's scope is limited. The experiments use only SST-2 and MRPC, one RoBERTa-base model, and a small number of clients. More experiments on other tasks, such as language generation tasks and coding tasks, more model families, such as Qwen and Llama, and more client numbers, are needed.
  3. The proposed method lacks explanatory proof. How can it be proved that the proposed method truly captures the shared and specific parts? Some experimental proofs or theoretical explanations are needed.
Questions for authors
  1. Are the results averaged over several random seeds and client splits? Please report standard deviations or confidence intervals.
  2. How does FedRDAC perform on more GLUE tasks, generation tasks, larger language models, and more clients?
  3. How were the “inspired” baselines tuned, and can you compare with official implementations when they exist?
  4. What is the memory cost of keeping local LoRA branches and heads on clients, compared with other personalized FL methods?
Reviewer_cCv8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FedRDAC, a personalized federated LoRA framework designed to address data heterogeneity through representation–decision dual adaptation (RDDA) and adaptive global–local coordination (AGLC). RDDA enables client-specific adaptation at both representation and decision levels, while AGLC coordinates shared and local components using adaptive gating and module-specific learning rates. Experiments on SST-2 and MRPC demonstrate improvements in average and worst-client accuracy while reducing cross-client performance variability compared to existing baselines.

Critical strengths
  1. Insightful motivation: The paper identifies an interesting personalization coverage–coordination gap in existing federated LoRA methods.

  2. Clear presentation: The paper is well-written, with a clear motivation, logical structure, and easy-to-follow methodology.

  3. Consistent improvements: The experimental results demonstrate improvements in average and worst-client accuracy across the evaluated tasks.

Critical weaknesses
  1. The proposed "Coverage–Coordination Gap" is mainly supported by conceptual arguments. More empirical observations or theoretical analysis would help validate its existence and significance.

  2. The paper compares FedRDAC with "inspired" versions of FedALT, FedDPA, and FedSA rather than their original implementations. Comparisons with the original methods would provide stronger evidence of its effectiveness.

  3. The ablation analysis of RDDA and AGLC is conducted only on SST-2. Additional experiments on MRPC or other tasks would strengthen the conclusions.

  4. The evaluation is restricted to two relatively small classification datasets (SST-2 and MRPC) using RoBERTa-base. Experiments with larger models and more diverse tasks would better demonstrate generalizability.

  5. The proposed framework mainly combines existing techniques, including local LoRA, personalized task heads, adaptive gating, and module-specific learning rates. The authors should further clarify the novelty beyond their integration.

Questions for authors
  1. Is there any empirical evidence demonstrating that representation-level personalization alone is insufficient, thereby motivating the proposed Coverage–Coordination Gap?
  2. Why are inspired implementations of FedALT, FedDPA, and FedSA used instead of their original implementations? How do these modifications affect the fairness of the comparisons?
  3. The ablation results show that RDDA with fixed fusion performs worse than decision-level adaptation alone in some metrics. Could the authors explain why combining representation- and decision-level adaptation does not consistently improve performance?
  4. How does the learned client-specific gate behave under different levels of data heterogeneity? Does it exhibit meaningful client-specific adaptation?
Reviewer_DiEpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FedRDAC, a representation–decision adaptation and global–local coordination framework for personalized federated LoRA. The framework employs a representation–decision dual adaptation mechanism to enable client-specific adaptation at both the feature representation and task-decision levels. Meanwhile, an adaptive global–local coordination mechanism dynamically adjusts the contributions of local components and supports module-specific optimization. Experimental results demonstrate the effectiveness and robustness of the proposed method.

Critical strengths
  1. Meaningful research problem and well-structured framework. The paper addresses a meaningful research problem and presents a clearly organized framework, in which RDDA and AGLC serve distinct yet complementary roles, forming a coherent client-side training pipeline.
Critical weaknesses
  1. Experimental setup, main results, heterogeneity analysis, hyperparameter sensitivity, and ablations are distributed across separate top-level sections, making the experimental narrative fragmented. Organizing these analyses as subsections within a unified experimental section would clarify their relationships and improve readability.
  2. The evaluation provides limited evidence for the statistical reliability and generalization of the reported improvements, leaving the robustness of the conclusions insufficiently established.
Questions for authors
  1. Section 5.4 selects the round with the highest Client Avg on the GLUE validation split. Could the authors report the numbers of training and partition seeds and uncertainty across independent runs? Are all clients evaluated on the same validation split or on client-specific subsets? If the latter, how are these subsets constructed, and what are their sample sizes and label distributions? Do the improvements persist at the final round or on evaluation data not used for checkpoint selection?

  2. Table 4 shows that LH-only numerically outperforms RDDA + Fixed Fusion, RDDA + Adaptive Gate, and RDDA + Module-Specific LR, while full FedRDAC performs best. Could the authors explain this pattern and clarify the interaction among these components? In particular, since the head learning rate differs between LH-only and full FedRDAC, does the full method retain its advantage against an LH-only baseline with a matched or fairly tuned head learning rate?

  3. Section 5.2 outlines the inspired baselines, but could the authors specify their exact modifications relative to the referenced methods and provide their component-wise training and aggregation rules? In particular, please clarify which parameters remain local, how classifier heads and LoRA factors are aggregated, and whether residual-corrected aggregation is used in each baseline. These details are needed to interpret the performance and communication comparisons.

  4. Could the authors report per-client gate trajectories together with effective local-update magnitudes, and examine their relationship with realized client heterogeneity and sample sizes? Since the scalar gate in Eq. (8) can be absorbed into a local LoRA factor at fixed parameters, what evidence shows that learning this scale provides an optimization benefit beyond a sufficiently tuned fixed-fusion baseline under matched learning-rate and training settings?

Reviewer_g8VQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FedRDAC for personalized federated LoRA, combining shared and client-local LoRA branches with a client-local classification head. A learned client-specific scalar gate controls the local adapter contribution, while module-specific learning rates support joint optimization; server aggregation adopts the existing FedEx-LoRA correction. Experiments on SST-2 and MRPC report improvements in average accuracy, worst-client accuracy, and cross-client variability, supplemented by heterogeneity, sensitivity, and ablation studies.

Critical strengths
  1. The framework provides a clear and practical combination of representation-level and decision-level personalization while keeping personalized parameters local. The paper explicitly distinguishes its client-side contribution from the adopted FedEx-LoRA aggregation mechanism.

  2. The reported improvements cover both average and worst-client performance on both tasks, rather than only aggregate accuracy. The SST-2 ablations also reveal that simply adding local components can worsen worst-client performance, providing useful evidence that optimization and coordination deserve attention.

Critical weaknesses
  1. The evaluation does not sufficiently establish the reliability of the improvements. Section 5.4 selects the best communication round using the same held-out GLUE validation split on which results are reported, without an independent final evaluation set. Repeated-run uncertainty and client evaluation-set sizes are also not reported. These omissions are consequential because worst-client accuracy can be particularly sensitive to small evaluation subsets.

  2. The experiments do not adequately isolate the contribution beyond combining established personalization components with optimizer settings. In Table 4, the local-head-only variant reaches 95.79% average accuracy, exceeding RDDA with fixed fusion, adaptive gating alone, or module-specific learning rates alone. However, the local-head-only variant is not evaluated with the same head learning rate as the complete method.

Questions for authors
  1. Can the authors report results across multiple training and partition seeds, with checkpoint selection separated from final evaluation?

  2. Does the complete method still outperform a shared-LoRA/local-head baseline when both use the same module-specific learning rates and comparable tuning budgets? A tuned fixed-fusion baseline would also help determine whether learning the gate is necessary.

  3. What precisely differs between the "inspired" baselines and their original methods, particularly their adapter composition, classifier handling, aggregation, and optimization? How were their hyperparameters selected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.