acceptodds
Under review as a conference paper at ICLR 2027

FedGPA: What to Adapt and What to Share in Personalized Federated Fine-Tuning

Abstract

Personalized federated fine-tuning allows clients to adapt pre-trained models to local data while benefiting from knowledge shared across clients, but repeatedly synchronizing model updates can incur substantial communication cost. Existing methods typically make adaptation and sharing decisions at the granularity of complete layers, adapters, or other coarse components, overlooking the fact that different attention heads and feed-forward network (FFN) units within the same Transformer layer can contribute differently to downstream adaptation. We propose FedGPA, a personalized federated fine-tuning framework that jointly determines adaptation placement and sharing roles at the granularity of intra-layer Transformer units. FedGPA constructs a unified, cost-aligned candidate space of attention heads and FFN clusters, estimates the benefit of each candidate for global sharing and local adaptation, and assigns each unit to a Global, Private, or Frozen role. To limit the overhead of fine-grained selection, FedGPA performs a one-time compressed role allocation; during subsequent federated training, only the always-shared task head and Global units are synchronized, while Private units are adapted and retained locally. Experiments with BERT-base and RoBERTa-base under non-IID federated settings show that FedGPA maintains strong final test performance while substantially improving communication efficiency, reducing the cumulative communication required to reach a target accuracy by up to 76.5% compared with the personalized FedDPA baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning, Federated Fine-Tuning, Personalized Federated Fine-Tuning, Communication Efficiency, Foundation Model Fine-Tuning, Intra-Layer Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bYjqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedGPA, a personalized federated fine-tuning framework that jointly determines which intra-layer Transformer units to adapt and which to share across clients. By assigning attention heads and FFN clusters to Global, Private, or Frozen roles, FedGPA aims to reduce communication while preserving personalization. Experiments on BERT-base and RoBERTa-base demonstrate competitive accuracy and substantial communication savings compared with existing baselines.

Critical strengths

1.The paper presents a meaningful approach to allocating adaptation and sharing roles at the intra-layer level, providing greater flexibility than conventional layer- or module-level methods. 2. FedGPA achieves competitive performance across multiple tasks while significantly reducing cumulative communication. The communication-to-target-accuracy evaluation is particularly relevant to practical federated learning.

Critical weaknesses
  1. Limited validation of the allocation mechanism. Although the paper emphasizes joint adaptation and sharing, the proposed allocation follows a Global-first greedy strategy. The advantages of this strategy over alternative allocation methods are not sufficiently demonstrated, making its specific contribution difficult to assess.
  2. Experimental limitations. The baselines use different trainable parameter budgets, making it difficult to isolate the benefits of fine-grained allocation. Moreover, the main experiments use only three random seeds and a single non-IID configuration, limiting the robustness of the conclusions.
Questions for authors
  1. Could the authors provide comparisons under matched trainable parameter budgets and explain the advantages of the Global-first allocation strategy?
  2. How robust is FedGPA under different levels of data heterogeneity? Could the authors also report the computational overhead introduced by unit selection and importance estimation?
1 comment

We thank the reviewer for the constructive comments. We have added matched-budget comparisons, an alternative Global–Private allocation strategy, heterogeneity experiments, and computational overhead measurements. The revisions are highlighted in blue.

W1 / Q1. Matched Budgets and Global-First Allocation

We have added allocation comparisons on MNLI under the same K_G = 1, K_P = 4 budget (Section 5.3, Table 4). FedGPA and Random allocation both use approximately 0.986M active parameters and 2.281 GiB of total communication. Across three seeds, FedGPA achieves higher mean final accuracy (91.94% versus 91.57%) and lower communication to the 88% target (0.610 versus 0.728 GiB).

We also compare FedGPA with FedDPA-r13 under near-matched active-parameter budgets (986K versus 961K) in Appendix D, Table 9. FedDPA-r13 achieves 0.87 percentage points higher final accuracy in the evaluated seed, whereas FedGPA reduces communication to the 85%, 88%, and 89% targets by 56.0%, 39.3%, and 38.0%, respectively.

Regarding Global-first allocation, we introduce a Local-aware alternative that explicitly considers the remaining Private utility when selecting Global units (Appendix C). Local-aware improves the evaluated one-step utility proxy but does not improve mean final accuracy (91.69% versus 91.94%) or communication to the 88% target (0.724 versus 0.610 GiB). These comparisons provide evidence for importance-based unit selection and characterize the trade-off of the Global-first strategy.

A direct joint maximization of one-step Global and Private utilities can favor Private adaptation, since client-local updates avoid cross-client gradient cancellation. Under a shared-start SGD approximation, the average Private utility is no smaller than the Global utility, with the difference determined by cross-client gradient variance. Moreover, these one-step estimates do not capture the future transfer benefit of shared units. Global-first therefore prioritizes cross-client sharing utility under a fixed Global budget, while allowing clients to select Private units upon participation without requiring their scores during Global selection.

Further more, this design allows clients to initialize Private units upon participation without requiring their Private scores during Global selection. We have clarified this motivation in Section 4.2.

W2 / Q2. Data Heterogeneity and Computational Overhead

We have added MNLI experiments at label-Dirichlet concentrations α = 0.5 and 1.0, in addition to the main α = 0.1 setting (Appendix D, Table 10). At α = 0.5, FedGPA achieves 81.26% final accuracy versus 84.15% for FedDPA; at α = 1.0, the results are 75.58% versus 83.09%.

These results indicate that FedGPA's relative performance varies with the degree of data heterogeneity. One possible explanation is that the fixed Global budget K_G = 1 provides insufficient shared capacity as client label distributions become more balanced.

We have also measured the computational overhead of unit selection on four BERT-base tasks (Section 5.2 and Appendix E, Table 11). Selection, calibration, and delayed Private initialization account for 2.5–10.7% of the measured cumulative client-side computation time over 150 rounds. The allocation communication cost is approximately 0.100 GiB, or 4.4% of total communication.

For MNLI, the accumulated selection work is 1,205.06 seconds across client participations, comprising 413.33 seconds for Global probing, 603.83 seconds for Private probing, and 187.90 seconds for calibration.

Reviewer_i3vBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies communication-efficient personalized federated fine-tuning under heterogeneous client data. The authors propose FedGPA, which jointly determines what to adapt and what to share at the granularity of attention heads and FFN clusters. Specifically, FedGPA estimates the benefits of different units for global sharing and local adaptation, and assigns them to Global, Private, or Frozen roles. To reduce communication overhead, the method performs a one-time compressed role allocation and subsequently synchronizes only the shared task head and Global units. Experiments on BERT-base and RoBERTa-base show that FedGPA achieves competitive or improved final accuracy while reducing the cumulative communication required to reach target accuracy, with up to 76.5% savings compared with FedDPA.

Critical strengths
  1. The paper addresses a meaningful challenge in personalized federated fine-tuning by considering both adaptation placement and parameter sharing under heterogeneous client data.

  2. The proposed framework combines one-time role allocation with sparse parameter synchronization to reduce repeated communication during federated training.

Critical weaknesses
  1. FedGPA performs Global–Private role allocation only once, after the task-head warm-up, and keeps the selected roles fixed throughout subsequent federated training. However, the importance of attention heads and FFN units may change as training progresses, different clients participate, and local models adapt to their own data. Particularly with non-IID data, units selected as Global early on may no longer be suitable for sharing, while some initially frozen units may become more important later. This raises concerns about whether a fixed role allocation can remain effective throughout training.

  2. The paper emphasizes jointly deciding which units should be shared globally and which should be adapted locally. However, Equation (7) follows a Global-first strategy: Global units are selected based on their global importance scores before each client selects its Private units from the remaining candidates. This sequential approach does not directly consider the relative benefits of assigning the same unit to different roles. As a result, a unit that is particularly valuable for local adaptation could be assigned to Global first, potentially limiting the overall effectiveness of the allocation.

  3. Limited experimental diversity and insufficient support for broader claims.

Questions for authors
  1. Could the authors further analyze whether the fixed Global–Private role allocation remains effective throughout federated training? It would be helpful to compare it with periodic role reallocation and examine how different client participation patterns and degrees of data heterogeneity affect the stability of the selected roles.

  2. Could the authors further analyze whether the fixed Global–Private role allocation remains effective throughout federated training? It would be helpful to compare it with periodic role reallocation and examine how different client participation patterns and degrees of data heterogeneity affect the stability of the selected roles.

1 comment

We thank the reviewer for the detailed comments. We have added allocation strategy comparisons, importance diagnostics, and experiments under additional data heterogeneity settings. The changes are highlighted in blue in the revised manuscript.

W1 / Q1–Q2. Fixed Role Allocation and Its Stability

We have added a comparison between one-time allocation (Static) and periodic reallocation (Periodic-50) on MNLI over three seeds (Section 5.3, Table 4, and Figure 7). Periodic-50 repeats allocation at rounds 54 and 104, using score averaging and Private-state retention.

Static and Periodic-50 achieve similar final accuracy (91.94±2.39% and 91.86±2.30%, respectively). However, Periodic-50 incurs 10.2% more total communication due to repeated selection and role transitions. Figure 7 also shows transient validation accuracy drops following reallocation.

We further examine importance rankings at rounds 4, 54, and 104 using a fixed client cohort (Appendix C, Table 8). The Global Top-1 overlap between round 4 and either later round is zero, while the Private Top-4 overlap remains between 0.80 and 0.85. These results indicate that importance rankings change during training, although periodic reallocation does not improve mean final accuracy in the evaluated setting.

The fixed allocation therefore represents a trade-off between adaptation flexibility and repeated selection costs.

W2. Global-First Versus Joint Global–Private Allocation

Equation (7) uses sequential Global-first allocation rather than solving a simultaneous combinatorial assignment. The server first selects units with high estimated cross-client sharing utility, after which clients select Private units from the remaining candidates.

A direct joint maximization of one-step Global and Private utilities can favor Private adaptation, since client-local updates avoid cross-client gradient cancellation. Under a shared-start SGD approximation, the average Private utility is no smaller than the Global utility, with the difference determined by cross-client gradient variance. Moreover, these one-step estimates do not capture the future transfer benefit of shared units. Global-first therefore prioritizes cross-client sharing utility under a fixed Global budget, while allowing clients to select Private units upon participation without requiring their scores during Global selection.

Further more, this design allows clients to initialize Private units upon participation without requiring their Private scores during Global selection. We have clarified this motivation in Section 4.2.

To examine the opportunity cost of Global-first selection, we introduce Local-aware allocation, which evaluates each candidate Global unit together with the remaining Private utility of the selection cohort (Appendix C).

Local-aware changes the selected Global unit in all three MNLI seeds and improves the evaluated one-step utility proxy. However, it does not improve mean final accuracy: Local-aware obtains 91.69±2.76%, compared with 91.94±2.39% for Global-first. Its C_88% is also higher (0.724 versus 0.610 GiB; Table 4).

These results support Global-first as a practical allocation strategy under the evaluated conditions.

W3. Experimental Diversity and Generalizability

The main evaluation covers four GLUE tasks with BERT-base and AG News and QNLI with RoBERTa-base, using three seeds with matched data partitions and participation schedules (Tables 1–3).

We have additionally examined data heterogeneity on MNLI at Dirichlet concentrations α = 0.5 and 1.0, using seed 43 (Appendix D, Table 10). At α = 0.5, FedGPA and FedDPA achieve 81.26% and 84.15% final accuracy, respectively; at α = 1.0, they achieve 75.58% and 83.09%.

These results show that FedGPA's relative performance depends on the data partition. One possible explanation is that the fixed Global budget K_G = 1 provides insufficient shared capacity as client label distributions become more balanced.

Reviewer_iGsdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedGPA, a framework for personalized federated fine-tuning. Existing methods typically make “what to adapt” and “what to share” decisions at the granularity of complete layers, modules, or adapters. In contrast, FedGPA makes these decisions at the level of attention heads and FFN clusters within Transformer layers. It assigning each candidate unit to one of three roles: Global, Private, or Frozen. Global units are synchronized across clients, Private units are adapted locally, and Frozen units remain unchanged. The paper evaluate FedGPA on four GLUE tasks which using BERT-base and on AG News and QNLI using RoBERTa-base. It reports final accuracy and the cumulative communication required to reach target accuracy, and claim up to a 76.5% reduction in cumulative communication-compared with FedDPA.

Critical strengths

The paper clearly and specifically formulates the problem of personalized federated fine-tuning. It distinguishes between two decisions: which model components to adapt and which adapted components to share across clients. It also defines three roles—Global, Private, and Frozen—for attention heads and FFN clusters.

Critical weaknesses
  1. The experiments don’t sufficiently establish why the method works. The paper uses a small number of tasks to suggest that selecting fine-grained model components may be more effective than updating entire layers, and different tasks favor different types of components, and retaining some client-specific capacity can be beneficial. However, these observations are based on SST-2 and QNLI, the evidence is limited. The paper should avoid presenting observations from a small number of tasks.
  2. More direct comparisons are needed to identify the source of gains. The baseline methods and FedGPA don’t update or synchronize the same number of model parameters which may affect the results. It is unclear whether the improvements come from FedGPA’s unit selection strategy or simply from updating fewer parameters. The paper should include comparisons under the same adaptation-parameter budget to help distinguish the benefits of unit selection from those of using fewer trainable parameters.
Questions for authors
  1. The communication-cost account is unclear. Reducing communication is one of the paper’s main claimed benefits, but it is unclear which costs are included in the reported communication totals. It is also unclear whether communication costs for all comparison methods are calculated using the same account rules.
  2. A brief algorithm or step-by-step overview would makes the paper more clear.
2 comments

We thank the reviewer for the constructive comments. We have revised the manuscript and highlighted the changes in blue. Our responses are provided below.

W1. Limited Evidence for the Motivating Observations

We agree that the observations should be interpreted within the evaluated settings. We have extended the head/FFN allocation experiment to MNLI (Figure 3), where the preferred ratio is 0.50:0.50, compared with 0.75:0.25 on SST-2 and 0:1 on QNLI. This provides additional evidence of task-dependent allocation preferences.

The broader evaluation includes four GLUE tasks with BERT-base and AG News and QNLI with RoBERTa-base (Tables 1–3). We have also added three-seed allocation comparisons on MNLI (Table 4).

W2. Comparisons Under Matched Parameter Budgets

We have added two complementary comparisons.

First, we compare FedGPA with Random allocation on MNLI under the same Global/Private budget (K_G = 1, K_P = 4). Both use approximately 0.986M active parameters and 2.281 GiB of total communication. FedGPA achieves 91.94±2.39% final accuracy, compared with 91.57±2.32% for Random, and reduces C_88% from 0.728 to 0.610 GiB (Table 4).

Second, we compare FedGPA with FedDPA-r13 under near-matched active-parameter budgets on MNLI (986K versus 961K parameters). FedDPA-r13 achieves higher final accuracy (90.21% versus 89.34%), while FedGPA requires 56.0%, 39.3%, and 38.0% less communication to reach the 85%, 88%, and 89% targets, respectively (Appendix D, Table 9). This comparison uses one seed, and the shared/private parameter counts remain different.

These results distinguish the contribution of importance-based selection at matched capacity from the accuracy–communication trade-off between methods.

Q1. Communication Cost Accounting

We have clarified the accounting procedure in Section 4.1 and Appendix A, with additional results in Appendix B.

Communication is calculated as the sum of uplink and downlink payloads across participating clients, including task-head warm-up. FedGPA additionally accounts for compressed candidate statistics, calibration, role metadata, and delayed Private initialization, together with routine task-head and Global synchronization. Private residuals remain local. The other methods are accounted for according to their transmitted parameters and method-specific messages. The same payload boundaries and target-confirmation criterion are applied across methods.

We further provide a common tensor-only comparison. On BERT-base QNLI at the 93% target, FedGPA and FedDPA require 0.528 and 2.238 GiB, respectively, corresponding to a 76.42% reduction, compared with 76.47% under the recorded accounting (Appendix B).

Q2. Algorithm Overview

We have added a four-step procedure at the beginning of Section 4:

  1. Clients warm up the shared task head.
  2. Participating clients provide compressed statistics for Global scoring and calibration.
  3. The server selects Global units, and each client selects Private units upon its first participation at or after allocation.
  4. Clients update their selected units while synchronizing only the task head and Global units.

Figure 5 illustrates the framework, and Sections 4.2–4.3 describe the allocation and subsequent synchronization.

Reviewer_iGsdOpenReview ↗

Response the authors

Thanks for the response. Although the author has added additional experiments and explanations, the evidence supporting the fine-grained unit selection strategy’s core advantage is still limited. I maintain the original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.