- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FedGPA, a personalized federated fine-tuning framework that jointly determines which intra-layer Transformer units to adapt and which to share across clients. By assigning attention heads and FFN clusters to Global, Private, or Frozen roles, FedGPA aims to reduce communication while preserving personalization. Experiments on BERT-base and RoBERTa-base demonstrate competitive accuracy and substantial communication savings compared with existing baselines.
Critical strengths
1.The paper presents a meaningful approach to allocating adaptation and sharing roles at the intra-layer level, providing greater flexibility than conventional layer- or module-level methods. 2. FedGPA achieves competitive performance across multiple tasks while significantly reducing cumulative communication. The communication-to-target-accuracy evaluation is particularly relevant to practical federated learning.
Critical weaknesses
- Limited validation of the allocation mechanism. Although the paper emphasizes joint adaptation and sharing, the proposed allocation follows a Global-first greedy strategy. The advantages of this strategy over alternative allocation methods are not sufficiently demonstrated, making its specific contribution difficult to assess.
- Experimental limitations. The baselines use different trainable parameter budgets, making it difficult to isolate the benefits of fine-grained allocation. Moreover, the main experiments use only three random seeds and a single non-IID configuration, limiting the robustness of the conclusions.
Questions for authors
- Could the authors provide comparisons under matched trainable parameter budgets and explain the advantages of the Global-first allocation strategy?
- How robust is FedGPA under different levels of data heterogeneity? Could the authors also report the computational overhead introduced by unit selection and importance estimation?
We thank the reviewer for the constructive comments. We have added matched-budget comparisons, an alternative Global–Private allocation strategy, heterogeneity experiments, and computational overhead measurements. The revisions are highlighted in blue.
W1 / Q1. Matched Budgets and Global-First Allocation
We have added allocation comparisons on MNLI under the same K_G = 1, K_P = 4 budget (Section 5.3, Table 4). FedGPA and Random allocation both use approximately 0.986M active parameters and 2.281 GiB of total communication. Across three seeds, FedGPA achieves higher mean final accuracy (91.94% versus 91.57%) and lower communication to the 88% target (0.610 versus 0.728 GiB).
We also compare FedGPA with FedDPA-r13 under near-matched active-parameter budgets (986K versus 961K) in Appendix D, Table 9. FedDPA-r13 achieves 0.87 percentage points higher final accuracy in the evaluated seed, whereas FedGPA reduces communication to the 85%, 88%, and 89% targets by 56.0%, 39.3%, and 38.0%, respectively.
Regarding Global-first allocation, we introduce a Local-aware alternative that explicitly considers the remaining Private utility when selecting Global units (Appendix C). Local-aware improves the evaluated one-step utility proxy but does not improve mean final accuracy (91.69% versus 91.94%) or communication to the 88% target (0.724 versus 0.610 GiB). These comparisons provide evidence for importance-based unit selection and characterize the trade-off of the Global-first strategy.
A direct joint maximization of one-step Global and Private utilities can favor Private adaptation, since client-local updates avoid cross-client gradient cancellation. Under a shared-start SGD approximation, the average Private utility is no smaller than the Global utility, with the difference determined by cross-client gradient variance. Moreover, these one-step estimates do not capture the future transfer benefit of shared units. Global-first therefore prioritizes cross-client sharing utility under a fixed Global budget, while allowing clients to select Private units upon participation without requiring their scores during Global selection.
Further more, this design allows clients to initialize Private units upon participation without requiring their Private scores during Global selection. We have clarified this motivation in Section 4.2.
W2 / Q2. Data Heterogeneity and Computational Overhead
We have added MNLI experiments at label-Dirichlet concentrations α = 0.5 and 1.0, in addition to the main α = 0.1 setting (Appendix D, Table 10). At α = 0.5, FedGPA achieves 81.26% final accuracy versus 84.15% for FedDPA; at α = 1.0, the results are 75.58% versus 83.09%.
These results indicate that FedGPA's relative performance varies with the degree of data heterogeneity. One possible explanation is that the fixed Global budget K_G = 1 provides insufficient shared capacity as client label distributions become more balanced.
We have also measured the computational overhead of unit selection on four BERT-base tasks (Section 5.2 and Appendix E, Table 11). Selection, calibration, and delayed Private initialization account for 2.5–10.7% of the measured cumulative client-side computation time over 150 rounds. The allocation communication cost is approximately 0.100 GiB, or 4.4% of total communication.
For MNLI, the accumulated selection work is 1,205.06 seconds across client participations, comprising 413.33 seconds for Global probing, 603.83 seconds for Private probing, and 187.90 seconds for calibration.