- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FedPolicy, an RL-guided parameter redistribution framework for federated learning under statistical heterogeneity. Instead of uniformly redistributing the entire aggregated model to all clients, FedPolicy dynamically selects among three actions for each client: Full, Backbone-only, and Head-only redistribution. The selected global parameters are combined with the client's retained local parameters to initialize subsequent local training.
The proposed RL controller uses class-level confusion information, local and global validation accuracy, and client–global parameter divergence to determine redistribution actions. Its reward combines local validation improvement with global validation progress. Clients subsequently train their complete models and upload all parameters for server-side aggregation.
The paper evaluates FedPolicy on CIFAR-10, CIFAR-100, ISIC2019, and additional benchmarks under heterogeneous data distributions. The experiments include comparisons with existing federated learning methods, static and learned redistribution strategies, state and reward ablations, policy behavior analysis, robustness tests, and communication-efficiency measurements. The supplementary material also provides theoretical results concerning selective initialization and optimization stability.
Critical strengths
1. Clear framework and relatively comprehensive experimental evaluation.
The paper is generally well organized, and the proposed framework is easy to understand. The experimental evaluation is relatively comprehensive, covering multiple datasets, different levels of data heterogeneity, several RL algorithms, fixed redistribution strategies, state and reward ablations, and communication-efficiency measurements.
I particularly appreciate the comparisons between learned and static redistribution strategies, which provide useful evidence that adaptive parameter redistribution can improve performance under severe data heterogeneity.
2. Interesting analysis of learned redistribution behavior.
I find the policy behavior analysis in Figures 5–7 particularly interesting. The results suggest that the preferred redistribution strategy changes across training stages and heterogeneity levels. For example, Head-only redistribution is more frequent during early training, while Full redistribution becomes more common later. Under severe non-IID conditions, effective RL controllers tend to favor Backbone-only redistribution.
These observations provide useful insights into how global and local knowledge may be balanced dynamically during federated training. I consider this analysis one of the most meaningful aspects of the paper.
Critical weaknesses
1. Limited methodological novelty relative to existing adaptive partial-initialization methods.
(a) Overlap with prior personalized federated learning methods. The core idea of selectively incorporating global parameters while preserving client-specific local parameters has already been explored in personalized federated learning. In particular, PartialFed (NeurIPS 2021) investigates adaptive partial initialization, while FedALA (AAAI 2023) learns how to combine global and local parameters before client training. These approaches share a closely related motivation with FedPolicy.
(b) Limited distinction in the proposed parameter redistribution mechanism. Although FedPolicy differs by using an RL controller to choose among Full, Backbone-only, and Head-only redistribution actions, model decomposition and adaptive incorporation of global parameters are not fundamentally new ideas. The main methodological contribution therefore appears to be the application of RL to a predefined parameter-selection problem. Furthermore, considering the FedACG-style optimization mechanisms observed in the released implementation, I feel that the overall framework can be structurally summarized as: $ \text{FedPolicy} \approx \text{PartialFed-style Initialization} + \text{FedACG-style Optimization} + \text{RL Controller} $ This is a conceptual comparison rather than a claim that these methods are algorithmically identical. Nevertheless, I believe the paper should more clearly explain how this combination advances beyond existing adaptive initialization and parameter-sharing approaches, particularly what additional methodological contribution is introduced by the RL controller.
(c) Insufficient discussion of closely related literature. The related-work discussion focuses primarily on conventional personalized federated learning and partial model updates, but does not adequately address adaptive partial-initialization methods such as PartialFed and FedALA. These studies are particularly relevant to evaluating the novelty of the proposed redistribution mechanism.
2. Insufficient isolation and explanation of the mechanisms responsible for the reported improvements.
(a) Additional optimization mechanisms in the implementation. Upon examining the released implementation, I noticed that the default FedPolicy training procedure includes FedACG-style server lookahead and proximal regularization. Although FedACG is cited and discussed in the supplementary material, the roles of these components are not sufficiently explained in the main methodology, which emphasizes standard local optimization and aggregation with adaptive redistribution. This raises concerns about the consistency between the presented method and the actual training procedure.
(b) Unclear contribution of RL relative to additional optimization mechanisms. The experiments provide encouraging evidence that learned redistribution policies outperform fixed strategies under severe heterogeneity. However, the current evaluation does not sufficiently isolate the contribution of RL-guided redistribution from the FedACG-style optimization components. Controlled experiments with and without server lookahead and proximal regularization would help clarify their respective contributions.
(c) Limited explanation of whether the learned policy itself drives the gains. Figures 5–7 reveal meaningful redistribution patterns across training stages and heterogeneity levels. However, it remains unclear whether these patterns are the primary source of the reported improvements, or whether similar performance could be achieved through simpler dynamic schedules. Although the paper compares RL with static heuristics and a contextual bandit, these comparisons do not fully establish the importance of client-specific sequential RL decisions over training-stage-dependent redistribution.
Overall, while the empirical evaluation is relatively comprehensive and the results are promising, I believe the novelty and the independent contribution of the proposed RL mechanism require stronger justification.
Questions for authors
1. Relationship to existing adaptive partial-initialization methods (clarification).
How does FedPolicy differ fundamentally from existing adaptive partial-initialization methods, particularly PartialFed (NeurIPS 2021) and FedALA (AAAI 2023)? Beyond replacing their adaptive parameter-selection mechanisms with an RL controller, what is the main methodological advance introduced by FedPolicy?
2. Role of FedACG-style optimization in the implementation (clarification and experimental evidence).
The released implementation of FedPolicy appears to include FedACG-style server lookahead and proximal regularization. Could the authors clarify whether these components are enabled in the main experiments reported in Table 2 and the controller comparisons in Table 5? How are these mechanisms incorporated into the proposed methodology, and why are they not described more explicitly in Section 2?
3. Independent contribution of RL-guided redistribution (experimental evidence).
Could the authors provide a controlled ablation comparing FedPolicy with and without server lookahead and proximal regularization, while keeping the RL controller, local loss, data partitions, and training budget unchanged? This would help determine how much of the reported improvement is attributable specifically to adaptive redistribution.
4. Importance of client-specific RL decisions (experimental evidence).
Figures 5–7 show that the learned action distribution changes over training. Could the authors clarify whether the main performance gains come from client-specific decisions or from the overall evolution of redistribution preferences? A comparison with a simple training-stage-dependent schedule, or with a policy that preserves the same action proportions but randomly assigns actions to clients, would help establish the importance of client-specific state information.
5. Interpretation of the learned redistribution strategy (clarification).
Under severe non-IID conditions, the effective RL controllers increasingly favor Backbone-only redistribution, whereas Head-only redistribution is more common during early training. Could the authors provide a more detailed explanation of why these behaviors emerge and how they relate to the improvement in global-model performance?