acceptodds
Withdrawn from ICLR 2027

FedPolicy: An RL-Guided Redistribution Policy for Synergizing Local-Global Optimization in Federated Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Statistical heterogeneity remains a central challenge in federated learning, where uniform post-aggregation broadcast can overwrite client-specific structure and cause negative transfer. Existing approaches primarily address heterogeneity through local objectives, server aggregation, or personalization, while clients typically receive the same parameter block after aggregation. We propose FedPolicy, an RL-guided post-aggregation redistribution framework that treats the return path of the aggregated model as a client-specific decision problem. Rather than broadcasting the same update to every client, FedPolicy learns which part of the aggregated model should be transferred back to each client by selecting among full-model, backbone-only, and head-only parameter blocks. This formulation identifies post-aggregation redistribution as a previously underexplored control axis in federated optimization, improving the balance between global transfer and local specialization. Extensive experiments under heterogeneous federated settings show that FedPolicy consistently outperforms strong baselines across CIFAR-10, CIFAR-100, and ISIC2019, with the clearest gains appearing in the more challenging heterogeneous regimes. Across the settings in the main comparison, FedPolicy achieves an average relative gain of approximately over the strongest baseline, with the largest improvement reaching on ISIC2019 under severe heterogeneity; the supplementary FMNIST and UCI HAR study shows the same direction of improvement. It also converges faster and improves the cost-to-accuracy trade-off, while adding only 0.021 seconds of controller computation per round. These results identify client-specific post-aggregation redistribution as an effective and underexplored design dimension in heterogeneous federated learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Federated Learning, Data Heterogeneity, Reinforcement Learning, Policy-Guided Model Redistribution, Selective Parameter Sharing

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Uc2UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedPolicy, an RL-guided parameter redistribution framework for federated learning under statistical heterogeneity. Instead of uniformly redistributing the entire aggregated model to all clients, FedPolicy dynamically selects among three actions for each client: Full, Backbone-only, and Head-only redistribution. The selected global parameters are combined with the client's retained local parameters to initialize subsequent local training.

The proposed RL controller uses class-level confusion information, local and global validation accuracy, and client–global parameter divergence to determine redistribution actions. Its reward combines local validation improvement with global validation progress. Clients subsequently train their complete models and upload all parameters for server-side aggregation.

The paper evaluates FedPolicy on CIFAR-10, CIFAR-100, ISIC2019, and additional benchmarks under heterogeneous data distributions. The experiments include comparisons with existing federated learning methods, static and learned redistribution strategies, state and reward ablations, policy behavior analysis, robustness tests, and communication-efficiency measurements. The supplementary material also provides theoretical results concerning selective initialization and optimization stability.

Critical strengths

1. Clear framework and relatively comprehensive experimental evaluation.

The paper is generally well organized, and the proposed framework is easy to understand. The experimental evaluation is relatively comprehensive, covering multiple datasets, different levels of data heterogeneity, several RL algorithms, fixed redistribution strategies, state and reward ablations, and communication-efficiency measurements.

I particularly appreciate the comparisons between learned and static redistribution strategies, which provide useful evidence that adaptive parameter redistribution can improve performance under severe data heterogeneity.

2. Interesting analysis of learned redistribution behavior.

I find the policy behavior analysis in Figures 5–7 particularly interesting. The results suggest that the preferred redistribution strategy changes across training stages and heterogeneity levels. For example, Head-only redistribution is more frequent during early training, while Full redistribution becomes more common later. Under severe non-IID conditions, effective RL controllers tend to favor Backbone-only redistribution.

These observations provide useful insights into how global and local knowledge may be balanced dynamically during federated training. I consider this analysis one of the most meaningful aspects of the paper.

Critical weaknesses

1. Limited methodological novelty relative to existing adaptive partial-initialization methods.

(a) Overlap with prior personalized federated learning methods. The core idea of selectively incorporating global parameters while preserving client-specific local parameters has already been explored in personalized federated learning. In particular, PartialFed (NeurIPS 2021) investigates adaptive partial initialization, while FedALA (AAAI 2023) learns how to combine global and local parameters before client training. These approaches share a closely related motivation with FedPolicy.

(b) Limited distinction in the proposed parameter redistribution mechanism. Although FedPolicy differs by using an RL controller to choose among Full, Backbone-only, and Head-only redistribution actions, model decomposition and adaptive incorporation of global parameters are not fundamentally new ideas. The main methodological contribution therefore appears to be the application of RL to a predefined parameter-selection problem. Furthermore, considering the FedACG-style optimization mechanisms observed in the released implementation, I feel that the overall framework can be structurally summarized as: $ \text{FedPolicy} \approx \text{PartialFed-style Initialization} + \text{FedACG-style Optimization} + \text{RL Controller} $ This is a conceptual comparison rather than a claim that these methods are algorithmically identical. Nevertheless, I believe the paper should more clearly explain how this combination advances beyond existing adaptive initialization and parameter-sharing approaches, particularly what additional methodological contribution is introduced by the RL controller.

(c) Insufficient discussion of closely related literature. The related-work discussion focuses primarily on conventional personalized federated learning and partial model updates, but does not adequately address adaptive partial-initialization methods such as PartialFed and FedALA. These studies are particularly relevant to evaluating the novelty of the proposed redistribution mechanism.

2. Insufficient isolation and explanation of the mechanisms responsible for the reported improvements.

(a) Additional optimization mechanisms in the implementation. Upon examining the released implementation, I noticed that the default FedPolicy training procedure includes FedACG-style server lookahead and proximal regularization. Although FedACG is cited and discussed in the supplementary material, the roles of these components are not sufficiently explained in the main methodology, which emphasizes standard local optimization and aggregation with adaptive redistribution. This raises concerns about the consistency between the presented method and the actual training procedure.

(b) Unclear contribution of RL relative to additional optimization mechanisms. The experiments provide encouraging evidence that learned redistribution policies outperform fixed strategies under severe heterogeneity. However, the current evaluation does not sufficiently isolate the contribution of RL-guided redistribution from the FedACG-style optimization components. Controlled experiments with and without server lookahead and proximal regularization would help clarify their respective contributions.

(c) Limited explanation of whether the learned policy itself drives the gains. Figures 5–7 reveal meaningful redistribution patterns across training stages and heterogeneity levels. However, it remains unclear whether these patterns are the primary source of the reported improvements, or whether similar performance could be achieved through simpler dynamic schedules. Although the paper compares RL with static heuristics and a contextual bandit, these comparisons do not fully establish the importance of client-specific sequential RL decisions over training-stage-dependent redistribution.

Overall, while the empirical evaluation is relatively comprehensive and the results are promising, I believe the novelty and the independent contribution of the proposed RL mechanism require stronger justification.

Questions for authors

1. Relationship to existing adaptive partial-initialization methods (clarification).

How does FedPolicy differ fundamentally from existing adaptive partial-initialization methods, particularly PartialFed (NeurIPS 2021) and FedALA (AAAI 2023)? Beyond replacing their adaptive parameter-selection mechanisms with an RL controller, what is the main methodological advance introduced by FedPolicy?

2. Role of FedACG-style optimization in the implementation (clarification and experimental evidence).

The released implementation of FedPolicy appears to include FedACG-style server lookahead and proximal regularization. Could the authors clarify whether these components are enabled in the main experiments reported in Table 2 and the controller comparisons in Table 5? How are these mechanisms incorporated into the proposed methodology, and why are they not described more explicitly in Section 2?

3. Independent contribution of RL-guided redistribution (experimental evidence).

Could the authors provide a controlled ablation comparing FedPolicy with and without server lookahead and proximal regularization, while keeping the RL controller, local loss, data partitions, and training budget unchanged? This would help determine how much of the reported improvement is attributable specifically to adaptive redistribution.

4. Importance of client-specific RL decisions (experimental evidence).

Figures 5–7 show that the learned action distribution changes over training. Could the authors clarify whether the main performance gains come from client-specific decisions or from the overall evolution of redistribution preferences? A comparison with a simple training-stage-dependent schedule, or with a policy that preserves the same action proportions but randomly assigns actions to clients, would help establish the importance of client-specific state information.

5. Interpretation of the learned redistribution strategy (clarification).

Under severe non-IID conditions, the effective RL controllers increasingly favor Backbone-only redistribution, whereas Head-only redistribution is more common during early training. Could the authors provide a more detailed explanation of why these behaviors emerge and how they relate to the improvement in global-model performance?

Reviewer_rFpwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FedPolicy modifies the ordinary gathering and broadcasting in Fed to per-client and per-round decision. The server stills gather the complete client model, but the sending to clients depends on RL with options of complete model, backbone or head. It changes the initialization of local training only. A policy model learns how to determine the sending strategy.

Critical strengths

The paper is well-written and easy to follow. I like the idea of RL-based policy model to decide how to send parameters to clients based on the performance. This can not only reduce the communication cost, but also improves both the local and global performance.

Critical weaknesses

The global reward can not distinguish the fine-grained influence of each client policy decision. Although the policy model can learn how to raise the global performance during rounds of optimization with varying global reward, the efficiency is limited due to the reward is averaged across clients.

Questions for authors

What's the size and source of validation sets? Can we extend the similar idea to the FeD distilling methods like FedSSD and FedCAD?

Reviewer_tQRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedPolicy, an RL-guided federated learning framework that treats post-aggregation redistribution as an adaptive, client-specific decision problem. Instead of broadcasting the same aggregated global model to all selected clients, FedPolicy employs a DRL controller to select from three redistribution actions, namely Full model, Backbone only, or Head only, according to each client’s semantic error structure and optimization status. The controller state incorporates a soft confusion matrix, client and global validation accuracy, and client–global parameter divergence, while the reward combines local validation improvement and global validation progress. Experiments on CIFAR-10, CIFAR-100, ISIC2019, and supplementary FMNIST and UCI-HAR settings show that FedPolicy generally improves accuracy, convergence, and the communication–accuracy trade-off under heterogeneous data distributions.

Critical strengths
  1. The paper explores an under-investigated stage of federated optimization: how the aggregated model should be redistributed to heterogeneous clients. Rather than modifying local optimization or server aggregation, FedPolicy adaptively determines which parameter block should be transferred back to each client. This provides a clear methodological distinction from conventional aggregation-based and personalization-based FL methods.

  2. The experiments cover multiple datasets, several degrees and types of statistical heterogeneity, nine representative FL baselines, convergence behavior, controller alternatives, state/reward ablations, client dropout, aggregation compatibility, and communication/computational overhead. The main experiments also report results over multiple random seeds.

  3. The Full/Backbone/Head redistribution choices are intuitive and directly correspond to different trade-offs between global representation transfer and preservation of local specialization. The framework keeps the local optimization and aggregation rules unchanged, which makes the proposed redistribution mechanism relatively modular and potentially compatible with different FL algorithms.

Critical weaknesses
  1. The theoretical results mainly characterize prescribed selective-initialization actions under strong assumptions such as block-wise strong convexity, full participation, and one exact local step. They do not establish convergence of the RL controller, policy optimality, or superiority of the learned redistribution policy over full-model redistribution. This weakens the theoretical support for the complete FedPolicy framework.

  2. The global reward increment is shared among all participating clients even though the next global model is jointly determined by all selected-client actions. Therefore, a particular client action may be rewarded or penalized due to decisions made by other clients. In addition, the factorized per-client policy is trained using local transition tuples although the underlying environment transition is federation-level and coupled.

  3. In the main comparison, each baseline retains its configured local objective, while FedPolicy can benefit from a stronger confusion-aware loss. Although the paper provides a controlled CE/C3E experiment, this study is mainly conducted on CIFAR-10 and mostly with a single seed. Therefore, the headline gains cannot be fully attributed to the proposed redistribution mechanism alone.

  4. The controller state and reward use a server-held validation set, with thousands of labeled samples in some experiments. This is a nontrivial assumption, especially in cross-device FL where the server may not possess representative labeled data. A server-data-free variant should be more systematically evaluated.

  5. The main results report the peak global test accuracy over the entire communication trajectory rather than final-round or validation-selected performance. Such a metric can favor methods with higher temporal variance. Moreover, most controller, sensitivity, and ablation experiments are single-seed evaluations, which is insufficient for an RL-based method whose policy learning may exhibit considerable variance.

  6. The controller state contains a CXC soft confusion matrix, resulting in O(C^2) state and communication complexity. This may become problematic for large-class problems. Meanwhile, the method assumes a clear backbone/head decomposition and uses only Full, Backbone, and Head actions, which may not naturally generalize to architectures with adapters, encoder-decoder structures, large prediction heads, or more complex parameter-sharing schemes. The paper itself also acknowledges these architectural limitations.

Questions for authors
  1. How sensitive are the improvements to the availability and representativeness of the server-held labeled validation set?

  2. How does the controller address the credit-assignment problem caused by jointly aggregated client actions?

  3. Would the conclusions remain unchanged when using final-round or validation-selected accuracy rather than peak test accuracy?

Reviewer_5yAgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedPolicy learns which parameter blocks an aggregated global model should send to each participating client in federated learning. A shared client-conditioned controller selects full-model, backbone-only, or head-only redistribution using confusion statistics, validation performance, and local-global parameter divergence. Local training and model upload remain unchanged. Experiments examine heterogeneous image-classification settings, additional datasets, alternative controllers, local losses, and aggregation operators, with the largest reported gains under stronger heterogeneity.

Critical strengths

(1) The paper isolates redistribution as a useful intervention point rather than changing client selection, local optimization, or aggregation simultaneously. (2) The evaluation includes several meaningful controls. In particular, the CE/C3E reruns address the concern that the main improvements could arise entirely from the local loss, while the static, random, and contextual-bandit comparisons investigate the value of learning the redistribution policy..

Critical weaknesses

(1) The evidence for the necessity and stability of sequential RL is less convincing than the evidence for selective redistribution itself. The key controller comparisons are single-seed experiments, and a static backbone-only policy slightly outperforms SAC under mild Dirichlet heterogeneity. (2) Personally, I also feel introducing such method into FL field is a trival improvement and contrbution. (3)The contextual bandit performs poorly, but its result alone does not establish that delayed credit assignment is responsible for the RL advantage. A matched comparison with uncertainty estimates is needed to distinguish a robust policy-learning benefit from controller tuning or favorable trajectories. (4) The reported performance and efficiency summaries do not fully establish the operating-point claims. Accuracy is summarized by the maximum test accuracy over the training trajectory rather than a validation-selected or final checkpoint. The timing decomposition does not clearly account for all validation and state-construction work, and Table 17 reports a FedCA total of 21.79 seconds despite listed components of 20.92 and 0.087 seconds. The mean and FedAvg aggregation entries also require an explicit operator-level distinction before their results can be interpreted consistently.

Questions for authors

(1) Can the main learned-versus-static comparison be repeated across the main experiment seeds, with matched hyperparameter-search budgets? Please include the contextual bandit and a simple state-dependent heuristic, and report uncertainty for the advantage over the strongest non-RL policy. (2) Do the improvements persist when checkpoints are selected using the held-out validation set, and when reporting the final round or a fixed final-round window? Please clarify whether all compared methods have equivalent access to validation data for model selection and tuning. (3) Please give the exact aggregation weights used by Mean, FedAvg, and the Full-Only control, together with the corresponding partition and initialization identifiers. Which configuration differences explain the different full-redistribution results across Tables 3, 5, and 15? (4) Does the reported controller overhead include client validation, server validation, confusion-statistic construction, replay updates, and state assembly? Please correct or explain the FedCA timing total and provide a complete end-to-end timing scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.