- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to preserve and combine locally learned behaviors under heterogeneous offline experience in federated reinforcement learning. The proposed Collaborative Specialist Knowledge Distillation (CSKD) retains client-specific residual adaptations across communication rounds, optimizes local policies using Conservative Q-Learning (CQL), and jointly distills state–policy pairs from clients into a shared policy. Experiments on three D4RL MuJoCo tasks report higher average normalized scores than the compared federated baselines, with additional ablation, scalability, and communication analyses.
Critical strengths
-
The paper considers the preservation of locally acquired behaviors under heterogeneous offline experience in federated reinforcement learning. Maintaining persistent client-specific residual adaptations across communication rounds provides a reasonable mechanism for retaining local specialization during federated training.
-
CSKD achieves higher average normalized returns than the compared federated baselines across three D4RL MuJoCo tasks.
Critical weaknesses
1. Insufficient justification and limited methodological novelty of specialist knowledge integration.
The paper argues that model aggregation may weaken locally acquired behaviors, yet it does not sufficiently establish how the proposed distillation mechanism achieves effective complementary knowledge integration. Heterogeneous local experience does not necessarily imply reliable or beneficial expertise. In Eqs. (11)–(12), specialist integration primarily reduces to pooling state–policy pairs and performing multi-teacher distillation with equal client weighting, without explicitly identifying complementary behaviors, assessing teacher quality, or resolving conflicting and potentially harmful policies. Simply replacing parameter aggregation with joint policy imitation does not ensure that beneficial behaviors are selectively preserved or that the global policy achieves higher expected returns. Moreover, CSKD largely combines persistent local residual adaptation with conventional multi-teacher distillation, leaving its substantive methodological advance over existing approaches unclear.
2. Incomplete empirical validation and unclear federated data-sharing assumptions.
CSKD transmits states sampled directly from clients' private offline datasets alongside policy outputs (Eq. 10), without adequately discussing the associated privacy implications or data-sharing assumptions. The evaluation is limited to three D4RL MuJoCo tasks with repartitioned offline data, while the ablations do not directly examine heterogeneous teacher quality, conflicting policies, or negative transfer. Thus, the experiments do not sufficiently establish whether the reported improvements arise from beneficial specialist complementarity or from conventional joint distillation and residual preservation. These limitations weaken the evidence for the proposed mechanism and its applicability to realistic federated settings.
Questions for authors
-
Given the equal client weighting in Eq. (12), how does CSKD account for differences in teacher quality and conflicting local policies to mitigate potential negative transfer?
-
Unlike conventional federated model aggregation, Eq. (11) simply pools state–policy pairs from different clients, while Eq. (12) performs standard multi-teacher distillation. What specific mechanism, beyond pooling these targets and minimizing imitation loss, enables collaborative integration of complementary specialist behaviors, and how does this contribute to improving global policy returns?
-
Does the federated setting explicitly permit transmitting states sampled directly from clients' private offline datasets to the server (Eq. 10)? What are the corresponding data-sharing and privacy assumptions?
-
Could the authors clarify the environment and experimental configuration used in Figure 4, and explain why its normalized scores are substantially lower than those reported in the main experiments (Table 1)?