acceptodds
Withdrawn from ICLR 2027

Collaborative Specialist Knowledge Distillation for Federated Offline Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated offline reinforcement learning coordinates policies trained from decentralized offline experience, yet model-level aggregation can underutilize complementary behaviors acquired by individual clients. We propose Collaborative Specialist Knowledge Distillation (CSKD), which preserves client-specific specialization across communication rounds and transfers specialist behavior into a unified shared policy. Each client maintains a persistent residual adaptation and performs local Conservative Q-Learning, while the server composes state-policy targets contributed by multiple specialists and integrates them through policy-space distillation. On heterogeneous D4RL MuJoCo benchmarks, CSKD achieves normalized scores of 48.04 on HalfCheetah, 100.16 on Hopper, and 88.54 on Walker2d, with an average score of 78.91. Additional experiments examine repeated collaboration, client scalability, communication cost, and experience overlap, showing that specialist knowledge can be transferred effectively with substantially lower client-to-server payload than full-model exchange.

Primary area:
reinforcement learning
Keywords:
Federated Reinforcement Learning, Offline Reinforcement Learning, Knowledge Distillation, Policy Distillation, Specialist Policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kBWFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to preserve and combine locally learned behaviors under heterogeneous offline experience in federated reinforcement learning. The proposed Collaborative Specialist Knowledge Distillation (CSKD) retains client-specific residual adaptations across communication rounds, optimizes local policies using Conservative Q-Learning (CQL), and jointly distills state–policy pairs from clients into a shared policy. Experiments on three D4RL MuJoCo tasks report higher average normalized scores than the compared federated baselines, with additional ablation, scalability, and communication analyses.

Critical strengths
  1. The paper considers the preservation of locally acquired behaviors under heterogeneous offline experience in federated reinforcement learning. Maintaining persistent client-specific residual adaptations across communication rounds provides a reasonable mechanism for retaining local specialization during federated training.

  2. CSKD achieves higher average normalized returns than the compared federated baselines across three D4RL MuJoCo tasks.

Critical weaknesses

1. Insufficient justification and limited methodological novelty of specialist knowledge integration.

The paper argues that model aggregation may weaken locally acquired behaviors, yet it does not sufficiently establish how the proposed distillation mechanism achieves effective complementary knowledge integration. Heterogeneous local experience does not necessarily imply reliable or beneficial expertise. In Eqs. (11)–(12), specialist integration primarily reduces to pooling state–policy pairs and performing multi-teacher distillation with equal client weighting, without explicitly identifying complementary behaviors, assessing teacher quality, or resolving conflicting and potentially harmful policies. Simply replacing parameter aggregation with joint policy imitation does not ensure that beneficial behaviors are selectively preserved or that the global policy achieves higher expected returns. Moreover, CSKD largely combines persistent local residual adaptation with conventional multi-teacher distillation, leaving its substantive methodological advance over existing approaches unclear.

2. Incomplete empirical validation and unclear federated data-sharing assumptions.

CSKD transmits states sampled directly from clients' private offline datasets alongside policy outputs (Eq. 10), without adequately discussing the associated privacy implications or data-sharing assumptions. The evaluation is limited to three D4RL MuJoCo tasks with repartitioned offline data, while the ablations do not directly examine heterogeneous teacher quality, conflicting policies, or negative transfer. Thus, the experiments do not sufficiently establish whether the reported improvements arise from beneficial specialist complementarity or from conventional joint distillation and residual preservation. These limitations weaken the evidence for the proposed mechanism and its applicability to realistic federated settings.

Questions for authors
  1. Given the equal client weighting in Eq. (12), how does CSKD account for differences in teacher quality and conflicting local policies to mitigate potential negative transfer?

  2. Unlike conventional federated model aggregation, Eq. (11) simply pools state–policy pairs from different clients, while Eq. (12) performs standard multi-teacher distillation. What specific mechanism, beyond pooling these targets and minimizing imitation loss, enables collaborative integration of complementary specialist behaviors, and how does this contribute to improving global policy returns?

  3. Does the federated setting explicitly permit transmitting states sampled directly from clients' private offline datasets to the server (Eq. 10)? What are the corresponding data-sharing and privacy assumptions?

  4. Could the authors clarify the environment and experimental configuration used in Figure 4, and explain why its normalized scores are substantially lower than those reported in the main experiments (Table 1)?

Reviewer_Wg2MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Collaborative Specialist Knowledge Distillation (CSKD), which preserves client-specific specialization across communication rounds and transfers specialist behavior into a unified shared policy. As reported by the authors, the CSKD method outperforms all baselines in D4RL tasks.

Critical strengths

The idea of sharing specialist behaviour is interesting.

Critical weaknesses
  1. The idea of specialist policy is attractive. However by the formal definition, it is hard to see the difference between specialist policy and the policy as defined in the previous FRL work.
  2. The privacy analysis is missing. I am wondering if the proposed CSKD method is likely to cause some leak of privacy information of the clients.
  3. The baselines reported in the table1 are weak in the selected tasks. Please justify the selection of baselines.
  4. It is not clear why fig4 could lead to the conclusion in section5.6
Questions for authors
  1. By defining specialization, do you mean the domain knowledge of each client?
  2. Please justify the selection of baselines in table1.
Reviewer_3EkxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Collaborative Specialist Knowledge Distillation (CSKD) for federated offline RL. The motivation is valid and the proposed solution indeed improves the performance compared to several federated RL baselines. While the overall approach is an incremental RL work, its advantages should be highlighted more to outstand the others.

Critical strengths
  1. Clear motivation for preserving client-specific behavior
  2. Extensive results with meaningful ablations
Critical weaknesses
  1. Some details for re-implementation are missing.
  2. Communication advantages rely on full-model baselines
Questions for authors
  1. Is state matching only pooling client state–policy pairs and applying an equally weighted distillation loss? How does it differ from standard pooling multi-teacher distillation mechanism?
  2. Does “Individual Distillation” mean sequential updates, separate students, or different optimization budgets?
  3. Could you measure local-policy performance immediately before and after each server broadcast when shared policy changes?
  4. In section 5.6, Why are the displayed returns much lower than the main results? Which environment, client count, and initialization are used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.