acceptodds
Under review as a conference paper at ICLR 2027

A Framework for Enhancing Multi-agent Cooperation through Communication Policy Transfer

Abstract

By enabling agents to exchange complementary information, communication-based multi-agent reinforcement learning (MARL) substantially enhances coordination, spurring growing interest in transferring its cooperative knowledge to decentralized agents under centralized training with decentralized execution (CTDE). However, communication policies may exploit information unavailable during decentralized execution, causing policy discrepancy and negative transfer. We theoretically analyze this gap and derive a Q-smoothing condition requiring the communication Q-function to remain consistent across feasible observations of other agents under the same local observation. Based on this analysis, we propose Communication Smoothing Distillation (CSD). CSD learns cross-agent conditional observation distributions and constructs latent communication through distribution sampling. Gradient ascent is further employed to search for communication variations that maximize Q-value discrepancy, while a smoothing perturbation loss constrains Q-value. The communication-smoothed agents provide Q-value targets for CTDE students, enabling the transfer of cooperative knowledge while reducing policy discrepancy. Extensive evaluations on partially observable cooperative tasks demonstrate that CSD mitigates negative transfer, outperforms advanced communication policy transfer and CTDE methods, and can be integrated into existing value-decomposition algorithms.

Primary area:
reinforcement learning
Keywords:
Multi-agent system, Reinforcement Learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UU8hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Communication Smoothing Distillation (CSD) to transfer communication-enabled MARL policies into decentralized students. It characterizes when a communication policy can be represented using local information, learns cross-agent conditional latent distributions, and regularizes teacher Q-values against communication perturbations before distillation. Evaluation includes a predator–prey diagnostic, SMAC and GRF with restricted visibility, component ablations, and integration with multiple value-decomposition methods.

Critical strengths
  1. The paper explicitly addresses information-dependent teacher behavior that decentralized students cannot reproduce. The predator–prey comparison, including a variant without smoothing, supports the practical relevance of this problem and a performance benefit from the smoothing component.

  2. The restricted-visibility SMAC and GRF evaluations report competitive aggregate performance, and integration with VDN, QMIX, and Qatten shows that the framework is applicable across several value-decomposition architectures. These results provide meaningful empirical value.

Critical weaknesses
  1. The paper defines a joint search over mixture assignments and residual codes, but the code optimizes only residual codes with assignments fixed during the search. Its latent-message scale also includes between-component variance, unlike the paper's weighted component standard deviations; these constructions differ under the soft assignments enabled in the supplied configurations. Additionally, distribution learning uses a reconstruction aggregation and KL surrogate different from the stated variational objective.

  2. The proofs establish the existence of a local approximation under uniform smoothness over feasible joint observations. They do not establish that the finite latent search satisfies this assumption or that the optimized student inherits the approximation bound. The teacher–student success-rate gaps do not directly measure policy disagreement or Q variation.

Questions for authors
  1. The supplied learner optimizes residual codes while keeping mixture assignments fixed. Was this restricted search used for the results, and how does it correspond to the joint search specified by the method?

  2. The code uses a log-sum-exp reconstruction loss, weighted component-wise Gaussian KL terms, and a mixture-balancing penalty. What objective was optimized in the experiments, and what relationship to the stated variational bound is established for this surrogate?

  3. What establishes that finite latent perturbations satisfy the theory's uniform smoothness hypothesis and that the learned student, rather than merely an existing local function, satisfies the approximation bound?

  4. What reported evidence links the smoothing component's performance benefit specifically to reduced communication-dependent policy or Q discrepancy, rather than to a general regularization effect?

2 comments
AuthorsOpenReview ↗

Additional Experiments, Corrections to the Method Description, and Further Analysis of Communication Smoothing

Thank you for recognizing the value of our work. We have added SMAC and GRF ablations and corrected errors in the method description. Our responses follow.

Response to Q1:

Thank you for identifying this issue. Listing both mixture weights and residual codes as search variables was a writing oversight, corrected in Section 4.2.3 and Appendix B.2. The reported experiments perturb only communication samples, consistent with our motivation and submitted code. The GMM weights ww, means, and standard deviations define the conditional distribution, while the Sampler generates messages from the sender's actual observation. During perturbation search, distribution parameters and mixture assignments remain fixed; gradient ascent updates only residual codes to maximize teacher Q-value discrepancies.

Response to Q2:

Eq. (8) learns cross-agent conditional observation distributions through reconstruction and latent-distribution regularization. The implementation follows this modeling goal using a regularized GMM surrogate: log-sum-exp aggregates component reconstruction likelihoods; weighted component Gaussian KL provides an analytical upper bound on the mixture KL to the shared prior; and a balancing term encourages component utilization. Related treatments appear in prior work: [1] combines mixture negative log-likelihood with a joint-distribution KL that decomposes into weighted Gaussian KL and mixture-weight KL; [2] adds an entropy regularizer on average auxiliary assignments in a GMM-based generative classifier to encourage balanced assignments. Our implementation draws on these designs, and Appendix B.1 now explains the implementation details.

Response to Q3:

For latent perturbations, we clarify the connection between the maximum-discrepancy objective and Lemma 1. Fix the local information, write the teacher's Q-vector as Qi(ϵ)Q_i(\epsilon), and let JJ denote the maximum squared discrepancy in Eq. (14), with the original code ϵ^\hat\epsilon as reference. For any ϵ,ϵ′\epsilon,\epsilon' in the search domain, the triangle inequality gives

\|Q_i(\epsilon)-Q_i(\epsilon')\|_2 \le \|Q_i(\epsilon)-Q_i(\hat\epsilon)\|_2 +\|Q_i(\hat\epsilon)-Q_i(\epsilon')\|_2\\ \le 2\sqrt{J}. \end{aligned}$$ Thus, minimizing $J$ controls pairwise Q variation within this domain, linking the objective to Lemma 1. Finite-step search approximates this maximization. The added iteration study (Figure A7) shows that further search steps bring little additional success-rate improvement within the tested range. For the learned student, Lemma 1 characterizes the existence of a local approximation under its smoothing condition. CSD uses smoothing to approach this condition and Q-value distillation to train a local student to fit the teacher. We assess this training mechanism from two perspectives: 1. **Teacher and student performance is close.** Figure A5 and Table 4 show comparable SMAC success rates; the student's mean success rate is 2.6 percentage points higher than the teacher's. 2. **Ablations in three environments confirm the role of smoothing.** Removing smoothing lowers mean student success rates from 75.2% to 57.4% in Predator-Prey, from 77.6% to 44.0% across the two SMAC tasks, and from 55.8%/61.3% to 47.2%/50.8% on the two GRF tasks. These results support improved distillation and student learning. **Response to Q4 & W2:** We explain the smoothing mechanism and its connection to teacher-student differences from three perspectives: **First, the visualization in Figure A3 illustrates the meaning of communication perturbations.** With agent 2's local observation fixed, gradient search changes the positions of agent 1 and the target encoded in its messages. Smoothing constrains the resulting Q-value differences, reducing dependence on information unavailable to students and producing supervision better suited to CTDE students. **Second, Figure A4 examines communication-dependent Q variation.** We change the position of an agent that has found the target to generate different messages and compare the resulting Q-values. The smoothing loss reduces these differences, improving Q-value consistency across communication inputs. **Third, Figure 2 illustrates the effect on teacher and student performance.** Without smoothing, the teacher retains a near-perfect success rate, but its student's mean success rate is 57.4%. With smoothing, teacher performance decreases in some settings, while student success reaches 75.2%. This supports our design of constraining the communication teacher to improve knowledge transfer to decentralized students. We hope these responses address your concerns and welcome further discussion. [1] Jayaram Raghuram et al. Few-Shot Domain Adaptation for End-to-End Communication. ICLR, 2023. [2] Can Zhang and Gim Hee Lee. GeT: Generative Target Structure Debiasing for Domain Adaptation. ICCV, 2023.
Reviewer_UU8hOpenReview ↗

Thank you for your response

Although some of the concerns I raised have been addressed, I still believe that the paper essentially meets the minimum acceptance threshold for ICLR. Therefore, I will maintain my original score.

Reviewer_mx2xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates transferring knowledge from communication-enabled teachers to communication-free CTDE students. The authors argue that teachers may rely on information unavailable to students during execution, creating a representational mismatch and potentially causing negative transfer.

The paper characterizes when a communication policy can be represented by a local policy and introduces approximate policy and Q-value smoothing conditions. Its proposed framework, Communication Smoothing Distillation (CSD), learns conditional distributions over other agents’ observations, constructs latent communication messages, searches for perturbations that maximize Q-value discrepancies, and regularizes the teacher against these discrepancies before distilling its individual Q-values into students.

Experiments include a Predator–Prey diagnostic task, partially observable variants of SMAC and GRF, and integration with several value-decomposition methods.

Critical strengths

A strong communication teacher need not provide useful supervision to a student with less information. The distinction between teacher performance and transferability is important.

Rather than only modifying the distillation weight, CSD attempts to reduce the teacher’s dependence on information that cannot be reproduced during decentralized execution.

CSD performs strongly on several SMAC tasks. Its integration with VDN, QMIX, and Qatten suggests that its benefits are not restricted to a single mixing architecture.

Critical weaknesses

Equation (13) maximizes the squared difference between the perturbed and original Q-values, while Equation (14) starts the search from the original communication variables. For a fixed deterministic teacher, this gives ∇Γ∥Q(Γ)−Q(Γ0)∥22=0.\nabla_{\Gamma}\|Q(\Gamma)-Q(\Gamma^0)\|_2^2=0. Consequently, if the initial point is feasible, the update would remain at that point and produce no smoothing penalty.

The propositions characterize when a communication policy can be exactly or approximately represented by a decentralized policy. This is a helpful perspective, but exact representability is stronger than what is required for a teacher to provide useful learning signals. Similarly, consistency of raw Q-values is stronger than consistency of action selection: communication-dependent shifts shared by all actions can change Q-values without changing the greedy policy. The theory therefore motivates the proposed regularizer, but does not establish that this particular constraint is necessary for beneficial transfer.

The Predator–Prey experiment provides useful evidence of negative transfer, and the CSD-S comparison supports the value of smoothing in that setting. On SMAC and GRF, however, the reported ablations mainly compare distribution models and perturbation strategies while retaining the smoothing mechanism. These comparisons help assess design choices, but provide less direct evidence that smoothing itself accounts for the principal benchmark improvements.Including the existing no-smoothing variant on a representative main-benchmark task would strengthen the attribution of the gains.

Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Manuscript Corrections, Theoretical Clarifications, and Additional Ablation Studies

In response to your valuable comments, we have added ablation experiments and corrected errors in the manuscript. Our responses are as follows.

Response to Comment 1:

Thank you for pointing this out. This was a typographical error in the perturbation initialization formula, which we have corrected in Section 4.2.3 and Appendix B.2. The perturbation residual codes should be independently randomly initialized within the feasible domain, with the original communication kept as a fixed reference. Projected gradient ascent is then used to search for communication perturbations that maximize the Q-value discrepancy. This is consistent with our submitted code.

Response to Comment 2:

Thank you for recognizing our motivation. We acknowledge that consistency of raw Q-values imposes a stronger constraint than consistency of action selection, and we have added this limitation to the Conclusion. Lemma 1 directly connects communication Q-value smoothness to the existence of a local Q-function approximation, providing theoretical support for teacher smoothing and subsequent Q-value distillation. These conditions provide a theoretical basis for the direct Q-value distillation used in our method, motivating our use of the smoothing loss to constrain the teacher.

Your comment also suggests exploring transfer mechanisms that more directly capture action preferences. Even when a teacher cannot be fully represented by a student, its action selection can still provide useful policy guidance. For example, distilling relative preferences among actions could provide effective supervision under weaker consistency requirements. We consider this an interesting direction and will investigate its effectiveness in future work.

Response to Comment 3:

Thank you for this suggestion. We have added experiments with the no-smoothing variant CSD-S on SMAC and GRF. This variant retains conditional distribution modeling and teacher-student distillation, removing only the communication smoothing loss. The results below report the mean and standard deviation of student success rates:

BenchmarkTaskCSDCSD-S
SMAC8m_vs_9m89.8% ± 1.6%73.1% ± 5.2%
SMAC6h_vs_8z65.3% ± 8.6%14.8% ± 24.5%
GRFacademy_3_vs_1_with_keeper55.8% ± 12.6%47.19% ± 12.9%
GRFacademy_run_pass_and_shoot_with_keeper61.3% ± 33.7%50.8% ± 27.7%

Removing the smoothing loss reduces mean student success rates on all four tasks. Across the two SMAC tasks listed above, the average success rate decreases from approximately 77.6% to 44.0%, with a particularly pronounced reduction on 6h_vs_8z. These results directly demonstrate the contribution of communication smoothing to transfer performance on the main benchmarks. We have included the SMAC ablations in Section 5.4 and Figure 6, and the GRF results in Appendix C.4.2, Table 5.

We thank you again for your careful review. We hope these responses address your concerns and that you will reconsider your assessment of our submission.

Thank you for the additional experiments and clarifications. I appreciate the authors' efforts to address my concerns. However, I still have some remaining concerns regarding the proposed method. Therefore, I will maintain my original rating.

Reviewer_PpquOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the policy gap between communication-based teachers and decentralized students in cooperative multi-agent reinforcement learning. During training, teachers can use information received from other agents, while students can only rely on their own local observations during execution. As a result, directly distilling the teacher's policy into the student may lead to poor performance or even negative transfer.

To address this problem, the authors propose Communication Smoothing Distillation (CSD). The method first uses a Gaussian mixture model (GMM) to learn cross-agent conditional latent distributions. A learned Sampler is then used to generate communication messages. Next, the authors apply gradient-based perturbation search to find changes in communication messages that cause large differences in the teacher's Q-values. A smoothing loss is used to reduce these differences, making the teacher less sensitive to changes in communication. Finally, the Q-values from the smoothed teacher are distilled into CTDE students, which make decisions using only their local observation histories.

Critical strengths

The proposed method addresses the problem of negative transfer during policy distillation, providing a clear motivation.

The paper provides theoretical conditions under which communication-based policies and their Q-functions can be represented by decentralized policies.

CSD achieves improvements over several baselines on partially observable SMAC tasks.

Critical weaknesses
  1. The connection between communication smoothing and the theoretical conditions needs further validation.

CSD performs only a limited number of perturbation steps within the learned GMM latent space. It is unclear whether these perturbations can adequately represent the possible changes in other agents' observations in the actual environment. Therefore, it is difficult to determine how well the smoothing process satisfies the conditions discussed in the theoretical analysis.

Although the results show that learning is more beneficial when the teacher is less sensitive to changes in communication, it could also limit the teacher's ability to use information from other agents. The authors should investigate how smoothing affects the sensitivity of Q-value, teacher performance, and student distillation, which would clarify this point.

  1. The individual contributions of the GMM and Sampler are unclear.

CSD uses both a GMM and a learned Sampler to construct latent communication messages for the smoothing process. However, the current ablation studies mainly compare GMM with VAE. They do not separately examine the benefits of the GMM and Sampler. It remains unclear why both components are needed and how each contributes to the effectiveness of communication smoothing.

Questions for authors

Q1. CSD performs communication perturbations mainly within the latent space learned by the GMM. How do the authors determine whether these perturbations adequately reflect the possible changes in other agents' observations in the actual environment?

Q2. Would stronger communication smoothing limit the teacher's ability of using useful information from other agents?

Q3. What are the respective roles of the GMM and Sampler in constructing latent communication messages? Since the current ablation studies mainly compare GMM with VAE, could the authors further examine how each component contributes to the effectiveness of communication smoothing?

3 comments
AuthorsOpenReview ↗

Additional Visualizations and Ablation Studies to Further Clarify Our Method

We thank you for recognizing the motivation and effectiveness of our work. We have added visualization and ablation experiments to address your questions.

Response to Q1:

We address this question through visualizations of the learned distributions and communication perturbations, together with ablation experiments.

First, Appendix C.2 visualizes the learned conditional observation distributions. In Predator-Prey, we draw 10,000 samples from agent 2's predicted conditional observation distribution given agent 1's local information, decode their positions, and count their frequencies in each grid cell. When agent 2 is outside agent 1's observation range, the distribution spreads across unobserved regions; upon entry, it concentrates near the true position. This supports the model's ability to provide meaningful conditional observation distributions for perturbation search.

Second, Figure A3 visualizes communication perturbations. With agent 2's local observation fixed, we perturb agent 1's communication through gradient search and decode the resulting messages. Agent 1's reconstructed position moves away from its original position shared with the target, while the encoded target position also changes within the feasible range. Enforcing Q-value consistency under these perturbations constrains the teacher's dependence on information unavailable to the student.

Third, we add no-smoothing ablations on SMAC and GRF, reported in Section 5.4, Figure 6, and Appendix C.4.2, Table 5, respectively:

EnvironmentTaskCSDCSD-S
SMAC8m_vs_9m89.8% ± 2.0%73.1% ± 5.2%
SMAC6h_vs_8z65.3% ± 8.6%14.8% ± 24.5%
GRFacademy_3_vs_1_with_keeper55.8% ± 12.6%47.2% ± 12.9%
GRFacademy_run_pass_and_shoot_with_keeper61.3% ± 33.7%50.8% ± 27.7%

These results show that smoothing based on the constructed communication perturbations improves knowledge transfer across environments, providing indirect support for our approach.

Response to Q2:

Our method introduces the smoothing loss precisely to constrain the teacher's dependence on information unavailable to students, thereby providing supervision better suited to CTDE students. Stronger communication smoothing can indeed limit the teacher's ability to use useful information from other agents. Section 5.1, Figure 2, illustrates this effect: compared with CSD-S, which excludes the smoothing loss, the CSD teacher has a slightly lower success rate, while its student's success rate increases from 44.7% to 67.3% (Appendix C.1, Table 2).

The plug-and-play experiments in Section 5.3 further demonstrate the value of this design. As shown in Figure 5, integrating CSD with VDN, QMIX, and Qatten improves student success rates for all three methods on both tasks. Teachers trained with smoothing constraints can thus still provide useful cooperative knowledge to guide students relying only on local information, thereby improving value-decomposition methods. To address your concern about Q-value sensitivity, we also examine the effects of different communication inputs on Q-values in Figure A4. The results show that the smoothing loss reduces the Q-value differences arising when another agent observes the target.

Response to Q3:

The GMM and Sampler jointly support CSD's communication mechanism and smoothing constraints. The GMM models the sender's conditional observation distribution given the receiver's local information, providing a distribution for communication construction and perturbation search. The Sampler combines the sender's current observation with the GMM parameters to generate communication samples and preserves the sender's information through observation reconstruction training.

This design follows Lemma 1: when a communication teacher maintains approximately consistent Q-values across joint observations sharing the same local observation, its Q-function can be approximated using local information alone. Accordingly, we translate other agents' observation variations into latent communication variations and constrain the resulting Q-value differences through the smoothing loss. Both are necessary, interconnected components of our distribution-guided communication architecture, serving conditional distribution modeling and actual observation transmission, respectively. The original manuscript already includes ablations replacing the GMM with a VAE and gradient-based perturbation search with random perturbations (CSD-R), examining the choices of distribution model and perturbation strategy. We are also happy to conduct a separate Sampler ablation to assess its contribution to communication smoothing and knowledge transfer more directly. We will make every effort to complete this experiment and report the results during the discussion period.

We hope these responses address your concerns and welcome suggestions for additional ablation experiments.

We have additionally conducted an ablation study on the Sampler by removing its reconstruction loss and using only the RL objective to guide sampling from the GMM and message transmission. Student success rates decrease to 79.9% ± 2.8% on 8m_vs_9m and 16.3% ± 4.3% on 6h_vs_8z. These results indicate that observation reconstruction supervision helps the Sampler learn to effectively transmit observation information between agents. We hope this additional experiment addresses your concerns about our method design.

Reviewer_PpquOpenReview ↗

response

Thanks for the response and new results. I don't have other questions.

Comments on the paper

1 comment

We appreciate the reviewers' recognition of our research motivation and the performance of our method. In response to their comments, we have revised the manuscript and added experiments to address the main concerns. These updates are highlighted in blue in the manuscript and summarized below.

  1. Additional visualizations and mechanism analyses. We added conditional observation distribution visualizations and decoded communication perturbations to illustrate how latent communication variations represent changes in other agents' observations. We also evaluated Q-value sensitivity to different communication inputs, showing that smoothing reduces communication-induced Q-value differences. Furthermore, we examined the effect of perturbation search steps and discussed the trade-off between teacher performance and knowledge transfer to students.

  2. Additional component ablations. We added no-smoothing ablations on SMAC and GRF, where removing the smoothing loss reduces mean student success rates across all four evaluated tasks. We also completed a Sampler ablation that removes its reconstruction loss and uses only the RL objective to guide communication sampling. The resulting performance decreases highlight the contribution of reconstruction supervision to communication learning and the overall method.

  3. Corrections to the method description. We corrected the descriptions of perturbation initialization, search variables, and latent-message scale, and supplemented the loss implementation details in the appendix.

We hope these revisions and additional experiments address the main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.