acceptodds
Under review as a conference paper at ICLR 2027

Multi-Agent Reinforcement Learning with On-Policy Distillation

Abstract

Centralized training with decentralized execution (CTDE) exploits privileged information during training, yet decentralized multi-agent reinforcement learning policies often remain driven by sparse environmental rewards. Online policy distillation can provide dense supervision, but applying it to MARL introduces observation mismatch between a globally informed teacher and local students, as well as policy learning mismatch between their continuously evolving policies. We propose MAOPD, an online policy distillation framework that jointly trains an autoregressive centralized teacher and decentralized parallel students. Agent distillation bridges observation mismatch by using a local prefix predictor to construct locally realizable decision representations that match privileged teacher features. Action distillation addresses the structural policy mismatch by factorizing the teacher's joint policy into agent-wise conditional targets on student-generated trajectories. The teacher and students further share most policy parameters and co-adapt online through separate reinforcement-learning objectives, alleviating evolving policy mismatch without separate teacher pre-training. Across 26 tasks from multiple MARL benchmarks (i.e., RWARE, SMAX, and MPE), MAOPD achieves the highest aggregate performance under matched environment-interaction budgets and ranks first or tied for first on 22 tasks. Ablations confirm the complementary benefits of both distillation objectives and the online co-adaptation mechanisms.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Centralized Training with Decentralized Execution, On-Policy Distillation, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kkiCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAOPD, an on-policy distillation framework for cooperative MARL that jointly trains a centralized autoregressive teacher and decentralized students. It combines feature-level agent distillation, action-level policy distillation, and shared-parameter co-training, and is evaluated on 26 tasks across RWARE, SMAX, and MPE.

Critical strengths

The empirical evaluation is broad and reasonably rigorous, with 10 seeds, multiple benchmark suites, matched environment-interaction budgets, and extensive ablations. The method consistently outperforms strong MARL and distillation baselines, suggesting that the proposed training scheme is practically effective.

Critical weaknesses

The main weakness is limited methodological novelty. The approach largely combines known ingredients including privileged-information distillation, feature matching, KL-based policy distillation, parameter sharing, and concurrent PPO training, without a strong new theoretical or conceptual principle. The claimed learning-efficiency advantage is also not fully convincing because comparisons mainly normalize environment interactions rather than total computation, while MAOPD trains both teacher and student pathways.

Questions for authors

How does MAOPD compare with MAPPO, HAPPO, and MAT under matched wall-clock time or compute budget? What performance remains if the same shared teacher–student backbone and concurrent PPO training are retained but both proposed distillation mechanisms are removed?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kkiC. (Part1)

We greatly appreciate the time and effort you devoted to reviewing this work. Below, we address each of your concerns in detail.

Weaknesses

W1. The main weakness is limited methodological novelty. The approach largely combines known ingredients including privileged-information distillation, feature matching, KL-based policy distillation, parameter sharing, and concurrent PPO training, without a strong new theoretical or conceptual principle. The claimed learning-efficiency advantage is also not fully convincing because comparisons mainly normalize environment interactions rather than total computation, while MAOPD trains both teacher and student pathways.

We acknowledge that individual components, such as feature matching, KL distillation, parameter sharing, and concurrent PPO training, are established techniques. However, to the best of our knowledge, MAOPD is the first to introduce on-policy distillation into MARL, enabling student-on-policy distillation from a privileged centralized teacher to independently executing decentralized students.

Our key insight is that directly applying OPD to MARL is insufficient due to observation mismatch and policy learning mismatch. MAOPD addresses these challenges through a coordinated design of agent distillation, action distillation, and online teacher–student co-adaptation, rather than simply adding OPD to an existing MARL framework. As demonstrated in Fig. 6, simply introducing OPSD into MARL (naive OPSD) performs substantially worse than MAOPD. Furthermore, our additional experiments addressing reviewer Nyyw (W1/Q1) show that MAGPO-OPD (MAGPO with on-policy-distillation) and other MAGPO variants incorporating MAOPD's modules also fail to reproduce MAOPD's performance. These results demonstrate that simply applying existing methods directly to MARL fails to achieve effective student on-policy distillation.

Moreover, integrating established techniques to address new challenges is a common and valuable approach to methodological innovation. For example, DQN extends Q-learning with deep neural networks, a target network, and experience replay; Rainbow integrates double Q-learning, prioritized replay, dueling networks, multi-step learning, distributional RL, and noisy networks. These works have made significant contributions despite building on existing techniques. Similarly, MAOPD's novelty lies not in the individual components, but in their problem-driven integration to enable effective student-on-policy distillation in MARL, as supported by our empirical analyses and ablation studies.

We have listed the ablation study conducted and their results in the following tables.

Component / DesignLocation
Agent / Action DistillationFigure. 4
Prefix PredictorFigure. 5 (left)
Teacher / Student PPOFigure. 5 (middle)
Shared parametersFigure. 5 (right)
Naive On-Policy DistillationFigure. 6 (left)
Student- vs. Teacher-generated TrajectoriesFigure. 11 (middle)
MethodMPE simple_spread_5ag ReturnSMAX smacv2_10_units Win Rate(%)
MAOPD−16.68 ± 0.1482.62 ± 2.70
w/o Agent-Distillation−17.83 ± 0.6877.38 ± 2.18
w/o Action-Distillation−20.10 ± 0.6180.63 ± 3.45
w/o Dual-Distillation−22.58 ± 0.5974.50 ± 3.03
w/o Teacher PPO−22.89 ± 1.0656.42 ± 8.20
w/o Student PPO−20.42 ± 0.7574.63 ± 4.41
w/o Shared parameters−19.57 ± 0.5666.12 ± 4.51
Teacher-generated Trajectories−19.02 ± 0.4170.40 ± 3.17
Naive On-Policy Distillation−25.38 ± 0.8036.80 ± 2.50
MethodMPE simple_spread_5ag ReturnMPE simple_spread_10ag Return
MAOPD−16.68 ± 0.14−38.81 ± 0.62
w/o Prefix-Predictor−17.98 ± 0.37−41.10 ± 0.73

Together, these results support MAOPD as a problem-driven methodological contribution, rather than a simple combination of existing techniques.

AuthorsOpenReview ↗

Response to Reviewer kkiC. (Part2)

The claimed learning-efficiency advantage is also not fully convincing because comparisons mainly normalize environment interactions rather than total computation, while MAOPD trains both teacher and student pathways.

We agree that matching environment interactions alone does not establish computational efficiency. To address this concern, we conduct additional wall-clock comparisons with MAPPO, HAPPO, and MAT under identical hardware conditions.

Specifically, the wall-clock budget is defined as the time MAOPD requires to complete 20M total environment interactions (including both teacher and student rollouts). We compare performance under both matched wall-clock time and matched environment steps (20M).

SMAX smacv2_10_units (Win Rate ↑)

MethodMatched Wall-Clock TimeMatched Env. Steps
MAPPO78.44 ± 6.33%78.50 ± 2.67%
HAPPO43.13 ± 9.06%48.69 ± 4.25%
MAT75.63 ± 7.03%69.19 ± 2.41%
MAOPD82.62 ± 2.70%82.62 ± 2.70%

RWARE tiny-4ag (Episode Return ↑)

MethodMatched Wall-Clock TimeMatched Env. Steps
MAPPO28.02 ± 2.2126.36 ± 3.20
HAPPO27.37 ± 3.2018.05 ± 1.92
MAT25.67 ± 5.5225.78 ± 1.79
MAOPD38.53 ± 3.8638.53 ± 3.86

MPE simple_spread_5ag (Episode Return ↑)

MethodMatched Wall-Clock TimeMatched Env. Steps
MAPPO−20.80 ± 0.54−22.49 ± 0.40
HAPPO−20.41 ± 1.24−23.29 ± 0.64
MAT−22.04 ± 0.89−31.08 ± 1.03
MAOPD−16.68 ± 0.14−16.68 ± 0.14

These results demonstrate that MAOPD consistently achieves the best performance under both evaluation protocols. Its advantages under matched wall-clock time further validate its practical learning efficiency, complementing the sample-efficiency results reported in the paper.

Questions

Q1. How does MAOPD compare with MAPPO, HAPPO, and MAT under matched wall-clock time or compute budget? What performance remains if the same shared teacher–student backbone and concurrent PPO training are retained but both proposed distillation mechanisms are removed?

Matched Wall-Clock Time. We provide additional comparisons with MAPPO, HAPPO, and MAT-Dec under matched wall-clock time and environment-interaction budgets in our response to W1. The results show that MAOPD consistently outperforms all three baselines under both settings, further supporting its practical learning efficiency.

Shared Backbone + Concurrent PPO without Distillation. This exact ablation is already included in Fig. 4 as "w/o dual-distillation." It removes both Agent Distillation and Action Distillation while retaining the shared teacher–student backbone and concurrent PPO training (i.e., α=β=0\alpha=\beta=0 in Eq. 8). Across MPE, SMAX, and RWARE, this variant consistently underperforms full MAOPD, demonstrating that the improvements cannot be attributed solely to parameter sharing or concurrent PPO. The two proposed distillation mechanisms provide additional benefits beyond these existing training components.

AuthorsOpenReview ↗

Anticipating Further Discussions with Reviewer kkiC

Dear Reviewer,

The authors believe that the revised manuscript and our responses have addressed the concerns regarding:

  • the methodological novelty and contributions of MAOPD beyond existing techniques;
  • the computational efficiency of MAOPD under matched wall-clock budgets;
  • the performance of MAOPD when both distillation mechanisms are removed.

We would be grateful if you would take these revisions into account in your final assessment. Should any questions remain, we would be pleased to provide further clarification.

Reviewer_Z8KkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an online policy distillation framework, that trains an autoregressive centralized teacher and students, all sharing parameters and co-adapting online through separate RL objectives, without requiring separate teacher pre-training This proposal (a) bridges observation mismatch by using a students’ shared prefix predictor to construct local representations of students’ unobserved features (coordination information), aligned with the teachers’ privileged “global” observations; and (b) addresses the teacher-students structural policy mismatch by factorizing the teacher’s joint policy into agent-wise conditional targets on student-generated trajectories. The proposed framework, MAODP, is evaluated in MARL benchmarks, showing its merits, while ablations confirm the complementary benefits of both distillation objectives and the online co-adaptation mechanisms.

Critical strengths

This is a well motivated and concisely presented work. Contributions claimed are supported by experimental evidence and ablation studies. Authors bridge observation mismatch between students and teacher while addressing the teacher-students structural policy mismatch by factorizing the teacher’s joint policy into agent-wise conditional targets on student-generated trajectories.

Critical weaknesses

Authors miss a comparison with closely-related works such as SMPE^2 [1], LIAM[2] and SIDE [3]. These works constrain the innovation of this proposal.

[1] Andreas Kontogiannis, Konstantinos Papathanasiou, Yi Shen, Giorgos Stamou, Michael M. Zavlanos, and George Vouros. 2025. Enhancing cooperative multi-agent reinforcement learning with state modelling and adversarial exploration. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267. JMLR.org, Article 1235, 31437–31466.

[2] Papoudakis, G., Christianos, F., and Albrecht, S. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210–19222, 2021.

[3] Xu, Z., Bai, Y., Li, D., Zhang, B., and Fan, G. Side: State inference for partially observable cooperative multi-agent reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1400–1408, 2022.

After the rebuttal: I am satisfied from the responses and revision provided by authors.

Questions for authors
  • Can the teacher be generalized into a non-autoregressive one? What would be the consequences of this in your architecture?
  • Can you provide a brief qualitative comparison with each of the related works indicated?

After the rebuttal: I am satisfied from the responses and revision provided by authors.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Z8Kk. (Part1)

Thank you for your thorough review and helpful remarks. Below, we address each of your concerns in detail.

Weaknesses

W1. Authors miss a comparison with closely-related works such as SMPE^2 [1], LIAM[2] and SIDE [3]. These works constrain the innovation of this proposal. [1] Andreas Kontogiannis, Konstantinos Papathanasiou, Yi Shen, Giorgos Stamou, Michael M. Zavlanos, and George Vouros. 2025. Enhancing cooperative multi-agent reinforcement learning with state modelling and adversarial exploration. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267. JMLR.org, Article 1235, 31437–31466. [2] Papoudakis, G., Christianos, F., and Albrecht, S. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210–19222, 2021. [3] Xu, Z., Bai, Y., Li, D., Zhang, B., and Fan, G. Side: State inference for partially observable cooperative multi-agent reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1400–1408, 2022.

Thank you for highlighting these relevant works. We agree that they are related to MAOPD and have added them to the Related Work section, together with additional experimental comparisons in the Appendix.

All methods use the same total environment-interaction budget and are run with 10 random seeds.

Qualitative Comparison. These works primarily focus on learning representations of missing information, whereas MAOPD focuses on transferring coordination knowledge from a privileged centralized teacher actor to decentralized students through student-on-policy distillation. Specifically:

MethodMain Approach
SMPE²Models unobserved states from local histories and enhances exploration through adversarial exploration.
LIAMLearns representations of other agents by reconstructing their observations and actions from local information.
SIDEInfers global state information from local observations to improve value decomposition.
MAOPD (Ours)Employs student-on-policy distillation to transfer privileged decision features and action distributions from a centralized teacher actor to decentralized students through dense supervision.

Performance Comparison. We additionally compare MAOPD with these three methods across RWARE, SMAX, and MPE:

taskMAOPDSMPE$^2$LIAMSIDE
RWARE tiny-4ag38.53 ± 3.8611.83 ± 2.1614.08 ± 1.721.60 ± 1.61
SMAX smacv2_10_units82.62% ± 2.70%53.80% ± 7.00%44.00% ± 0.62%70.60% ± 1.91%
MPE simple_spread_5ag−16.68 ± 0.14−17.26 ±2.13−27.39 ± 4.50−27.11 ± 0.48

MAOPD consistently outperforms all three methods across these tasks, demonstrating the effectiveness of transferring centralized policy knowledge through dense decision-level supervision.

SMPE²-based Replacement. SMPE² is the most recent of these works, and its original paper already includes comparisons with LIAM and SIDE. We therefore further investigate its differences from MAOPD through a controlled replacement experiment.

Specifically, we introduce MAOPD - Prefix + SMPE$^2$, which replaces the original Prefix Predictor and Agent Distillation with SMPE²-style belief modeling. The learned belief representation is projected into a prefix token for the student decoder, while SMPE²'s auxiliary state-modeling objectives, filtered critic, and SimHash-based exploration rewards are incorporated into student training. We retain MAOPD's Action Distillation and joint teacher–student PPO training.

taskMAOPDMAOPD - Prefix + SMPE$^2$
RWARE tiny-4ag38.53 ± 3.8624.86 ± 5.13
SMAX smacv2_10_units82.62% ± 2.70%79.63% ± 1.44%
MPE simple_spread_5ag−16.68 ± 0.14−18.05 ± 0.82

The SMPE²-based variant underperforms MAOPD across all three tasks. These results suggest that, within MAOPD, our Prefix Predictor and Agent Distillation more effectively transfer coordination-relevant decision information encoded in the teacher's privileged global context and preceding agents' actions than SMPE²-style state modeling.

AuthorsOpenReview ↗

Response to Reviewer Z8Kk. (Part2)

Questions

Q1. Can the teacher be generalized into a non-autoregressive one? What would be the consequences of this in your architecture?

Yes, MAOPD can be extended to non-autoregressive centralized teachers. Such a variant is called MAOPD(non-auto).

MAOPD(non-auto), our non-autoregressive teacher uses a centralized SABLE encoder with temporal memory to process all agents' observations, followed by an MLP that directly outputs a joint action distribution over KNK^N actions. The joint action is sampled in a single step without autoregressive decoding. For action distillation, we condition the joint distribution on the preceding actions recorded in student trajectories and marginalize over the remaining agents' actions, obtaining a K−K-dimensional conditional action distribution for each agent. These distributions serve as agent-wise distillation targets without requiring sequential teacher decoding. The dual-distillation objectives and concurrent PPO training remain applicable, while decentralized student network (and execution) remains unchanged.

We conduct two sets of experiments: the first compares MAOPD (non-auto) with MAGPO, and the second compares MAOPD with MAGPO. In each setting, MAGPO’s guider/teacher uses the same architecture as the teacher of the corresponding MAOPD variant. The results are presented below.

Non-autoregressive centralized teacher: MAOPD (non-auto)

TaskMAOPD (non-auto) StudentMAOPD (non-auto) TeacherMAOPD Teacher − MAOPD StudentMAGPO Student
tiny-2ag20.36 ± 1.7120.49 ± 1.49+0.1319.18 ± 0.33
tiny-4ag26.18 ± 3.0829.20 ± 3.14+3.0219.26 ± 3.92
small-4ag-hard11.06 ± 1.4111.70 ± 1.00+0.644.79 ± 1.01

Autoregressive centralized teacher: MAOPD

TaskMAOPD StudentMAOPD TeacherMAOPD Teacher − MAOPD StudentMAGPO Student
tiny-2ag20.93 ± 0.4120.84 ± 0.34−0.0817.98 ± 1.24
tiny-4ag38.53 ± 3.8643.88 ± 0.77+5.3530.57 ± 3.07
small-4ag-hard11.63 ± 0.6913.22 ± 0.90+1.6011.51 ± 0.65

The results show that MAOPD remains effective with non-autoregressive teachers, as students achieve performance close to their teachers and consistently outperform MAGPO under the same teacher architecture.

However, non-autoregressive teachers exhibit consistently weaker performance across all three tasks, resulting in correspondingly lower student performance. One possible explanation is that non-autoregressive teachers face a trade-off. Directly modeling the joint action distribution incurs an exponentially growing output space and makes agent-wise distillation to decentralized students difficult. Alternatively, parallel factorized policies (factoring the action space) avoid this complexity but cannot explicitly capture dependencies between agents' actions. In contrast, autoregressive teachers factorize joint decisions into tractable agent-wise conditional distributions, enabling more effective coordinated exploration.

Q2. Can you provide a brief qualitative comparison with each of the related works indicated?

We thank the reviewer for this question. Please refer to our response to W1, where we provide a qualitative comparison of MAOPD with SMPE², LIAM, and SIDE, highlighting their key methodological differences.

AuthorsOpenReview ↗

Anticipating Further Discussions with Reviewer Z8Kk

Dear Reviewer,

The authors believe that the revised manuscript and our responses have addressed the concerns regarding:

  • the relationship, methodological differences, and novelty of MAOPD relative to SMPE², LIAM, and SIDE;
  • the applicability of MAOPD to non-autoregressive centralized teachers and its architectural and performance implications.

We would be grateful if you would take these revisions into account in your final assessment. Should any questions remain, we would be pleased to provide further clarification.

Reviewer_NyywOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAOPD, an on-policy distillation framework for MARL that addresses observation and policy learning mismatches between centralized teachers and decentralized students. It combines feature distillation through a prefix predictor, action distillation on student-generated trajectories, and online teacher–student co-adaptation.

Experiments on 26 tasks across RWARE, SMAX, and MPE show that MAOPD achieves the best or tied-best performance on 22 tasks. Ablation studies further evaluate the effectiveness of its main components.

Critical strengths
  1. The paper addresses observation and policy mismatches between centralized teachers and decentralized students through feature distillation, action distillation, and online co-adaptation, providing a practical approach to leveraging centralized information for decentralized policy learning.

  2. The experiments cover 26 tasks across three MARL benchmarks under matched total environment-interaction budgets. MAOPD achieves strong overall performance, particularly in sparse-reward settings, supported by reasonably comprehensive ablation studies.

Critical weaknesses
  1. MAGPO already employs a centralized autoregressive guider and agent-wise conditional KL distillation. Although the paper compares MAOPD against MAGPO and includes several component ablations, it does not directly establish how much improvement comes specifically from student-generated distillation trajectories versus the additional prefix predictor, parameter sharing, and online optimization mechanisms. Furthermore, the marginal benefit of learning a centralized actor over a controlled critic-only CTDE alternative remains unclear. These missing comparisons make it difficult to assess the necessity and novelty of the proposed framework.

  2. The original CTDS method distills centralized Q-values into decentralized Q-functions and updates its teacher and student concurrently. However, the manuscript characterizes CTDS differently across the related-work and experimental sections, including describing it as behavioral distillation. Similarly, MAT is introduced as a centralized-execution method, while the appendix indicates that a decentralized implementation is used. Without clearer descriptions of the actual baseline implementations and modifications, it is difficult to determine whether the reported comparisons faithfully represent the referenced methods under comparable execution constraints.

Questions for authors
  1. MAGPO already uses an autoregressive guider and agent-wise conditional KL objectives. Could the authors provide a controlled MAGPO variant that uses student-generated trajectories for distillation while preserving comparable optimization settings and total environment-interaction budgets? This would help isolate the contribution of student-grounded distillation from the additional architectural and training modifications in MAOPD.

  2. Can the authors compare MAOPD with a critic-only CTDE variant using the same decentralized student architecture and a comparable training budget? Such a comparison would help clarify whether learning a centralized actor provides benefits beyond centralized value estimation and additional training supervision.

  3. The original CTDS method performs Q-value distillation and updates its teacher and student concurrently. Why is CTDS described as behavioral distillation in Section 4.1, and does the experimental implementation follow the original algorithm or a modified version? Similarly, could the authors clarify which MAT implementation is used in Table 1, what information is available to each agent during execution, and whether action generation depends on other agents' current actions?

  4. Figure 5 suggests that the Prefix Predictor provides relatively limited performance improvements. Moreover, it remains unclear whether this module actually learns the claimed coordination-relevant information or whether its benefits primarily arise from additional model capacity. Could the authors provide a parameter-matched baseline or further analyze the learned prefix representations to validate the module's specific contribution and design rationale?

  5. On SMAX 5m_vs_6m, MAOPD achieves a substantially lower mean win rate than MAT and exhibits considerable variance across seeds. Could the authors report the corresponding teacher and student learning curves and per-seed results? If the teacher performs substantially better than the students, additional analysis of the feature- and action-distillation objectives on this task would help determine whether the degradation results from unsuccessful knowledge transfer or insufficient teacher performance.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Nyyw. (Part1)

Thanks for your time and effort in reviewing this work. We address your concerns as follows.

Weaknesses

W1. MAGPO already employs a centralized autoregressive guider and agent-wise conditional KL distillation. Although the paper compares MAOPD against MAGPO and includes several component ablations, it does not directly establish how much improvement comes specifically from student-generated distillation trajectories versus the additional prefix predictor, parameter sharing, and online optimization mechanisms. Furthermore, the marginal benefit of learning a centralized actor over a controlled critic-only CTDE alternative remains unclear. These missing comparisons make it difficult to assess the necessity and novelty of the proposed framework.

We thank the reviewer for the constructive comments. We have added the corresponding results to the appendix.

Existing Component Ablations. We have conducted extensive ablations to isolate the contributions of MAOPD's components:

The conducted experiments are listed as follows.

Component / DesignLocation
Agent / Action DistillationFigure. 4
Prefix PredictorFigure. 5 (left)
Teacher / Student PPOFigure. 5 (middle)
Shared parametersFigure. 5 (right)
Naive On-Policy DistillationFigure. 6 (left)
Student- vs. Teacher-generated TrajectoriesFigure. 11 (middle)

The following table demonstrates the individual contributions of the proposed mechanisms.

MethodMPE simple_spread_5ag ReturnSMAX smacv2_10_units Win Rate(%)
MAOPD−16.68 ± 0.1482.62 ± 2.70
w/o Agent-Distillation−17.83 ± 0.6877.38 ± 2.18
w/o Action-Distillation−20.10 ± 0.6180.63 ± 3.45
w/o Dual-Distillation−22.58 ± 0.5974.50 ± 3.03
w/o Teacher PPO−22.89 ± 1.0656.42 ± 8.20
w/o Student PPO−20.42 ± 0.7574.63 ± 4.41
w/o Shared parameters−19.57 ± 0.5666.12 ± 4.51
Teacher-generated Trajectories−19.02 ± 0.4170.40 ± 3.17
Naive On-Policy Distillation−25.38 ± 0.8036.80 ± 2.50
MethodMPE simple_spread_5ag ReturnMPE simple_spread_10ag Return
MAOPD−16.68 ± 0.14−38.81 ± 0.62
w/o Prefix-Predictor−17.98 ± 0.37−41.10 ± 0.73

Controlled Comparisons with MAGPO. We further conduct component-level comparisons using MAGPO as the base framework. The five variants are configured as follows:

VariantModification Relative to NAGPO
MAGPO-OPDShifts the Student's policy KL distillation from guider-generated to student-generated trajectories, while keeping all other MAGPO components and objectives unchanged.
MAGPO + Student PPORetains all original MAGPO components and objectives, while adding student-side PPO and value learning on student-generated trajectories.
MAGPO + Agent DistillationRetains all original MAGPO components and objectives, while adding feature-level MSE distillation between the Student's representations and the Guider's intermediate features.
MAGPO + PredictorRetains the original MAGPO components and objectives, and adds a local token predictor after the Student GRU, whose predicted token is fused with the GRU features before the original policy network.
MAGPO + Agent Distillation + PredictorRetains all original MAGPO components and objectives, while combining the Agent Distillation and Predictor modifications described above.

All variants use the same PPO settings and the same total environment-interaction budget, and are run with 10 random seeds.

MethodMPE simple_spread_5ag ReturnSMAX smacv2_10_units Win Rate(%)
MAOPD−16.68 ± 0.1482.62 ± 2.70
MAGPO−18.35 ± 0.4376.54 ± 2.91
MAGPO-OPD−21.09 ± 0.4469.28 ± 5.19
MAGPO + Student PPO−22.68 ± 0.5971.02 ± 3.35
MAGPO + Predictor−18.35 ± 0.5276.31 ± 3.26
MAGPO + Agent distillation−19.37 ± 0.5077.45 ± 3.16
MAGPO + Agent distillation + Predictor−18.94 ± 0.4076.16 ± 2.85

Learning curves can be seen here. These results on MPE and SMAX show that MAOPD outperforms all these variants. In particular, simply replacing MAGPO's distillation trajectories with student-generated trajectories does not reproduce MAOPD's gains. This supports the importance of our jointly designed mechanisms rather than any single modification to MAGPO.

AuthorsOpenReview ↗

Response to Reviewer Nyyw. (Part2)

Furthermore, the marginal benefit of learning a centralized actor over a controlled critic-only CTDE alternative remains unclear. These missing comparisons make it difficult to assess the necessity and novelty of the proposed framework.

Centralized Actor over Critic-only CTDE. Unlike controlled critic-only CTDE methods (MAPPO and HAPPO), MAOPD additionally learns a centralized teacher actor and its associated critic. Their architectural differences are summarized below:

MethodDecentralized ActorStudent CriticCentralized Teacher ActorTeacher Critic
MAPPO✓✓——
HAPPO✓✓——
MAOPD✓✓✓✓

MAOPD also outperforms MAPPO and HAPPO on three representative tasks:

MethodMPE simple_spread_5ag ReturnSMAX smacv2_10_units Win Rate(%)
MAOPD−16.68 ± 0.1482.62% ± 2.70%
MAPPO−22.49 ± 0.4078.50% ± 2.67%
HAPPO−23.29 ± 0.6448.69% ± 4.25%

To further isolate the benefit of centralized actor learning from architectural differences, we introduce the following architecture-controlled variants:

VariantTraining ObjectiveControlled Configuration
MAOPD student-criticDecentralized student actor + Student criticRetains the student backbone, prefix predictor, and centralized critic; removes the teacher actor and teacher-dependent objectives.
MAOPD student-critic + teacher-criticDecentralized student actor + Student critic + Teacher criticExtends critic-only by retaining an additional teacher-side centralized critic.
MAOPD w/o Dual DistillationDecentralized student actor + Student critic + Centralized teacher actor + Teacher criticBased on MAOPD, removes both agent and action distillation objectives.

All variants use the same PPO settings and the same total environment-interaction budget, and are run with 10 random seeds.

MethodMPE simple_spread_5ag ReturnSMACv2 10_units Win Rate(%)
MAOPD−16.68 ± 0.1482.62 ± 2.70
MAOPD student-critic−22.64 ± 0.4441.37 ± 8.32
MAOPD student-critic + teacher-critic−24.45 ± 0.9549.46 ± 10.36
MAOPD w/o dual-distillation−22.59 ± 0.6074.28 ± 3.48

Learning curves can be seen here. MAOPD outperforms all three variants on MPE and SMAX. Together, these comparisons support the benefits of jointly learning a centralized actor and transferring its privileged coordination knowledge through dense feature-level and action-level distillation, beyond conventional critic-only CTDE.

W2. The original CTDS method distills centralized Q-values into decentralized Q-functions and updates its teacher and student concurrently. However, the manuscript characterizes CTDS differently across the related-work and experimental sections, including describing it as behavioral distillation. Similarly, MAT is introduced as a centralized-execution method, while the appendix indicates that a decentralized implementation is used. Without clearer descriptions of the actual baseline implementations and modifications, it is difficult to determine whether the reported comparisons faithfully represent the referenced methods under comparable execution constraints.

We thank the reviewer for identifying the inconsistencies in our baseline descriptions. We have added the corresponding results to the appendix.

CTDS. As correctly noted by the reviewer, CTDS concurrently trains centralized and decentralized Q-functions through Q-value distillation. We have corrected the inaccurate description in Sec. 4.1 from "CTDS directly distills the behavior of a centralized teacher into decentralized students" to "CTDS concurrently trains centralized and decentralized Q-functions through Q-value distillation." We have also clarified the corresponding descriptions in the Introduction and Sec. 2.

MAT. For a fair comparison under decentralized execution constraints, Table 1 uses decentralized MAT, the decentralized variant introduced in the original MAT paper, which we refer to as MAT throughout our experiments. During execution, each agent selects actions independently using only its local observations/history, without access to global information or other agents' current actions. In contrast, centralized MAT (denoted MAT-CTCE) exploits additional information during execution, making a direct comparison with decentralized methods unfair under different execution constraints.

Regarding Appendix B.1.4 (Table 3), all three methods are evaluated under decentralized execution. Specifically, MAT refers to decentralized MAT, while MAT-MAGPO and MAT-Ours use MAT-CTCE as their guider and teacher, respectively, with only their decentralized student policies retained for execution. We have clarified these definitions and updated the corresponding descriptions and table captions in revised PDF.

AuthorsOpenReview ↗

Response to Reviewer Nyyw. (Part3)

Questions

Q1. MAGPO already uses an autoregressive guider and agent-wise conditional KL objectives. Could the authors provide a controlled MAGPO variant that uses student-generated trajectories for distillation while preserving comparable optimization settings and total environment-interaction budgets? This would help isolate the contribution of student-grounded distillation from the additional architectural and training modifications in MAOPD.

We thank the reviewer for the suggestion. As discussed in our response to W1, we conduct controlled comparisons to isolate the contribution of student-generated distillation trajectories.

Specifically, we introduce MAGPO-OPD, which replaces MAGPO's guider-generated distillation trajectories with student-generated trajectories while keeping the remaining MAGPO components and optimization settings unchanged. All methods use the same total environment-interaction budget.

MethodMPE simple_spread_5ag ReturnSMAX smacv2_10_units Win Rate(%)
MAOPD−16.68 ± 0.1482.62 ± 2.70
MAGPO−18.35 ± 0.4376.54 ± 2.91
MAGPO-OPD−21.09 ± 0.4469.28 ±5.19

Additional learning curves show that MAGPO_OPD underperforms the original MAGPO on both tasks, while MAOPD consistently achieves the best performance. These results demonstrate that simply adopting student-generated trajectories is insufficient to reproduce MAOPD's gains. These findings, together with the component-level comparisons in our response to W1, support that MAOPD effectively adapts OPD to MARL through its integrated design, rather than merely changing the source of distillation trajectories.

Q2. Can the authors compare MAOPD with a critic-only CTDE variant using the same decentralized student architecture and a comparable training budget? Such a comparison would help clarify whether learning a centralized actor provides benefits beyond centralized value estimation and additional training supervision.

We thank the reviewer for the suggestion. As detailed in our response to W1, we conduct controlled experiments to examine the benefits of centralized actor learning beyond centralized value estimation.

Specifically, we introduce three MAOPD variants that retain the same student backbone, Prefix Predictor, and centralized student critic, while differing in their teacher-side components and training objectives:

VariantTraining ObjectiveControlled Configuration
MAOPD student-criticDecentralized student actor + Student criticRetains the student backbone, prefix predictor, and centralized critic; removes the teacher actor and teacher-dependent objectives.
MAOPD student-critic + teacher-criticDecentralized student actor + Student critic + Teacher criticExtends critic-only by retaining an additional teacher-side centralized critic.
MAOPD w/o Dual DistillationDecentralized student actor + Student critic + Centralized teacher actor + Teacher criticBased on MAOPD, removes both agent and action distillation objectives.
MAOPDDecentralized student actor + Student critic + Centralized teacher actor + Teacher criticFull framework with agent and action distillation

All variants use the same PPO settings and the same total environment-interaction budget, and are run with 10 random seeds.

MethodMPE simple_spread_5ag ReturnSMAX smacv2_10_units Win Rate(%)
MAOPD−16.68 ± 0.1482.62 ± 2.70
MAOPD student-critic−22.64 ± 0.4441.37 ± 8.32
MAOPD student-critic + teacher-critic−24.45 ± 0.9549.46 ± 10.36
MAOPD w/o Dual Distillation−22.59 ± 0.6074.28 ± 3.48

Learning curves can be seen here. The results show that MAOPD substantially outperforms both critic-only variants, even with additional teacher-side value supervision, as well as the variant without dual distillation. These findings indicate that centralized actor learning combined with feature- and action-level distillation provides benefits beyond centralized value estimation and additional training supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.