- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MAOPD, an on-policy distillation framework for cooperative MARL that jointly trains a centralized autoregressive teacher and decentralized students. It combines feature-level agent distillation, action-level policy distillation, and shared-parameter co-training, and is evaluated on 26 tasks across RWARE, SMAX, and MPE.
Critical strengths
The empirical evaluation is broad and reasonably rigorous, with 10 seeds, multiple benchmark suites, matched environment-interaction budgets, and extensive ablations. The method consistently outperforms strong MARL and distillation baselines, suggesting that the proposed training scheme is practically effective.
Critical weaknesses
The main weakness is limited methodological novelty. The approach largely combines known ingredients including privileged-information distillation, feature matching, KL-based policy distillation, parameter sharing, and concurrent PPO training, without a strong new theoretical or conceptual principle. The claimed learning-efficiency advantage is also not fully convincing because comparisons mainly normalize environment interactions rather than total computation, while MAOPD trains both teacher and student pathways.
Questions for authors
How does MAOPD compare with MAPPO, HAPPO, and MAT under matched wall-clock time or compute budget? What performance remains if the same shared teacher–student backbone and concurrent PPO training are retained but both proposed distillation mechanisms are removed?
3 comments
Response to Reviewer kkiC. (Part2)
The claimed learning-efficiency advantage is also not fully convincing because comparisons mainly normalize environment interactions rather than total computation, while MAOPD trains both teacher and student pathways.
We agree that matching environment interactions alone does not establish computational efficiency. To address this concern, we conduct additional wall-clock comparisons with MAPPO, HAPPO, and MAT under identical hardware conditions.
Specifically, the wall-clock budget is defined as the time MAOPD requires to complete 20M total environment interactions (including both teacher and student rollouts). We compare performance under both matched wall-clock time and matched environment steps (20M).
SMAX smacv2_10_units (Win Rate ↑)
| Method | Matched Wall-Clock Time | Matched Env. Steps |
|---|---|---|
| MAPPO | 78.44 ± 6.33% | 78.50 ± 2.67% |
| HAPPO | 43.13 ± 9.06% | 48.69 ± 4.25% |
| MAT | 75.63 ± 7.03% | 69.19 ± 2.41% |
| MAOPD | 82.62 ± 2.70% | 82.62 ± 2.70% |
RWARE tiny-4ag (Episode Return ↑)
| Method | Matched Wall-Clock Time | Matched Env. Steps |
|---|---|---|
| MAPPO | 28.02 ± 2.21 | 26.36 ± 3.20 |
| HAPPO | 27.37 ± 3.20 | 18.05 ± 1.92 |
| MAT | 25.67 ± 5.52 | 25.78 ± 1.79 |
| MAOPD | 38.53 ± 3.86 | 38.53 ± 3.86 |
MPE simple_spread_5ag (Episode Return ↑)
| Method | Matched Wall-Clock Time | Matched Env. Steps |
|---|---|---|
| MAPPO | −20.80 ± 0.54 | −22.49 ± 0.40 |
| HAPPO | −20.41 ± 1.24 | −23.29 ± 0.64 |
| MAT | −22.04 ± 0.89 | −31.08 ± 1.03 |
| MAOPD | −16.68 ± 0.14 | −16.68 ± 0.14 |
These results demonstrate that MAOPD consistently achieves the best performance under both evaluation protocols. Its advantages under matched wall-clock time further validate its practical learning efficiency, complementing the sample-efficiency results reported in the paper.
Questions
Q1. How does MAOPD compare with MAPPO, HAPPO, and MAT under matched wall-clock time or compute budget? What performance remains if the same shared teacher–student backbone and concurrent PPO training are retained but both proposed distillation mechanisms are removed?
Matched Wall-Clock Time. We provide additional comparisons with MAPPO, HAPPO, and MAT-Dec under matched wall-clock time and environment-interaction budgets in our response to W1. The results show that MAOPD consistently outperforms all three baselines under both settings, further supporting its practical learning efficiency.
Shared Backbone + Concurrent PPO without Distillation. This exact ablation is already included in Fig. 4 as "w/o dual-distillation." It removes both Agent Distillation and Action Distillation while retaining the shared teacher–student backbone and concurrent PPO training (i.e., in Eq. 8). Across MPE, SMAX, and RWARE, this variant consistently underperforms full MAOPD, demonstrating that the improvements cannot be attributed solely to parameter sharing or concurrent PPO. The two proposed distillation mechanisms provide additional benefits beyond these existing training components.
Anticipating Further Discussions with Reviewer kkiC
Dear Reviewer,
The authors believe that the revised manuscript and our responses have addressed the concerns regarding:
- the methodological novelty and contributions of MAOPD beyond existing techniques;
- the computational efficiency of MAOPD under matched wall-clock budgets;
- the performance of MAOPD when both distillation mechanisms are removed.
We would be grateful if you would take these revisions into account in your final assessment. Should any questions remain, we would be pleased to provide further clarification.
Response to Reviewer kkiC. (Part1)
We greatly appreciate the time and effort you devoted to reviewing this work. Below, we address each of your concerns in detail.
Weaknesses
We acknowledge that individual components, such as feature matching, KL distillation, parameter sharing, and concurrent PPO training, are established techniques. However, to the best of our knowledge, MAOPD is the first to introduce on-policy distillation into MARL, enabling student-on-policy distillation from a privileged centralized teacher to independently executing decentralized students.
Our key insight is that directly applying OPD to MARL is insufficient due to observation mismatch and policy learning mismatch. MAOPD addresses these challenges through a coordinated design of agent distillation, action distillation, and online teacher–student co-adaptation, rather than simply adding OPD to an existing MARL framework. As demonstrated in Fig. 6, simply introducing OPSD into MARL (naive OPSD) performs substantially worse than MAOPD. Furthermore, our additional experiments addressing reviewer Nyyw (W1/Q1) show that MAGPO-OPD (MAGPO with on-policy-distillation) and other MAGPO variants incorporating MAOPD's modules also fail to reproduce MAOPD's performance. These results demonstrate that simply applying existing methods directly to MARL fails to achieve effective student on-policy distillation.
Moreover, integrating established techniques to address new challenges is a common and valuable approach to methodological innovation. For example, DQN extends Q-learning with deep neural networks, a target network, and experience replay; Rainbow integrates double Q-learning, prioritized replay, dueling networks, multi-step learning, distributional RL, and noisy networks. These works have made significant contributions despite building on existing techniques. Similarly, MAOPD's novelty lies not in the individual components, but in their problem-driven integration to enable effective student-on-policy distillation in MARL, as supported by our empirical analyses and ablation studies.
We have listed the ablation study conducted and their results in the following tables.
Together, these results support MAOPD as a problem-driven methodological contribution, rather than a simple combination of existing techniques.