- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on the exploitation–exploration trade-off problem. It combines entropy-based methods and divergence-regularized methods and introduces a pseudo-expert policy trained on well-performing trajectories. The proposed method automatically switches between imitating the pseudo-expert and exploring actions. Finally, the authors test the algorithm's performance on SMAC, SMACv2, and GRF environments.
Critical strengths
-
The paper's writing is easy to follow, and its structure is well organized.
-
The idea is easy to understand, and the implementation is easy to follow.
-
The visualization experiments are helpful for understanding the underlying mechanism of the proposed algorithm.
Critical weaknesses
- The novelty of the idea is a bit poor. Learning from self-generated trajectories is not a new approach. There are some existing studies that have explored this direction. I have provided two related studies published in 2021 and 2022, and I believe there may be more studies from the past four years that have explored this direction. However, the authors do not fully introduce such related studies or compare their approach with existing work, weakening the novelty of this paper.
[1] Zhang Y, Yang Q, Li D, et al. A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility[J]. Applied Energy, 2022, 323: 119543. [2] Luo S, Kasaei H, Schomaker L. Self-imitation learning by planning[C]//2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 4823-4829.
-
There is a potentially important issue behind the self-generated trajectory mechanism: relying on early-stage exploration to identify pseudo-expert strategies presents a challenge. If the initial exploration is insufficient, particularly in environments with sparse rewards, and a good strategy is not found, then such pseudo-expert strategies will be unable to provide adequate strategic guidance.
-
I have some concerns about the reliability of the experimental results. On the 6h_vs_8z map in Figure 4, the baselines MAPPO and RODE perform very badly, with win rates below 30%. However, in the original MAPPO paper, both MAPPO and RODE can achieve win rates of at least 60% within 3M training steps. There is a large gap between the authors' reported results and the existing results, especially considering that the authors use the default settings provided in the original papers for the baselines.
-
In Section 4.2, the test environments (3m and 2s3z) under sparse-reward settings are too easy. As I mentioned earlier, would this algorithm fail in a slightly more complex environment with sparse rewards?
Questions for authors
Why is R_{thr} set to the task-dependent maximum return R_{max} in SMAC and GRF environments? On some difficult maps, especially considering the sparse-reward setting, few trajectories satisfy this criterion.
6 comments
Response #2 to Reviewer A6iw
W2. There is a potentially important issue behind the self-generated trajectory mechanism: relying on early-stage exploration to identify pseudo-expert strategies presents a challenge. If the initial exploration is insufficient, particularly in environments with sparse rewards, and a good strategy is not found, then such pseudo-expert strategies will be unable to provide adequate strategic guidance.
AW2. Thank you for raising this important question. We agree that learning from self-generated experience depends on discovering informative trajectories through exploration. This motivates BMARL’s mechanism for balancing exploration and exploitation. When desirable trajectories have not yet been found, BMARL continues exploring through entropy-regularized actor updates. A further challenge in cooperative MARL is that agents may discover a successful strategy during training but subsequently fail to retain it. BMARL uses a separately learned pseudo-expert to preserve and reuse such rare desirable experience, while maintaining exploration on undesirable samples.
As discussed in AQ1, unsuccessful experience also contributes to learning. Episodes that do not satisfy the success criterion still support critic learning when sampled from the replay buffer. Furthermore, when the desirable buffer is constructed using return-based priority, as described in Section 3.2, unsuccessful trajectories with relatively high returns can also contribute to training the pseudo-expert. When a sampled trajectory is labeled desirable, BMARL uses that policy to guide the actor update. Thus, unsuccessful experience can influence both the critic and the pseudo-expert, while desirability gating determines when the actor receives this guidance.
BMARL combines reward-driven actor–critic learning with conditional pseudo-expert guidance. This allows it to exploit rare successful experience while continuing to learn from unsuccessful episodes and explore for better strategies. We hope this clarifies the contribution and strengths of BMARL.
Response #3 to Reviewer A6iw
W3. I have some concerns about the reliability of the experimental results. On the 6h_vs_8z map in Figure 4, the baselines MAPPO and RODE perform very badly, with win rates below 30%. However, in the original MAPPO paper, both MAPPO and RODE can achieve win rates of at least 60% within 3M training steps. There is a large gap between the authors' reported results and the existing results, especially considering that the authors use the default settings provided in the original papers for the baselines.
AW3. We agree that the gap on 6h_vs_8z warrants a closer comparison, as it is one of the super-hard tasks in SMAC and trained models often exhibit high variability across random seeds. For our evaluation, we used the original implementations and the settings specified in the code or manuscripts, where available.
For MAPPO, the original paper reports several results on this map depending on the critic input. To clarify, we used the environment-provided state alone (EP) for all experiments, as this is the default setting in SMAC and SMACv2. Around 3M environment steps, its supplementary Figure 8 shows a win rate of approximately 20% with EP, compared with roughly 55–60% for the FP and AS variants. There is still a gap between their reported performance and ours. We believe this difference is attributable to the high variability across random seeds discussed in Section 5.2 of the original MAPPO paper.
To determine whether this performance gap had also been observed in other studies, we searched the literature. Other studies have also reported lower MAPPO performance on this map, with win rates similar to those observed in our experiments.
-
E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning [1], reporting a win rate near 0% for the standard MLP-based MAPPO baseline within the reported 2M training steps. (Figure 7, p. 7: https://arxiv.org/pdf/2308.11842#page=9.)
-
JaxMARL: Multi-Agent RL Environments and Algorithms in JAX [2], reporting a win rate near 0% at 3M steps for MAPPO in the original SMAC environment. (Figure 19, p. 24: https://arxiv.org/pdf/2311.10090#page=24.)
For RODE, our original runs used the original implementation and the default settings presented in the paper. We also repeated the experiments in a different computing environment, but obtained worse performance, with a win rate below 5% at 3M training steps.
To determine whether this performance gap had also been observed in other studies, we searched the literature. Other studies have also reported lower RODE performance on this map, with win rates similar to those observed in our experiments. Examples include the following.
-
AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning [3], reporting a win rate near 0% at 3M steps. (Figure 2, p. 6: https://arxiv.org/pdf/2412.15700v2#page=6.)
-
Grounded Answers for Multi-agent Decision-making Problem through Generative World Model [4], reporting a win rate of approximately 10% at 3M steps. (Figure 8(b), p. 24: https://arxiv.org/pdf/2410.02664v1#page=24.)
-
Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning [5], reporting a win rate of approximately 2–3% at 3M steps. (Figure 2, p. 9: https://arxiv.org/pdf/2010.04740v2#page=9.)
[1] Dingyang Chen and Qi Zhang. “E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning.” Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, 7640–7666, 2024.
[2] Alexander Rutherford, Benjamin Ellis, Matteo Gallici, et al. “JaxMARL: Multi-Agent RL Environments and Algorithms in JAX.” Advances in Neural Information Processing Systems, 37, 50925–50951, 2024.
[3] Guangchong Zhou, Zeren Zhang, and Guoliang Fan. “AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, 39(21), 22919–22927, 2025.
[4] Zeyang Liu, Xinrui Yang, Shiguang Sun, Long Qian, Lipeng Wan, Xingyu Chen, and Xuguang Lan. “Grounded Answers for Multi-agent Decision-making Problem through Generative World Model.” Advances in Neural Information Processing Systems, 37, 46622–46652, 2024.
[5] Navid Naderializadeh, Fan H. Hung, Sean Soleyman, and Deepak Khosla. “Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning.” arXiv preprint arXiv:2010.04740, 2020.
Response #1 to Reviewer A6iw
W1. The novelty of the idea is a bit poor. Learning from self-generated trajectories is not a new approach. There are some existing studies that have explored this direction. I have provided two related studies published in 2021 and 2022, and I believe there may be more studies from the past four years that have explored this direction. However, the authors do not fully introduce such related studies or compare their approach with existing work, weakening the novelty of this paper.
[1] Zhang Y, Yang Q, Li D, et al. A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility[J]. Applied Energy, 2022, 323: 119543.
[2] Luo S, Kasaei H, Schomaker L. Self-imitation learning by planning[C] 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 4823-4829.
AW1. We thank the reviewer for highlighting these studies. We do not claim to be the first to propose learning from self-generated experience through imitation. Our contribution concerns a specific question in cooperative MARL: when should a team use its past behavior to guide its decentralized policies, and when should those policies continue exploring? Imitating the highest-return episodes collected early in training can reinforce a strategy that is only locally successful. BMARL addresses this through a regularized actor objective. It trains a separate pseudo-expert on selected experience and uses the team’s episode outcome to decide when that prior should guide each actor. The actors continue to learn from the team critic in both cases.
Appendix A.3 already discusses successful-trajectory reuse, and our experiments include representative state-of-the-art MARL baselines that exploit episodic memory and informative experience. According to the reviewer's suggestion, we reviewed the mentioned literature and we expanded Appendix A.3 to clarify these methodological distinctions and the scope of prior evaluations. To clarify, Luo et al. [1] use planner-generated demonstrations and Q-filtered imitation, while Zhang et al. [2] use self-generated expert knowledge for electricity pricing. Extending these single-agent formulations to cooperative MARL requires specifying how team outcomes guide decentralized actors under partial observability and changing teammate policies. Deng et al. [3] address cooperative MARL through planned rollout prefixes and template-based TD-target aggregation. Their published evaluation does not report a dedicated sparse-reward study, and the authors identify settings requiring substantial early exploration as future work. On the other hand, BMARL controls guidance through the actor’s regularization prior, and our sparse-reward experiments evaluate this mechanism when successful trajectories are rare.
We also added a direct test of whether cloning the same elite experience would suffice. Under a matched training budget, the baseline applies an auxiliary BC loss directly to each actor using the same number of elite samples per update and the same advantage weights as BMARL. We selected its BC coefficient through the study in Appendix D.3 of the revised manuscript. As shown in Figure 10 of Section 4.4 (Ablation study), direct BC did not consistently match BMARL. On
3m_sparse, it failed to learn a winning strategy, as did FOP and BMARL without policy regularization. This comparison shows that how the actors use elite experience matters even when the amount of elite data is matched.AW2 also presents the contribution points and strengths of BMARL.
[1] Sha Luo, Hamidreza Kasaei, and Lambert Schomaker. “Self-Imitation Learning by Planning.” 2021 IEEE International Conference on Robotics and Automation (ICRA), 4823–4829, 2021.
[2] Yang Zhang, Qingyu Yang, Donghe Li, and Dou An. “A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility.” Applied Energy, 323, 119543, 2022.
[3] Yue Deng, Zirui Wang, and Yin Zhang. “Improving Multi-agent Reinforcement Learning with Stable Prefix Policy.” Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), 49–57, 2024.