acceptodds
Under review as a conference paper at ICLR 2027

Balanced Multi-Agent Reinforcement Learning via Desirability-Gated Pseudo-Expert Regularization

Abstract

The exploitation–exploration trade-off remains a central challenge in complex cooperative multi-agent tasks, where agents must discover coordinated joint behaviors from local observations. Entropy- or divergence-regularized actor-critic methods partly alleviate this challenge, but they often provide only weak guidance for learning complex team behaviors. Exploiting expert behaviors via behavior cloning can reproduce such behaviors, yet it usually assumes access to expert demonstrations and is therefore difficult to apply in online settings. To address this issue, we propose Balanced Multi-agent Reinforcement Learning (BMARL), a desirability-gated policy-regularization framework that balances exploration and exploitation using a pseudo-expert learned online from desirable trajectories. BMARL introduces a hybrid divergence-regularized objective: on desirable trajectories, the learning policy is regularized toward the pseudo-expert prior, while on undesirable trajectories, the objective reverts to maximum-entropy exploration. The framework is compatible with standard MARL backbones and mixer choices, and we provide theoretical analyses of policy evaluation and improvement. The proposed method is evaluated on various MARL tasks, including dense- and sparse-reward settings, and achieves further performance improvements over state-of-the-art methods.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, entropy-based exploration, policy regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A6iwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the exploitation–exploration trade-off problem. It combines entropy-based methods and divergence-regularized methods and introduces a pseudo-expert policy trained on well-performing trajectories. The proposed method automatically switches between imitating the pseudo-expert and exploring actions. Finally, the authors test the algorithm's performance on SMAC, SMACv2, and GRF environments.

Critical strengths
  1. The paper's writing is easy to follow, and its structure is well organized.

  2. The idea is easy to understand, and the implementation is easy to follow.

  3. The visualization experiments are helpful for understanding the underlying mechanism of the proposed algorithm.

Critical weaknesses
  1. The novelty of the idea is a bit poor. Learning from self-generated trajectories is not a new approach. There are some existing studies that have explored this direction. I have provided two related studies published in 2021 and 2022, and I believe there may be more studies from the past four years that have explored this direction. However, the authors do not fully introduce such related studies or compare their approach with existing work, weakening the novelty of this paper.

[1] Zhang Y, Yang Q, Li D, et al. A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility[J]. Applied Energy, 2022, 323: 119543. [2] Luo S, Kasaei H, Schomaker L. Self-imitation learning by planning[C]//2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 4823-4829.

  1. There is a potentially important issue behind the self-generated trajectory mechanism: relying on early-stage exploration to identify pseudo-expert strategies presents a challenge. If the initial exploration is insufficient, particularly in environments with sparse rewards, and a good strategy is not found, then such pseudo-expert strategies will be unable to provide adequate strategic guidance.

  2. I have some concerns about the reliability of the experimental results. On the 6h_vs_8z map in Figure 4, the baselines MAPPO and RODE perform very badly, with win rates below 30%. However, in the original MAPPO paper, both MAPPO and RODE can achieve win rates of at least 60% within 3M training steps. There is a large gap between the authors' reported results and the existing results, especially considering that the authors use the default settings provided in the original papers for the baselines.

  3. In Section 4.2, the test environments (3m and 2s3z) under sparse-reward settings are too easy. As I mentioned earlier, would this algorithm fail in a slightly more complex environment with sparse rewards?

Questions for authors

Why is R_{thr} set to the task-dependent maximum return R_{max} in SMAC and GRF environments? On some difficult maps, especially considering the sparse-reward setting, few trajectories satisfy this criterion.

6 comments
AuthorsOpenReview ↗

Response #1 to Reviewer A6iw

W1. The novelty of the idea is a bit poor. Learning from self-generated trajectories is not a new approach. There are some existing studies that have explored this direction. I have provided two related studies published in 2021 and 2022, and I believe there may be more studies from the past four years that have explored this direction. However, the authors do not fully introduce such related studies or compare their approach with existing work, weakening the novelty of this paper.

[1] Zhang Y, Yang Q, Li D, et al. A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility[J]. Applied Energy, 2022, 323: 119543.

[2] Luo S, Kasaei H, Schomaker L. Self-imitation learning by planning[C] 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 4823-4829.

AW1. We thank the reviewer for highlighting these studies. We do not claim to be the first to propose learning from self-generated experience through imitation. Our contribution concerns a specific question in cooperative MARL: when should a team use its past behavior to guide its decentralized policies, and when should those policies continue exploring? Imitating the highest-return episodes collected early in training can reinforce a strategy that is only locally successful. BMARL addresses this through a regularized actor objective. It trains a separate pseudo-expert on selected experience and uses the team’s episode outcome to decide when that prior should guide each actor. The actors continue to learn from the team critic in both cases.

Appendix A.3 already discusses successful-trajectory reuse, and our experiments include representative state-of-the-art MARL baselines that exploit episodic memory and informative experience. According to the reviewer's suggestion, we reviewed the mentioned literature and we expanded Appendix A.3 to clarify these methodological distinctions and the scope of prior evaluations. To clarify, Luo et al. [1] use planner-generated demonstrations and Q-filtered imitation, while Zhang et al. [2] use self-generated expert knowledge for electricity pricing. Extending these single-agent formulations to cooperative MARL requires specifying how team outcomes guide decentralized actors under partial observability and changing teammate policies. Deng et al. [3] address cooperative MARL through planned rollout prefixes and template-based TD-target aggregation. Their published evaluation does not report a dedicated sparse-reward study, and the authors identify settings requiring substantial early exploration as future work. On the other hand, BMARL controls guidance through the actor’s regularization prior, and our sparse-reward experiments evaluate this mechanism when successful trajectories are rare.

We also added a direct test of whether cloning the same elite experience would suffice. Under a matched training budget, the baseline applies an auxiliary BC loss directly to each actor using the same number of elite samples per update and the same advantage weights as BMARL. We selected its BC coefficient through the study in Appendix D.3 of the revised manuscript. As shown in Figure 10 of Section 4.4 (Ablation study), direct BC did not consistently match BMARL. On 3m_sparse, it failed to learn a winning strategy, as did FOP and BMARL without policy regularization. This comparison shows that how the actors use elite experience matters even when the amount of elite data is matched.

AW2 also presents the contribution points and strengths of BMARL.

[1] Sha Luo, Hamidreza Kasaei, and Lambert Schomaker. “Self-Imitation Learning by Planning.” 2021 IEEE International Conference on Robotics and Automation (ICRA), 4823–4829, 2021.

[2] Yang Zhang, Qingyu Yang, Donghe Li, and Dou An. “A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility.” Applied Energy, 323, 119543, 2022.

[3] Yue Deng, Zirui Wang, and Yin Zhang. “Improving Multi-agent Reinforcement Learning with Stable Prefix Policy.” Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), 49–57, 2024.

AuthorsOpenReview ↗

Response #2 to Reviewer A6iw

W2. There is a potentially important issue behind the self-generated trajectory mechanism: relying on early-stage exploration to identify pseudo-expert strategies presents a challenge. If the initial exploration is insufficient, particularly in environments with sparse rewards, and a good strategy is not found, then such pseudo-expert strategies will be unable to provide adequate strategic guidance.

AW2. Thank you for raising this important question. We agree that learning from self-generated experience depends on discovering informative trajectories through exploration. This motivates BMARL’s mechanism for balancing exploration and exploitation. When desirable trajectories have not yet been found, BMARL continues exploring through entropy-regularized actor updates. A further challenge in cooperative MARL is that agents may discover a successful strategy during training but subsequently fail to retain it. BMARL uses a separately learned pseudo-expert to preserve and reuse such rare desirable experience, while maintaining exploration on undesirable samples.

As discussed in AQ1, unsuccessful experience also contributes to learning. Episodes that do not satisfy the success criterion still support critic learning when sampled from the replay buffer. Furthermore, when the desirable buffer is constructed using return-based priority, as described in Section 3.2, unsuccessful trajectories with relatively high returns can also contribute to training the pseudo-expert. When a sampled trajectory is labeled desirable, BMARL uses that policy to guide the actor update. Thus, unsuccessful experience can influence both the critic and the pseudo-expert, while desirability gating determines when the actor receives this guidance.

BMARL combines reward-driven actor–critic learning with conditional pseudo-expert guidance. This allows it to exploit rare successful experience while continuing to learn from unsuccessful episodes and explore for better strategies. We hope this clarifies the contribution and strengths of BMARL.

AuthorsOpenReview ↗

Response #3 to Reviewer A6iw

W3. I have some concerns about the reliability of the experimental results. On the 6h_vs_8z map in Figure 4, the baselines MAPPO and RODE perform very badly, with win rates below 30%. However, in the original MAPPO paper, both MAPPO and RODE can achieve win rates of at least 60% within 3M training steps. There is a large gap between the authors' reported results and the existing results, especially considering that the authors use the default settings provided in the original papers for the baselines.

AW3. We agree that the gap on 6h_vs_8z warrants a closer comparison, as it is one of the super-hard tasks in SMAC and trained models often exhibit high variability across random seeds. For our evaluation, we used the original implementations and the settings specified in the code or manuscripts, where available.

For MAPPO, the original paper reports several results on this map depending on the critic input. To clarify, we used the environment-provided state alone (EP) for all experiments, as this is the default setting in SMAC and SMACv2. Around 3M environment steps, its supplementary Figure 8 shows a win rate of approximately 20% with EP, compared with roughly 55–60% for the FP and AS variants. There is still a gap between their reported performance and ours. We believe this difference is attributable to the high variability across random seeds discussed in Section 5.2 of the original MAPPO paper.

To determine whether this performance gap had also been observed in other studies, we searched the literature. Other studies have also reported lower MAPPO performance on this map, with win rates similar to those observed in our experiments.

  • E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning [1], reporting a win rate near 0% for the standard MLP-based MAPPO baseline within the reported 2M training steps. (Figure 7, p. 7: https://arxiv.org/pdf/2308.11842#page=9.)

  • JaxMARL: Multi-Agent RL Environments and Algorithms in JAX [2], reporting a win rate near 0% at 3M steps for MAPPO in the original SMAC environment. (Figure 19, p. 24: https://arxiv.org/pdf/2311.10090#page=24.)

For RODE, our original runs used the original implementation and the default settings presented in the paper. We also repeated the experiments in a different computing environment, but obtained worse performance, with a win rate below 5% at 3M training steps.

To determine whether this performance gap had also been observed in other studies, we searched the literature. Other studies have also reported lower RODE performance on this map, with win rates similar to those observed in our experiments. Examples include the following.

  • AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning [3], reporting a win rate near 0% at 3M steps. (Figure 2, p. 6: https://arxiv.org/pdf/2412.15700v2#page=6.)

  • Grounded Answers for Multi-agent Decision-making Problem through Generative World Model [4], reporting a win rate of approximately 10% at 3M steps. (Figure 8(b), p. 24: https://arxiv.org/pdf/2410.02664v1#page=24.)

  • Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning [5], reporting a win rate of approximately 2–3% at 3M steps. (Figure 2, p. 9: https://arxiv.org/pdf/2010.04740v2#page=9.)

[1] Dingyang Chen and Qi Zhang. “E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning.” Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, 7640–7666, 2024.

[2] Alexander Rutherford, Benjamin Ellis, Matteo Gallici, et al. “JaxMARL: Multi-Agent RL Environments and Algorithms in JAX.” Advances in Neural Information Processing Systems, 37, 50925–50951, 2024.

[3] Guangchong Zhou, Zeren Zhang, and Guoliang Fan. “AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, 39(21), 22919–22927, 2025.

[4] Zeyang Liu, Xinrui Yang, Shiguang Sun, Long Qian, Lipeng Wan, Xingyu Chen, and Xuguang Lan. “Grounded Answers for Multi-agent Decision-making Problem through Generative World Model.” Advances in Neural Information Processing Systems, 37, 46622–46652, 2024.

[5] Navid Naderializadeh, Fan H. Hung, Sean Soleyman, and Deepak Khosla. “Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning.” arXiv preprint arXiv:2010.04740, 2020.

Reviewer_dC6sOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Balanced Multi-Agent Reinforcement Learning (BMARL), which uses successful self-generated trajectories to guide cooperative policy learning without external demonstrations. An advantage-weighted behavior cloning objective trains a pseudo-expert from selected trajectories. A desirability gate then switches between pseudo-expert regularization and entropy-based exploration. The authors derive policy evaluation and improvement results for a fixed-prior regularized-control problem. Experiments on SMAC, SMACv2, GRF, SMAX, Overcooked, and MABrax report improvements over several actor-critic and value-based baselines, particularly in challenging sparse-reward settings.

Critical strengths
  1. The desirability-gated prior provides a practical way to exploit successful self-generated experience while retaining entropy-based exploration on other trajectories. The method does not require external demonstrations and can be incorporated into existing actor-critic training pipelines.

  2. The empirical evaluation covers several cooperative MARL benchmarks, including challenging sparse-reward tasks. The component ablations and policy-distribution diagnostics provide useful evidence about pseudo-expert guidance.

Critical weaknesses
  1. Proposition 3.1 and the policy improvement results assume a fixed prior and an exact regularized policy update. In BMARL, desirability is assigned retrospectively from completed trajectories, while the deployed actor does not condition on this label. The same local history may therefore induce different regularization targets.

  2. Eq. (2) uses a sum of agent-wise regularization terms, whereas the critic target in Eq. (19) uses their average. This changes the effective regularization strength. Eq. (18) optimizes individual critics, while Eq. (13) permits an arbitrary mixer.

  3. BMARL combines successful-trajectory selection, advantage-weighted behavior cloning, KL regularization, and pseudo-expert smoothing. The No-PE and No-DG ablations do not establish whether the separate pseudo-expert and binary gate outperform direct self-imitation using the same selected experience. The Soft-DG comparison requires clarification.

Questions for authors
  1. How does the fixed-prior policy improvement analysis apply when desirability labels are determined after episode completion and the deployed actor does not condition on them?

  2. Eq. (7) and Appendix F.3 specify a uniform fallback prior before pseudo-expert readiness, whereas Appendix F.2 describes a target-policy fallback. Which rule is implemented in the experiments?

  3. Could the authors compare BMARL against a capacity-matched self-imitation baseline that uses the same desirable buffer and advantage weights but applies imitation directly to the actor?

  4. Are episodic returns normalized before computing Soft-DG in Eq. (38)? What fraction of training samples receive intermediate gate values? Additionally, how are the reported win-rate means aggregated across seeds and evaluation checkpoints?

7 comments
AuthorsOpenReview ↗

Response #1 to Reviewer dC6s

W1. Proposition 3.1 and the policy improvement results assume a fixed prior and an exact regularized policy update. In BMARL, desirability is assigned retrospectively from completed trajectories, while the deployed actor does not condition on this label. The same local history may therefore induce different regularization targets.

Q1. How does the fixed-prior policy improvement analysis apply when desirability labels are determined after episode completion and the deployed actor does not condition on them?

AW1 and AQ1. Thank you for the important question. Under CTDE, the outcome of each completed episode is available during training. We assign one desirability label to the episode and use it at every sampled timestep. This label uniquely determines the prior for each training sample. Thus, even when two episodes share a local history, their outcomes can intentionally provide different guidance: a desirable episode guides the actor toward successful coordination through the pseudo-expert, while an undesirable episode uses the uniform prior to encourage exploration. At execution, the actor uses only its local history. Proposition 3.1 applies to the prior selected for each training sample; Appendix B analyzes exact policy improvement under a fixed prior.

W2-1. Eq.(2) uses a sum of agent-wise regularization terms, whereas the critic target in Eq. (19) uses their average. This changes the effective regularization strength.

AW2-1. Following the public FOP implementation (https://github.com/PKU-RL/FOP-DMAC-MACPF/blob/main/src/learners/fop_learner.py), we average the per-agent actor objectives in Eq. (18) so that the loss scale does not grow simply with the number of agents. For fixed nn, this normalization leaves the actor-loss minimizer unchanged. In Eq. (19), we likewise average the per-agent log-policy and log-prior terms while retaining the mixed critic on the team-return scale. Since the joint policy and prior factorize across agents, this averaged regularizer is exactly equivalent to using coefficients α/n\alpha/n and β/n\beta/n on the joint log probabilities in Eq. (2), where α\alpha and β\beta are the per-agent coefficients used in training. Put simply, using α\alpha and β\beta in the average in Eq. (19) has the same effect as using α/n\alpha/n and β/n\beta/n in the sum in Eq. (2). We also add some explanation for this in Section 3.4 of the revised manuscript.

W2-2. Eq. (18) optimizes individual critics, while Eq. (13) permits an arbitrary mixer.

AW2-2. Equation (19) estimates the regularized value of the current joint policy by evaluating a sampled joint action with the centralized mixer. This evaluation applies to any mixer that returns a joint action value and does not require its Boltzmann target to factorize. For the decentralized actor update, we extend the individual KL projection in Eq. (8) of FOP (Zhang et al., 2021) by incorporating the desirability-gated prior into each agent’s local target. Applying Proposition 3.1 to this target, with the local utility qiq_i learned through team TD, yields the per-agent objective in Eq. (18).

AuthorsOpenReview ↗

## Response #2 to Reviewer dC6s

W3. BMARL combines successful-trajectory selection, advantage-weighted behavior cloning, KL regularization, and pseudo-expert smoothing. The No-PE and No-DG ablations do not establish whether the separate pseudo-expert and binary gate outperform direct self-imitation using the same selected experience. The Soft-DG comparison requires clarification.

Q3. Could the authors compare BMARL against a capacity-matched self-imitation baseline that uses the same desirable buffer and advantage weights but applies imitation directly to the actor?

AW3 and AQ3. Thank you for the constructive feedback. Although the existing PE/gating ablations support our design, we agree with the reviewer that a direct comparison with an auxiliary BC/self-imitation loss applied to the actor using the same elite data was missing. Following the reviewer's suggestion, we conducted an additional ablation study under a matched training budget, using the same number of elite samples per training step, as presented in Figure 10 of Section 4.4 in the revised manuscript. We also conducted a hyperparameter study to select an appropriate coefficient for the auxiliary loss, with the results reported in Appendix D.3. The results show that directly applying BC to the actor does not consistently match the performance of the proposed method. In particular, the direct-BC baseline failed to learn a winning strategy on the 3m_sparse task, similar to FOP and BMARL with no policy regularization (PR). These results suggest that introducing BC as an auxiliary loss may interfere with reward-driven policy optimization, whereas BMARL incorporates imitation-based guidance through a separate pseudo-expert prior.

Q2. Eq. (7) and Appendix F.3 specify a uniform fallback prior before pseudo-expert readiness, whereas Appendix F.2 describes a target-policy fallback. Which rule is implemented in the experiments?

AQ2. Our original presentation emphasized how the uniform prior encourages exploration on undesirable trajectories and how the pseudo-expert guides learning from desirable ones. For brevity, we described the target-policy fallback in the implementation details, but Eq. (7) and Appendix F.3 stated the pre-readiness rule too broadly. The fallback applies only when a desirable trajectory is sampled before the pseudo-expert is ready. With a return-based buffer D⋆\mathcal{D}^{\star} and success-based desirability labels, the pseudo-expert can become ready before the first success, in which case the target-policy fallback is never needed. We have clarified when this fallback applies in Eq. (7) and Appendices F.2-F.3.

AuthorsOpenReview ↗

Response #3 to Reviewer dC6s

Q4. Are episodic returns normalized before computing Soft-DG in Eq. (38)? What fraction of training samples receive intermediate gate values? Additionally, how are the reported win-rate means aggregated across seeds and evaluation checkpoints?

AQ4. Thank you for the thorough review. We first clarify that R(T)R(\mathcal{T}) is a typo and is not the normalized return value, but episodic returns are converted to running empirical percentile ranks in [0,1][0,1] compared to samples in the replay buffer D\mathcal{D}. Hence, to avoid the confusion, we explicitly define pt(T)=percentilet(R(T))p_t(\mathcal{T})=\textrm{percentile}_t(R (\mathcal{T})) with R(T)R(\mathcal{T}) as the return value following the convention. Then, the soft-gating value becomes

g(T)=clip⁡!(pt(T)−qlowqhigh−qlow,0,1)g(\mathcal{T})=\operatorname{clip}\\!\left(\frac{p_t(\mathcal{T})-q_{\mathrm{low}}}{q_{\mathrm{high}}-q_{\mathrm{low}}},0,1\right).

To quantify how frequently intermediate gate values are assigned, we conducted additional experiments on the same tasks. The following table reports the average fraction of training samples assigned intermediate gate values, along with the average fraction of desirable samples.

Table 1. Soft-DG statistics at different training steps (single seed case). All values are percentages. The binary rate refers to the fraction of desirable samples in a minibatch.

tenvt_{\mathrm{env}}Intermediate fraction (2s3z (sparse))Binary rate (2s3z (sparse))Win rate (2s3z (sparse))Intermediate fraction (6h_vs_8z)Binary rate (6h_vs_8z)Win rate (6h_vs_8z)
1M2.9480.0000.00097.9710.1546.250
2M81.19195.512100.00097.4891.5403.125
3M58.18698.852100.00099.2842.77212.500

Table 1 illustrates two different training patterns under Soft-DG. In the reported single-seed run on 2s3z (sparse) , the minibatch binary success rate increases from 0% at 1M steps to around 98.9% at 3M steps, while the evaluation win rate reaches 100% at 2M steps and remains at that level. The intermediate fraction decreases to 58.2% at 3M steps as successful experience becomes more prevalent in the training minibatches.

On 6h_vs_8z, however, the intermediate fraction remains above 97%, while the minibatch binary success rate stays below 3% and the win rate reaches about 12.5% at 3M steps. Thus, return-based soft gating assigns intermediate guidance predominantly to unsuccessful training samples, and a large intermediate fraction does not translate into a high win rate. This pattern is consistent with the risk that pseudo-expert guidance on high-return but unsuccessful trajectories can reinforce suboptimal coordination and lead to premature convergence.

reported win-rate

As stated in Section 4, unless otherwise specified, we evaluate the current policy using 32 test episodes at each evaluation step to measure the win rate or average return. We then average the results across four independent runs with different seeds. Thus, a total of 32×4=12832 \times 4 = 128 test episodes are used to compute the mean win rate at each evaluation step. This evaluation protocol is widely adopted in the MARL community. However, unlike conventional practice, where random seeds may vary across methods, we use the same set of four consecutive random seeds (e.g., 1,2,3,4\\{1,2,3,4\\}) to ensure fair comparisons.

We hope these responses address your concerns and questions. Please let us know if any further clarification is needed.

Reviewer_6sKoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Balanced Multi-Agent Reinforcement Learning (BMARL), a framework that uses a desirability-gated pseudo-expert prior to balance exploration and exploitation in cooperative MARL. A pseudo-expert is trained through advantage-weighted behavior cloning on selected online trajectories. The learning policy is regularized toward this prior on desirable trajectories, while undesirable trajectories use a uniform prior to encourage entropy-based exploration. The paper provides fixed-prior policy evaluation and improvement analyses and evaluates the method on several cooperative benchmarks, including SMAC, SMACv2, GRF, and additional environments in the appendix.

Critical strengths
  1. The paper addresses a useful problem: successful coordination can be difficult to discover, and making better use of these occasional successes is a sensible direction for improving training efficiency.

  2. The method learns its teacher from online experience rather than requiring external demonstrations. This makes the approach relevant to tasks where expert multi-agent behavior is difficult to obtain.

  3. The gate provides a clear motivation for treating successful and unsuccessful experience differently. It also gives the paper a concrete hypothesis to investigate through the component ablations.

  4. The evaluation is extensive, which covers several environments and includes both dense- and sparse-reward settings.

Critical weaknesses

The contribution relative to direct self-imitation is not fully isolated.

Learning from one's own good trajectories is established, for example in [1]. The existing ablations support roles for the pseudo-expert and gate, but do not show whether the same elite data and advantages used in a direct actor BC/self-imitation auxiliary loss would recover most of the gain. A matched-backbone comparison would better establish the value of distilling a separate gated prior.

Several loss and implementation definitions are inconsistent.

Equation (2) sums joint log-policy/prior terms, whereas Equation (19) averages their per-agent contributions with a factor of 1/n. The normalization of the critic and actor utilities needs to explain this difference. In addition, Equation (7) and Appendix F.3 specify a uniform fallback before pseudo-expert readiness, while Appendix F.2 describes a target-policy fallback. These choices affect the actual algorithm and should be reconciled. The claimed reduction to ordinary maximum-entropy control also needs care when the number of available actions varies by state: the uniform-prior term is then state-dependent, even though it is action-independent locally.

minor: The font size of the text in Fig. 11 and 12 is too small, which affects readability.

[1] Self-Imitation Learning, ICML 2018.

Questions for authors
  1. Would a direct weighted-BC auxiliary loss achieve comparable results under the same data and training budget? A comparison on a small number of representative tasks would be particularly helpful.
5 comments
AuthorsOpenReview ↗

Response #1 to Reviewer 6sKo

W1. The contribution relative to direct self-imitation is not fully isolated. Learning from one's own good trajectories is established, for example in [1]. The existing ablations support roles for the pseudo-expert and gate, but do not show whether the same elite data and advantages used in a direct actor BC/self-imitation auxiliary loss would recover most of the gain. A matched-backbone comparison would better establish the value of distilling a separate gated prior.

[1] Self-Imitation Learning, ICML 2018.

AW1. Thank you for the constructive feedback. Although the existing PE/gating ablations support our design, we agree with the reviewer that a direct comparison with an auxiliary BC/self-imitation loss applied to the actor using the same elite data was missing. Following the reviewer's suggestion, we conducted an additional ablation study under a matched training budget, using the same number of elite samples per training step, as presented in Figure 10 of Section 4.4 in the revised manuscript. We also conducted a hyperparameter study to select an appropriate coefficient for the auxiliary loss, with the results reported in Appendix D.3. The results show that directly applying BC to the actor does not consistently match the performance of the proposed method. In particular, the direct-BC baseline failed to learn a winning strategy on the 3m_sparse task, similar to FOP and BMARL with no policy regularization (PR).

AuthorsOpenReview ↗

Response #2 to Reviewer 6sKo

W2. Several loss and implementation definitions are inconsistent.

W2-1. Equation (2) sums joint log-policy/prior terms, whereas Equation (19) averages their per-agent contributions with a factor of 1/n1/n. The normalization of the critic and actor utilities needs to explain this difference.

AW2-1. Following the public FOP implementation (https://github.com/PKU-RL/FOP-DMAC-MACPF/blob/main/src/learners/fop_learner.py), we average the per-agent actor objectives in Eq. (18) so that the loss scale does not grow simply with the number of agents. For fixed nn, this normalization leaves the actor-loss minimizer unchanged. In Eq. (19), we likewise average the per-agent log-policy and log-prior terms while retaining the mixed critic on the team-return scale. Since the joint policy and prior factorize across agents, this averaged regularizer is exactly equivalent to using coefficients α/n\alpha/n and β/n\beta/n on the joint log probabilities in Eq. (2), where α\alpha and β\beta are the per-agent coefficients used in training. Put simply, using α\alpha and β\beta in the average in Eq. (19) has the same effect as using α/n\alpha/n and β/n\beta/n in the sum in Eq. (2). We also add some explanation for this in Section 3.4 of the revised manuscript.

W2-2. In addition, Equation (7) and Appendix F.3 specify a uniform fallback before pseudo-expert readiness, while Appendix F.2 describes a target-policy fallback. These choices affect the actual algorithm and should be reconciled.

AW2-2. Our original presentation emphasized how the uniform prior encourages exploration on undesirable trajectories and how the pseudo-expert guides learning from desirable ones. For brevity, we described the target-policy fallback in the implementation details, but Eq. (7) and Appendix F.2 stated the pre-readiness rule too broadly. The fallback applies only when a desirable trajectory is sampled before the pseudo-expert is ready. With a return-based buffer D⋆\mathcal{D}^{\star} and success-based desirability labels, the pseudo-expert can become ready before the first success, in which case the target-policy fallback is never needed. We have clarified when this fallback applies in Eq. (7) and Appendices F.2-F.3.

W2-3. The claimed reduction to ordinary maximum-entropy control also needs care when the number of available actions varies by state: the uniform-prior term is then state-dependent, even though it is action-independent locally.

AW2-3. Thank you for the constructive review. With a uniform prior throughout and a fixed horizon, Eq. (2) is equivalent to the standard maximum-entropy objective up to a policy-independent constant when every state has the same number of feasible joint actions. As the reviewer pointed out, when that number varies across states, the cumulative prior term can depend on the trajectory, so the two objectives can differ. During the actor update, however, the uniform-prior term is constant across feasible actions at each sampled history and contributes zero to the policy gradient. Most importantly, on undesirable trajectories, BMARL therefore recovers the standard maximum-entropy actor gradient for the same local critic, encouraging exploration. We have revised Section 3.3 to limit our equivalence claim to this actor update.

AuthorsOpenReview ↗

## Response #3 to Reviewer 6sKo

W3. minor: The font size of the text in Fig. 11 and 12 is too small, which affects readability.

AW3. Thank you for your thorough review. We enlarged the labels, legends, and annotations of Figs. 11–12 in the revised manuscript.

Q1. Would a direct weighted-BC auxiliary loss achieve comparable results under the same data and training budget? A comparison on a small number of representative tasks would be particularly helpful.

AQ1. Thank you for your constructive review. As noted in AW1, we have included additional ablation experiments in Section 4.4 of the revised manuscript, with implementation details provided in Appendix D.3, following the reviewer's suggestion. The results show that directly applying BC to the actor does not consistently match the performance of the proposed method. In particular, the direct-BC baseline failed to learn a winning strategy on the 3m_sparse task, similar to FOP and BMARL with no policy regularization (PR). These results suggest that introducing BC as an auxiliary loss may interfere with reward-driven policy optimization, whereas BMARL incorporates imitation-based guidance through a separate pseudo-expert prior.

We hope our responses have adequately addressed your concerns. We would be happy to provide further clarification if needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Dear Reviewers,

Thank you for your constructive and thorough feedback. We have revised the manuscript and conducted additional experiments to respond to reviewers' questions and comments. BMARL's contribution is to control how self-generated experience guides cooperative policy learning: a separate pseudo-expert learns from selected team experience, and episode-level desirability determines when it guides decentralized actors. The actors continue to learn from the team critic while retaining entropy-based updates on undesirable samples. This automatic balancing between exploration and exploitation enables BMARL to achieve strong performance in both dense- and sparse-reward settings.

Summary of revisions and additional results

  • Direct self-imitation comparison. We added an auxiliary weighted-BC baseline under a matched training budget, with the same number of elite samples per update and the same advantage weights. Direct BC does not consistently match BMARL's performance (Section 4.4, Figure 10). The implementation and BC-coefficient study appear in the new Appendix D.3.

  • Additional sparse-reward evaluation. We added 8m_sparse to test BMARL on a more complex sparse-reward task (Section 4.2, Figure 6). Our response also reports final win rates and learning efficiency across seeds.

  • Objective and implementation clarifications. We explain the agent-wise normalization and its relationship to the joint regularization coefficients (Section 3.4). Before pseudo-expert readiness, desirable samples use the target-policy prior and undesirable samples use the uniform prior. This rule is now stated consistently in Eq. (7) and Appendices F.2–F.3. We also restrict the maximum-entropy equivalence claim to the actor gradient for the same local critic (Section 3.3).

  • Diagnostics and evaluation details. We clarify in Appendix K that Soft-DG uses empirical percentile ranks of episodic returns. Our responses provide additional statistics on 2s3z (sparse) and 6h_vs_8z, and explain baseline configurations and the aggregation of evaluation results across seeds.

  • Related works. We have expanded Appendix A.3 to clarify the connections and differences relative to self-imitation and successful-trajectory reuse.

We have also improved the readability of Figures 11–12.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.