- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how global information available during centralized training can help agents learn coordination under decentralized execution in cooperative multi-agent reinforcement learning. It examines different ways of using privileged coordination advice: directly guiding the policy through imitation or behavioral regularization, or supervising its internal representation through auxiliary prediction. When the advice provides only partial coordination information and leaves behaviors such as movement and positioning unspecified, directly encouraging the policy to follow it may not be the most effective approach.
To this end, the paper introduces FPAS, which uses an auxiliary prediction head sharing the policy encoder to predict coordination targets derived from global information using each agent’s local observation history. The auxiliary task aims to promote coordination-relevant representations, while action selection remains the responsibility of the reinforcement learning policy. Experiments on SMACv2, MPE, and Overcooked, together with controls on supervision pathways and target content, investigate the method’s mechanism and applicability. FPAS outperforms MAPPO across all three SMACv2 races. Further results suggest that imitation can be more effective when the advice defines a strong policy that decentralized agents can reproduce in closed-loop interaction. When the advice specifies only part of the required coordination behavior, auxiliary prediction can help agents extract useful information while reinforcement learning learns the remaining behavior. The paper also examines combining reinforcement learning with behavioral regularization. Overall, the work treats the supervision interface through which privileged coordination advice enters policy learning as an important design choice and empirically investigates the effectiveness and applicability of different interfaces.
Critical strengths
- FPAS offers a clear and simple approach to using incomplete coordination advice through auxiliary prediction, while leaving action selection to reinforcement learning.
- Experiments across multiple environments and controlled analyses examine the method’s effectiveness, mechanism, and the conditions favoring different supervision routes.
Critical weaknesses
- The behavioral-supervision comparison leaves uncertainty about the source of the gains. The paper does not clearly report loss-weight tuning and annealing results for the availability-matched SMACv2 comparison, limiting confidence that the observed advantage reflects the supervision interface rather than the particular behavioral-regularization configuration.
- Broader generality remains uncertain. The RL experiments use MAPPO and limited task configurations, the main SMACv2 targets rely on hand-designed rules, and the supervision-route selection diagnostic lacks independent prospective validation.
Questions for authors
Question 1: In the action-availability-matched comparison between the two supervision routes on SMACv2, how were the loss weights and their schedules selected? Were the weight sweeps and annealing experiments reported in the main text also conducted under this matched condition? If so, does the performance gap between the two routes persist?
Question 2: Is “closed-loop attainability” intended only to characterize how well behavioral imitation performs, or does it also suggest that imitation difficulties partly arise from insufficient local information? If the latter, which experiments in the paper distinguish the effects of insufficient local information from those of errors compounding during closed-loop execution?
Thank you for the candid response and the additional experiments.
Q1. I appreciate that the authors ran the availability-matched sweep and reported the outcome directly. The result materially changes the interpretation: tuned behavioral regularization achieves a higher reported mean (0.6075 at weight 0.05) than the matched auxiliary route at weight 0.50 (0.5735). Thus, the original gap does not establish an advantage of the auxiliary interface that persists after tuning behavioral regularization.
The abstract’s loss-matched claim and Section 4.4’s statement that the behavioral route remains below MAPPO need to be qualified accordingly. The broader “partial advice → predict” framing also requires qualification, since behavioral regularization can remain effective with partial advice. Although the route-selection diagnostic is explicitly formulated for pure imitation, its scope should be kept distinct from conclusions about RL with behavioral regularization.
The auxiliary weight was not swept under the matched condition, and the new results lack standard deviations or per-seed scores. The best behavioral setting also lies at the lower boundary of the tested grid. These limitations preclude a firm ranking of the two routes after comparable tuning.
Q2. The distinction between local inferability and closed-loop execution is clearer, and I am satisfied with the acknowledgement of the confounds in the DAgger comparison. However, the MPE masking experiment measures target predictability and the FPAS gain, rather than imitation performance. It therefore does not isolate insufficient local information as a cause of imitation failure. This distinction should be reflected in the revised interpretation of closed-loop attainability.
Generality. I appreciate the acknowledgement of the study’s scope. However, no new evidence addresses generality across MARL algorithms, reliance on handcrafted coordination targets, or independent prospective validation of the route-selection diagnostic, so these concerns remain.
Overall, the evidence supports FPAS as a simple method that improves over MAPPO in the tested settings. However, the central interpretation of the SMACv2 supervision-interface comparison requires substantial revision. I maintain my rating for now and am willing to reconsider after reviewing a revised manuscript that consistently incorporates the new results and appropriately narrows the claims.