acceptodds
Under review as a conference paper at ICLR 2027

Supervision Interfaces for Privileged Coordination Advice in Cooperative MARL

Abstract

Cooperative multi-agent reinforcement learning commonly adopts the Centralized Training with Decentralized Execution (CTDE) paradigm, in which global infor- mation is used during training while execution relies only on local observations. However, global information typically influences the actor indirectly through a centralized critic, leaving unclear how privileged coordination advice should be delivered to decentralized actors. We propose that privileged advice can supervise behavior directly through imitation or policy regularization, or supervise repre- sentation while reinforcement learning remains responsible for action selection, allowing the actor to absorb privileged coordination structure without directly im- itating the advised behavior. We introduce Factorized Privileged Action Super- vision (FPAS), which maps privileged coordination decisions to per-agent targets and trains each decentralized actor to predict them through a train-only auxiliary head. FPAS requires no learned teacher and adds no execution-time overhead. On SMACv2, FPAS outperforms MAPPO across all three races (+8.2/ + 12.8/ + 5.3 pp; paired p < 0.03 in each race, with 15/15 paired seed wins), with two races held out from tuning. Matched control experiments show that the gains are associated with the prediction pathway and decision-relevant target content. A loss-matched intervention further sharpens this conclusion: the same advice and cross-entropy degrade learning when applied through policy logits, but recover the FPAS gain when applied through the auxiliary prediction head. Target interventions further show that consistency, local inferability, and decision content are important prop- erties of effective representation supervision. Across MPE, SMACv2, and Over- cooked, imitation is strongest when the advice defines a strong and locally attain- able policy, whereas prediction is more reliable when the advice provides only partial coordination structure or when direct execution is constrained. These re- sults show that the supervision route through which privileged information enters the actor is itself an important design variable in CTDE.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, cooperative MARL, centralized training with decentralized execution, privileged information, auxiliary representation learning, imitation learning, coordination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rwoaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how global information available during centralized training can help agents learn coordination under decentralized execution in cooperative multi-agent reinforcement learning. It examines different ways of using privileged coordination advice: directly guiding the policy through imitation or behavioral regularization, or supervising its internal representation through auxiliary prediction. When the advice provides only partial coordination information and leaves behaviors such as movement and positioning unspecified, directly encouraging the policy to follow it may not be the most effective approach.

To this end, the paper introduces FPAS, which uses an auxiliary prediction head sharing the policy encoder to predict coordination targets derived from global information using each agent’s local observation history. The auxiliary task aims to promote coordination-relevant representations, while action selection remains the responsibility of the reinforcement learning policy. Experiments on SMACv2, MPE, and Overcooked, together with controls on supervision pathways and target content, investigate the method’s mechanism and applicability. FPAS outperforms MAPPO across all three SMACv2 races. Further results suggest that imitation can be more effective when the advice defines a strong policy that decentralized agents can reproduce in closed-loop interaction. When the advice specifies only part of the required coordination behavior, auxiliary prediction can help agents extract useful information while reinforcement learning learns the remaining behavior. The paper also examines combining reinforcement learning with behavioral regularization. Overall, the work treats the supervision interface through which privileged coordination advice enters policy learning as an important design choice and empirically investigates the effectiveness and applicability of different interfaces.

Critical strengths
  1. FPAS offers a clear and simple approach to using incomplete coordination advice through auxiliary prediction, while leaving action selection to reinforcement learning.
  2. Experiments across multiple environments and controlled analyses examine the method’s effectiveness, mechanism, and the conditions favoring different supervision routes.
Critical weaknesses
  1. The behavioral-supervision comparison leaves uncertainty about the source of the gains. The paper does not clearly report loss-weight tuning and annealing results for the availability-matched SMACv2 comparison, limiting confidence that the observed advantage reflects the supervision interface rather than the particular behavioral-regularization configuration.
  2. Broader generality remains uncertain. The RL experiments use MAPPO and limited task configurations, the main SMACv2 targets rely on hand-designed rules, and the supervision-route selection diagnostic lacks independent prospective validation.
Questions for authors

Question 1: In the action-availability-matched comparison between the two supervision routes on SMACv2, how were the loss weights and their schedules selected? Were the weight sweeps and annealing experiments reported in the main text also conducted under this matched condition? If so, does the performance gap between the two routes persist?

Question 2: Is “closed-loop attainability” intended only to characterize how well behavioral imitation performs, or does it also suggest that imitation difficulties partly arise from insufficient local information? If the latter, which experiments in the paper distinguish the effects of insufficient local information from those of errors compounding during closed-loop execution?

1 comment

Thank you for the candid response and the additional experiments.

Q1. I appreciate that the authors ran the availability-matched sweep and reported the outcome directly. The result materially changes the interpretation: tuned behavioral regularization achieves a higher reported mean (0.6075 at weight 0.05) than the matched auxiliary route at weight 0.50 (0.5735). Thus, the original gap does not establish an advantage of the auxiliary interface that persists after tuning behavioral regularization.

The abstract’s loss-matched claim and Section 4.4’s statement that the behavioral route remains below MAPPO need to be qualified accordingly. The broader “partial advice → predict” framing also requires qualification, since behavioral regularization can remain effective with partial advice. Although the route-selection diagnostic is explicitly formulated for pure imitation, its scope should be kept distinct from conclusions about RL with behavioral regularization.

The auxiliary weight was not swept under the matched condition, and the new results lack standard deviations or per-seed scores. The best behavioral setting also lies at the lower boundary of the tested grid. These limitations preclude a firm ranking of the two routes after comparable tuning.

Q2. The distinction between local inferability and closed-loop execution is clearer, and I am satisfied with the acknowledgement of the confounds in the DAgger comparison. However, the MPE masking experiment measures target predictability and the FPAS gain, rather than imitation performance. It therefore does not isolate insufficient local information as a cause of imitation failure. This distinction should be reflected in the revised interpretation of closed-loop attainability.

Generality. I appreciate the acknowledgement of the study’s scope. However, no new evidence addresses generality across MARL algorithms, reliance on handcrafted coordination targets, or independent prospective validation of the route-selection diagnostic, so these concerns remain.

Overall, the evidence supports FPAS as a simple method that improves over MAPPO in the tested settings. However, the central interpretation of the SMACv2 supervision-interface comparison requires substantial revision. I maintain my rating for now and am willing to reconsider after reviewing a revised manuscript that consistently incorporates the new results and appropriately narrows the claims.

Reviewer_W6QJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FPAS, which uses privileged coordination advice to supervise decision-relevant representations rather than directly guiding actor behavior. Through systematic experiments, it investigates when behavioral supervision or representation supervision is more effective.

Critical strengths
  1. A meaningful perspective on privileged supervision. The paper distinguishes behavioral and representation supervision based on advice completeness and executability. Controlled experiments further demonstrate how their relative effectiveness changes as more task-relevant behavior falls outside the provided advice.
  2. Simple and potentially extensible design with comprehensive evaluation. FPAS introduces a lightweight auxiliary prediction head without redesigning the underlying MARL algorithm, requiring neither a learned teacher nor additional execution-time computation. Extensive experiments across SMACv2, MPE, and Overcooked support its effectiveness.
Critical weaknesses
  1. Limited methodological novelty. FPAS closely resembles existing auxiliary representation learning methods. Its main contribution appears to lie in predicting coordination decisions and comparing supervision pathways rather than introducing a fundamentally new learning mechanism. The distinction from prior auxiliary learning approaches remains insufficiently established.
  2. Strong dependence on handcrafted coordination targets. The main experiments rely on task-specific heuristics, such as focus-fire and low-health rules in SMACv2 and Hungarian assignment in MPE. Since FPAS depends on target quality and consistency, its applicability to tasks without well-defined coordination rules remains unclear.
  3. Insufficient analysis of representation quality and policy performance. Frozen-encoder probes demonstrate improved encoding of coordination-related information, but the relationship between auxiliary prediction quality and final policy performance remains unclear. Better target prediction does not necessarily translate into higher returns, limiting the mechanistic explanation.
Questions for authors
  1. Could the authors provide direct comparisons with representative auxiliary representation learning methods under matched targets and training configurations to better isolate FPAS's methodological contribution?
  2. Could the authors investigate the relationship between auxiliary prediction accuracy, prediction loss, and policy returns, potentially using targeted representation ablations to assess the contribution of learned coordination features?
  3. Could the authors further evaluate FPAS with automatically generated or less task-specific coordination targets to assess its dependence on handcrafted heuristics?

If all the above issues are solved, I will consider raising my rating.

Reviewer_ijEiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how agents in MARL can benefit from information available during training but unavailable when they act independently for coordination in the execution time. It proposes FPAS, which asks each agent to predict the advice from a rule or planner while still learning its actions through reinforcement learning. The authors conduct experiments in StarCraft, MPE, and Overcooked environments to show the effectiveness of the method and compare the imitation and prediction approaches.

Critical strengths

None (at least with the current writing of the paper)

Critical weaknesses

1- The study appears to be tailored to the specific benchmarks considered, and does not provide significant insight for a general MARL problem. The discussion of the experimental results focuses excessively on environment-specific interpretations. More importantly, the coordination targets, as the core of the proposed method, are manually designed for the evaluated environments (Section 3.3). The paper does not provide any insight into how these targets could be defined or generalized to new environments.

2- I believe the paper requires a thorough revision of its writing, particularly in the results section. In its current form, substantial portions of the paper are completely incomprehensible. As an illustrative example, consider section 4.2. It is unclear what the overall objective of the evaluations is, which quantities are being measured, how these quantities are precisely defined, and what general scientific conclusions can be drawn from the reported results.

Questions for authors

Could you explain how the coordination targets should be defined for a general MARL problem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.