acceptodds
Under review as a conference paper at ICLR 2027

BEAD: Blind-spot Entity-aware Advantage Distillation for Cooperative Multi-Agent Reinforcement Learning

Abstract

Centralized Training with Decentralized Execution (CTDE) is widely adopted in cooperative Multi-agent Reinforcement Learning. Building on CTDE, Centralized Teacher with Decentralized Student (CTDS) exploits global information to construct teachers, but indiscriminate use of such information may degrade teacher decision quality. We propose Blind-spot Entity-aware Advantage Distillation (BEAD), a centralized-teacher decentralized-student framework that selectively exploits global information for effective knowledge transfer. BEAD represents the global state as entities and employs gated blind-spot compensation to selectively incorporate locally unobserved entities into the privileged teacher. In addition, BEAD introduces explicit advantage distillation anchored to the student's value estimates, transferring the teacher's relative action preferences while preserving the student's own value learning. Theoretical analysis shows that BEAD preserves action-selection preferences, while experiments on challenging multi-agent reinforcement learning benchmarks demonstrate consistent improvements over strong CTDE and teacher-student baselines.

Primary area:
reinforcement learning
Keywords:
Multi‑Agent Reinforcement Learning, CTDE, Knowledge Distillation, Partial Observability, Advantage Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EpfGOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BEAD (Blind-spot Entity-aware Advantage Distillation), a centralized-teacher, decentralized-student framework for cooperative multi-agent RL. It uses gated attention to incorporate globally available entities that individual agents cannot observe into the teacher’s representations, then distills relative action advantages, rather than raw Q-values, into the decentralized student. Experiments on SMAC and SMACv2 show that BEAD outperforms strong CTDE and teacher–student baselines, particularly in challenging asymmetric-coordination scenarios. Ablation studies further demonstrate the benefits of both the gating mechanism and advantage-based distillation..

Critical strengths
  1. The paper is well-written, clear, and easy to follow. I enjoyed reading it.
  2. The use of attention-based gating in the teacher appears novel, and the intuition behind the approach is clear- not all globally-observable information is useful to a given agent.
  3. The paper tests on challenging settings on SMAC and SMACv2 maps.
Critical weaknesses
  1. Insufficient statistical reporting: The most critical weakness, in my view, is that the paper does not report the number of random seeds used, and the results are presented without standard deviations. The authors should evaluate their main experiments across at least 5–7 random seeds and report the variability of the results. Without this, it is impossible to assess whether the reported results are statistically meaningful [1].
  2. Analysis: The paper does not analyze what the attention and gating mechanisms actually learn to select or suppress, e.g., via attention-weight visualizations or a case study of which blind-spot entities are prioritized in a given state. Without such analysis, the mechanism remains an uninterpreted transformation: it could be performing the claimed entity-level relevance filtering, or it could be learning some other function entirely that happens to help performance. An example where gating helps would substantiate the selectivity claim made in the paper.
  3. Comparison to relevant methods: The authors lack any mechanistic or qualitative analysis comparing what their gated blind-spot attention actually selects or suppresses against what prior selective approaches (e.g., PTDE’s agent-specific representations) do, relying instead on aggregate downstream performance as the sole evidence of improved selectivity.

Without a more detailed analysis, the benefit of the proposed mechanism cannot be distinguished from that of a general attention/gating mechanism, preventing me from giving a higher score. I would be happy to raise my score if the authors can adequately address those issues.

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

Questions for authors
  1. Statistical significance and seed variability: The paper reports win rates (e.g., 56.25%, 65.63%) without specifying the number of random seeds or reporting variance or confidence intervals. Given the known run-to-run variability in SMAC, how many seeds were used, and are the reported improvements over the baselines statistically robust?
4 comments

Dear authors, I would like to thank you for the effort put into your response.

Regarding the original manuscript, for a paper proposing a new algorithm and relying primarily on empirical evaluation, reporting results from only a single seed, without clearly stating the actual number of seeds used, is problematic in terms of both soundness and clarity. An approach such as BEAD cannot be properly evaluated on this basis. That said, I would be satisfied if the authors revised the manuscript to report standard deviations for all graphs and complete results over 5–7 seeds. However, from the response and revised manuscript, it appears that the graphs remain largely unchanged, with complete multi-seed results provided only for some maps.

I understand that the discussion period is limited and that you had only a week to respond. However, since this is essential to evaluating the paper's main claims, these results should ideally have been prepared before submission.

Regarding the analysis, examining only two training seeds on limited states per seed is insufficient to establish that the observed effects are systematic and consistent. A convincing mechanistic analysis should be conducted across more training seeds and report aggregate statistics and variability to rule out seed-dependent effects. As I mentioned in my original review, the paper is well written and presents a simple, clean idea, which I found compelling as a motivation for the proposed attention modification. However, since the algorithmic novelty on the teacher side ultimately amounts to adding an attention mechanism, which is already widely used, the authors need to provide a detailed analysis demonstrating why this particular modification is effective to support the novelty of their approach.

I wish you the best with the paper, but given the remaining concerns, I cannot justify a higher score in its current state.

Dear Reviewer,

Thank you for your further feedback. We understand your concerns regarding the rigor of the multi seed evaluation and the justification of our teacher side contribution. We address these two points below.

1. Complete five seed evaluation and reporting of variability.

We agree that reporting results from a single training seed was insufficient and that the number of seeds should have been stated clearly. Following our previous response, we have completed evaluations over five seeds for all experiments in the main text, including the complete SMAC and SMACv2 comparisons and the ablation studies.

The revised manuscript now reports means and sample standard deviations across five independent training seeds, and the training curves display the mean across seeds with shaded regions indicating ±1 standard deviation. The additional MPE and Google Research Football evaluations also report results over five seeds. Unless explicitly stated otherwise, experiments use five independent training seeds.

These revisions extend the evaluation over multiple seeds to the complete main benchmark comparison rather than only selected maps. BEAD achieves the highest mean decentralized win rate on six of the eight SMAC and SMACv2 scenarios. Its largest mean improvements over HPN are observed on 3s5z_vs_3s6z and 6h_vs_8z, reaching 24.69 and 25.63 percentage points, respectively. We believe these expanded results provide substantially stronger empirical support for BEAD’s effectiveness while explicitly documenting variability across training runs. We acknowledge that this evaluation should have been included in the original submission.

2. Teacher side contribution and its distinction from PTDE.

We would like to clarify that we do not claim the Transformer architecture or attention mechanism itself as our teacher side novelty. As summarized in the Introduction and explained in the subsection on gated blind spot compensation, our contribution is gated blind spot compensation. This design identifies globally available but locally unobserved entities, aggregates their information for each agent, and adaptively regulates its incorporation into the local decision pathway.

As discussed in our previous response, this design differs from the PTDE GIP teacher in three specific respects.

(i) Information eligibility. PTDE constructs an agent personalized latent representation from the flattened global state through a projection conditioned on the local input. BEAD explicitly applies a visibility mask to restrict privileged candidates to active, locally unobserved entities. It therefore distinguishes information that complements the local observation from information already available to the agent.

(ii) Entity weighting. PTDE personalizes global information through a learned transformation of global state coordinates. BEAD instead uses the local embedding as a query to assign explicit attention weights to eligible blind spot entities. This separates entity eligibility from weighting for each agent, rather than combining both within a transformation of the flattened global state.

(iii) Injection strength and location. PTDE concatenates its personalized global latent with the local representation after the recurrent update. BEAD applies a learned elementwise gate to the pooled blind spot context and injects it additively before the recurrent update. This introduces explicit control over how strongly privileged context influences the recurrent decision representation.

These architectural distinctions are complemented by the experimental results. Across five training seeds, BEAD achieves higher mean decentralized performance than PTDE on all eight SMAC and SMACv2 scenarios, as well as on MPE and Google Research Football. The gate ablation further shows that learned gating improves mean teacher win rate from 62.50 ± 13.62% without gating to 81.25 ± 14.32%, supporting the importance of regulating context injection beyond attention based aggregation alone.

Our matched entity interventions additionally connect entity weighting to changes in the teacher’s action values, while the controlled gate interventions illustrate how context injection affects subsequent decisions. Together, the structural comparison, component ablations, and benchmark results support gated blind spot compensation as a distinct teacher design. The contribution lies in the coordinated treatment of entity eligibility based on visibility, weighting for each agent, and gated injection, rather than the use of attention alone.

Thank you again for your careful reading and constructive feedback. We sincerely hope that our response and the revised manuscript will encourage you to reconsider your assessment of our work.

Dear authors,

Multi-seed evaluation is an integral part of an empirical algorithmic paper. It should have been in the original submission rather than added over the discussion period after repeated requests, and I do not view this favorably.

I do not dispute the motivation or the core idea behind the work. My main concern is that the algorithmic instantiations of the idea (e.g., attention and gating) are not sufficiently analyzed or shown to be connected to the underlying motivation. Since mechanisms such as gating and attention are widely used, I believe the paper needs to demonstrate that these mechanisms are specifically connected to and support the proposed novel idea.

In the current state, I do not feel comfortable assigning a score above 2. I will raise my score to 2, and given the higher scores from the other reviewers, this may be sufficient for the paper. However, I cannot argue in favor of acceptance.

Dear Reviewer,

Thank you for considering our revisions and for raising your score. We appreciate your recognition of the motivation and core idea. We also acknowledge that the evaluation across multiple seeds should have been included in the original submission.

We would like to briefly emphasize our two contributions.

1. Gated blind spot compensation. Our contribution is the coordinated design that explicitly identifies locally unobserved entities, weights their relevance to each agent, and adaptively regulates their injection into the local decision pathway. This connects privileged information to the agent’s specific information gap, beyond simply adding attention or gating.

2. Student value anchored explicit advantage distillation. We transfer the teacher’s relative action preferences while retaining the student’s own TD objective for value learning. Our theoretical analysis establishes that mean centering preserves pairwise action preferences and decomposes action value matching error into advantage disagreement and mean value disagreement, providing a principled justification for this objective.

Together, these contributions address both how to selectively exploit privileged information and how to transfer decision relevant knowledge to decentralized students. We hope this clarifies the specific novelty of our approach.

Thank you again for your careful assessment and your willingness to revise the score.

Reviewer_AAtPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BEAD uses gating to selectively fill gaps in the teacher’s local observations and distills the teacher’s action advantages into students that execute in a decentralized manner. It improves performance on cooperative multi-agent tasks, but the ablation evidence for its key mechanisms remains insufficient.

Critical strengths
  1. The motivation is clear and well-founded. The authors argue that indiscriminately providing the teacher with the global state introduces information that is either already locally observable or irrelevant to the current decision. Global knowledge should therefore be supplied selectively according to each agent’s information gaps.
  2. The method is relatively compact and preserves decentralized execution. The additional processing of global information occurs primarily in the teacher during training, while the deployed students still require only local information.
  3. The experimental results are valuable. HPN serves as the backbone of the main implementation, so outperforming HPN provides stronger evidence for the value of the added modules than merely outperforming standard QMIX. The method also achieves substantial gains in difficult scenarios.
Critical weaknesses
  1. Statistical reliability is inadequately reported, which is one of the most important issues. The paper does not report the number of random seeds, means and variances across runs, or confidence intervals, nor does it specify the number of episodes used in each evaluation. The absence of error bands in the learning curves is particularly problematic for a MARL paper evaluated on SMAC.
  2. The contributions of teacher-generated data and distillation supervision are not disentangled. BEAD collects data using a teacher with privileged information, so some of the gains may come from better exploration and training data. Without a control that uses the same teacher-collected data but trains the student solely through TD learning, the overall improvement cannot be attributed entirely to knowledge distillation.
Questions for authors

See the Weaknesses.

1 comment

Dear Reviewer,

We would like to share a brief update regarding your first concern about statistical reliability.

Since our initial response, we have completed five seed evaluations for all eight SMAC and SMACv2 scenarios, extending the results beyond the four maps previously reported. The revised manuscript also includes five seed results for the main ablation studies and the additional MPE and Google Research Football evaluations.

All these results are now reported as means and sample standard deviations across five independent training seeds. We have also updated the learning curves to display the mean across seeds with ±1 standard deviation shading, making variability across training runs explicit.

The manuscript now clearly specifies the evaluation protocols. The main SMAC and SMACv2 comparisons use two million environment steps per run and 32 episodes per evaluation. For SMAC, we report the last evaluation within the training budget. For SMACv2, we average the last five evaluations within each run before calculating statistics across seeds. MPE and Google Research Football use one million environment steps per run and 64 episodes per evaluation.

These complete results replace the partial benchmark statistics in our initial response and are included directly in the revised manuscript. BEAD achieves the highest mean decentralized win rate on six of the eight SMAC and SMACv2 scenarios, providing broader empirical support while explicitly documenting variability.

Thank you again for highlighting the importance of statistical reliability. We hope this update is helpful when considering our response and the revised manuscript, and we would welcome any further comments you may have.

Reviewer_TTdhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes BEAD, a Centralized Training with Decentralized Execution (CTDE) framework for cooperative Multi-Agent Reinforcement Learning (MARL). It addresses the issues that providing privileged teachers with the full global state introduces redundant information, and that distilling raw action-values can unnecessarily constrain the student's value representation. BEAD introduces two main components: (1) Gated blind-spot compensation, which extracts entity-level global information not present in the agent's local observation and uses a learned gate to adaptively filter this context before fusing it with the teacher's local representation. (2) Student-value-anchored explicit advantage distillation, which transfers the teacher's relative action preferences (advantages) rather than raw Q-values. Theoretical analysis (Theorem 1) shows that mean-centering over available actions isolates preference disagreements from mean-value discrepancies. Validation is performed on the SMAC and SMACv2 benchmarks, demonstrating that BEAD significantly outperforms strong baselines (e.g., QMIX, HPN, CTDS, PTDE) on challenging, heterogeneous, and procedurally generated scenarios.

Critical strengths

Highly effective and well-motivated gated compensation: The empirical demonstration that indiscriminately injecting global information degrades teacher performance (Figure 5) strongly validates the necessity of the proposed gating mechanism. The gate allows the teacher to selectively exploit locally unobserved entities, which proves crucial for resolving asymmetric and complex coordination tasks.

Strong theoretical grounding for Advantage Distillation: Theorem 1 elegantly decomposes the Q-value matching error into advantage matching error and mean-value discrepancy. This provides a solid theoretical justification for why distilling advantages preserves the relative action-selection structure needed for decentralized execution while remaining compatible with the student's TD-based value learning.

Critical weaknesses

Dependency on Entity-Structured States: The "blind-spot" extraction relies on the global state being represented as a set of entities. This limits the method's out-of-the-box applicability to environments with unstructured observations, such as pixel-based Atari games or continuous control tasks where entities are not explicitly defined in the state space (as acknowledged in the conclusion).

Limited Diversity of Benchmarks: The evaluation is heavily concentrated on StarCraft-based environments (SMAC and SMACv2). While these are standard, they share similar mechanics (unit attributes, partial observability via vision ranges). Validation on other MARL benchmarks (e.g., Google Research Football, MPE, or Hanabi) would strengthen the claims of generalizability.

Questions for authors

How does the gating mechanism and blind-spot entity extraction scale with a very large number of entities in the global state? Does the computational overhead of the gate become a bottleneck in complex environments?

In Figure 2 (3s5z_vs_3s6z), the BEAD student significantly outperforms the BEAD teacher (56.25% vs 37.5%). Why does the student surpass the teacher? Is it due to the student's TD loss acting as a regularizer, or is the teacher overfitting to the privileged blind-spot context during execution?

The method relies on explicit entity embeddings. How could BEAD be adapted for environments where the global state is high-dimensional and unstructured (e.g., raw pixels), perhaps by using attention mechanisms over spatial features to define "blind-spot entities"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.