acceptodds
Withdrawn from ICLR 2027

Recoverable Coordination Distillation for Decentralized Multi-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Centralized training with decentralized execution (CTDE) allows multi-agent learners to exploit privileged information during training, but deployed agents must act from local inputs. Existing approaches may attempt to reconstruct missing teammate information, although much of this information can be irrelevant to the current decision. We propose Recoverable Coordination Distillation (RCD), which instead predicts the locally recoverable effect of privileged information on a teacher's action preferences. A frozen teacher defines this coordination effect as the centered difference between its action logits with and without privileged input. A local predictor estimates the conditional mean and uncertainty of this effect, while a reliability-aware fusion mechanism regulates its influence on the decentralized policy. At deployment, all computations use only execution-available local information, without the teacher, privileged inputs, or communication. We characterize the locally recoverable component of the teacher effect and derive a return bound relating prediction errors to policy performance. Across seven tasks from MPE, LBF, and RWARE, RCD achieves higher mean final return than our MAPPO-based MAE baseline in all task–input comparisons. With local history, it also achieves higher mean learning-curve AUC than the Local baseline on all seven tasks. These results support recovering decision-relevant coordination effects rather than reconstructing missing information for decentralized multi-agent control.

Primary area:
reinforcement learning
Keywords:
Multi-agent Reinforcement Learning, Recoverable Coordination Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CQH1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Recoverable Coordination Distillation (RCD), which predicts the locally recoverable effect of privileged information on a centralized teacher’s action preferences rather than reconstructing the missing information itself. The method estimates both the mean and uncertainty of this effect and integrates it into a decentralized MAPPO policy. Experiments are conducted on seven MPE, LBF, and RWARE tasks.

Critical strengths

The core idea is intuitive and reasonably novel: transferring the decision effect of privileged information is more targeted than reconstructing full hidden observations. The paper also includes several useful ablations and comparisons beyond the main baseline.

Critical weaknesses

The definition of the “coordination effect” is questionable because it subtracts a privilege-blocked forward pass from a teacher that was trained with privileged information. This restricted forward pass can be out-of-distribution, so the residual may reflect network artifacts rather than a meaningful effect of privileged information. The empirical advantage is also not sufficiently strong. RCD consistently beats the authors’ MA2E implementation, but MA2E is often much worse than the plain Local baseline. RCD itself does not consistently outperform Local, and the uncertainty component is not directly validated through calibration or prediction-error analysis.

Questions for authors

How do the authors justify that the privilege-blocked teacher output provides a valid counterfactual reference rather than an out-of-distribution network response? Can the authors directly evaluate whether the predicted uncertainty is calibrated and correlated with actual coordination-effect prediction error?

Reviewer_hmfFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Recoverable Coordination Distillation (RCD) for decentralized multi-agent reinforcement learning. A frozen teacher provides supervision through the difference between its action logits with and without privileged information. Each student predicts this difference and its uncertainty from local inputs, then uses reliability-aware fusion to guide action selection.

Critical strengths

The motivation is clear and reasonable: predicting how missing information affects action preferences may be more useful than reconstructing the missing teammate information itself.

Critical weaknesses

The empirical benefits are task-dependent, and the ablations provide mixed support for the core design. Final-return gains are concentrated in a few tasks. RCD performs worse than Local on SpreadXY-4 and achieves similar final returns on the warehouse tasks. Although learning-curve AUC improves across all seven tasks with history, the residual target does not consistently outperform full-logit prediction, and explicit reliability modulation does not consistently outperform a learned gate with fixed reliability. These results suggest useful gains in some settings, but do not yet establish a robust advantage for the proposed mechanisms.

Questions for authors

Could the authors compare RCD with closely related methods such as PTDE and PC3D under matched local inputs and training budgets, to assess whether RCD provides benefits beyond existing centralized-to-decentralized distillation methods?

Reviewer_F8gbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RCD, a teacher-student method for CTDE multi-agent RL. Instead of reconstructing missing teammate information at execution time (as in MA²E), RCD predicts the locally recoverable effect of privileged information on a frozen privileged teacher's action preferences. A local predictor estimates the conditional mean and variance of this effect via a diagonal Gaussian NLL, while the predicted variance is converted into a scalar reliability score that gates a residual fusion of the projected prediction into the student's feature. Experiments cover seven tasks from MPE, LBF, and RWARE, with current-observation and history-conditioned inputs.

Critical strengths
  1. The motivation is reasonable: predicting a decision-relevant consequence of missing information can be more useful than reconstructing details of that information.
  2. The experiments are rigorous: they share the same MAPPO backbone, matched seeds, and identical execution-time information contracts across the compared methods. The improvements on SpreadBlindfold and LBF-10×10 are also promising.
Critical weaknesses
  1. The supervision target is insufficiently validated. The teacher is trained with privileged information but queried with that pathway disabled. The resulting residual may reflect masking artifacts rather than useful coordination guidance. Freezing the teacher does not resolve this concern.
  2. The theory does not justify the distinctive design choices. The analysis combines standard conditional-expectation identities with a sensitivity bound around a fixed hypothetical reference. It establishes neither improvement over Local nor an advantage of residual targets or gradient separation. Proposition 1 is immediate (σ(X,h(X))=σ(X)\sigma(X, h(X)) = \sigma(X) for deterministic hh). Theorem 1 follows directly from the standard performance-difference identity and an assumed uniform Lipschitz condition in (μ,ξ)(\mu, \xi). The constants Lμ,LξL_\mu, L_\xi are not quantified for the proposed architecture, limiting the bound’s explanatory value for RCD’s specific design.
  3. Baseline fairness concerns. The MA²E baseline is a re-implementation adapted to MAPPO with pretraining budget 0 and TMA2E=1T_{MA^2E}=1 in the history-free case. However, the original MA²E relies on masked-autoencoder pretraining, so the main-table comparison is against a weakened version of the baseline.
  4. The empirical gains are limited. RCD improves final-return means over Local in only 4/7 current-observation tasks and 5/7 history tasks.
  5. This paper significantly reduced the page margins, which may be considered a desk rejection.
Questions for authors

RCD requires a full privileged-teacher training run plus two teacher forward passes per student transition. Can the authors provide analysis of time, sample complexity, or sensitivity to teacher quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.