- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents MSEMM, a model-based multi-agent method under the centralized-training-decentralized-execution paradigm. Drawing on components used in single-agent model-based actor-critic RL such as simplicial representations, latent space world models and RSSMs, this paper constructs a permutation-invariant (and equivariant where relevant) system for training multiple cooperating agents in complex scenarios.
The paper demonstrates the utility of the proposed method on SMAC and MAMuJoCo simulated multi-agent tasks, outperforming a suite of benchmarks both in asymptotic performance and in sample efficiency.
Critical strengths
The choice to build the internal representation of the various components of the system around permutation invariance is compelling. The permutation invariant (and equivariant) construction of the architecture could be the most significant contribution of this paper.
In addition, the experimental section seems comprehensive and covers a variety of baselines plus a good evaluation of design choices (model architecture and hyperparameters). Results seem to clearly point to the superiority of the proposed model compared to the baselines and various ablations.
Critical weaknesses
My central issue with this paper is clarity of presentation. The paper is hard to read in multiple sections: notation is overloaded and cramped in many places, which quickly becomes hard to follow. In addition, Figure 1 (and to a lesser extent Figs. 2-3) is confusing and hard to read; while subsequent text explains the various components, the figures appear earlier in the paper and are inadequately explained in the captions.
Questions for authors
- As I understand it (sec. 4.1), each agent holds a RSSM for each of the other entities. So, network sizes grow exponentially with the number of agents, and have to be pre-defined per task given numbers of agents (no ad-hoc cooperation is possible in this framework)?
- What is the architecture of the mixing network? It is unclear how the global state is mixed in with the prediction representation, beyond the loss objective.
- Following the previous question, how is the latent imagined trajectory encoded? It is unclear what the benefit of the auxiliary world model prediction is (beyond the demonstrated improvement in performance). Could the authors elaborate on the structure of this component?
- I’m curious about the generalization capabilities of MSEMM: how different are the SMAC test scenarios from the training ones?