acceptodds
Under review as a conference paper at ICLR 2027

Structured Representation Learning in Model-Based Multi-Agent Reinforcement Learning

Abstract

In this paper, we propose a novel value-based multi-agent reinforcement learning (MARL) method that integrates structured representation learning into world model imagination. We use latent states from recurrent and representation models to build the agent networks, injecting permutation-invariant and permutation-equivariant inductive biases into the networks and constraining the network embeddings to a product of probability simplices. These well-structured representations of the agents transfer smoothly across different team compositions. Moreover, we present a world model as the imagination module of our framework and apply the simplicial embeddings to the model latents. The world model supplies the mixing network with imagined trajectories, guiding agent learning through well-shaped representations and improving sample efficiency. Across different MARL benchmarks, our method demonstrates consistent gains over value-based and actor-critic baseline methods in both sample efficiency and asymptotic performance. We perform ablation experiments, design-space studies, and analyses of learned representations to highlight the importance of the main framework components within a model-based multi-agent paradigm.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, model-based reinforcement learning, structured representation learning, multi-agent systems, value-based reinforcement learning, world models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aeY2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents MSEMM, a model-based multi-agent method under the centralized-training-decentralized-execution paradigm. Drawing on components used in single-agent model-based actor-critic RL such as simplicial representations, latent space world models and RSSMs, this paper constructs a permutation-invariant (and equivariant where relevant) system for training multiple cooperating agents in complex scenarios.

The paper demonstrates the utility of the proposed method on SMAC and MAMuJoCo simulated multi-agent tasks, outperforming a suite of benchmarks both in asymptotic performance and in sample efficiency.

Critical strengths

The choice to build the internal representation of the various components of the system around permutation invariance is compelling. The permutation invariant (and equivariant) construction of the architecture could be the most significant contribution of this paper.

In addition, the experimental section seems comprehensive and covers a variety of baselines plus a good evaluation of design choices (model architecture and hyperparameters). Results seem to clearly point to the superiority of the proposed model compared to the baselines and various ablations.

Critical weaknesses

My central issue with this paper is clarity of presentation. The paper is hard to read in multiple sections: notation is overloaded and cramped in many places, which quickly becomes hard to follow. In addition, Figure 1 (and to a lesser extent Figs. 2-3) is confusing and hard to read; while subsequent text explains the various components, the figures appear earlier in the paper and are inadequately explained in the captions.

Questions for authors
  1. As I understand it (sec. 4.1), each agent holds a RSSM for each of the other entities. So, network sizes grow exponentially with the number of agents, and have to be pre-defined per task given numbers of agents (no ad-hoc cooperation is possible in this framework)?
  2. What is the architecture of the mixing network? It is unclear how the global state is mixed in with the prediction representation, beyond the loss objective.
  3. Following the previous question, how is the latent imagined trajectory encoded? It is unclear what the benefit of the auxiliary world model prediction is (beyond the demonstrated improvement in performance). Could the authors elaborate on the structure of this component?
  4. I’m curious about the generalization capabilities of MSEMM: how different are the SMAC test scenarios from the training ones?
Reviewer_Urz4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper glues three existing ideas together for value-based multi-agent RL: HPN-style entity networks that are invariant/equivariant to agent permutations, simplicial embeddings that force action representations onto a probability simplex, and a small learned world model that imagines extra rollouts to feed the centralized mixer. Two short proofs confirm the permutation properties. Results beat prior methods on SMAC, SMACv2, and Multi-Agent MuJoCo, with thorough ablations and a check that the world-model latents track the true global state.

Critical strengths

Empirically thorough: three benchmarks (SMAC, SMACv2, MAMuJoCo), seven ablations isolating each component, a design-space sweep, plus behavioral analysis and a latent-space correlation check—well above the typical MARL paper.

Critical weaknesses

The individual pieces are each well-established in prior work—entity-factored networks, simplicial embeddings, and learned imagination have all been explored separately—and the main contribution here is bringing them together under one value-based MARL framework. The gap between this combination and the sum of its parts could be spelled out more clearly.

Questions for authors

What is the marginal gain from the world model over a strong entity-factored baseline (HPN-QMIX) that already has PI/PE?

Reviewer_BoD2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MSEMM, a value-based multi-agent reinforcement learning (MARL) method designed for Centralized Training with Decentralized Execution (CTDE). The method combines permutation-consistent agent action representations with centralized world-model predictive learning to mitigate sparse supervision and limited generalization across varying team compositions. On the agent side, by applying Simplicial Embeddings (SEM) before the linear output head, the network produces a bounded per-agent state-action representation while guaranteeing permutation invariant (PI) on entity-independent actions and permutation equivariance (PE) on entity-targeted actions. On the centralized side, an SEM-augmented world model generates multi-step imagined rollouts from joint latent states, feeding the trajectory encoding into the monotonic mixing network alongside the global state during training without incurring execution-time overhead. Experiments on multiple benchmarks show consistent gains in sample efficiency and win rates over existing value-based, actor-critic, and model-based baselines.

Critical strengths

First, using world-model rollouts to condition the centralized mixing network is a clean, practical design choice for value-based CTDE. It avoids combinatorial planning over the joint action space and adds zero computational overhead during decentralized execution.

Second, MSEMM constructs an intermediate state-action representation ϕtoa(a)\phi_t^{oa}(a). This makes it straightforward to apply geometric constraints (SEM) and auxiliary local losses while preserving exact PI and PE guarantees. This explicit representation structure provides a more flexible interface for shaping internal features than enforcing permutation symmetries directly on the output weights via hypernetworks as done in prior work.

In addition, the empirical evaluation on SMAC (especially the five super-hard maps) and SMACv2 is strong.

Critical weaknesses

First, there are several incomplete and inconsistent mathematical formulations in Sections 4.3 and 4.4.

  • In Line 338, the agent dynamics loss is written as Ldyn(ψ,θ)=∑i=1nMSE((ϕtoa,i),sg(v~t+1i))\mathcal{L}_{dyn}(\psi, \theta) = \sum_{i=1}^n \text{MSE}((\phi_t^{oa,i}), \text{sg}(\tilde{v}_{t+1}^i)). I wonder if there is a missing predictor network around (ϕtoa,i)(\phi_t^{oa,i}). Because ϕtoa,i∈(ΔV−1)L\phi_t^{oa,i} \in (\Delta^{V-1})^L lies on a product of probability simplices after SEM normalization whereas v~t+1i\tilde{v}_{t+1}^i is an unnormalized RNN backbone state, taking the raw MSE between them is dimensionally and geometrically inconsistent.
  • In Line 317 of the world model rollout, the paper sets s^t+1:=zt+1s^\hat{s}_{t+1} := z_{t+1}^{\hat{s}}, where zt+1s^z_{t+1}^{\hat{s}} is already SEM-normalized. However, Figure 3 and Line 312 show that at the start of a rollout step, s^\hat{s} is passed through LN(enc(⋅))\text{LN}(\text{enc}(\cdot)) and SEM\text{SEM}. It is unclear how multi-step rollouts (t+1→t+kt+1 \to t+k) avoid double-encoding an already simplicial vector.
  • In Line 330, the stop-gradient targets s^t+1rep\hat{s}_{t+1}^{rep} and ϕt+1rep\phi_{t+1}^{rep} in LLS(θ)\mathcal{L}_{LS}(\theta) are never formally defined. In addition, Ldyn(ψ,θ)\mathcal{L}_{dyn}(\psi, \theta) and Lrew(ψ,θ)\mathcal{L}_{rew}(\psi, \theta) are written as functions of the world model parameters θ\theta, even though their formulas never depend on it.

Second, there are several questions regarding the evaluation design:

  • The NO-WM ablation does not disentangle the effect of multi-step imagination from auxiliary representation learning. In Section 5.2, NO-WM is described simply as removing the world model. The main text does not specify which auxiliary prediction losses remain active when the world model is removed, nor whether network capacity is controlled for. To separate the effect of the imagination process from that of auxiliary representation learning, a more targeted baseline should retain the predictive objectives and comparable model capacity while simply removing the imagined rollout features ρt\rho_t from the mixing network input.
  • Model-based MARL baselines (MAG, MAMBA, MABL) are evaluated on SMACv1 (Figure 4) but are omitted from SMACv2 (Figure 5) and MAMuJoCo (Figure 6).
  • To apply MSEMM to continuous control in MAMuJoCo, the authors discretize each joint's torque range into 30 bins (Appendix C.3). It is unclear whether the competing value-based baselines (e.g., QMIX, QMIX-CIA) were evaluated under the exact same discretization scheme.

Third, I inspected the code provided in the supplementary material and was unable to reproduce the experimental results. Beyond missing basic execution instructions and dependency configurations, the codebase does not fully implement the core method proposed in the paper. Moreover, the paper provides no analysis of training wall-clock time or computational overhead.

My initial recommendation is weak acceptance. The core architectural idea, combining PI/PE simplicial state-action embeddings on the agent side with imagined rollout conditioning in the centralized mixer, is well-motivated for CTDE value decomposition and is backed by strong empirical results and thorough ablations across SMAC and SMACv2. However, the mathematical presentation needs to be tightened, and questions regarding the evaluation detail and computational cost need to be addressed during the discussion period.

Questions for authors
  1. In Section 4.1, how are the per-entity states hti[j]h_t^i[j] and zti[j]z_t^i[j] updated when entity jj moves out of agent ii's sight range or dies?

  2. In Section 4.4, what is the exact architecture of the missing prediction head mapping ϕtoa,i\phi_t^{oa,i} to v~t+1i\tilde{v}_{t+1}^i in Ldyn\mathcal{L}_{dyn}? Also, how are s^t+1rep\hat{s}_{t+1}^{rep} and ϕt+1rep\phi_{t+1}^{rep} computed, and how does the world model handle s^t+1:=zt+1s^\hat{s}_{t+1} := z_{t+1}^{\hat{s}} at rollout steps 2…k2 \dots k without applying LN(enc(⋅))\text{LN}(\text{enc}(\cdot)) and SEM\text{SEM} twice?

  3. In MAMuJoCo (Figure 6), were all value-based baselines trained under the exact same N=30N=30 torque discretization? How are the entity classes (ownown, allyally, advadv) assigned when observations consist of neighboring robot joints within distance two in the kinematic graph?

  4. Since I was unable to reproduce the training and evaluation from the supplementary code, could the authors report the wall-clock training time and GPU memory footprint required to obtain the results in the paper, and clarify how MSEMM's advantage over existing methods (such as HPN-QMIX and QMIX) holds up when accounting for computational cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.