acceptodds
Withdrawn from ICLR 2027

Multi-Agent Contrastive World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cooperative multi-robot control is usually learned from a hand-designed reward and often relies on communication between robots at execution. We drop both assumptions and present the Multi-Agent Contrastive World Model (MACWM), which combines a contrastive goal-conditioned critic with an action-conditioned world model that predicts the critic's own latent representation. A critic alone gives the actor a weak signal, because at a high control rate one command barely changes which goals the team reaches. MACWM therefore lets the critic score short rollouts of the world model inside the actor's training objective, and each robot executes its policy on its own observation. We train it alongside reward-free, communication-free baselines under one protocol on a benchmark of cooperative tasks for pairs of quadruped robots in Isaac Lab. Across nine tasks MACWM is the only one of these methods whose every training run reaches the goal, and over a wide band of success thresholds it fails less often than either contrastive baseline. With three to five robots our method leads both contrastive baselines on every seed of cooperative box pushing, and its policies transfer to two real quadrupeds without further training.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, goal-conditioned reinforcement learning, contrastive reinforcement learning, world models, reward-free reinforcement learning, decentralized execution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FEsTOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MACWM, a multi-robot cooperative learning method that combines a contrastive goal-conditioned critic with an action-conditioned latent world model. The critic learns goal reachability from actual trajectories and hindsight goals, while the world model predicts the latent representations used by the critic and provides the policy with training signals that account for the consequences of joint actions through short-horizon differentiable imagination. The training does not rely on hand-designed task rewards, and at deployment time each robot generates actions based only on its own observations and goals, without running the world model or exchanging policy messages.

Overall, I have a positive view of this work and consider it a reasonable and effective cooperative learning scheme; the remaining issues mainly concern clarification of implementation details and explanation of some experimental phenomena.

Critical strengths
  1. MACWM uses a critic to evaluate short-horizon imagined trajectories, enabling policy updates to account for the effects of continuous joint actions and reasonably addressing the weak learning signal from single-step actions in high-frequency control.

  2. On nine dual-robot tasks, MACWM achieves a higher average success rate than MACRL and ICRL, supporting the overall effectiveness of this design; removing the imagination term on PushBox reduces the success rate, indicating that the world model provides valuable training signals for strongly cooperative tasks; MACWM also maintains its lead in tasks with an increased number of robots.

  3. The paper cross-examines the world model and exploration noise, and analyzes the critic's ability to distinguish actions, providing richer evidence for understanding when the imagination term is effective.

  4. The world model and centralized critic are used only during training, and each robot runs its policy independently at execution time; compared with the baselines, this further improves physical deployability.

Critical weaknesses
  1. The actions in the imagination phase are all sampled from the same replay rather than generated step by step based on predicted observations. In my view, this is an intentional design choice by the authors, but there seems to be a lack of experiments validating this design.

2.The default imagination horizon is five steps, and the paper has not yet adequately explained why such a short horizon can significantly improve tasks requiring sustained cooperation, such as PushBox.

  1. in Table 12, the average success rate under pink noise increased from 0.629 to 0.655, but the result under Gaussian noise decreased from 0.521 to 0.451. The analysis of this significant contrast seem still insufficient.
Questions for authors
  1. Are the imagined actions across the entire segment sampled from the same policy distribution using independent noise, and is it intentional not to use the decoded predicted observations to update the action distribution, so as to reduce model error or training overhead? Could the authors add experiments to clarify this?

  2. It is suggested to add an ablation study on the horizon.

  3. Under the two types of exploration noise, are there observed differences in world model prediction error or in the number of successful cooperative trajectories? This could help explain the reversal in the direction of the imagination term's benefit in Table 12.

Reviewer_wgp3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Multi-Agent Contrastive World Model (MACWM), a method for cooperative multi-robot control. MACWM learns a goal-conditioned policy without relying on hand-designed rewards or communication during execution. This is achieved by combining a contrastive critic with a world model to update the policy. The proposed method is evaluated against state of the art baselines on multiple cooperative tasks from a novel benchmark.

Critical strengths

This paper aims to address a relevant challenge in multi-robot control, i.e., how to learn cooperative policies that do not rely on communication during execution or hand-designed reward signals. In this regard, using world models to address this problem is a valuable and underexplored research direction.

Critical weaknesses
  1. The experiments do not fully support MACWM. Despite achieving a higher success mean, MACWM does not consistently outperform MACRL and ICRL on the majority of the tasks (Table 2), and the high standard deviations make it difficult to establish the significance of the performance gaps without an appropriate statistical test. Importantly, the ablation in Section 4.3 shows that the core contribution of MACWM provides a performance improvement only in PushBox (one task out of nine). This is further highlighted by the scaling experiment (Section 4.4), where the results of MACWM are indistinguishable from MACRL and ICRL in Gate and Door. Combined with the much higher (more than 2×2\times, as reported in Table 10) wall-clock time compared to the baselines mentioned above, the current results do not fully support adopting MACWM for multi-robot control problems.

  2. The presentation makes the novelty of MACWM difficult to assess. In particular, except for the summary in Table 1, there is no clear distinction between the contribution of MACWM compared to reward-free and communication-free baselines such as MACRL and ICRL. Moreover, the "Problem Setting" (Section 3.1) is reported under Section 3 ("Method"), but seems very similar to the "Multi-Agent Goal-Reaching Problem" introduced in ICRL [1]. The authors should explicitly clarify the difference between the two formulations, or, if it is not a novel contribution, include it as a preliminary.

[1] Nimonkar, Chirayu, et al. "Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration." arXiv preprint arXiv:2509.10656 (2025).

Questions for authors
  1. Considering the results achieved by MACWM without a world model (Appendix F.1) and with Gaussian noise (Appendix F.2), is the main improvement in performance due to the imagined actor loss or the pink noise?

  2. What are the novel contributions of MACWM compared to other baselines that do not use communication mechanisms or hand-designed rewards, such as MACRL or ICRL?

  3. How does the problem setting introduced in Section 3.1 differ from the setting introduced in ICRL?

Other minor questions about clarity:

  • What is the functional form of the goal encoder fGf_G?

  • What are the differences between the various metrics reported in Table 2?

Reviewer_CQ1ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Multi-Agent Contrastive World Models (MACWM), a method for cooperative multi-robot control without hand-designed task rewards or communication between agents at execution time. The approach combines a centralized contrastive goal-conditioned critic, an action-conditioned latent world model, and decentralized policies. The world model is trained with AC-CPC, value-equivalence, and observation-reconstruction losses to predict the critic's latents. During actor training, the critic scores K-step open-loop rollouts of the world model, which gives each sampled command a gradient through its effect on the next few steps. This addresses the weak one-step signal at a 50 Hz control rate. Execution is decentralized with a shared policy. The paper also contributes a multi-slice hindsight sampler, uses pink exploration noise, and builds a benchmark of nine cooperative two-Go1 tasks in Isaac Lab.

Critical strengths
  • This paper introcues a well-motivated method for a hard and practical setting. Dropping both task rewards and execution-time communication matches real constraints on multi-robot deployment. The explanation of why one-step contrastive critics give a weak signal at high control rates, and how scoring short world-model rollouts addresses it (supported by the 3–4× wider value spread in Fig. 4), is clear and convincing. PushBox, the task that needs joint action, shows the intended effect: +0.36 from the imagined term, and all 12 paired larger-team runs ahead of MACRL.
  • It presents a technically meaningful combination of contrastive goal-conditioned RL and multi-agent world modeling. The paper addresses the important problem of cooperative control without hand-designed rewards or explicit inter-agent communication. Using a centralized world model to provide multi-step, goal-conditioned gradients to decentralized policies is well motivated. The model predicts the contrastive critic's latent representations and uses value-equivalence and observation-reconstruction objectives to support imagined rollouts.
  • the authors show substantial results on difficult coordination tasks. The paper evaluates nine tasks, eight seeds per method, several reward-free baselines, and additional reward-based reference methods. MACWM achieves the highest aggregate success and performs particularly well on Door, Herding, and PushBox. The PushBox results provide the strongest support for the imagined actor objective: removing it substantially reduces performance, and the proposed method also performs favorably in larger-team PushBox experiments. The demonstration of zero-shot transfer to real quadrupeds is additionally encouraging.
Critical weaknesses
  • The imagined world-model term, contributes little on average. Setting η = 0 changes the nine-task mean by only +0.03, and on eight of nine tasks the change is within seed noise (−0.08 to +0.06). Much of MACWM's advantage over MACRL seems to come from other changes: pink noise raises MACWM by 0.20, and the multi-slice sampler raises success by 0.37 on the four tasks measured. The overall gain over MACRL (0.630 vs. 0.581) is smaller than the within-method seed spread (per-task SD ~0.18–0.19). The paper's framing and title emphasize the world model, but the evidence supports a benefit mainly on PushBox
  • The main algorithms are compared at equal environment-step budgets, but MACWM requires substantially greater training computation: Table 10 reports a median of 8.23 hours per run versus 3.05 hours for MACRL. Given the modest average improvement, performance under a matched wall-clock or compute budget is important to establish practical value.
  • Furthermore, the real-robot PushBox environment was fitted during MACWM's own trials, and the authors explicitly state that the hardware results do not provide a fair ranking of methods. The simulation results with randomized starts under the hardware constraints also show that the relative ranking can change. Consequently, the hardware experiments demonstrate feasibility of transfer, but not robust superiority over the baselines.
Questions for authors
  • Do MACRL and ICRL use the multi-slice hindsight sampler (Sec. 3.3) and the logsumexp penalty? If not, please report them with these fixes so the comparison isolates the world model.
  • When does the world model actually improve control? Can the authors provide a more systematic analysis of why the imagined actor objective substantially improves PushBox but offers limited or negative gains on most other tasks?
  • How do the methods compare under matched computation? Given the approximately 2.7× training-time difference between MACWM and MACRL, can the authors provide compute-matched learning curves or compare the best performance attainable by the baselines under an equivalent training budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.