acceptodds
Withdrawn from ICLR 2027

Artemis: Geometry-Grounded Multi-Agent Driving World Models with Shared 3D State and Progressive Memory Update

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent video world models have witnessed the paradigm shift from single-agent to multi-agent involvements, which can reveal more complicated dynamics and cross-agent interaction in the real world. However, existing approaches commonly adopt implicit inter-agent communications via cross attention, which lack explicit geometry constraints and unified 3D state, thereby leading to poor multi-view consistency and struggling with recovering out-of-sight agents. In addition, most of them assume a static background, failing to represent uncontrolled background dynamics. To address these problems, we propose Artemis: a geometry-grounded multi-agent world model with explicit memory sharing. An explicit 3D world map is reconstructed from multi-agent observations to enforce a unified 3D state across agents, offering high cross-view consistency. Specifically, an action-guided geometric injection module is developed to simultaneously render decomposed foreground-background control maps, which are then injected into a diffusion transformer through a designed GeoAdapter block. Compared to previous methods assuming static-only background, our GeoAdapter can also distinguish uncontrolled non-agent dynamics, which are conditioned on their own multi-frame history positions to provide consistent motion cues. Keyframes selected from progressive video rollouts are used to progressively update the reconstructed 3D world maps. To effectively capture complex dynamic patterns, we curate a novel dataset sampled from the CARLA simulator called MA-CARLA, with scalable agent numbers, physical plausibility, and abundant interaction modes. Extensive experiments demonstrate the superiority of our proposed method in terms of visual fidelity and cross-view consistency in the generated videos. In addition, our Artemis can support simultaneous multi-modal rollouts with both 2D video and 3D point map maintenance, scale to scenarios beyond two agents, and flexibly switch between single-camera or multi-camera setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-agent World Model, Geometry-grounded video generation, Memory sharing, Autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xttLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Artemis, a multi-agent driving world model, which generates synchronized future videos for several vehicles in a shared scene from initial observations and actions by single agents.

Artemis builds on top of existing components and combines them: specifically, starting from the set of observations assembled by the multiple agents, it reconstructs a shared static 3D point-cloud map with LingBot-Map, then renders some control maps along the planned trajectory of each agent, dividing vehicles from the background, and injects these into the pretrained Wan2.2-5B diffusion transformer through a novel GeoAdapter module, alongside a cross-agent attention layer. During autoregressive rollout, keyframes from generated videos update the shared map, and a pose memory re-initializes vehicles about to enter an agent's view.

The paper also introduces MA-CARLA, a multi-agent driving dataset rendered with CARLA engine, providing about 11K sequences with four camera views per vehicle. Experimental results on this benchmark compare Artemis with Solaris (both zero-shot and fine-tuned) and an OASIS-inspired channel-concatenation baseline. Artemis reports the best PSNR, SSIM, LPIPS, and subject consistency.

Critical strengths

I believe the main strengths of this paper are:

  • Sound and principled idea. I agree with the authors that modelling interaction between agents' observations explicitly has the potential to overcome failures of prior multi-agent frameworks solely based on implicit communication through attention. By building on estimated geometry, Artemis makes cross-view consistency a property of the conditioning signal.

  • The paper introduces a new benchmark, MA-CARLA. It is a synchronized multi-agent driving dataset with 11,283 sequences across eight towns and 17 weather presets, covering two- and three-agent interactions, moving non-agent actors, and four camera views per vehicle. This gives the field a driving-specific setting for evaluating multi-agent world models, where earlier work relied (mostly) on game scenes.

Critical weaknesses

Unfortunately, several weaknesses undermine the aforementioned strengths:

  • Comparisons with the selected baselines do not validate the main claims. The paper argues that explicit 3D state beats implicit cross-agent association, but it compares only against Solaris and an OASIS-like baseline. Neither has a 3D component. MultiWorld and ShareVerse are mentioned among related work, yet not compared or discussed. Although the authors state that recent multi-agent models are "mostly still close-sourced", this does not hold for ShareVerse and MultiWorld (both provide code and weights). Furthermore, no ablation replaces the explicit map with an implicit 3D prior either (e.g., from VGGT).

  • Results do not fully support the claims. Authors claim that Artemis gets "almost double SSIM", but this is true only against zero-shot Solaris, which is out of domain. Once fine-tuned, the gap with Solaris becomes much smaller. Furthermore, fine-tuned Solaris exposes higher background consistency (0.953 vs 0.949) and imaging quality (0.581 vs 0.568), and subject consistency differs by 0.006. In the absence of any variance study with different seeds, the improvement in subject consistency is not statistically meaningful. Concerning ablation studies: removing the static map update leaves PSNR, SSIM, and LPIPS essentially unchanged (17.7429 vs 17.7429, 0.5666 vs 0.5662, 0.3092 vs 0.3085). The long-horizon argument fundamentally builds on this update, yet clips are only 81 frames at a quite low resolution (256×416). The "separate static map" variant isn't explained in the text, and the effect sizes are small throughout.

  • The need for MA-CARLA is not clarified or demonstrated. Although ShareVerse is cited among related work, no reference is given to the dataset it proposes and releases. There is no explanation of what MA-CARLA adds with respect to it and, furthermore, there is no reference to a possible release of the MA-CARLA benchmark upon acceptance. Furthermore, it seems no ground-truth depth, poses and masks are provided, therefore with no experiments that measure how estimation errors propagate or compare with ground-truth geometry, which a simulator could supply.

Questions for authors

Answering the following questions could convince me to revise my initial assessment:

  • Comparison with MultiWorld/ShareVerse. MultiWorld uses a frozen VGGT prior for global state and is criticized in the introduction for lacking geometric cues. Why is it not compared, given that its code is public? Comparing with it, or replacing the explicit map with a VGGT-derived implicit 3D prior, is definitely necessary to test the central claim. The same applies to ShareVerse, which is very relevant to this work. Since code and models are available, it should be compared against Artemis. Please also clarify to which recent multi-agent models the statement "are mostly still close-sourced" refers.

  • Statistical reliability. Please report variance across seeds for Table 1 and Table 2. Several differences are very small, such as subject consistency (0.902 vs 0.896) and most ablation rows. Furthermore, finetuned Solaris is better on background consistency (0.953 vs 0.949) and imaging quality (0.581 vs 0.568). This should be discussed.

  • MA-CARLA contribution and release. What does MA-CARLA offer beyond the CARLA dataset released with ShareVerse (e.g., dynamic actors, three agents, motion variety)? A comparison between the two datasets is necessary. Will MA-CARLA be released, and under what terms? Furthermore, the three-agent pool is excluded from reported experiments and shown only in Fig. 8b. Quantitative comparisons would also be necessary

Reviewer_4erzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Artemis, an interactive multi-agent driving world model grounded in an explicit 3D world state. To tackle the inconsistencies common to implicit inter-agent attention mechanisms, the authors reconstruct and incrementally maintain a shared point-cloud world map across agents. Future rollouts are steered by 24-channel control maps that disentangle static backgrounds, action-controlled 3D Gaussian foreground proxies, and uncontrolled background dynamics, which are then injected into a video DiT through a GeoAdapter module. The authors also contribute MA-CARLA, a multi-agent, multi-camera driving dataset collected in CARLA across various interaction modes and weather presets.

Critical strengths

Extending interactive world models to multi-agent driving scenarios is a valuable and challenging problem, as single-agent setups fail to reflect true collaborative or reactive behaviors in the real world. Introducing an explicit, shared 3D geometric state rather than relying solely on implicit cross-agent feature exchange is a conceptually sensible and well-motivated direction. The curated MA-CARLA dataset, featuring multi-camera views and uncontrolled background actors, also provides a meaningful testbed for this problem setting.

Critical weaknesses

However, the experimental validation falls short of supporting the core claims. The paper repeatedly stresses that existing methods suffer from poor multi-view consistency, yet no experiment directly verifies cross-agent consistency. Generating separate video rollouts for each agent's viewpoint does not guarantee that agents observe mutually consistent physical states, and neither the qualitative examples nor the quantitative metrics measure inter-agent agreement. The reported subject and background consistency metrics evaluate only the temporal stability within a single video stream, which is an entirely different property. Furthermore, the ablation results in Table 2 show negligible margins across most metrics, leaving it questionable whether modules like proxy depth, ray-maps, or static map updates offer meaningful improvements. In Table 1, the proposed method performs on par with a finetuned Solaris baseline, raising questions about whether all baselines received fair and balanced tuning on the target dataset.

The presentation also requires substantial improvement. The method section is not self-contained, introducing technical terminology well before, or entirely without proper definitions. For example, in line 300, the authors mention constructing per-block control maps by rendering static background geometry, action-controlled foreground proxies, and history-conditioned control maps, yet none of these terms (what constitutes a block, how background geometry is parameterized, how proxies are constructed and rendered, or what history maps contain) are defined at that point. Similarly, the 3D buffer mentioned in line 286 is introduced without explaining its data structure, update rules, or inference-time behavior. Figure 4 fails to clarify these gaps: the pipeline from input actions to the GeoAdapter remains ambiguous, leaving the exact definition of "degraded render", the channel layout, and the injection mechanism into the DiT completely unclear. Significant rewriting is needed to make the technical approach clear and reproducible.

Questions for authors

None.

Reviewer_yxXpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Artemis, a multi-agent driving world model that uses a shared 3D representation to improve cross-view consistency. The method reconstructs a global 3D map from multiple agents' observations and uses geometry-based control signals to guide video generation. It also introduces a progressive memory update mechanism to maintain the world state during autoregressive rollouts. The authors further build the MA-CARLA dataset and show improvements in video quality and consistency over existing multi-agent world models.

Critical strengths
  • Maintaining a consistent shared world across multiple agents is an important problem. Using explicit 3D geometry instead of relying only on cross-attention is a reasonable and promising direction.
  • The paper presents a fairly complete pipeline, covering 3D reconstruction, geometry-guided generation, and memory updates. The newly collected MA-CARLA dataset is also a useful contribution, and the improvements in video generation quality are encouraging.
Critical weaknesses
  • Progressive map updating is one of the key contributions, but removing it barely changes the results in Table 2. PSNR remains identical, and SSIM is even slightly higher without the update. This makes it unclear how much the proposed memory update actually helps.
  • The paper emphasizes shared 3D states and cross-agent consistency, but most quantitative evaluations are still based on image quality and perceptual consistency. Direct 3D geometric evaluations, such as cross-agent object position or relative pose errors, are missing. Also, the proposed framework mainly combines existing reconstruction, geometry conditioning, and video generation techniques, so stronger evidence is needed to establish the benefit of the overall design.
  • The generated videos look surprisingly close to the GT in Figures 6 and 9. Since the method renders future geometric control maps using action-derived trajectories, it is unclear how much of the result comes from the video model itself. I am also concerned about possible overfitting to the CARLA scenes and motion patterns. More importantly, Eq. (3) seems to allow target-time observations during offline preprocessing. The authors should clarify whether any future GT information enters the conditioning pipeline during training or evaluation, and provide results on unseen towns or trajectories.
Questions for authors
  • Why does removing the static map update have almost no impact on performance in Table 2? Could the authors provide longer rollouts or revisit scenarios where the benefit of progressive memory updating becomes more evident?
  • Since shared 3D consistency is the main motivation, could the authors provide direct geometric evaluations across different agents, rather than mainly relying on perceptual metrics?
  • How does the method handle errors accumulated from reconstructing generated frames and adding them back to the shared map? Also, could the authors provide quantitative results on scalability beyond two agents?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.