acceptodds
Withdrawn from ICLR 2027

Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Egocentric world models predict first-person observations conditioned on an agent’s actions, but most focus on a single agent. Real embodied settings often involve multiple agents that act and interact within a shared environment. Existing multi-agent world models generate observations for multiple agents, but rely on coarse actions like locomotion, camera control, or discrete commands, leaving fine-grained embodied interactions underexplored. We formulate embodied multi-agent world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared-environment consistency, and consistent propagation of interaction-induced state updates. We propose ME-World, which jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents’ target-view poses, and grounds generation with shared environment memory. We train and evaluate on real and synthetic multi-agent data and introduce shared-world consistency metrics for environment, update, and identity consistency. Experiments show \ours improves shared-world consistency, action control, identity preservation, and video quality over existing methods.

Primary area:
generative models
Keywords:
World model, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9QuGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ME-World, a video world model that jointly generates synchronized egocentric observations of multiple interacting agents. The paper introduces a framework combining joint multi-stream denoising, view-aligned conditioning on all agents’ motions, and shared environment memory, and benchmarks with metrics for environment and identity consistency. Validation uses separate models trained on curated CoMind recordings and a newly rendered synthetic dataset, with quantitative evaluation primarily on two-agent clips. Comparisons with released baselines, ME-World achieves improvements in cross-view consistency and action alignment. Qualitative examples further demonstrate three-agent and longer autoregressive generation.

Critical strengths
  • A meaningful problem formulation beyond single egocentric agent generation. The paper targets fine-grained interactions whose actions and outcomes must agree across multiple egocentric views, rather than merely producing videos for each agent separately, which is interesting and meaningful. This is consequential because cross-view agreement is a fundamental requirement for shared-world simulation, and the formulation brings together action alignment, environment consistency, and interaction-induced updates in a concrete, evaluable setting.

  • Broad experimental coverage across datasets, baselines, and ablations. The paper evaluates on both real CoMind recordings and a substantial synthetic corpus of paired clips, with 100 paired evaluation clips in each domain. Component ablations separately examine joint generation, shared action conditioning, and shared environment memory, while evaluation covers cross-view consistency, camera and action control, and video quality. Controlled tests also validate the proposed consistency metrics. It provides complementary evidence for the framework’s effectiveness and the roles of its main components.

Critical weaknesses
  • Limited novelty beyond combining established components. Joint generation is implemented through token concatenation and shared self-attention; shared action conditioning uses view-projected poses and camera-ray encodings; and environment memory combines geometric reprojection with shared reference images. These appear to be adaptations of established techniques rather than substantially new mechanisms. Although the conditions are tailored to the egocentric setting, the paper does not convincingly establish a deeper interaction-specific design.
  • The main comparisons do not adequately isolate architectural gains from training and conditioning advantages. ME-World is trained on task-specific paired egocentric data and receives all agents’ motion conditions and shared observation history, whereas the main baselines use released checkpoints without task-specific fine-tuning, and several generate the two streams independently with more limited action or environment inputs. Consequently, improvements—particularly in counterpart action alignment and cross-view consistency—may partly reflect additional information and task adaptation rather than superior modeling. The same-backbone, same-data MultiWorld adaptation and internal ablations mitigate this concern, but do not replace a strong independently generated baseline trained with the same data and supplied with equivalent conditions.
Questions for authors

What I am mainly concerned about is the novelty of the framework and its scalability going forward. I will consider whether to revise my score based on the other reviewers' opinions and the authors' responses in the rebuttal.

Reviewer_AM3dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies synchronized egocentric video generation for multiple interacting agents, conditioned on their future motions, initial observations, text prompts, and shared observation history. ME-World combines joint denoising of multiple ego streams, target-view pose conditioning for all agents, and shared environment memory consisting of warped historical observations and clean reference tokens. The model is fine-tuned from Cosmos-Predict2.5 2B and evaluated separately on real CoMind recordings and synthetic human-interaction data. The paper introduces environment, state-update, and identity consistency metrics and reports improvements over existing models, adapted multi-agent baselines, and ablated variants.

Critical strengths
  1. The paper addresses a meaningful interaction-aware generation problem. Fine-grained actions must simultaneously determine an agent’s own observations and its appearance in other agents’ views. The proposed combination of shared action conditioning, joint generation, and environment memory provides a coherent approach to this requirement. The real and synthetic evaluations make this a useful contribution beyond demonstrations in a single domain.

  2. Controlled comparisons provide evidence that coupling the streams is beneficial. Table 2 compares adapted multi-agent mechanisms using the same backbone, training data, and shared action conditioning. Moreover, comparing variants (7) and (8) in Table 4 isolates joint generation under otherwise shared conditions: environment consistency improves from 0.433 to 0.468, and update consistency from 0.422 to 0.466. These results support the effectiveness of cross-stream information exchange rather than attributing all improvements to additional conditioning or domain-specific fine-tuning.

Critical weaknesses
  1. The state-update evaluation does not sufficiently establish the paper’s central claim about interaction-induced state changes. In Appendix B.1.1, update regions are selected using the discrepancy between each model’s generated output and warped history, with a per-frame 80th-percentile threshold. Consequently, different methods can be evaluated on different regions, and the selected changes need not correspond to actual interaction-induced updates; they can also reflect reprojection errors or generation artifacts. Cross-view feature agreement in these regions measures consistency, but does not establish whether the appropriate object-state transition occurred. Table 5 usefully demonstrates sensitivity to introduced conflicts, but does not test consistent yet incorrect outcomes. This limitation matters because consistent propagation of interaction-induced updates is a central claimed contribution. Evaluation on common, model-independent update regions, supplemented by object-state correctness, would substantially strengthen the evidence.

  2. The real-data conditioning pipeline is insufficiently reconciled with the stated action interface. Section 3.3 describes constructing view-specific conditions from other agents’ 3D poses and relative camera geometry. Appendix C.2 instead describes extracting 2D body, hand, and facial landmarks using Sapiens2 because CoMind lacks complete whole-body annotations. It is unclear whether the real benchmark constructs all views from a unified 3D action sequence or independently extracts conditions from each future target video. The latter can provide target-view visibility and occlusion information beyond the stated action interface. Using target-video pose annotations is not inherently invalid, but this distinction materially affects whether the experiments demonstrate generation from specified multi-agent actions or reconstruction from view-specific observed motion conditions.

Questions for authors
  1. Clarification: For the real benchmark, are partner pose conditions extracted independently from each future target RGB stream, or projected from a shared 3D pose sequence? Please explain how the pipeline in Appendix C.2 implements the interface described in Section 3.3, including treatment of occluded and undetected joints.

  2. Evidence affecting the assessment: Does ME-World retain its state-update advantage when all methods are evaluated on the same update regions defined independently of their outputs? Can you provide an object-state evaluation or a diagnostic distinguishing correct updates from cross-view-consistent but incorrect outcomes?

  3. Clarification: How are the different supported output lengths—41 frames for AnyView-DVS, 61 for EgoSim, and 77 for ME-World—handled in the quantitative comparison? Are metrics, particularly FVD, computed over a common temporal extent?

Reviewer_1AhmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The approach extends egocentric video generation from one person to two persons. For evaluating realistic interactions of two persons, new metrics are proposed. The approach is evaluated on synthetic data and real data.

Critical strengths

Extending egocentric video generation of one person to two persons is a valid extension. Introducing new metrics for measuring the consistency of the generated egocentric views of both persons is also valid. The introduction clearly describes the contribution.

Critical weaknesses

Paper is not technically sound and the method is not sufficiently described:

While the introduction is well written, the method (Section 3) is poorly written. The description of the method just sketches the idea without providing details or describing the components of the method in an understandable way. The description of the method is also highly inconsistent with the LLM generated images. For example:

  • The paper uses the term "action", but it presumably refers to poses.

  • The model takes a text prompt as input, but it is never described what the text prompts are.

  • The network architecture is not described at all. It builds on Cosmos (Ali et al. 2025), but not more details are given. It mentions that positional embeddings are used, but not where they are added. The shared action conditioning is also not described in an understandable way. The description is superficial. It is mentioned that Plücker ray maps are used, but not how. The description of the shared environment memory is also not described in a way such that it is understandable what exactly has been implemented since the entire description is superficial. Given the information provided in the paper, it is impossible to reimplement the approach or to understand what has been implemented. In case of a resubmission to another venue, the entire Section 3 needs to be rewritten from scratch.

  • Figure 3 is highly inconsistent with the text in Section 3. Fig. 3 claims that the shared environment is stored as 3D reconstruction of the scene, but this is inconsistent with Section 3.4 that claims that the approach stores "selected history images ... in their original". The components shown in Fig. 3 are not described in Section 3.

The newly proposed metrics are not solid and not useful for comparisons. Instead of measuring consistency across the views, the metrics require many pre-processing steps like masking, warping and feature matching. These steps make the metrics highly sensitive to the pre-processing, which is very problematic. Fig. 10 actually shows that the masking does not work well. It is also not convincing that DINO features are used instead of comparing the consistency of the views directly. The identity consistency metric is not useful at all. It is also only reported on the synthetic dataset. The proposed metric need to be revised based on practicality.

The results are not convincing. The synthetic dataset is of very low rendering quality as shown in the suppl. video. The real dataset does not seem to have many realistic interactions. Reporting the results on the Nymeria dataset would strengthen the evaluation. There are not enough qualitative results in the suppl. video. The very few sample are also of low quality and not convincing.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.