acceptodds
Under review as a conference paper at ICLR 2027

World-Ego Modeling for Embodied Video Generation in Long-Horizon Navigation-Manipulation Tasks

Abstract

Embodied video world models typically capture both scene evolution and the robot's behavior, which we refer to as the world and the ego, respectively. The world and the ego exhibit different underlying dynamics: world prediction relies primarily on visual history and emphasizes scene stability, whereas ego prediction relies more strongly on the current instruction and emphasizes accurate instruction following. Modeling both components within a single generation stream can entangle these different dependencies, making it difficult to specialize the prediction of either component. Consequently, it becomes difficult to simultaneously maintain scene consistency and accurate instruction following, particularly in long-horizon navigation-manipulation tasks. In this paper, we propose to decompose an embodied video into the world and the ego and disentangle their generation processes. Specifically, we define the world as the background and currently unmanipulated objects, and the ego as the robot and currently manipulated objects. Based on this definition, we develop the World-Ego Model (WEM), which combines a vision-language state predictor using role-conditioned attention (RCA) and asymmetric query budgets with a semantic-routed mixture-of-experts (SR-MoE) diffusion generator. To enable rigorous evaluation, we further construct HTEWorld, a dataset and benchmark for long-horizon embodied video generation with hybrid navigation-manipulation tasks, providing 125K training video clips comprising over 4.5M frames with fine-grained instructions, together with 300 multi-turn evaluation trajectories covering over 2K instructions. Extensive experiments show that WEM achieves state-of-the-art performance on HTEWorld while remaining competitive on existing manipulation-oriented evaluations.

Primary area:
generative models
Keywords:
Embodied World Models, Video Generation, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_b7C8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WEM for long-horizon embodied video generation with interleaved navigation and manipulation. Its main role is to separate world and ego, so scene consistency and instruction following are not entangled in a single generation stream. Authors also build the HTEWorld dataset and benchmark.

Critical strengths

S1. The proposed solution is sound and shows clear potential. Defining world and ego by interaction role (background and unmanipulated objects versus the robot and manipulated objects) is a promising direction for embodied tasks.

S2. The evaluation is thorough. Experiments cover HTEWorld, WorldArena, and real-world DROID, with ablations of the state predictor, SR-MoE, assignment proxies, and Neighbor-Expanded Assignment; the dataset itself—125K training clips and 300 multi-turn trajectories with interleaved navigation-manipulation annotation—is a potential contribution to the community.

Critical weaknesses

W1. Separating controllable from uncontrollable dynamics and ego-motion from object dynamics has precedents in world models and video generation; the contribution here is more a systematic engineering instantiation than a novel principle. The empirical margins are also thin: on WorldArena, WEM's EWMScore is 58.10 versus CogVideoX's 57.90 (+0.20), while action-conditioned Ctrl-World reaches 59.70.

W2. The central claim is not directly proved, and the component it relies on is unreliable. The paper targets interleaved navigation and manipulation, but the real-world evaluation (DROID) covers only tabletop manipulation, so the long-horizon navigation-manipulation setting lacks real-world evidence.

Questions for authors

The paper’s central claim targets interleaved navigation and manipulation, but the real-world evaluation on DROID covers only tabletop manipulation. Could the authors either provide real-world evidence for the long-horizon navigation–manipulation setting?

1 comment

We thank Reviewer b7C8 for the careful and constructive assessment. We appreciate the reviewer’s recognition of our World-Ego decomposition and the HTEWorld benchmark, as well as the positive assessment of our experimental evaluation. We address the weaknesses (W) and question (Q) below.

W1: Principle Novelty and WorldArena Results.

Principle Novelty. The contribution of this work is to investigate the distinct dynamics of environment evolution and robot interaction in long-horizon tasks with interleaved navigation and manipulation, and to decompose embodied videos into the World and the Ego according to their current interaction roles. We further identify that the two exhibit different conditioning dependencies. Modeling them jointly in a single generation stream makes it difficult to simultaneously maintain scene consistency and instruction following. Therefore, we separately model their evolution to mitigate the entanglement between these different conditioning dependencies.

WorldArena Results. The comparison with CogVideoX (58.10 vs. 57.90) does not isolate the contribution of World-Ego decomposition, as the two methods use different video-generation backbones. To provide a more controlled comparison, we additionally evaluate PAN on WorldArena using the same Wan2.2 backbone as WEM. WEM achieves an EWMScore of 58.10, outperforming PAN (56.65) by 1.45 points. This result further demonstrates the effectiveness of World-Ego decomposition in short-horizon tabletop manipulation tasks.

W2/Q: Real-world Long-horizon Navigation-Manipulation Generation.

To further evaluate the proposed World-Ego decomposition in real-world long-horizon navigation-manipulation generation, we conduct additional experiments on the real-world Galaxea Open-World dataset [1], which contains real-world trajectories involving both navigation and manipulation.

Experimental Setup. We select 3,637 episodes from the Galaxea Open-World dataset, comprising approximately 90K video clips. Following an episode-level split, 3,337 episodes are used for training, while 300 episodes (approximately 6.5K video clips) are held out for testing, with no overlap between the two sets. From the 300 held-out episodes, we construct 500 multi-turn evaluation trajectories, each containing 3–12 instruction rounds and involving both navigation and manipulation, resulting in approximately 5K instruction rounds in total. We compare WEM with PAN [2], using Wan2.2 as the video-generation backbone for both methods. We follow the same training and evaluation protocols as those used for HTEWorld in Appendix B.

Table R1: Results on Galaxea Open-World. PAN and WEM use the Wan2.2 backbone. Higher EWMScore is better.

ModelEWMScore ↑
PAN53.59
WEM57.18

Results and Analysis. WEM achieves an EWMScore of 57.18, outperforming PAN (53.59) by 3.59 points. These results provide real-world evidence for the effectiveness of the proposed World-Ego decomposition in long-horizon navigation-manipulation generation. We include these additional experiments in Appendix E.3.


We hope these new controlled experiments and clarifications address the concerns and welcome further feedback.

References

[1] Tao Jiang et al. Galaxy Open-World Dataset and G0 Dual-System VLA Model. In 2026 IEEE International Conference on Robotics and Automation (ICRA), pp. 20749–20756, 2026.

[2] Jiannan Xiang et al. PAN: A World Model for General, Interactable, and Long-Horizon World Simulation. arXiv:2511.09057, 2025.

Reviewer_GAdzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WEM, which separates world and ego conditioning states and routes video tokens to specialized generation experts. The ego comprises the robot and currently manipulated objects, so object roles can change across interactions. WEM combines role-conditioned VLM queries, semantic mask prediction, and overlapping routing near boundaries. The paper also introduces HTEWorld for multi-turn navigation-manipulation video generation and evaluates on WorldArena and DROID.

Critical strengths

1、The decomposition is operationally meaningful, and the shared early blocks plus specialized late blocks provide a sensible way to preserve global interaction while allowing role-specific parameters. Neighbor-expanded routing directly addresses boundary discontinuities. 2、The controlled PAN comparisons are useful. WEM improves HTEWorld EWMScore from 58.40 to 61.48 with Wan2.2 and from 56.58 to 60.11 with Wan2.1; the same-backbone DROID comparison also favors WEM (Tables 1 and 3). 3、The evaluation and ablations address several obvious confounds: mask supervision alone, state-predictor removal, token separation, expert specialization, and assignment type. The appendix also validates predicted masks, evaluates six multi-turn diagnostics, and reports correlations with human ratings. It explicitly distinguishes video generation from closed-loop robot control.

Critical weaknesses
  1. Additional model capacity remains a confound in attributing the benefit to role specialization. WEM duplicates the final six of thirty DiT blocks and adds a semantic head. The PAN plus mask-supervision control addresses extra labels, but does not match expert parameter capacity. Under semantic assignment, removing expert specialization reduces EWMScore from 61.48 to 58.85, while removing token separation only reduces it to 61.09 (Table 5). This suggests that the additional specialized parameter paths account for much of the gain, but does not establish why semantic specialization is preferable to a capacity-matched generic alternative. Parameter counts, active compute, and an appropriate two-branch control would clarify the contribution.
  2. The long-horizon and statistical evidence is still mostly aggregate. The 300 evaluation trajectories are selected from only fifty held-out parent episodes across ten tasks (Appendix A.1), so trajectory outcomes may be dependent. No confidence intervals or training-seed variation are reported for the main improvements or the smaller ablation differences. Aggregate EWMScore and the six diagnostics also do not show how the advantage changes with instruction round or navigation-manipulation transition count. The existing human ratings are valuable, but only metric correlations are reported; per-system human scores would provide a more direct validation of the claimed improvement.
  3. Several details that determine the decomposition are not sufficiently specified for reproduction. The interaction-consistency procedure lists proximity, overlap, motion, confidence, and temporal cues without giving an executable decision rule. The spatial expansion radius and exact history sampling or truncation policy also need to be specified. These details affect which objects are treated as ego, expert boundary behavior, and access to distant history. In addition, the motion-based alternative relies on approximate homography compensation, so its lower score is evidence about the implemented proxy, not a clean comparison of semantic versus motion-based decompositions.
Questions for authors
  1. How much of the gain remains against a model with comparable total parameters and active computation, such as a generic two-branch generator? Please report parameter and inference-cost differences for WEM and PAN.
  2. Can you provide paired uncertainty estimates clustered by parent episode, together with performance versus instruction round? From the existing human study, what are the per-system mean ratings and uncertainty for perceived completion and realism?
  3. Please specify the interaction-role labeling rule, neighbor-expansion radius, and history sampling/truncation policy. How much does assignment error increase when an object changes between world and ego roles?
4 comments
AuthorsOpenReview ↗

Response to W1/Q1 (Ablating the effect of additional model capacity on WEM)

W1 & Q1: Ablating the effect of additional model capacity on WEM.

(1) Parameter- and compute-matched comparisons. To isolate the effect of additional model capacity on WEM's performance, we introduce three baselines matched to WEM in parameter count and inference computation.

  • Wider single-stream: We retain the other components of WEM but remove the World-Ego expert architecture and token routing to construct a single-stream generator. We widen the final six DiT blocks: attention width 3072 →6144, attention heads 24 → 48, and FFN intermediate width 14336 → 28672, matching WEM in parameter count and inference computation. The first 24 blocks, residual/token dimension (3072), and per-head dimension (128) remain unchanged.

  • Randomly routed two-expert MoE: We retain WEM's World-Ego expert architecture but replace semantic routing with random routing. Specifically, we randomly shuffle the World-Ego token assignments before routing and route tokens according to the shuffled assignments, while preserving the original token sequence order.

  • Non-semantic two-expert MoE: We retain WEM's World-Ego expert architecture but remove token routing. Specifically, we feed the complete token sequence output by the Preceding Expert to both the World Expert and Ego Expert, and combine their outputs through element-wise averaging.

We conduct these experiments on HTEWorld. All three baselines use Wan2.2-TI2V-5B [1] as the backbone and are trained using the same training data and configuration as WEM (Wan2.2).

Table R2-1: Parameter- and compute-matched comparisons on HTEWorld. All variants use Wan2.2. Higher EWMScore is better.

ModelEWMScore ↑
PAN58.40
Wider single-stream59.38
Randomly routed two-expert MoE56.65
Non-semantic two-expert MoE58.59
WEM61.48

Analysis. Under matched parameter and computation budgets, WEM achieves a higher EWMScore (61.48) than the wider single-stream generator (59.38) and the two generic two-branch controls, namely the randomly routed two-expert MoE (56.65) and the non-semantic two-expert MoE (58.59). These results indicate that although additional model capacity can provide some performance gains, WEM's improvement primarily comes from World-Ego semantic specialization, rather than simply increasing model capacity or adopting a two-expert architecture.

(2) Parameter and inference-cost differences. Compared with PAN [2], WEM increases the parameter count of the video generator by 23.2% (including the Semantic Head) and inference computation by approximately 21.8%. Considering the complete model, the parameter count and inference computation increase by 18.6% and 17.5%, respectively. Although WEM has higher model capacity and inference computation than PAN, the results in Table R2-1 show that its performance gains primarily come from World-Ego semantic specialization, rather than the additional model capacity and computation themselves.

AuthorsOpenReview ↗

Response to W2/Q2 ( Statistical reliability and human evaluation)

W2 & Q2: Statistical reliability and human evaluation.

(1) Paired uncertainty estimates clustered by parent episode. To quantify the statistical uncertainty of our results, we conduct paired uncertainty analysis for the main comparison between WEM (Wan2.2) and PAN (Wan2.2), as well as the comparison between WEM and w/o Token Separation in Table 5, where the performance difference is relatively small. Specifically, we perform 10,000 bootstrap resamples using the 50 held-out parent episodes in HTEWorld as the sampling units. For each resample, we retain all evaluation trajectories belonging to the sampled episodes, calculate the paired EWMScore differences between the two models on the same trajectories, and average these differences to obtain the mean paired difference. We report the mean paired difference and its 95% percentile confidence interval across the bootstrap resamples.

Table R2-2: Paired uncertainty estimates with parent-episode clustering.

ComparisonMean paired difference95% CI
WEM − PAN+3.0796[2.5691, 3.7566]
WEM − w/o Token Separation+0.3939[0.0063, 0.7670]

WEM achieves a mean paired improvement of 3.0796 points over PAN, with a 95% confidence interval entirely above zero, indicating that its performance advantage remains stable after accounting for correlations among trajectories from the same parent episode. In comparison, the mean improvement of WEM over w/o Token Separation is relatively small, at 0.3939 points (95% CI: [0.0063, 0.7670]). This is because w/o Token Separation retains WEM's core design of separately modeling the world and ego using different generation parameters and conditioning states, while removing only token separation within the experts. This further supports the importance of World-Ego specialization, with Token Separation providing additional gains by reducing interference between tokens of different roles.

(2) Performance versus instruction round. To further analyze generation performance across instruction rounds, we collect all video chunks corresponding to each instruction round across the evaluation trajectories and calculate the EWMScore of PAN and WEM on these chunks. As shown in Table R2-3, PAN and WEM perform comparably in the early rounds. However, as the instruction round progresses, the performance gap gradually increases. From the fifth round onward, WEM consistently outperforms PAN by 1.96–4.01 points. In particular, at the twelfth round, PAN's EWMScore drops to 56.20, whereas WEM maintains a score of 59.74, outperforming PAN by 3.54 points. These results indicate that WEM better maintains generation quality, scene consistency, and instruction following across successive instruction rounds.

Table R2-3: EWMScore at different instruction rounds. Higher is better.

Model3456789101112
PAN61.7260.9259.6758.4358.2657.9857.7057.2857.0556.20
WEM61.5661.3461.6361.8361.7361.4961.2161.2960.4559.74

(3) Per-system human ratings. We report the human ratings of the seven systems in Table R2-4 across two five-point evaluation dimensions: Perceived Task Completion and Visual Realism. Specifically, we first average the ratings from 15 annotators for each trajectory and then calculate the mean across the 300 evaluation trajectories for each system. We obtain the 95% confidence intervals through 10,000 parent-episode-clustered bootstrap resamples, using the 50 held-out parent episodes as the sampling units. In each resample, we sample parent episodes with replacement, retain all evaluation trajectories belonging to the sampled episodes, and recompute the mean rating for each system. The 95% percentile confidence intervals are calculated from the resulting bootstrap estimates, reflecting trajectory-sampling uncertainty conditional on the fixed set of annotators. With the Wan2.2 backbone, WEM improves over PAN by 0.183 points in Perceived Task Completion and 0.194 points in Visual Realism. These results show that WEM not only achieves better automatic evaluation metrics but also improves perceived task completion and visual realism according to human evaluations.

Table R2-4: Per-system human evaluation results, reported as mean [95% CI].

SystemPerceived Task Completion ↑Visual Realism ↑
WoW-7B3.305 [3.263, 3.348]3.370 [3.326, 3.414]
Cosmos-Predict2.5-2B3.307 [3.266, 3.348]3.435 [3.394, 3.477]
Cosmos-Predict2.5-14B3.437 [3.396, 3.476]3.375 [3.335, 3.415]
PAN (Wan2.1)3.533 [3.492, 3.573]3.418 [3.373, 3.463]
WEM (Wan2.1)3.610 [3.566, 3.655]3.612 [3.569, 3.655]
PAN (Wan2.2)3.567 [3.525, 3.609]3.548 [3.502, 3.593]
WEM (Wan2.2)3.750 [3.708, 3.793]3.742 [3.700, 3.783]
AuthorsOpenReview ↗

Response to W3/Q3 (Implementation details)

W3 & Q3: Implementation details.

(1) Interaction-role labeling rule. Based on the original dataset annotations and the procedures described in Appendices A, we obtain per-frame instance segmentation masks, optical-flow information, and trajectory-level candidate manipulated objects. However, candidate manipulated objects refer to objects that have been manipulated at some point in a trajectory and do not necessarily belong to the Ego in every frame. The purpose of the interaction-role labeling rule is to determine the actual role of each object in each frame, which we achieve through interaction-consistency filtering. Specifically, we first filter candidate instance masks based on mask confidence, effective area, and connected-component size. We then determine whether an object is involved in the current interaction based on two types of evidence:

  • Gripper-contact evidence: We first use proximity to identify candidate objects near the gripper, and then check whether their masks overlap with the gripper region.
  • Motion-consistency evidence: We examine whether the candidate object exhibits consistent motion with the robot or gripper.

A candidate object is considered currently manipulated only when both the gripper-contact and motion-consistency criteria satisfy their respective thresholds. After making these frame-wise decisions, we further apply temporal consistency filtering to suppress transient role changes and errors caused by unstable tracking. Finally, the robot, gripper, and currently manipulated objects are assigned to the Ego, while all remaining pixels are assigned to the World. Thus, the same object can dynamically switch between the World and Ego according to its interaction state. This pipeline reuses existing instance segmentation, tracking, and motion information without requiring additional model inference. We have added a detailed description of this procedure and executable pseudocode to the revised Appendix A.3 to improve reproducibility.

(2) Neighbor-expansion radius. In Section 3.3, Neighbor-Expanded Assignment uses a spatial expansion radius of one latent token. Specifically, for each of the World and Ego token index sets, we include spatial neighbors within one token of the original region boundary in the same latent frame in the expanded set. Consequently, tokens near the World-Ego boundary are assigned to both the World Expert and Ego Expert, whereas tokens farther from the boundary remain assigned to their respective experts. This overlapping assignment provides cross-boundary spatial context to both experts, reducing generation discontinuities and temporal flickering near the boundary. We have added this implementation detail to the revised Section 3.3.

(3) History sampling/truncation policy. We follow PAN's multi-turn generation history construction. For the kk-th generation round, the State Predictor receives the initial observation, the previously retained instruction-video history, and the current instruction. We do not perform additional frame-level sampling on historical videos; instead, the complete retained video chunks are directly provided to the State Predictor. Regarding history truncation, we retain at most 12 historical video chunks and their corresponding instructions. When the history exceeds this limit, the oldest instruction-video pairs are discarded sequentially, while the initial observation is always preserved. We have clarified this history input and truncation policy in the revised Appendix B.2.

(4) Interpretation of the motion-based comparison. The reliability of proxy acquisition during data annotation is itself an important consideration when evaluating an assignment pipeline. Our semantic-based pipeline offers an advantage in this regard: World-Ego proxies can be reliably obtained using commonly available annotations, off-the-shelf models, and predefined filtering rules. In contrast, the motion-based proxy in our comparison may be affected by the approximation errors introduced by homography compensation.

AuthorsOpenReview ↗

Response to W3/Q3 (Assignment error under dynamic role transitions)

(5) Assignment error under dynamic role transitions. The original Appendix E.1 reports the overall World-Ego assignment accuracy of the Semantic Head on the HTEWorld validation set. Building on this evaluation, we conduct an additional stratified analysis on all evaluation trajectories to assess assignment quality during dynamic role transitions. Specifically, using the ground-truth World-Ego masks, we identify the time points at which an object's semantic role changes and partition the evaluation frames into two groups: frames containing an object role change are categorized as Transition, while all remaining frames are categorized as Stable. Under the same prediction setting, we then calculate the mean mIoU of the Semantic Head's predicted masks for each group.

Table R2-5: World-Ego assignment quality on stable and transition frames.

Frame categoryProportion of evaluation framesmIoU ↑
Stable95.2%81.6%
Transition4.8%75.8%

The mIoU is 81.6% on stable frames and 75.8% on transition frames, corresponding to a decrease of 5.8 percentage points. This indicates that semantic assignment becomes more challenging when objects switch between world and ego roles. Nevertheless, the Semantic Head still achieves an mIoU of 75.8% on these more challenging transition frames, demonstrating its ability to adapt to dynamic changes in object interaction roles.


We hope these new controlled experiments and clarifications address the concerns and welcome further feedback.

References

[1] Team Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[2] Jiannan Xiang et al. PAN: A World Model for General, Interactable, and Long-Horizon World Simulation. arXiv:2511.09057, 2025.

Reviewer_wRq3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the World-Ego Model (WEM), which decomposes embodied videos into a world component containing the background and unmanipulated objects, and an ego component containing the robot and manipulated objects. WEM uses role-conditioned state queries and a semantic-routed mixture-of-experts generator to model the two components separately. The paper also introduces HTEWorld, a benchmark for long-horizon navigation-manipulation video generation. Experiments on HTEWorld, WorldArena, and DROID show improved video quality, scene consistency, and instruction following.

Critical strengths
  1. WEM provides a clear and intuitive decomposition of embodied videos into world and ego components, with specialized state extraction and generation modules for their different contextual dependencies.

  2. The paper introduces HTEWorld, a large-scale benchmark for long-horizon, multi-turn navigation-manipulation video generation, and provides extensive evaluation on both the new benchmark and existing datasets.

Critical weaknesses
  1. The main improvement is not isolated from increased model capacity. WEM duplicates the final six DiT blocks into separate World and Ego Experts, while the world-ego state decomposition alone improves EWMScore only from 58.40 to 58.69. Without a parameter- and compute-matched wider single-stream or non-semantic MoE baseline, it is unclear whether the gain to 61.48 comes from world-ego specialization or simply additional expert capacity.

  2. Evidence for long-horizon hybrid navigation-manipulation is largely limited to the authors’ new simulated benchmark. WorldArena and DROID mainly evaluate manipulation-oriented video generation, and no downstream planning or control experiment is provided. Thus, improvements in video metrics and perceived task completion do not establish that WEM provides a more useful world model for embodied decision-making.

Questions for authors
  1. Please provide parameter- and compute-matched comparisons with a wider single-stream generator and a non-semantic or randomly routed two-expert MoE. These controls are necessary to determine whether the gains arise from world-ego specialization rather than increased expert capacity.

  2. Please clarify whether WEM-generated videos improve downstream planning, action selection, or policy evaluation. Such evidence would materially strengthen the claim that WEM is an effective embodied world model beyond improving video-generation metrics.

3 comments
AuthorsOpenReview ↗

Response to W1/Q1 (Parameter- and compute-matched comparisons)

We thank Reviewer wRq3 for the careful and constructive assessment. We appreciate the reviewer’s recognition of our world-ego decomposition and the HTEWorld benchmark. We address the weaknesses (W) and questions (Q) below.

W1 & Q1: Parameter- and compute-matched comparisons.

To isolate the effect of increased model capacity on WEM's performance gain, we introduce three additional baselines matched to WEM in parameter count and computation.

  • Wider single-stream: We retain the other components of WEM but remove the World-Ego expert architecture and token routing to construct a single-stream generator. We widen the final six DiT blocks: attention width 3072 →6144, attention heads 24 → 48, and FFN intermediate width 14336 → 28672, matching WEM in parameter count and inference computation. The first 24 blocks, residual/token dimension (3072), and per-head dimension (128) remain unchanged.

  • Randomly routed two-expert MoE: We retain WEM's World-Ego expert architecture and routing mechanism, but replace semantic routing with random routing. Specifically, we randomly shuffle the World-Ego token assignments before routing and route tokens according to the shuffled assignments, while preserving the original token sequence order. The random assignments remain fixed throughout all denoising steps within each video chunk.

  • Non-semantic two-expert MoE: We retain WEM's World-Ego expert architecture but remove token routing. Specifically, we feed the complete token sequence output by the Preceding Expert to both the World Expert and Ego Expert, and combine their outputs through element-wise averaging.

We conduct these experiments on HTEWorld. All three baselines use Wan2.2-TI2V-5B [1] as the backbone and are fully trained using the same training data and configuration as WEM (Wan2.2).

Table R3-1: Parameter- and compute-matched comparisons on HTEWorld. All variants use Wan2.2. Higher EWMScore is better.

ModelEWMScore ↑
PAN58.40
Wider single-stream59.38
Randomly routed two-expert MoE56.65
Non-semantic two-expert MoE58.59
WEM61.48

Analysis. The wider single-stream baseline achieves an EWMScore of 59.38, improving over PAN by 0.98 points, but remains 2.10 points below WEM. This indicates that simply increasing model capacity is insufficient to explain WEM's performance gain. The non-semantic two-expert MoE achieves 58.59, only 0.19 points higher than PAN, indicating that introducing two experts without World-Ego-based token routing provides only marginal improvement. The randomly routed two-expert MoE achieves 56.65, 1.75 points lower than PAN and 4.83 points lower than WEM, further demonstrating that disrupting the correspondence between token assignments and their World-Ego roles degrades generation performance. Overall, these parameter- and compute-matched comparisons demonstrate that WEM's performance gains arise primarily from world-ego specialization rather than increased expert capacity.

AuthorsOpenReview ↗

Response to W2/Q2 (Downstream planning and action selection)

W2 & Q2: Downstream planning and action selection.

To investigate whether WEM is an effective embodied world model, we follow the simulative reasoning setting in World Reasoning Arena [3] and conduct a closed-loop skill planning experiment using the simulation scenes and object assets from BEHAVIOR-1K in the OmniGibson simulator. This experiment evaluates whether WEM-generated future videos improve downstream planning and action selection.

Experimental design. Given the robot's initial head-camera observation, the final task goal, and a predefined set of atomic skills, the robot uses a vision-language model (VLM) to plan by repeatedly selecting and executing skills until the task goal is satisfied or the maximum number of planning rounds KK is reached. The predefined skill set covers various atomic skills, including MOVE_TO, PICK_UP, PLACE_IN, WAIT, and others. Each skill has a corresponding textual description that can be directly interpreted by the VLM and is mapped to low-level robot actions through a predefined program. After each skill execution, the robot receives a new head-camera observation and updates its execution history for the next planning round.

We compare three methods:

  • VLM-only: At each planning round, the VLM directly selects the next skill based on the current head-camera observation, task goal, execution history, and candidate skills, without using a video generation model.

  • PAN-based planning: At each planning round, the VLM first proposes three candidate atomic skills and generates corresponding video-conditioning text for each candidate. PAN predicts a future head-camera video for each candidate based on its corresponding video-conditioning text. The VLM then compares these counterfactual futures and selects the next skill. The robot executes only the selected skill, while the unselected predicted branches are discarded.

  • WEM-based planning: We follow the same planning procedure as PAN-based planning, but use WEM to predict the future head-camera videos corresponding to the three candidate skills. The VLM compares the predicted futures and selects the next skill, which is then executed by the robot.

Experimental setup. We design 20 different tasks, each with 10 randomly initialized initial states, resulting in 200 evaluation trajectories. All methods use the same atomic skill set, VLM, and skill executor. We use Qwen3.8-Flash [2] as the VLM.

Table R3-2: Closed-loop planning results on 200 evaluation episodes across 20 tasks. Higher Task Success Rate indicates better performance.

MethodSuccessful EpisodesTask Success Rate ↑
VLM-only95/20047.5%
PAN116/20058.0%
WEM134/20067.0%

Analysis. Compared with VLM-only, WEM improves the task success rate from 47.5% to 67.0%, an increase of 19.5 percentage points. This demonstrates that WEM-generated videos improve downstream planning and action selection. Furthermore, compared with PAN, WEM improves the task success rate from 58.0% to 67.0%, an additional gain of 9.0 percentage points. These results demonstrate that WEM not only improves video-generation metrics but also enhances the practical utility of generated videos for downstream embodied planning and action selection. We include this experiment in Section 5.2 and Appendix E.4 and show in Fig. 8 an example of how WEM-generated future videos assist the VLM in skill selection.

We hope these new controlled experiments address the concerns and welcome further feedback.

References

[1] Team Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[2] An Yang et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.

[3] PAN Team et al. World Reasoning Arena. arXiv:2603.25887, 2026.

The response addresses my main concerns through matched baselines and a downstream planning experiment. These additions strengthen the paper, and I maintain my rating of 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To AC: Rebuttal Summary

Dear AC, SAC, PC,

Thank you very much for your time and for supporting the review process. We sincerely appreciate the reviewers' constructive feedback and valuable suggestions. During the rebuttal, we addressed the main concerns raised by all reviewers and updated the manuscript accordingly, with newly added content highlighted in blue.

The reviewers recognized the clear and intuitive World-Ego decomposition, the value of the HTEWorld benchmark, and the comprehensive experimental evaluation. Their main concerns were:

  • Lack of parameter- and compute-matched baselines.
    • We added three matched baselines, demonstrating that WEM outperforms the strongest baseline by 2.10 EWMScore points.
  • Lack of downstream planning evaluation.
    • We added a closed-loop skill planning experiment, where WEM achieves 67.0% task success rate, outperforming VLM-only (47.5%) by 19.5 percentage points and PAN (58.0%) by 9.0 percentage points.
  • Lack of real-world long-horizon navigation-manipulation evidence.
    • We added experiments on the real-world long-horizon navigation-manipulation dataset Galaxea Open-Worl*, where WEM outperforms PAN by 3.59 EWMScore points.

Below is a concise summary of our responses.


Reviewer wRq3 (Initial Score: 3)

  • Concern: Lack of parameter- and compute-matched baselines.
    • We added three matched baselines: Wider single-stream, Randomly routed two-expert MoE, and Non-semantic two-expert MoE. WEM achieves an EWMScore of 61.48, outperforming the strongest baseline (59.38) by 2.10 points. (Table R3-1)
  • Concern: Lack of downstream planning and action selection evaluation.
    • We added a closed-loop skill planning experiment, where WEM achieves a task success rate of 67.0%, outperforming VLM-only (47.5%) by 19.5 percentage points and PAN (58.0%) by 9.0 percentage points. (Table R3-2, Section 5.2, Appendix E.4)

We appreciate the reviewer’s follow-up confirmation that these additions address the main concerns and strengthen the paper.


Reviewer GAdz (Initial Score: 3)

  • Concern: Lack of parameter- and compute-matched baselines.
    • Parameter- and compute-matched comparisons. We added three baselines: Wider single-stream, Randomly routed two-expert MoE, and Non-semantic two-expert MoE. WEM achieves an EWMScore of 61.48, outperforming the strongest baseline (59.38) by 2.10 points. (Table R2-1)
  • Concern: Insufficient statistical analysis and per-system human evaluation.
    • Paired uncertainty estimates clustered by parent episode. We added 10,000 parent-episode-clustered bootstrap resamples. WEM achieves a mean paired improvement of 3.0796 points over PAN (95% CI: [2.5691, 3.7566]) and 0.3939 points over w/o Token Separation (95% CI: [0.0063, 0.7670]). (Table R2-2)
    • Performance versus instruction round. We added EWMScore across instruction rounds. (Table R2-3)
    • Human ratings. We added human evaluation results. With Wan2.2, WEM improves over PAN by 0.183 in Perceived Task Completion and 0.194 in Visual Realism. (Table R2-4)
  • Concern: Insufficient implementation details for reproducibility.
    • Interaction-role labeling rule. We clarified the labeling rule based on gripper-contact evidence, motion-consistency evidence, and temporal consistency filtering, and added executable pseudocode. (Appendix A.3)
    • Neighbor-expansion radius. We specified a spatial expansion radius of one latent token. (Section 3.3)
    • History sampling/truncation policy. We clarified that at most 12 historical video chunks are retained. (Appendix B.2)
  • Concern: Assignment errors under dynamic role transitions.
    • We added a stratified evaluation, where the Semantic Head achieves an mIoU of 81.6% on stable frames and 75.8% on transition frames. (Table R2-5)

Reviewer b7C8 (Initial Score: 3)

  • Concern: Principle novelty and WorldArena results.
    • Principle novelty. We clarified that WEM exploits the distinct dynamics of environment evolution and robot interaction by decomposing embodied videos according to semantic interaction roles and decoupling their generation processes.
    • WorldArena results. We added a controlled comparison with PAN on WorldArena. WEM achieves an EWMScore of 58.10, outperforming PAN (56.65) by 1.45 points.
  • Concern: Lack of real-world long-horizon navigation-manipulation evidence.
    • We added experiments on the real-world long-horizon navigation-manipulation dataset Galaxea Open-World. WEM achieves an EWMScore of 57.18, outperforming PAN (53.59) by 3.59 points. (Table R1, Appendix E.3)

We sincerely appreciate all reviewers for their constructive feedback and valuable suggestions. We are especially grateful to the AC, SAC, and PC for their time and dedication throughout the review process. We believe the additional experiments and analyses have substantially strengthened the paper.

Thank you very much for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.