acceptodds
Withdrawn from ICLR 2027

Who Acts, Where, and How: Unified World Model With Heterogeneous Controls

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce a unified video world model for compositional control over viewpoint, embodiment, and action. Joint learning is hindered by inconsistent control representations and sparse paired supervision. We align partially annotated controls from 18 datasets spanning robot demonstrations, human egocentric recordings, and camera-motion videos through reference-camera geometry, a unified action template, and a shared action encoder. This provides consistent physical semantics for learning from complementary sources. Visual context supplies embodiment cues without requiring explicit identifiers at inference. A hybrid Mixture-of-Transformers and Mixture-of-Experts architecture couples pretrained vision-language conditioning with noise-specialized generation. We pretrain with supervised image-to-video learning and cross-segment self-supervised video prediction, then adapt the high-noise expert using limited cross-view pairs for video re-shooting. The low-noise expert remains frozen, preserving refinement learned from heterogeneous pretraining. Experiments show strong visual fidelity and control accuracy in human and robot action-conditioned generation, camera-controlled generation, and video re-shooting. Qualitative examples further illustrate compositional action–camera control and human-to-robot transfer.

Primary area:
generative models
Keywords:
Embodied World Models, Video Generation, Action Conditioning, Camera Control, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Cm8HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a video world model on 18 heterogeneous data sources to support action control, camera control, and visually specified embodiment. Camera and effector trajectories are expressed in a reference-camera coordinate system, with a common action template and shared encoder accommodating missing annotations. A frozen VLM conditions noise-specific generation experts. Single-video pretraining is followed by cross-view adaptation of the high-noise expert while retaining the low-noise expert. Evaluation covers human and robot action-conditioned generation, camera-controlled generation, and video re-shooting; composition and human-to-robot transfer are illustrated qualitatively.

Critical strengths
  1. The control interface addresses a practical obstacle to heterogeneous world-model training. Reference-camera motion coordinates, local articulation fields, and learned null embeddings give different annotations a reasonably clear physical interpretation. Keeping missing controls distinct from stationary controls is particularly sensible. Combining these sources in one model is useful even when the individual architectural components are established.

  2. The reported task-level results are strong across several settings. For example, video re-shooting reaches 164.03 FVD versus 384.27 for TrajectoryCrafter, while also improving camera errors and PSNR (Table 3). The human and robot experiments include relevant specialized baselines rather than only general video generators. This breadth gives the system contribution some empirical substance.

Critical weaknesses
  1. The main compositional-control claim is not directly established by the quantitative evaluations. Separate action and camera benchmarks do not show that both can be changed independently while preserving the other, or that this works for held-out combinations. Figure 5 is a qualitative illustration, and the human-to-robot examples in Figure 8 additionally use a rendered target robot image, as Appendix A.3 correctly explains. These are useful demonstrations, but there is no quantitative transfer or unseen-embodiment evaluation. Since this capability is the main distinction from existing controlled generators, the gap materially limits the contribution supported by the experiments.

  2. The experiments do not isolate the two central design choices well enough. The encoder ablation keeps reference-camera alignment fixed, so it tests parameter sharing rather than the benefit of geometric alignment. The noise-expert ablation is not a comparison of high-noise-only paired adaptation against adapting both experts. Consequently, the proposed explanations for cross-domain transfer and preserving refinement remain plausible rather than demonstrated. Reproduction is also difficult: Appendix A describes the procedure but does not give numerical training budgets, source mixture sizes, routing threshold, or sampling settings needed to assess these comparisons.

Questions for authors
  1. Is there quantitative evidence for simultaneous action-camera control on held-out combinations, reporting both control errors and preservation of the factor held fixed? For embodiment transfer, which target embodiments were absent from training, and what conditions, including rendered initial images, were provided? A comparison with a joint-control method such as Uni3C would help locate the contribution.

  2. Can you provide controlled comparisons with unaligned source coordinates and with both experts adapted on the same paired data, holding data and training budget fixed? Please also specify the routing threshold, training steps and mixture sizes, and inference resolution, frame count, denoising steps, and guidance settings.

I currently lean toward weak rejection. The system and single-task results are promising, but stronger evidence for composition and the claimed alignment/adaptation mechanisms is needed for acceptance.

Reviewer_G7WGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that viewpoint, action, and embodiment should be modeled jointly rather than as separate camera-control and action-conditioned lines. It aligns controls from 18 heterogeneous datasets into a unified reference-camera frame: camera and end-effector trajectories relative to a canonical first-frame coordinate, plus a shared action encoder/template with an adaptive learned embedding for missing and unknown embodiment, no explicit embodiment IDs at inference

Critical strengths

The reference-camera parameterization plus shared action templates are precisely what makes 18 datasets with inconsistent poses, joints, and action formats jointly trainable, and the classifier-free treatment of missing controls is pragmatic — this is the most reusable contribution

Five held-out suites with distinct sources and consistent metrics, plus ablations of noise experts, the shared action encoder, and cross-segment self-supervision, cover most of the claims rather than one cherry-picked benchmark.

Critical weaknesses

The headline novelty is the least tested. The claim that embodiment is supplied by visual context and the human-to-robot transfer are shown only qualitatively in the appendix; there is no quantitative cross-embodiment table and no control condition with visual context removed.

Breadth dilutes per-task strength and scale fairness is unclear. On reshooting the model is not uniformly best, and the system is trained on 18 datasets with a Cosmos3-Nano backbone while baselines such as Wan2.2/Cosmos3 are general pretrained models. The data advantage and the amount of "limited paired videos" are never quantified, so the margins and reproducibility are hard to judge.

Questions for authors

see weakness

Reviewer_YKKKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an interactive video world model that jointly and compositionally controls who acts (embodiment), from where it is observed (viewpoint/camera), and how it acts (action trajectory). The model uses a MoT + MoE architecture with separate high-noise and low-noise denoising experts, trained in two stages: single-video I2V/V2V pretraining, followed by paired adaptation that updates only the high-noise expert on limited synchronized videos. Experiments cover robot and human action-conditioned generation, camera-controlled I2V, and video re-shooting, with ablations on self-supervision, the shared action encoder, and noise-specific experts.

Critical strengths

(1) The problem formulation is practically meaningful. Unifying embodiment, viewpoint, and action as composable controls is a sensible direction for interactive world modeling. The shared 3D reference-camera representation, unified action template, and shared encoder are reasonable attempts to absorb heterogeneous data without embodiment-specific action encoders.

(2) The paper evaluates multiple task types, i.e., robot action, human egocentric action, camera-controlled I2V, and video re-shooting, and includes ablations for cross-segment self-supervision, the shared action encoder, and noise-specific experts.

Critical weaknesses
  1. The validation is incomplete and fairness is unclear, which weakens the strong claims of “best on most metrics” and unified-framework advantage. Table 2 shows only the header, so the main camera-controlled I2V results cannot be verified; Table 1 formatting is difficult to parse. There are no variance estimates or significance tests across runs.

  2. it is unclear whether baselines receive the same control inputs, training data, and comparable training budgets. Compositional action–camera control is shown qualitatively but lacks a quantitative strong baseline. As presented, it is difficult to determine whether gains are robust and attributable to control alignment rather than data/model scale.

  3. Key methodological statements are ambiguous or inconsistent, affecting reproducibility and the credibility of the central claims. Eq. 1 appears to be a typo in the target variable. In Eq. 5, the low-noise branch conditions are inconsistent with the stated high/low-noise experts. Appendix Fig. 8 uses an additional rendered robot initial image for human-to-robot examples, raising consistency questions with the main setting.

Questions for authors

Please refer to paper weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.