acceptodds
Under review as a conference paper at ICLR 2027

WoCA: World Simulation with Compositional Assets

Abstract

World simulation aims to model how environments evolve and predict alternative futures with different possible outcomes. Representing these possibilities calls for a model that retains object appearance, captures how geometric states evolve, and renders the resulting observations. In this paper, we propose World Simulation with Compositional Assets (WoCA), a new object-centric world simulation approach with three core components: a set of reusable appearance assets, a trajectory diffusion model, and an asset-conditioned video renderer. We extract reusable object appearance tokens from reference observations and bind them to time-varying poses. A separate trajectory diffusion model predicts future object states and interactions from history states and visual context. To render these assets under new states, our video renderer combines global token conditioning with localized spatial features, connecting each object's stored appearance to its location in the generated video. This disentangled factorization makes appearance and motion separately addressable: object appearances can be reassigned while pose trajectories are held fixed, and trajectories can be edited while the same appearance assets are maintained. It also provides an explicit object memory that remains available across recursive state predictions for long-horizon world simulation. WoCA therefore brings video generation closer to an editable world representation where persistent entities can participate in multiple possible futures. Experiments on MOVi-D and Waymo Open show improved video quality over Neural Assets under appearance and trajectory control and demonstrate long-horizon simulation with persistent appearance assets.

Primary area:
generative models
Keywords:
object-centric world simulation, controllable world simulation, world representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vTUuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors notice the challenge of preserving object identity while independently controlling appearance and motion in world simulation. To address it, they propose WoCA, an object-centric framework comprising persistent appearance assets, a trajectory diffusion model, and an asset-conditioned video renderer, disentangling appearance and motion. Appearance features extracted from reference images are paired with object poses, while the trajectory model predicts future 3D states, and the renderer combines global conditioning with localized spatial features. Experiments on MOVi-D and Waymo Open show improved video quality over Neural Assets across controlled rendering tasks, alongside a long-horizon generation example.

Critical strengths
  1. The paper addresses an important problem in generative world simulation: preserving object identity while consistently associating appearance and motion with the correct objects. Reliable control over these factors is valuable for scenario editing and exploring alternative futures, especially in rare or difficult-to-capture settings where synthetic data could broaden training coverage and help improve the robustness of downstream models.

  2. The proposed method is rational and well-motivated. Maintaining reusable appearance assets independently of evolving object states supports appearance replacement, trajectory editing, and recursive prediction with a shared visual reference, and Global tokens provide access to object appearances and poses, while localized spatial features associate this information with the intended image regions in a complementary manner.

  3. The evaluation is comprehensive, covering original generation, appearance swapping, and trajectory editing on both synthetic object-interaction and real-world driving data

Critical weaknesses
  1. Separating appearance and motion at the input level does not guarantee faithful control in the generated video. Training primarily relies on reconstruction of original videos, without explicit supervision enforcing appearance preservation under edited trajectories.

  2. The evaluation emphasizes video quality rather than editing fidelity. The reported metrics do not directly establish whether appearances are assigned to the correct objects, trajectories are accurately followed, or unedited content remains unchanged. The reference videos used for PSNR, SSIM, and LPIPS under Swap/Edit also require clarification.

  3. The main comparison is largely limited to Neural Assets, implemented with the video backbone’s temporal components removed. Some improvements may therefore reflect the benefits of temporal modeling rather than the proposed conditioning design specifically.

  4. The framework depends on substantial geometric annotation, including track identities, object boxes, and calibrated cameras; motion prediction additionally requires initial 3D object states. It is better to show how robust it is to the noise and how to deal with the error

Questions for authors

The computational efficiency of the full simulation pipeline remains unclear. The trajectory model uses 50 sampling steps for each four-frame continuation, while video rendering uses 40 sampling steps per window. Although these sampling configurations are provided, end-to-end latency, throughput, and inference memory usage are not reported.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vTUu

Thank you for your valuable feedback! Below, we address your concerns point by point.

Weakness 1. Separating appearance and motion at the input level does not guarantee faithful control in the generated video.

Our method learns appearance reuse across poses through source-to-target reconstruction: reference appearance is paired with target-frame geometry, while foreground masking prevents the conditioning image from directly supplying the target objects’ appearance. This formulation explicitly enforces appearance preservation under edited trajectories, which is one of the core contributions of our method. We have also empirically verified the resulting control on edited outputs in new Table 1 and new Figure 3 in the updated manuscript.

Weakness 2. Evaluation of editing fidelity and clarification of PSNR, SSIM, and LPIPS under Swap/Edit.

The reference for PSNR, SSIM and LPIPS is the original GT video, since counterfactual GT videos are unavailable. These metrics characterize full-reference visual fidelity, reflecting both preservation of unchanged scene content and differences introduced by intentional edits, rather than directly measuring editing correctness. Below, we complement these results with explicit identity-preservation and spatial-control evaluations.

We have added identity preservation and spatial control evaluations in new Figure 3 in the updated manuscript. The results show that WoCA achieves the highest identity similarity and box Recall@ 0.5 on both datasets. WoCA also achieves the highest box mIoU on MOVi-D; on Waymo Open, its mIoU is slightly below the new VACE baseline, while providing higher identity similarity and recall. These results support improved identity preservation and object placement across both synthetic and real-world scenes.

We have also conducted an additional ablation study to isolate the effect of video editing by evaluating unedited and edited outputs against the same edited target boxes in new Table 1 in the updated manuscript. Applying the edit increases box mIoU from 0.2975 to 0.5962 and Recall@ 0.5 from 14.93% to 82.02%, demonstrating that the generated objects follow the requested changes. Meanwhile, identity similarity remains close to that of unedited reconstruction evaluated under original targets (0.5854 vs. 0.6158).

Together, these results support WoCA's ability to modify object motion while largely retaining source identity, without requiring counterfactual ground-truth videos for evaluation.

Weakness 3. Insufficient baselines.

As suggested by Reviewer 6NGh, We have added a video-editing baseline VACE in updated Figure 2 and new Figure 3 in the updated manuscript. For fair comparison, we finetune the official VACE checkpoint with LoRA and construct coarse RGB conditioning videos by placing reference-object crops at the prescribed projected boxes over the initial background. These videos are supplemented with source/donor references and a fixed prompt per task; no future ground-truth RGB frames are provided at inference. The results show that VACE underperforms WoCA in terms video quality, identity preservation and spatial control.

Weakness 4. The framework depends on substantial geometric annotation.

Our approach follows the problem formulation from Neural Assets, which does require geometric inputs, but their source does not necessarily need to be human annotation: estimated tracks, 3D boxes, and camera parameters can in principle use the same conditioning interface after consistent coordinate alignment. Our contribution establishes controllable simulation from these inputs, making integration with automatic perception a natural extension. As already acknowledged in the limitations section in our manuscript, reducing annotation dependence remains an important future work direction.

Question 1. Computational efficiency of the full simulation pipeline.

We have profiled controlled video rendering on Waymo Open using a single NVIDIA H20 GPU under matched generation settings in new Table 2 in the updated manuscript. The results show that our method only has modest increase in latency and peak memory compared to the native Wan2.2 backbone.

We hope that we have adequately addressed your concerns. Please let us know if you have any further questions.

Reviewer_jfCuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WoCA is an object-centric framework that separates persistent appearance assets from time-varying object geometry. It extracts object appearance tokens from reference images, conditions a Wan-based video renderer through global attention and localized spatial features, and trains a separate diffusion model to predict future 3D object states. Cross-frame reconstruction with foreground masking encourages the renderer to use the appearance assets.

Experiments on MOVi-D and Waymo evaluate reconstruction, appearance swapping, and trajectory editing against Neural Assets. Ablations support the conditioning and training designs, while a qualitative 200-frame example demonstrates recursive generation.

Critical strengths
  • Persistent appearance assets provide an explicit and reusable conditioning interface. Keeping object appearance separate from evolving geometry enables appearance swaps and trajectory edits within one renderer. Reusing the same assets across windows is also a reasonable mechanism for reducing dependence on previously generated appearances.

  • The training objective addresses a concrete conditioning shortcut. Masking foreground objects in the conditioning image encourages reconstruction from external reference assets rather than direct appearance propagation from the first frame. Cross-frame supervision provides this training signal without requiring paired edited videos.

Critical weaknesses
  • The formulation depends heavily on externally supplied object structure and geometry. Training requires tracked identities, 3D bounding boxes, and calibrated cameras. Inference still requires reference boxes and, for motion prediction, observed 3D seed states and a supplied camera path. Thus, object discovery, correspondence, and initial geometric reconstruction are assumed rather than learned. The fixed object roster and dimensions further restrict scene evolution. These are substantial limitations for a broadly applicable world simulator, and robustness to imperfect or incomplete geometric inputs is not established.

  • The evaluation primarily establishes controlled rendering rather than world simulation. The main quantitative experiments use supplied original or edited trajectories. The learned dynamics model lacks quantitative forecasting, interaction, and recursive-rollout evaluation, while long-horizon evidence is a qualitative example. Moreover, the principal baseline is Neural Assets, an image-synthesis method, so improved video metrics do not establish competitiveness against strong video-based conditioning approaches. Direct measurements of appearance identity and trajectory adherence are also needed to substantiate the claimed disentanglement.

Questions for authors
  • Which parts of the formulation can accommodate uncertain object identities, missing 3D seed states, or newly entering objects? Which require changing the representation or training procedure rather than simply improving input annotations?

  • How accurately does the dynamics model predict future positions, orientations, and interactions relative to trajectory-prediction baselines? How do these errors accumulate during recursive rollout?

  • What reference targets are used for PSNR, SSIM, and LPIPS under appearance swaps and trajectory edits? How do these metrics distinguish successful editing from preserving the original video, and can direct identity-preservation and trajectory-adherence measurements be provided?

  • My central concern is that the dependence on externally supplied object structure substantially restricts the method’s applicability, while the current evidence does not establish the dynamics capabilities needed to justify its broader world-simulation framing.

1 comment
AuthorsOpenReview ↗

Response to Reviewer jfCu

Thank you for your valuable comments!

Weakness 1 and Question 4. The formulation depends heavily on externally supplied object structure and geometry.

Our method follows the problem formulation from Neural Assets, which does require geometric inputs, but their source does not necessarily need to be human annotation: estimated tracks, 3D boxes, and camera parameters can in principle use the same conditioning interface after consistent coordinate alignment. Our contribution establishes controllable simulation from these inputs, making integration with automatic perception a natural extension. As already acknowledged in the limitations section in our manuscript, reducing annotation dependence remains an important future work direction.

Weakness 2 and Question 2. How accurate is the dynamics model?

Our dynamics model aims to generate plausible, diverse future trajectories: we don't directly measure the accuracy of the dynamics model since agreement with a single recorded continuation is not a complete measure of success. For world simulation, these trajectories should support coherent, high-quality videos. The quality and temporal coherence of videos rendered from sampled trajectories therefore provide an indirect, system-level perspective on the usefulness of the learned dynamics. WoCA’s superior video quality in the supplied-trajectory setting further supports the effectiveness of its trajectory-conditioned rendering for world simulation.

Weakness 2. Insufficient baselines.

As suggested by Reviewer 6NGh, We have added a video-editing baseline VACE in updated Figure 2 and new Figure 3 in the updated manuscript. For fair comparison, we finetune the official VACE checkpoint with LoRA and construct coarse RGB conditioning videos by placing reference-object crops at the prescribed projected boxes over the initial background. These videos are supplemented with source/donor references and a fixed prompt per task; no future ground-truth RGB frames are provided at inference. The results show that VACE underperforms WoCA in terms video quality, identity preservation and spatial control.

Question 1. Which parts of the formulation can accommodate uncertain object identities, missing 3D seed states, or newly entering objects? Which require changing the representation or training procedure rather than simply improving input annotations?

Our method already supports temporarily missing poses and entry/re-entry of known objects: unavailable pose conditions are masked while appearance assets are retained. Better tracking or 3D estimation can provide consistent identities and missing seed states through the existing interface. Explicit identity uncertainty, however, requires extending the asset-track association mechanism and its training; prediction without any usable 3D seed requires state initialization; and autonomous introduction of previously unseen objects requires asset/track creation and training to model object entry. These extensions go beyond improving annotations, but can retain WoCA’s core separation of persistent appearance from evolving geometric state.

Question 3. What reference targets are used for PSNR, SSIM, and LPIPS under appearance swaps and trajectory edits? How do these metrics distinguish successful editing from preserving the original video?

The reference for PSNR, SSIM and LPIPS is the original GT video, since counterfactual GT videos are unavailable. These metrics characterize full-reference visual fidelity, reflecting both preservation of unchanged scene content and differences introduced by intentional edits, rather than directly measuring editing correctness. Below, we complement these results with explicit identity-preservation and spatial-control evaluations.

We have added identity preservation and box control evaluations in new Figure 3 in the updated manuscript. The results show that WoCA achieves the highest identity similarity and box Recall@ 0.5 on both datasets. WoCA also achieves the highest box mIoU on MOVi-D; on Waymo Open, its mIoU is slightly below the new VACE baseline, while providing higher identity similarity and recall. These results support improved identity preservation and object placement across both synthetic and real-world scenes.

We have also conducted an additional ablation study to isolate the effect of video editing by evaluating unedited and edited outputs against the same edited target boxes in new Table 1 in the updated manuscript. Applying the edit increases box mIoU from 0.2975 to 0.5962 and Recall @ 0.5 from 14.93% to 82.02%, demonstrating that the generated objects follow the requested changes. Meanwhile, identity similarity remains close to that of unedited reconstruction evaluated under original targets (0.5854 vs 0.6158).

Together, these results support WoCA's ability to modify object motion while largely retaining source identity, without requiring counterfactual ground-truth videos for evaluation.

Reviewer_6NGhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WoCA, an object-centric world simulation framework that separates object appearance from dynamics. Appearance tokens extracted from a reference frame form a persistent asset bank, and a trajectory diffusion model predicts future 3D object boxes that can be rolled out recursively. A Wan2.2-based renderer then renders the assets under the given poses. Experiments on MOVi-D and Waymo Open show better video quality than Neural Assets on reconstruction, appearance swapping, and trajectory editing.

Critical strengths
  • Clear pipeline design: Separating persistent appearance from evolving 3D state is well motivated, and enables appearance swapping and trajectory editing within one framework.
  • Simple training recipe: Masking objects in the first frame forces the renderer to use the asset bank, which allows training on ordinary annotated videos without edited data pairs.
Critical weaknesses
  • Missing critical quantitative evaluation for editing and simulation claims: While editing and simulation are presented as the main applications of this pipeline, there is no quantitative evaluation of the performance on these two tasks. PSNR, SSIM, and LPIPS are reported for Swap and Edit, but they presumably compare against the original, unedited video and thus cannot reflect how well the edit is performed. Task-specific measures, e.g., identity similarity or alignment with the edited boxes, are missing.
  • Insufficient baselines: The only baseline is Neural Assets, an image model with its temporal components removed, and how it produces a video is not described. No video-based pipeline is compared, although swapping and trajectory editing can be done with mask-based video editing, e.g., VACE [1], which is also built on Wan, or the methods discussed in the paper (VideoSwap, MotionCtrl, FlexAM). It is therefore unclear what the asset-pose representation offers over direct video editing.
  • Unclear benefit for downstream tasks: How does the generated data augment downstream tasks in driving scenarios, such as 2D and 3D object detection? The diversity of the swapped or edited outcomes is also not analyzed.

References [1] Jiang et al., "VACE: All-in-One Video Creation and Editing", ICCV 2025.

Questions for authors
  1. How does WoCA work on videos without any existing human annotation, e.g., using 3D boxes and camera parameters estimated by off-the-shelf models? Does it still work robustly?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6NGh

Thank you for your valuable feedback! Below, we address your concerns point by point.

Weakness 1. Task-specific measures, e.g., identity similarity or alignment with the edited boxes, are missing.

We have added identity preservation and box control evaluations in new Figure 3 in the updated manuscript. The results show that WoCA achieves the highest identity similarity and box Recall @ 0.5 on both datasets. WoCA also achieves the highest box mIoU on MOVi-D; on Waymo Open, its mIoU is slightly below the new VACE baseline, while providing higher identity similarity and recall. These results support improved identity preservation and object placement across both synthetic and real-world scenes.

We have also conducted an additional ablation study to isolate the effect of video editing by evaluating unedited and edited outputs against the same edited target boxes in new Table 1 in the updated manuscript. Applying the edit increases box mIoU from 0.2975 to 0.5962 and Recall @ 0.5 from 14.93% to 82.02%, demonstrating that the generated objects follow the requested changes. Meanwhile, identity similarity remains close to that of unedited reconstruction evaluated under original targets (0.5854 vs. 0.6158).

Together, these results support WoCA's ability to modify object motion while largely retaining source identity.

Weakness 2. Insufficient baselines: swapping and trajectory editing can be done with mask-based video editing, e.g., VACE, which is also built on Wan.

We have added the requested video-editing baseline VACE in updated Figure 2 and new Figure 3 in the updated manuscript. For fair comparison, we finetune the official VACE checkpoint with LoRA and construct coarse RGB conditioning videos by placing reference-object crops at the prescribed projected boxes over the initial background. These videos are supplemented with source/donor references and a fixed prompt per task; no future ground-truth RGB frames are provided at inference. The results show that VACE underperforms WoCA in terms video quality, identity preservation and spatial control.

Weakness 3. Benefit for downstream tasks.

The core contribution of this work is the methodology part, i.e., developing a world simulation approach that can vary object appearance and motion independently, enabling targeted scene generation with intended 2D/3D box labels. Our swapping and trajectory-editing experiments demonstrate this controllability on both synthetic and real-world datasets, while identity and box-control metrics verify preservation of source identity and alignment with requested geometry. These capabilities provide a concrete basis for, e.g., driving-data augmentation. As evaulating downstream utility is out of the scope of this work, we leave quantifying downstream task gains for future work.

Question 1. How does WoCA work on videos without any existing human annotation?

WoCA follows the problem formulation from Neural Assets, which does require geometric inputs, but their source does not necessarily need to be human annotation: estimated tracks, 3D boxes, and camera parameters can in principle use the same conditioning interface after consistent coordinate alignment. Our contribution establishes controllable simulation from these inputs, making integration with automatic perception a natural extension. As already acknowledged in the limitations section in our manuscript, reducing annotation dependence remains an important future work direction.

We hope that we have adequately addressed your concerns. Please let us know if you have any further questions.

Thanks authors for prepare a detailed rebuttal. I believe the identity preservation and box control evaluations could strength the paper. However, my concern for benefit for downstream tasks still remain unsolved. Thus, I will keep my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their valuable feedback and constructive comments, which have helped us significantly improve the clarity and quality of our manuscript.

We have addressed each reviewer's concerns separately below their respective review. We have also updated our manuscript to include additional results and details requested by the reviewers, with changes highlighted in blue:

  1. We have run an additional video editing baseline VACE and include its video quality results in updated Figure 2.
  2. We have quantitatively measured the identity-preservation and spatial-control abilities of all compared methods (new Figure 3). We also conduct an additional ablation study to isolate the effect of video editing by evaluating unedited and edited outputs against the same edited target boxes (new Table 1).
  3. We have profiled computational efficiency of controlled video rendering under matched generation settings for our method against its Wan2.2 backbone (new Table 2).
  4. We have updated Figure 1 to present a clearer illustration for the core ideas in our method.

We hope that we have adequately addressed all reviewers' concerns. Please let us know if you have any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.