acceptodds
Under review as a conference paper at ICLR 2027

Infusing Motion Intelligence into Structured 3D Worlds for 4D Scene Generation

Abstract

Generating a 4D scene from a single image requires endowing a structured 3D world with meaningful and coherent dynamics. Existing image-to-4D methods predominantly introduce dynamics through video generation and subsequently reconstruct 3D scenes. While video models provide strong motion priors, their dynamics are formed in 2D space, making scene geometry, physical constraints, and fine-grained motion control difficult to explicitly access and verify. In contrast, explicit 3D representations provide a structured world in which geometry, physical conditions, and motion states can be directly represented and manipulated, but lack the rich motion intelligence required to translate semantic instructions into meaningful dynamics. We propose SceneWeaver, a 3D-native 4D generation framework that bridges this gap through a Coordinator-Executor-Verifier architecture. Given an image and a prompt, a coordinator interprets the motion intent and translates it into structured conditions describing where the subject should move, how it should move, and how the environment should affect its motion. Two specialized executors then generate the corresponding global world-space trajectory and local articulated motion within the explicit 3D scene. Specifically, a potential-augmented Neural ODE, which is geometry- and physics-aware, generates the global trajectory by jointly modeling interpreted motion control, scene geometry, and environmental effects, while an LLM-based motion planner produces fine-grained articulated motion from structured motion descriptions. Crucially, the explicit 3D representation also enables numerical verification of intermediate motion states. A verifier identifies geometric, physical, and motion-related failures, localizes root causes and enables targeted repair through coordinator. This establishes a closed-loop generation in which 3D structure serves not only as the intermediate results, but also as the basis for verification and recovery. Experiments demonstrate that SceneWeaver shows promising controllability, motion quality in generated 4D scenes, offering an alternative paradigm for image-to-4D task.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D scene generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_taraOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SceneWeaver proposes a 3D-native framework for image-to-4D generation. Instead of generating a video and reconstructing 3D afterward, it builds an explicit 3D world and synthesizes dynamics directly within it. The framework uses a Coordinator–Executor–Verifier architecture: an LLM Coordinator translates the prompt into trajectory control, fine-grained motion description, and environmental conditions; a Global Trajectory Executor uses a potential-field-augmented Neural ODE with SDF-based terrain constraints; a Local Locomotion Executor uses an LLM planner to produce joint-level articulated pose edits; and a Verifier checks trajectory, local motion, and composed scene, triggering targeted repair. Experiments use a 60-sample WorldScore-derived benchmark against image-to-4D and video generation methods.

Critical strengths

1.Well-motivated 3D-native formulation. Generating dynamics directly in an explicit 3D world is a sensible alternative to video-first 4D generation, offering better geometric consistency, controllability, and physical grounding.

2.Useful motion decomposition. Separating global world-space trajectory from local articulated locomotion is a clean design that allows independent control over “where” and “how” the subject moves.

3.Closed-loop verification and targeted repair. Maintaining structured intermediate 3D states and repairing only the responsible stage is a promising agentic paradigm, more efficient than full regeneration.

4.Physical/geometric grounding. The Neural ODE with learned residual physics, SDF terrain potential, and environmental conditions is technically reasonable and more interpretable than purely latent video dynamics.

Critical weaknesses

1.Critical presentation failure. Table 1 is garbled and missing numerical results for most metrics. The main quantitative claims cannot be verified. As presented, the paper’s central empirical evidence is unreadable.

2.Insufficient experimental scope. The benchmark has only 60 samples derived from WorldScore. There is no evaluation on standard 4D generation benchmarks, no ground-truth physical metrics (e.g., contact, momentum, penetration rates), and no failure/repair statistics.

3.Baseline fairness is unclear. Video generation methods are not designed for explicit 3D control, while image-to-4D methods may be limited in motion range. Comparisons lack compute-matched analysis, and other 3D-native or agentic 4D methods are not compared.

4.Heavy dependence on external models. SAM3, Stable Diffusion Inpaint, Hunyuan3D, AniGen, and GPT-5.6 sol are all used. Error propagation across this pipeline is not analyzed, and reproducibility is questionable without a clear code/model release.

Questions for authors

Please see weakness

Reviewer_hvayOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SceneWeaver, an image-to-4D framework that generates global trajectories and local articulated motion within an explicit 3D scene. A Coordinator–Executor–Verifier architecture interprets instructions, checks generated motion, and performs targeted repairs.

Critical strengths
  1. Separating global trajectories from local articulation enables independent control and verification, providing a useful alternative to video-first generation.

  2. Targeted repair reduces generation time from 17.8 to 10.3 minutes compared with full regeneration while maintaining comparable or better motion scores.

Critical weaknesses
  1. The methodological novelty is limited. The framework mainly combines existing LLM planning, articulated animation, and potential-field dynamics. Without a matched analytic-dynamics baseline, the benefit of the learned Neural ODE remains unclear.

  2. The experimental evidence is limited. Evaluation covers only 60 examples, and the main physical metrics rely on LLM judgments rather than numerical measurements. Comparisons mainly involve video-first methods, leaving the advantage over closer structured or agentic 3D/4D approaches insufficiently established.

Questions for authors
  1. What benefit does the learned Neural ODE provide over the analytic dynamics used to generate its training data, under the same controls, scene geometry, and repair setup?

  2. Since the verifier already computes penetration and floating measures, could the authors report these numerically across the benchmark to complement the LLM-based physical scores?

  3. Eq. (2) and the algorithm use a negative terrain-force sign, while Appendix Eq. (13) uses a positive sign. Which formulation is implemented under the stated SDF convention?

Reviewer_1Z5kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SceneWeaver, a single-image 4D generation framework based on explicit 3D scenes. Through a Coordinator–Executor–Verifier architecture, the method decomposes language instructions into global trajectories, local joint motions, and environmental conditions; it generates motions using a Neural ODE and an LLM-based Motion Planner, respectively, and performs verification and local repair via 3D geometric and physical constraints. Experiments demonstrate advantages in motion controllability, trajectory quality, and environment awareness.

Critical strengths

The explicit 3D motion generation framework is reasonably sound and effective. Decoupling global trajectories from local joint motions and using explicit 3D representations to support fine-grained control and geometric constraint verification provides a valuable technical route for Image-to-4D.

Critical weaknesses
  • Limited core novelty. The system mainly combines existing 3D reconstruction, LLM-based motion planning, Neural ODE trajectory generation, and geometric verification. Although practical, it is unclear what substantive new method is introduced for motion modeling or 4D generation, and the experiments do not isolate the framework’s contribution beyond integrating existing components.
  • Motion plausibility and scene consistency are not fully convincing. Decoupling dynamic subjects from static environments and separately generating global trajectories and local joint motions cannot ensure alignment with scene constraints. Supplementary results show slightly unnatural trajectories and interactions, with insufficient modeling/validation of contact, lighting, shadows, and visual fusion. Despite controllability gains, limited visual quality weakens the claim of high-quality 4D scene generation.
Questions for authors
  • Could the authors further clarify the specific update rules for acceleration, gravity, and physical residuals in the Neural ODE, as well as the implementation actually used in experiments? I am confused about why the sign of the SDF potential field differs between the main text and the appendix (Equations (2) and (13)).
  • Could the authors add objective physical evaluations based on explicit 3D states, such as collision rate, contact error, and trajectory error, to further validate the effectiveness of Physics-aware Motion Generation?
  • Regarding methodological novelty, could the authors clarify what the paper’s core innovations are relative to existing methods in motion modeling, scene consistency, and controllability, beyond integrating existing modules into a complete system?
  • Related Work: The discussion of recent geometry-aware 4D generation and Agentic World Generation could be further improved, e.g., MoGe4D, Diff4Splat, PerpetualWonder, Comp4D, and works involving structured world control and interactive generation such as World Narrative Model and AlayaWorld.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.