acceptodds
Withdrawn from ICLR 2027

Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video world models aim to preserve scene structure and predict how dynamic objects evolve beyond visual observations. We present Kepler4D, a framework for future video generation through explicit 4D scene state evolution. Given a monocular video, Kepler4D constructs a shared 3D representation of background geometry, object motion histories, coarse spatial supports, and semantic context. Chain-of-Motion summarizes observed motion and uses a vision-language model to select structured speed and heading decisions and decide whether to bound object-center height from below. A deterministic rollout converts these decisions into future object trajectories for inspection and editing before synthesis. We render the evolving proxies into geometric controls for a pretrained video generator, separating coarse object motion from the synthesis of appearance and articulation. Experiments on real-world videos demonstrate that Kepler4D enables controllable object motion and plausible future rollout while preserving scene consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Controllable Video Generation, Future Video Prediction, Video World Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_igBhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a system that extends a video filmed by a single camera and lets users change how objects move next. It builds a 3D background and records each object’s past positions and rough size. Chain-of-Motion uses a vision-language model to choose the object’s speed, direction, and whether it should stay above a certain height. Simple rules turn these choices into future paths, which guide a pretrained video generator. The authors also test the system on 55 DL3DV scenes, 38 DAVIS videos, and tasks where users edit motion through language. The results are promising, and the paper tests how different components affect performance. However, the experiments do not clearly show whether the system predicts object motion more accurately or follows motion-editing instructions better.

Critical strengths
  1. A useful, editable motion representation. The shared 3D representation offers a practical way to inspect and edit object motion before expensive video synthesis. Separating object-center trajectories from appearance generation also makes the system easy to understand. The same trajectory can be rendered under different camera paths, which is a useful capability beyond giving the video generator only a text command. The paper explains the limits of coarse supports rather than treating them as detailed physical models.

  2. Encouraging results with helpful ablation studies. The complete pipeline improves several reported measures across different settings. On automatic continuation, it achieves PSNR 14.921 and Flow-EPE 15.220, compared with 14.090 and 15.400 for VerseCrafter. Adding rendered trajectories to the motion description reduces FID from 60.49 to 48.76 in Table 4. Table 6 further shows that motion history and semantic context matter within this pipeline. These results support the usefulness of explicit spatial guidance, even though they do not fully isolate forecasting quality.

Critical weaknesses

1: Chain-of-Motion chooses from a few speed and direction options, and fixed rules turn these choices into future paths. It is unclear whether this is enough to predict motion that changes with the scene, such as stopping before an obstacle or responding to another moving object.

2: Chain-of-Motion needs a comparison with simpler methods. The paper does not show whether it is better than simply keeping the current speed and direction or continuing the observed turn. These alternatives should use the same scene and video generator. The speed estimate also ignores time gaps when observations are missing, which may overestimate speed during occlusion.

3: It is unclear how consistent the improvements are. The automatic continuation test uses 38 selected DAVIS videos, but the paper does not fully explain their selection or report results for each video. The Flow-EPE improvement over VerseCrafter is small, from 15.400 to 15.220. Results across individual videos and several random seeds would help show whether this gain is reliable.

4: Consistency across camera views needs a direct test. The paper shows examples from different views, but does not directly measure whether they agree on the same moving scene. The DL3DV test has no moving foreground, while the DAVIS tests use each video’s estimated camera path. Rendering the same predicted motion from several cameras would help check whether object identity and motion remain consistent.

Questions for authors

Please refer to the weakness section.

Reviewer_dkbqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Kepler4D reconstructs scene geometry and object motion histories from a video, uses a VLM to select motion decisions, and converts them into editable future trajectories through deterministic rollout. The trajectories then guide a pretrained VerseCrafter generator. The main addition is the interface from observed motion and language instructions to future object trajectories.

Critical strengths

Separating motion decisions, trajectories, and synthesis makes the controls easy to inspect and edit. Figure 5 and the supplementary video show different motion responses to different instructions. The ablations also support the usefulness of motion descriptions, history, and rendered trajectories. The paper is clear about the components it reuses and the limitations of its metrics.

Critical weaknesses
  1. The reliability of language-guided object control is not adequately established. TurnErr in Table 3 measures camera yaw, not object turning, and cannot distinguish left from right. The authors acknowledge this, but the qualitative examples do not establish how reliably objects follow commands across scenes. AngleErr in Table 5 is more relevant, yet it evaluates generation from supplied reference or perturbed trajectories. It does not test the full path from a language instruction to the resulting object motion.

  2. It is unclear what CoM adds over simple motion extrapolation. Tables 4 and 6 show that motion information and trajectory guidance help the pipeline. They do not show whether the VLM decisions improve on constant-velocity or historical-turn extrapolation with the same reconstruction and generator. This matters because the synthesis interface comes from VerseCrafter and the rollout already uses substantial hand-specified rules. Table 2 offers limited evidence here: Flow-EPE improves from 15.400 to 15.220, while the larger RotErr improvement concerns camera following rather than object forecasting.

Questions for authors
  1. How is VerseCrafter configured in Table 2 when no future object trajectory is supplied: static objects, extrapolated motion, or no trajectory conditioning? Does it use the same reconstruction and sampling settings as Kepler4D?
  2. How many video–instruction pairs are used for language control, and which object categories, turning directions, and angles do they cover?
Reviewer_tnSKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Kepler4D combines reconstructed background geometry, tracked object-center histories, coarse object supports, and semantic context in a shared 3D scene state. Chain-of-Motion uses a pretrained VLM to choose speed, heading, and optional center-height constraints; deterministic rollout converts these decisions into editable future trajectories. Rendered geometric controls guide pretrained VerseCrafter/Wan2.1 video synthesis under an externally supplied future camera path. Experiments cover 55 DL3DV static scenes, 38 DAVIS continuation sequences, language-guided motion edits, and trajectory/component ablations. The contribution is an inference-time integration of explicit motion planning and controllable video synthesis.

Critical strengths
  1. The explicit intermediate trajectories provide a useful, inspectable interface linking observed motion, language edits, and camera-conditioned video synthesis. The division between coarse object displacement and generated appearance/articulation is clearly explained.
  2. The reported results support practical utility: Table 2 improves FID (303.80 to 202.64) and Overall (0.7773 to 0.8029) over VerseCrafter. Table 4 shows benefits from motion descriptions and rendered trajectories; Table 6 provides scene-state ablations. The appendix gives substantial implementation detail and openly explains metric and representation limitations.
Critical weaknesses
  1. Direct validation of the central forecasting and language-control capabilities remains incomplete. Table 2 Flow-EPE mixes foreground and camera motion, while RotErr measures camera rotation. Table 3 TurnErr uses peak absolute camera yaw, so it cannot establish object left/right or requested-angle compliance (pp. 20-21); the authors acknowledge this limitation. Table 5 does measure object-direction adherence, but uses supplied source-derived trajectories rather than testing automatic forecasting or the full language-to-motion pipeline. The qualitative examples are useful, but do not close this quantitative gap.

  2. CoM's benefit over simple motion extrapolation is insufficiently isolated. Tables 4 and 6 support conditioning/component choices, but do not compare CoM against constant-velocity, constant-turn, or simple semantic-rule trajectories with the same renderer and generator. The Table 2 Flow-EPE gain over VerseCrafter is about 1.2%, without paired uncertainty estimates. Exact sequence/split lists and language-case counts are also missing. These omissions limit attribution and reproducibility. I favor weak rejection pending direct motion/control evaluation and matched trajectory baselines; the concerns appear addressable.

Questions for authors
  1. With the same generator and renderer, how does CoM compare with constant-velocity, constant-turn, and simple semantic-rule rollout? Report foreground trajectory errors and command success, including signed turn direction, alongside video quality. Explain coordinate alignment and reference uncertainty; DAVIS does not supply true 3D trajectories.

  2. Appendix A.3 uses displacement/(retained centers - 1), then applies velocity per rollout frame without correcting observation gaps. Could occlusion inflate speed? Please compare elapsed-time normalization and quantify affected tracks.

  3. Please supply the 38 sequence IDs, prefix/future splits, language-case counts, paired per-sequence uncertainty, seed sensitivity, and representative failures. Provide actual CoM prompts/responses and fallback rates; Figure 6 is explicitly schematic. Clarify the reconstruction and alignment procedure for Table 5 reference trajectories and supplied camera poses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.