acceptodds
Under review as a conference paper at ICLR 2027

4DStreamCtrl: Interactive Video Generation with Online 4D Control

Abstract

Generative video models now synthesize footage nearly indistinguishable from reality. Their promise as interactive tools hinges on fine-grained control of how objects and the camera move over time, yet each existing approach captures only part of this: camera-parameter methods steer the viewpoint but cannot move objects, 2D-trajectory methods act in the image plane and ignore depth and occlusion, and recent 3D methods add geometry but run only offline at a fixed length. In particular, none combines 3D-consistent control of both camera and objects with real-time, streaming generation. Here we show that camera motion, object trajectories, and depth can be unified into a single 3D point-track representation, from which one model performs joint camera and object control, depth editing, and motion transfer in a single forward pass. To learn this interface at scale, we mine in-the-wild video for 3D motion supervision, yielding OpenVidHD-Motion3D, and encode it with a lightweight Geometric Motion Head that plugs into a pretrained video diffusion model. Because this encoder is temporally separable, we distill the model into a causal streaming student that generates arbitrarily long video in four denoising steps at memory independent of length. This unified design surpasses prior camera-only, 2D, and offline-3D methods in motion-control precision while covering modalities they address only in isolation. 4DStreamCtrl runs at 20 FPS on a single high-end GPU for 480p video and stays temporally coherent over hundreds of frames, enabling, to our knowledge, interactive 4D-controllable streaming generation for the first time. More broadly, grounding generation in explicit 3D geometry with efficient causal inference points toward interactive world models with closed-loop spatiotemporal control, from controllable simulators to real-time visual imagination for embodied agents.

Primary area:
generative models
Keywords:
Controllable video generation, world model, streaming, 4D

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jHwnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents 4DStreamCtrl, a framework for real-time interactive video generation with 3D-aware motion control. The method encodes point trajectories and depth information through a Geometric Motion Head and injects them into a pretrained video diffusion model via channel concatenation. It further adopts the existing Self-Forcing framework to enable causal streaming generation.

Critical strengths
  1. The method achieves promising motion-control accuracy and real-time generation speed (20 FPS).
  2. The paper provides reasonable experimental comparisons and ablation studies to demonstrate the effectiveness of 3D-aware conditioning.
Critical weaknesses
  1. Limited technical novelty. The paper largely extends MotionStream by replacing 2D trajectory conditioning with 3D-aware motion conditioning. The proposed method encodes track identities and depth using standard feature rasterization, temporal convolutions, and channel concatenation, while the streaming distillation follows existing Self-Forcing methods without substantial modifications. Although the additional geometric information improves motion control, the overall approach is relatively straightforward and lacks substantial methodological innovation.
  2. Limited demonstration of challenging motion. Although the paper emphasizes 3D-aware control and long-term streaming generation, the qualitative examples, particularly the long-video results in Figure A1, exhibit relatively limited motion. It is therefore difficult to assess whether the proposed method can maintain accurate control and temporal consistency under more challenging object and camera dynamics.
Questions for authors
  1. Beyond incorporating 3D geometric information into MotionStream-style conditioning, what is the key methodological contribution of this work?
  2. Can the authors provide qualitative results involving more substantial object and camera motion to better demonstrate the advantages of 3D-aware control?
2 comments
AuthorsOpenReview ↗

Response to Reviewer jHwn

Beyond adding 3D information to MotionStream-style conditioning, what is the key contribution?

Please see Common Response 2. In short: (i) a temporally separable control head that gives the bidirectional teacher and the causal student the consistent control signal, enabling streaming distillation and mid-generation editing; (ii) depth control of what 2D tracks leave undetermined, most notably occlusion order (Fig. 6) and parallax, which MotionStream-style 2D conditioning cannot specify; (iii) a large-scale in-the-wild 3D-track dataset with camera parameters that makes this control learnable on open-domain video. We will restate the contributions in these terms.

Figure A1 shows limited motion; can you show more challenging dynamics?

Yes. We add some generated videos with high-dynamics sequences in the supplementary material:

  1. two objects crossing with depth-order reversal, where the control signal alone decides who occludes whom (2D control cannot do this);
  2. a 90° camera orbit around a car (large parallax, continuous disocclusion);
  3. long-term object control and human control jointly.

Also see more generated videos in this anonymous link: https://anonymous.4open.science/w/4DStreamCtrl/

Thanks for providing additional examples. However, I am still not fully convinced that the current demonstrations sufficiently highlight the unique advantages of 4D control over existing 2D-based methods. For example, the two-object crossing case with depth-order reversal could potentially also be handled by 2D trajectories with visibility information. Meanwhile, the demonstrated motions still seem relatively limited in magnitude.

I would encourage the authors to include more challenging examples, such as humans performing complex motions (e.g., dancing and turning around), objects undergoing large rotations (e.g., 180° rotations that reveal previously unseen sides), and cameras freely navigating through complex scenes along non-trivial trajectories. Such examples would provide much stronger evidence of the practical advantages of 4D control and its potential for interactive world models.

I also have concerns about the applicability of the method to embodied scenarios. In the robot brushing example, after motion transfer, the brush appears to lose contact with the object it is supposed to interact with. This suggests that the current approach may struggle to preserve basic physical interactions, which is important for applications involving embodied agents.

Please highlight the newly added results and corresponding changes in blue in the revised manuscript, and clearly indicate the relevant figures or sections in your response.

Reviewer_euP7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce 4DStreamCtrll, a video-generation system that combines object trajectories, camera motion and depth in one control interface, it constructs approximately 400K training clips with 3D tracks and camera parameters, then trains a motion-conditoning module for a pretrained Wan video model, the authors distill this model into a streaming version that generated video in fewer denoising steps with a fixed-size memory window.

Critical strengths

Strengths:

  • In my opinion the unified control interface is useful and supported by ablations, combining depth with point trajectories provides a practical way to control the camera and object motion together, and the paper shows that by adding 3D information this reduces motion error, which supports the benefit of the geometric input beyond simply using a larger model backbone.On the other hand, the generated videos look great.

  • The streaming model seems to preserve much of the teacher's control accuracy. The paper also reports latency and memory usage and demonstrates changing controls during generation. Together, these results support a potentially useful interactive system.

Critical weaknesses

Weaknesses:

  • I think the evaluation does not clearly establish the claimed 3D control accuracy. Most metrics measure image similarity and the paper does not clearly define how the main motion metric (EPE) is computed (whether it measures 2D or 3D error?). Also, I see that there are no separate quantitative measurements of camera-pose accuracy, depth accuracy, or correct occlusion ordering.

  • Some other important details regarding the online operation remain unclear. Appendix B.1 rescales depth values using the minimum and maximum in each sample. However, the paper claims that processing separate chunks gives the same result as processing the whole video. On the other hand, It also does not specify the GPU model or which processing steps are included in the timing, and methods are compared at different resolutions, which I think it makes the speed comparison difficult to judge fairly.

Questions for authors

Please refer to weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer euP7

How is EPE computed — 2D or 3D?

EPE is a 2D image-space metric, following the MotionStream protocol. We will state this definition in the main text, and we now additionally report a true 3D-track EPE in camera space (Common Response 1(b)), together with camera-pose and occlusion-ordering metrics (1(a), 1(c)).

Appendix B.1 rescales depth with per-sample min/max, which seems inconsistent with the claim that chunked processing equals full-sequence processing.

B.1 is accurate but incomplete, and the two claims are compatible: depth rescaling is a preprocessing step outside the generation loop, and the resulting affine map is fixed before the first chunk is produced and never recomputed per chunk.

Concretely there are two regimes. In training and in offline replay evaluation (Table 1), the control signal is a complete pre-recorded 3D track set, so the min/max are taken over that whole set — this is what B.1 describes, and all baselines receive the same pre-recorded signal. In interactive streaming, no future track exists: the map is instead derived once from the metric depth map of the reference frame and reused for every chunk and every later user edit. In both regimes the map is constant throughout generation, and the encoder has no temporal receptive field crossing chunk boundaries (per-frame scatter-add + non-overlapping stride-4 Conv3D), so chunk-wise and full-sequence encoding are bit-identical. The bit-identical claim concerns the encoder, not the choice of normalization statistics. We will state both regimes explicitly in B.1.

Hardware, what the timing includes, and resolution-mismatched speed comparison.

  • Hardware/setting: a single NVIDIA H100 80GB, bf16, batch size 1, 480p (480×832), 4 denoising steps, chunk = 4 latent frames.
  • What is timed: per-chunk wall clock covering motion-head encoding of the current tracks, the 4 generator forward passes, the KV-cache context update, and VAE decoding to RGB. It excludes JPEG encoding and one-time model loading and reference-frame depth estimation. We will add this breakdown as a table (encode / denoise / KV / decode).
  • Resolution: all quality metrics in Table 1 are already resolution-matched — every video, ours and baselines, is resized to 832×480 before PSNR/SSIM/LPIPS/EPE, following MotionStream's protocol. What our table omits is each method's generation resolution, which is what makes the FPS column hard to read. We will add it as an explicit column:
MethodBackboneGen. resolutionFPS ↑
Image ConductorAnimateDiff256P2.98
Go-With-The-FlowCogVideoX-5B480P0.60
Diffusion-As-ShaderCogVideoX-5B480P0.29
ATIWan 2.1-14B480P0.23
MotionStream TeacherWan 2.1-1.3B480P0.79
MotionStream CausalWan 2.1-1.3B480P16.7
MotionStream TeacherWan 2.2-5B720P0.74
MotionStream CausalWan 2.2-5B720P10.4
MotionStream Causal (re-measured)Wan 2.2-5B480P19.5
Ours TeacherWan 2.2-5B480P0.84
Ours CausalWan 2.2-5B480P20.6

The 10.4 FPS previously placed next to our number is a 720P measurement, which we will label as such. Re-measured at 480p on the same H100, MotionStream Wan 2.2-5B reaches 19.5 FPS against our 20.6 FPS. The two are essentially on par, as expected: they share the backbone, the 4-step causal student and the resolution, and differ only by one lightweight Conv3D depth branch. We will revise Sec. 4.1 accordingly — our claim is not a speed advantage but that adding 3D/depth control costs essentially nothing at inference, so 4D controllability need not be traded against interactive speed.

Reviewer_rE5nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents 4DStreamCtrl, an interactive video generation model that uses camera-space 3D point tracks (2D positions + per-point depth) as a control signal. The tracks are encoded by a lightweight Geometric Motion Head attached to a pretrained video diffusion model, which is trained on in-the-wild videos pseudo-labeled with an off-the-shelf 3D tracker. The model is then distilled into a few-step causal student with self-forcing, so it can stream long videos in real time while the user keeps editing the tracks.

Critical strengths
  • The motivation makes sense, and the video demos are impressive. The method works especially well in cases like occlusion, where 3D ambiguity really matters.
  • Extending 2D motion control along the depth axis, and bringing this 4D controllability into a streaming model built on self-forcing, is useful.
  • The technical details of the Geometric Motion Head are useful in practice.
Critical weaknesses
  • The contribution does not go much beyond combining existing techniques. Adding depth to 2D motion control and plugging it into a self-forcing + DMD pipeline feels like a natural combination. It reads more like an application than a solution to a new technical challenge or a new insight for the field.
  • The quantitative evaluation does not fully back up the claims. There is no metric for camera control accuracy, and PSNR is somewhat lower than several baselines. The explanation given for PSNR sounds reasonable, but a masked PSNR outside occluded regions would make it more convincing.
  • (Minor) The left part of the overview in Fig. 1 is hard to take in at a glance.
Questions for authors
  • Could you report a quantitative metric for camera control accuracy, and a masked PSNR that excludes occluded regions, to support the explanation for the lower PSNR?

---- Update ------

Thank you for the detailed response and the additional experiments. The new evaluations address my concerns regarding camera control accuracy.

Regarding novelty, I agree that making these components work together effectively is non-trivial. That said, the broader research insights from this composition and training recipe still seem fairly specific to this particular application setting. In addition, as Reviewer jHwn pointed out, the current results mainly showcase simple scenarios rather than challenging cases that prior combinations could not handle, which leaves the full advantage of the 4D formulation less clear.

1 comment
AuthorsOpenReview ↗

Response to Reviewer rE5n

Contribution is a combination of existing techniques.

While the individual components are known, making them into a unified streaming 4D control interface, and unlocking the capabilities it enables, is non-trivial. We kindly refer the reviewer to Common Response 2 for the key designs that make this possible. We will revise the contribution paragraph accordingly.

No camera-control metric; PSNR lower than some baselines; a masked PSNR would be more convincing.

We fully agree and have added both, see Common Response 1(a) and 1(d). On the PSNR gap itself: PSNR rewards pixel-level agreement with one particular ground-truth video, so a generator that synthesizes plausible content in regions the control signal does not constrain (disocclusions, newly entering background under camera motion) is penalized even when its motion is correct. The masked PSNR isolates exactly this effect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.