- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents 4DStreamCtrl, a framework for real-time interactive video generation with 3D-aware motion control. The method encodes point trajectories and depth information through a Geometric Motion Head and injects them into a pretrained video diffusion model via channel concatenation. It further adopts the existing Self-Forcing framework to enable causal streaming generation.
Critical strengths
- The method achieves promising motion-control accuracy and real-time generation speed (20 FPS).
- The paper provides reasonable experimental comparisons and ablation studies to demonstrate the effectiveness of 3D-aware conditioning.
Critical weaknesses
- Limited technical novelty. The paper largely extends MotionStream by replacing 2D trajectory conditioning with 3D-aware motion conditioning. The proposed method encodes track identities and depth using standard feature rasterization, temporal convolutions, and channel concatenation, while the streaming distillation follows existing Self-Forcing methods without substantial modifications. Although the additional geometric information improves motion control, the overall approach is relatively straightforward and lacks substantial methodological innovation.
- Limited demonstration of challenging motion. Although the paper emphasizes 3D-aware control and long-term streaming generation, the qualitative examples, particularly the long-video results in Figure A1, exhibit relatively limited motion. It is therefore difficult to assess whether the proposed method can maintain accurate control and temporal consistency under more challenging object and camera dynamics.
Questions for authors
- Beyond incorporating 3D geometric information into MotionStream-style conditioning, what is the key methodological contribution of this work?
- Can the authors provide qualitative results involving more substantial object and camera motion to better demonstrate the advantages of 3D-aware control?
2 comments
Thanks for providing additional examples. However, I am still not fully convinced that the current demonstrations sufficiently highlight the unique advantages of 4D control over existing 2D-based methods. For example, the two-object crossing case with depth-order reversal could potentially also be handled by 2D trajectories with visibility information. Meanwhile, the demonstrated motions still seem relatively limited in magnitude.
I would encourage the authors to include more challenging examples, such as humans performing complex motions (e.g., dancing and turning around), objects undergoing large rotations (e.g., 180° rotations that reveal previously unseen sides), and cameras freely navigating through complex scenes along non-trivial trajectories. Such examples would provide much stronger evidence of the practical advantages of 4D control and its potential for interactive world models.
I also have concerns about the applicability of the method to embodied scenarios. In the robot brushing example, after motion transfer, the brush appears to lose contact with the object it is supposed to interact with. This suggests that the current approach may struggle to preserve basic physical interactions, which is important for applications involving embodied agents.
Please highlight the newly added results and corresponding changes in blue in the revised manuscript, and clearly indicate the relevant figures or sections in your response.
Response to Reviewer jHwn
Please see Common Response 2. In short: (i) a temporally separable control head that gives the bidirectional teacher and the causal student the consistent control signal, enabling streaming distillation and mid-generation editing; (ii) depth control of what 2D tracks leave undetermined, most notably occlusion order (Fig. 6) and parallax, which MotionStream-style 2D conditioning cannot specify; (iii) a large-scale in-the-wild 3D-track dataset with camera parameters that makes this control learnable on open-domain video. We will restate the contributions in these terms.
Yes. We add some generated videos with high-dynamics sequences in the supplementary material:
Also see more generated videos in this anonymous link: https://anonymous.4open.science/w/4DStreamCtrl/