- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper addresses the challenge of jointly controlling camera and object motion in video generation. It proposes GT2, which first composes camera and object motions in a unified coordinate system and generates a coarse preview of their combined visual effects. This preview is then used as supervision to optimize a LoRA module at inference time. The optimized LoRA is applied during the early denoising steps to achieve coordinated control. Experiments show that GT2 improves joint motion controllability while preserving high visual quality.
Critical strengths
-
Camera and object motions are coupled in image space, making joint camera–object control a practically valuable problem to explore. The paper composes both types of control in a unified coordinate system, explicitly handles occlusions, and uses the resulting coarse preview as supervision during the early denoising stages. The overall framework is coherent, and the motivation behind each component is clearly articulated.
-
The proposed method does not rely on large-scale annotated video datasets. Instead, it performs optimization directly at test time, reducing data requirements while improving transferability across different video generation backbones.
-
In the main joint-control experiments, GT2 outperforms all compared methods in RotErr, TransErr, and ObjMC, while achieving an Overall Score close to the best-performing method.
Critical weaknesses
-
Monocular point-cloud projection, Gaussian object cores, masked latent reconstruction, and LoRA-based test-time optimization are all well established in prior work. The main contribution therefore lies in their systematic integration. Moreover, the current method supports only translational object motion. Generating each video requires approximately 612 seconds on two 48GB GPUs, which limits its practical applicability.
-
The conclusions regarding joint control are primarily based on a self-constructed benchmark consisting of only 100 samples manually selected and annotated by the authors.
Questions for authors
Could the authors provide additional joint-control experiments on external datasets and release the complete benchmark manifest?
6 comments
Q4. Why only 100 curated samples? Please provide external joint-control experiments and the complete benchmark manifest.
A4. Thanks for raising this concern. Each joint-control sample requires a source image, object masks, reconstructed scene geometry, and full camera and per-object trajectories in one 3D coordinate system. The trajectories are manually specified through our 3D annotation interface, then supplied consistently to all methods (Appendix C). Camera-only and segmentation datasets do not directly provide these paired controls, so expansion requires per-scene geometry and control preparation, rather than simply collecting more images.
Given this preparation effort, we initially curated 100 fully specified cases for a common evaluation protocol, covering three image sources, 26% multi-object scenes, and varied motion magnitudes (Tables 7–8 in the original verson; Table 8–9 in the revision). We also report five generation seeds in Appendix F, Table 10 (Table 11 in the revision). We will expand the benchmark with more independent scenes, object configurations, and joint-motion conditions.
For external camera-control evidence, Appendix H, Table 12(Table 14 in the revision) evaluates 100 Sekai videos. The requested external joint-control evaluation is a separate experiment. Specifically, we further conduct an experiment on external DAVIS-test dataset, which contains much more complicated object motions and camera motions. The results are shown below. Our method still achieves superior performance.
| Method | TransErr ↓ | RotErr ↓ | ObjMC ↓ |
|---|---|---|---|
| Symphomotion | 0.489 | 1.351 | 5.367 |
| VerseCrafter | 0.246 | 0.941 | 5.248 |
| GT2 | 0.226 | 0.922 | 4.898 |
For reproducibility, we will release the complete benchmark manifest with sample identifiers and image sources, prompts, masks, camera/object trajectories, and generation/evaluation configurations.
Dear Reviewer,
As the discussion phase is approaching its end, we hope that the above clarifications and additional experiments have addressed your concerns. If you find our responses satisfactory, we would greatly appreciate your consideration of updating the score to reflect the additional results and discussion. We remain happy to clarify any remaining points during the discussion phase.
Thank you again for your time and consideration.
Q1. The components are established; what is the contribution beyond their integration?
A1. Thanks for raising this concern. Our contribution is a geometry-guided test-time optimization framework for joint camera–object control in pretrained video generators without task-specific offline training. We do not claim point-cloud projection, Gaussian object cores, masked reconstruction, or LoRA themselves as new contributions.
This is non-trivial because simply combining projection, masked reconstruction, and LoRA leaves projection-hole artifacts in the latent target and allows background pixels to dominate object supervision. To address this, we complete the preview before latent encoding while excluding filled regions from supervision, and separately normalize reliable background and object losses. These designs address target contamination and unbalanced control supervision within the joint optimization (Secs. 2.1–2.2).
Q2. The current method supports only translational object motion.
A2. Thanks for raising this point. GT2 supports independently prescribed 3D translation trajectories for multiple objects under simultaneous camera movement. Each control specifies the full per-frame path, including depth-direction motion (Appendix C), allowing object motion to be specified independently of viewpoint changes. Our focus is this joint trajectory-control capability; explicit object rotations and non-rigid transformations are not supported by the current interface (Appendix O).
To enable more flexible object motion, we further explore a cut-and-drag extension of our test-time optimization framework. Specifically, we cut out selected regions from the input image and animate them along user-specified trajectories, optionally incorporating in-plane rotation and scaling, to construct a coarse motion preview. We then use this preview and its corresponding foreground and valid-background masks to guide our existing test-time optimization, allowing the generative model to refine the coarse edits into a coherent video. This extension enables direct image-space motion control without requiring 3D reconstruction. We present a qualitative case study with demo videos in the newly submitted supplementary material (
cut_drag_demos.mp4) and Appendix.P of the revised paperQ3. Generating a video takes 612 seconds on two 48 GB GPUs, limiting practical applicability.
A3. Thanks for raising this concern. The same-backbone comparison isolates the additional cost of our control: on the same two-GPU setup, latency increases from 360 s to 612 s and peak memory from 25 GB to 40 GB per GPU (Appendix J, Table 15 in the original version; Tale 18 in the revision). In return, RotErr, TransErr, and ObjMC decrease by approximately 55%, 56%, and 27%, with comparable Overall Scores (Table 2). Thus, the additional 252 s per video improves joint motion control without task-specific offline training.
For system-level context, 81-frame joint generation takes 612 s on two 48 GB GPUs with GT2’s Wan-5B backbone at 1280 × 704, versus over 3600 s on eight 48 GB GPUs with VerseCrafter’s Wan-14B at 1280 × 720 (Table 14 in the original version; Table 17 in the revision). GT2 also has lower control errors (Table 1). These are different-backbone systems, so this comparison describes their reported resource requirements. GT2 targets controllable video synthesis rather than real-time interaction; the reported latency excludes model loading, preview construction, and VAE encoding/decoding (Appendix B).
We further evaluate the inference cost of WorldForge, a training-free camera-only control baseline using the same Wan2.2-5B backbone, under the same two-GPU setup. WorldForge with Wan2.2-5B requires 742 seconds for inference, with a peak memory usage of 35 GB per GPU. In comparison, our method achieves superior camera controllability, as shown in Table 1, while additionally enabling joint camera–object motion control. Moreover, our method reduces inference latency to 612 seconds, at the cost of only a modest increase in peak GPU memory usage (approximately 5 GB per GPU).