- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents GenCine, a framework for jointly controlling camera motion and local object motion in video generation from a single image. The method lifts the input image into a coarse 3D point cloud, allowing users to specify camera trajectories and transformations of local motion handles in a shared world coordinate system. These controls are rasterized into background XYZ, current foreground XYZ, and foreground identity maps. The maps are encoded using a pretrained VAE and incorporated into a Wan video generator through a side branch and LoRA adapters. This paper's design separates the current world-space coordinates of controlled regions from their persistent identities, while local rigid handles support coordinated motion. Experiments demonstrate applications involving joint camera–object motion, local deformation, and articulated motion, with comparisons against several controllable video generation methods.
Critical strengths
-
The problem has clear practical relevance. Jointly authoring camera and object motion is an important capability for controllable video generation. A shared world coordinate system helps users specify depth changes, rotations, and relative motion that can be difficult to express unambiguously through 2D dragging.
-
The conditioning representation is well motivated. Separating current position from persistent identity provides complementary cues for motion and correspondence across frames. Using a shared coordinate normalization for the foreground and background also provides an explicit reference for interpreting object motion relative to the scene.
-
Local motion handles offer useful authoring flexibility. Users can control different parts of an object without requiring a complete skeleton or a category-specific model. The examples in which sparse local controls induce motion beyond the directly controlled region suggest potential value for creative applications.
-
The experiments include relevant baselines and support some design choices. Comparisons include Wan-Move, ATI, Go-with-the-Track, VerseCrafter, and SymphoMotion. In Table 2, removing foreground XYZ increases R-LPIPS from 0.184 to 0.223, supporting the usefulness of explicit position conditioning within the proposed system. The paper also acknowledges limitations arising from incomplete single-image geometry, locally rigid approximations, and the absence of physical constraints.
Critical weaknesses
-
The incremental representation contribution is insufficiently validated. DaS already conditions generation on trajectories with fixed initial-coordinate colors, while FlexTraj encodes persistent trajectory and object identities. GenCine’s clearest representation-level distinction is separating current world-space XYZ from handle identity. However, comparisons with these encodings under matched settings and an ablation removing the identity map are missing, making the contribution difficult to isolate.
-
The evaluation does not directly test 3D motion disambiguation. The metrics mainly assess reconstruction quality, camera accuracy, and video consistency. They do not directly establish correct object depth, orientation, or 3D trajectories, particularly when different intended 3D motions produce similar 2D projections.
-
Some improvements over strong baselines are modest. On Camera Shooting, GenCine versus Go-with-the-Track achieves PSNR of 16.64 versus 16.61, rotation error of 3.70 versus 3.69, and translation error of 1.00 versus 1.07. Performance is competitive, but gains are small on some metrics.
-
The training–inference control discrepancy and robustness are insufficiently evaluated. Training handles may change membership and need not be rigid, whereas user-authored handles apply rigid transformations to fixed point sets. The effect of this discrepancy remains unclear, especially under large rotations, occlusion and reappearance, multiple coordinated handles, and multi-object interactions.
Questions for authors
-
Could the authors compare the proposed representation with DaS-style fixed initial-coordinate encoding and FlexTraj-style persistent identity encoding within the current backbone and training setup? Such a comparison would help isolate the contribution of current XYZ and independent identity conditioning.
-
How do control accuracy and identity preservation change when the foreground identity map is removed while current XYZ conditioning is retained? GenCine uses handle-level identities, whereas FlexTraj uses point-level trajectory identities. How does this difference in granularity affect rotation, occlusion, and local deformation?
-
Could the authors construct paired tests with similar 2D projections but different intended 3D motions, and directly evaluate object depth, orientation, and 3D trajectories to assess the benefit of current XYZ over existing correspondence encodings?
-
How do training regions with changing membership and potentially non-rigid motion generalize to user-specified rigid transformations of fixed point sets? How does performance change with the number of handles and objects, rotation magnitude, and degree of occlusion?
8 comments
Response to Reviewer MJ44 (2/2): direct 3D evaluation, margins, and training–inference controls
W2 / Q3. Direct evaluation of depth, orientation, and 3D trajectories.
We added direct compliance metrics on the 37 DAVIS Object Interactions clips (Table 1(b); protocols in App. A.2):
- EPE and PCK compare tracks in the generated video with the authored controls projected through the target camera.
- **RPE-T${3D}$ and RPE-R${3D}\Omega$ in the shared world frame. They fit a rigid pose to each handle and measure adjacent-frame translation (including depth, normalized by handle size) and rotation errors.
| Method | EPE↓ | PCK↑ | RPE-T$_{3D}$↓ | RPE-R_{3D}\ (°)↓ |
|---|---|---|---|---|
| ATI | 12.95 | 0.770 | 0.047 | 1.54 |
| Wan-Move | 19.50 | 0.629 | 0.040 | 1.46 |
| VerseCrafter | 61.71 | 0.287 | 0.046 | 1.56 |
| SymphoMotion | 77.55 | 0.334 | 0.096 | 2.02 |
| Go-w-Track | 8.39 | 0.914 | 0.040 | 1.45 |
| LeviTor | 73.15 | 0.342 | 0.092 | 1.93 |
| GenCine | 7.86 | 0.922 | 0.036 | 1.40 |
GenCine has the lowest errors on all four measures. Its 3D margins over the dense-track baselines are modest, even though those baselines receive dense ground-truth 2D tracks.
We have not built paired cases with matched 2D projections. The per-handle 3D errors address part of this question: they compare authored and generated relative poses in the world frame, so a motion with the wrong depth direction or rotation direction incurs a large error even when its 2D projection is similar.
W3. Modest improvements.
On Camera Shooting, GenCine is on par with Go-with-the-Track on PSNR (16.64 vs. 16.61) and rotation (3.70 vs. 3.69), and it has a lower translation error (1.00 vs. 1.07). Go-with-the-Track receives dense ground-truth foreground and background tracks, while GenCine uses only sparse 3D handles. GenCine also leads every direct control metric above.
The side-by-side videos (https://neuralcinema.github.io/#compare) show effects that averaged metrics miss. In Robotic arm and Goldfish, Go-with-the-Track moves only the controlled region, while GenCine also produces plausible motion of the connected arm segments and the uncontrolled fish.
Re-generating Table 1 for several seeds means re-running all seven methods, each built on a large video diffusion backbone, on all 75 examples. We are actively running them and trying to complete before the rebuttal deadline.
W4 / Q4. Training–inference control discrepancy and robustness.
During training, handles follow local tracks recovered from videos. Their membership changes as tracks become invalid or reappear, and the selected regions need not move rigidly. At inference, an authored handle moves a fixed point set rigidly.
Both settings render the same three guidance maps: current 3D positions plus persistent handle identities. The model therefore sees the same conditioning format. A rigid authored handle can be viewed as the easy end of the training distribution: its membership stays fixed and its points move consistently.
Training also varies the number and placement of handles and includes background-only clips, which covers partial artist control (Sec. 3.4, App. A.5).
The current qualitative evidence covers several of the stress conditions the reviewer lists:
- multiple coordinated handles on an articulated robotic arm (Sec. 4);
- an object rotating during a camera pullback (bus example);
- two objects crossing with occlusion and reappearance (Fig. 7).
We have not yet measured a systematic sweep. In the final version we will report performance against the number of handles and objects, rotation magnitude, and occlusion level. Limitations remain: rigid handles do not enforce physical interactions, and unseen surfaces are unmodeled (App. A.5, App. I).
We thank the reviewer again for the constructive questions, which directly shaped the identity ablation and the 3D evaluation.
Reply For Answers of Question 1: Thank you for the response and additional experiments. These results have addressed most of my concerns regarding Q1. Although these variants are not fully equivalent to the encodings used in FlexTraj or DaS, I acknowledge that the ablation retaining persistent identity while removing current XYZ can verify the additional contribution of current XYZ within the proposed framework. The new ablation in Table 2 of Sec. 4.2, together with the existing results, provides reasonably clear evidence for the contributions of current XYZ and foreground identity to control performance, as well as their complementary roles.
Reply For Answers of Question 2: Thank you for the response. The newly added Figure 7 addresses my concern about the role of identity: without identity conditioning, the characters in red and blue merge and ultimately fail to complete the specified position swap while with identity conditioning, they complete the specified swap. Regarding the distinction between handle-level and point-level identity in Q2, the authors also provided explanations covering rotation, occlusion, and local deformation. However, I would like to ask one more detail: Appendix E mentions that handles containing fewer than five points are filtered out during training. How many valid points does each handle typically contain during training and evaluation?
Response to Reviewer MJ44 (1/2): identity ablation, encodings, and identity granularity
We thank Reviewer MJ44 for their constructive feedback. We are glad to see that they found that "the problem has clear practical relevance," that jointly authoring camera and object motion "is an important capability for controllable video generation," and that "the conditioning representation is well motivated": "separating current position from persistent identity provides complementary cues for motion and correspondence across frames." We are also happy to find that the reviewer found that "local motion handles offer useful authoring flexibility," with examples that "suggest potential value for creative applications," and that "the experiments include relevant baselines and support some design choices." We now answer the questions and concerns raised by the reviewer.
Q2 (first part). Removing the identity map while retaining current XYZ.
We trained exactly this variant: w/o Fid, with identity removed from both branches and the same data, backbone, adaptation capacity, and budget (Sec. 4.2, Table 2).
Removing identity worsens control accuracy (EPE +6.2 px, PCK −0.15, higher camera errors) and identity preservation (the lowest subject consistency of all variants). In the two-character exchange of Fig. 7, the characters blend and end on their original sides without the identity map.
W1 / Q1. DaS-style and FlexTraj-style encodings.
These comparisons target one question: does separating current position from persistent identity help? Our matched ablations answer it directly, under the same backbone, data, and budget:
These variants are analogues, not reproductions, of DaS and FlexTraj. We believe exact reproductions are not needed to establish this point, for two reasons:
Q2 (second part). Handle-level versus point-level identity.
GenCine's identity labels a motion handle; the per-point geometry comes from the current foreground XYZ map. The difference in granularity affects the three cases the reviewer lists as follows: