acceptodds
Under review as a conference paper at ICLR 2027

Generative Cinematographer: Composing Camera and Object Motion in 3D

Abstract

Existing controllable video generation methods can model camera motion or object motion, but most operate through image-space controls such as 2D trajectories or sparse drag signals. These controls are fundamentally ambiguous: the same image-plane motion may correspond to multiple underlying 3D motions, making camera motion and object motion difficult to disentangle under viewpoint changes. We present Generative Cinematographer (GenCine), a controllable video generation framework that lifts a single image into an editable 3D scene scaffold where camera motion and foreground motion are authored jointly in a shared world frame. Our key observation is that world-space motion can be rasterized into dense correspondence maps that preserve persistent identities across viewpoints while remaining compatible with RGB-pretrained video diffusion models. The resulting representation supports sparse local 3D motion handles, coordinated camera–object choreography, and piecewise-rigid approximations of non-rigid motion without requiring physics simulators or category-specific priors. To preserve the generative prior of a pretrained video model, we freeze a Wan-based diffusion backbone and condition it through lightweight adaptation layers driven by the proposed correspondence representation. Experiments demonstrate coherent camera-relative motion, improved geometric consistency under viewpoint changes, and strong controllability across diverse real-world scenes.

Primary area:
generative models
Keywords:
Generative model, 3D control, Video generation model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MJ44OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GenCine, a framework for jointly controlling camera motion and local object motion in video generation from a single image. The method lifts the input image into a coarse 3D point cloud, allowing users to specify camera trajectories and transformations of local motion handles in a shared world coordinate system. These controls are rasterized into background XYZ, current foreground XYZ, and foreground identity maps. The maps are encoded using a pretrained VAE and incorporated into a Wan video generator through a side branch and LoRA adapters. This paper's design separates the current world-space coordinates of controlled regions from their persistent identities, while local rigid handles support coordinated motion. Experiments demonstrate applications involving joint camera–object motion, local deformation, and articulated motion, with comparisons against several controllable video generation methods.

Critical strengths
  1. The problem has clear practical relevance. Jointly authoring camera and object motion is an important capability for controllable video generation. A shared world coordinate system helps users specify depth changes, rotations, and relative motion that can be difficult to express unambiguously through 2D dragging.

  2. The conditioning representation is well motivated. Separating current position from persistent identity provides complementary cues for motion and correspondence across frames. Using a shared coordinate normalization for the foreground and background also provides an explicit reference for interpreting object motion relative to the scene.

  3. Local motion handles offer useful authoring flexibility. Users can control different parts of an object without requiring a complete skeleton or a category-specific model. The examples in which sparse local controls induce motion beyond the directly controlled region suggest potential value for creative applications.

  4. The experiments include relevant baselines and support some design choices. Comparisons include Wan-Move, ATI, Go-with-the-Track, VerseCrafter, and SymphoMotion. In Table 2, removing foreground XYZ increases R-LPIPS from 0.184 to 0.223, supporting the usefulness of explicit position conditioning within the proposed system. The paper also acknowledges limitations arising from incomplete single-image geometry, locally rigid approximations, and the absence of physical constraints.

Critical weaknesses
  1. The incremental representation contribution is insufficiently validated. DaS already conditions generation on trajectories with fixed initial-coordinate colors, while FlexTraj encodes persistent trajectory and object identities. GenCine’s clearest representation-level distinction is separating current world-space XYZ from handle identity. However, comparisons with these encodings under matched settings and an ablation removing the identity map are missing, making the contribution difficult to isolate.

  2. The evaluation does not directly test 3D motion disambiguation. The metrics mainly assess reconstruction quality, camera accuracy, and video consistency. They do not directly establish correct object depth, orientation, or 3D trajectories, particularly when different intended 3D motions produce similar 2D projections.

  3. Some improvements over strong baselines are modest. On Camera Shooting, GenCine versus Go-with-the-Track achieves PSNR of 16.64 versus 16.61, rotation error of 3.70 versus 3.69, and translation error of 1.00 versus 1.07. Performance is competitive, but gains are small on some metrics.

  4. The training–inference control discrepancy and robustness are insufficiently evaluated. Training handles may change membership and need not be rigid, whereas user-authored handles apply rigid transformations to fixed point sets. The effect of this discrepancy remains unclear, especially under large rotations, occlusion and reappearance, multiple coordinated handles, and multi-object interactions.

Questions for authors
  1. Could the authors compare the proposed representation with DaS-style fixed initial-coordinate encoding and FlexTraj-style persistent identity encoding within the current backbone and training setup? Such a comparison would help isolate the contribution of current XYZ and independent identity conditioning.

  2. How do control accuracy and identity preservation change when the foreground identity map is removed while current XYZ conditioning is retained? GenCine uses handle-level identities, whereas FlexTraj uses point-level trajectory identities. How does this difference in granularity affect rotation, occlusion, and local deformation?

  3. Could the authors construct paired tests with similar 2D projections but different intended 3D motions, and directly evaluate object depth, orientation, and 3D trajectories to assess the benefit of current XYZ over existing correspondence encodings?

  4. How do training regions with changing membership and potentially non-rigid motion generalize to user-specified rigid transformations of fixed point sets? How does performance change with the number of handles and objects, rotation magnitude, and degree of occlusion?

8 comments
AuthorsOpenReview ↗

Response to Reviewer MJ44 (1/2): identity ablation, encodings, and identity granularity

We thank Reviewer MJ44 for their constructive feedback. We are glad to see that they found that "the problem has clear practical relevance," that jointly authoring camera and object motion "is an important capability for controllable video generation," and that "the conditioning representation is well motivated": "separating current position from persistent identity provides complementary cues for motion and correspondence across frames." We are also happy to find that the reviewer found that "local motion handles offer useful authoring flexibility," with examples that "suggest potential value for creative applications," and that "the experiments include relevant baselines and support some design choices." We now answer the questions and concerns raised by the reviewer.

Q2 (first part). Removing the identity map while retaining current XYZ.

We trained exactly this variant: w/o Fid\mathbf{F}^{\mathrm{id}}, with identity removed from both branches and the same data, backbone, adaptation capacity, and budget (Sec. 4.2, Table 2).

Camera ShootingR-LPIPS↓Subj.↑Rot.↓Trans.↓EPE↓PCK↑
w/o Fbg\mathbf{F}^{\mathrm{bg}}0.2350.8895.760.58611.340.644
w/o Ffg\mathbf{F}^{\mathrm{fg}}0.2230.9123.550.39118.630.517
w/o Fid\mathbf{F}^{\mathrm{id}}0.2460.8784.200.45614.320.652
Full0.1840.9023.480.3858.140.804

Removing identity worsens control accuracy (EPE +6.2 px, PCK −0.15, higher camera errors) and identity preservation (the lowest subject consistency of all variants). In the two-character exchange of Fig. 7, the characters blend and end on their original sides without the identity map.

W1 / Q1. DaS-style and FlexTraj-style encodings.

These comparisons target one question: does separating current position from persistent identity help? Our matched ablations answer it directly, under the same backbone, data, and budget:

  • w/o Ffg\mathbf{F}^{\mathrm{fg}} keeps a persistent handle identity without current position. For moving points, this is the signal that DaS's fixed initial-XYZ colors and FlexTraj's track IDs mainly provide. Its tracking is the worst (EPE 18.63).
  • w/o Fid\mathbf{F}^{\mathrm{id}} keeps current position without identity (EPE 14.32).
  • Only their combination reaches 8.14, and it is best on all four control metrics.

These variants are analogues, not reproductions, of DaS and FlexTraj. We believe exact reproductions are not needed to establish this point, for two reasons:

  1. The ablations already show that both current position and identity are required for the best control.
  2. The encodings come with a different control unit. DaS and FlexTraj condition on dense or sparse point tracks. Our contribution centers on the handle-based motion representation: a user moves a few local regions rigidly, together with the camera, instead of specifying a trajectory for every point. Substituting their encodings would change the user input as well as the encoding.

Q2 (second part). Handle-level versus point-level identity.

GenCine's identity labels a motion handle; the per-point geometry comes from the current foreground XYZ map. The difference in granularity affects the three cases the reviewer lists as follows:

  • Rotation. A handle-level label does not encode orientation; the current XYZ of the handle's points does. Orientation is therefore specified in 3D rather than inferred from 2D displacements of point IDs. On Object Interactions, GenCine has the lowest rotation error, RPE-R$_{3D}$ = 1.40° (App. A.2).
  • Occlusion. One color identifies the handle whenever any part of it is visible, and depth ordering selects the visible handle where footprints overlap. The handle is re-identified when it reappears, without per-point track continuity. Fig. 7 includes two crossing characters with mutual occlusion.
  • Local deformation. Handle-level identity represents deformation piecewise, so finer deformation needs more handles. Point-level IDs can express arbitrary deformation, but they require dense trajectories that are hard to author. The robustness sweep we will add (see 2/2) varies the number of handles and probes this trade-off for our representation.
AuthorsOpenReview ↗

Response to Reviewer MJ44 (2/2): direct 3D evaluation, margins, and training–inference controls

W2 / Q3. Direct evaluation of depth, orientation, and 3D trajectories.

We added direct compliance metrics on the 37 DAVIS Object Interactions clips (Table 1(b); protocols in App. A.2):

  • EPE and PCK compare tracks in the generated video with the authored controls projected through the target camera.
  • **RPE-T${3D}$ and RPE-R${3D}∗∗reconstructeachgeneratedvideowithVGGT−** reconstruct each generated video with VGGT-\Omega$ in the shared world frame. They fit a rigid pose to each handle and measure adjacent-frame translation (including depth, normalized by handle size) and rotation errors.
MethodEPE↓PCK↑RPE-T$_{3D}$↓RPE-R_{3D}\ (°)↓
ATI12.950.7700.0471.54
Wan-Move19.500.6290.0401.46
VerseCrafter61.710.2870.0461.56
SymphoMotion77.550.3340.0962.02
Go-w-Track8.390.9140.0401.45
LeviTor73.150.3420.0921.93
GenCine7.860.9220.0361.40

GenCine has the lowest errors on all four measures. Its 3D margins over the dense-track baselines are modest, even though those baselines receive dense ground-truth 2D tracks.

We have not built paired cases with matched 2D projections. The per-handle 3D errors address part of this question: they compare authored and generated relative poses in the world frame, so a motion with the wrong depth direction or rotation direction incurs a large error even when its 2D projection is similar.

W3. Modest improvements.

On Camera Shooting, GenCine is on par with Go-with-the-Track on PSNR (16.64 vs. 16.61) and rotation (3.70 vs. 3.69), and it has a lower translation error (1.00 vs. 1.07). Go-with-the-Track receives dense ground-truth foreground and background tracks, while GenCine uses only sparse 3D handles. GenCine also leads every direct control metric above.

The side-by-side videos (https://neuralcinema.github.io/#compare) show effects that averaged metrics miss. In Robotic arm and Goldfish, Go-with-the-Track moves only the controlled region, while GenCine also produces plausible motion of the connected arm segments and the uncontrolled fish.

Re-generating Table 1 for several seeds means re-running all seven methods, each built on a large video diffusion backbone, on all 75 examples. We are actively running them and trying to complete before the rebuttal deadline.

W4 / Q4. Training–inference control discrepancy and robustness.

During training, handles follow local tracks recovered from videos. Their membership changes as tracks become invalid or reappear, and the selected regions need not move rigidly. At inference, an authored handle moves a fixed point set rigidly.

Both settings render the same three guidance maps: current 3D positions plus persistent handle identities. The model therefore sees the same conditioning format. A rigid authored handle can be viewed as the easy end of the training distribution: its membership stays fixed and its points move consistently.

Training also varies the number and placement of handles and includes background-only clips, which covers partial artist control (Sec. 3.4, App. A.5).

The current qualitative evidence covers several of the stress conditions the reviewer lists:

  • multiple coordinated handles on an articulated robotic arm (Sec. 4);
  • an object rotating during a camera pullback (bus example);
  • two objects crossing with occlusion and reappearance (Fig. 7).

We have not yet measured a systematic sweep. In the final version we will report performance against the number of handles and objects, rotation magnitude, and occlusion level. Limitations remain: rigid handles do not enforce physical interactions, and unseen surfaces are unmodeled (App. A.5, App. I).

We thank the reviewer again for the constructive questions, which directly shaped the identity ablation and the 3D evaluation.

Reply For Answers of Question 1: Thank you for the response and additional experiments. These results have addressed most of my concerns regarding Q1. Although these variants are not fully equivalent to the encodings used in FlexTraj or DaS, I acknowledge that the ablation retaining persistent identity while removing current XYZ can verify the additional contribution of current XYZ within the proposed framework. The new ablation in Table 2 of Sec. 4.2, together with the existing results, provides reasonably clear evidence for the contributions of current XYZ and foreground identity to control performance, as well as their complementary roles.

Reply For Answers of Question 2: Thank you for the response. The newly added Figure 7 addresses my concern about the role of identity: without identity conditioning, the characters in red and blue merge and ultimately fail to complete the specified position swap while with identity conditioning, they complete the specified swap. Regarding the distinction between handle-level and point-level identity in Q2, the authors also provided explanations covering rotation, occlusion, and local deformation. However, I would like to ask one more detail: Appendix E mentions that handles containing fewer than five points are filtered out during training. How many valid points does each handle typically contain during training and evaluation?

Reviewer_UN4fOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to let artists compose camera and object motion together in 3D from a single image, avoiding the ambiguity of image-space controls such as 2D trajectories or drag hints. Generative Cinematographer (GenCine) lifts the input image into a colored point-cloud scaffold. In this shared world frame, the artist authors a camera path and moves selected foreground regions as rigid motion handles, and several independent handles together approximate non-rigid motion. These controls are rasterized into dense correspondence maps: a background XYZ map and a foreground XYZ map that encode world-space positions, plus a foreground identity map that keeps each handle’s identity fixed across frames. The maps are encoded by the video VAE and condition a frozen Wan-based diffusion backbone through a lightweight side branch and LoRA adapters, trained on real and synthetic videos with motion controls recovered from each clip. The work shows that world-space motion, expressed as correspondence maps that a pretrained RGB video model can consume, separates camera motion from object motion without physics simulators or category-specific priors, bringing the shot-composition workflow of classical virtual cinematography to generative video.

Critical strengths

1. A clear motivation and a shared world frame for camera and object motion. The paper shows convincingly that 2D drags can be ambiguous, for example about the direction of a rotation or whether the camera or the object is moving. Authoring both camera paths and object trajectories in the same 3D world frame is a natural way to resolve this, and the background XYZ map gives the model a static reference for interpreting object motion while the camera moves. This matters to my assessment because joint camera and object control is a real need in practice, and the qualitative examples (such as the bus turning during a camera pullback) show that the method can handle it.

2. A simple, practical conditioning design. Encoding world coordinates and handle identities as image-like maps lets the frozen video VAE encode them directly, so a large pretrained model can be adapted with only a side branch and LoRA. The training cost is modest, and the backbone’s generative prior is preserved. This is consequential because it makes the approach easy to adopt and extend to other pretrained video models.

Critical weaknesses

1. Limited novelty over prior 3D-aware control methods. Most elements of the representation already exist. Diffusion as Shader renders 3D tracking videos colored by 3D coordinates for joint camera and object control. VerseCrafter renders background point clouds and object trajectories into conditioning maps. FlexTraj encodes persistent track identities. LeviTor* (CVPR 2025), which is not cited, already controls objects with sparse depth-aware control points and per-instance identities, and shows that the identity signal is critical, which is the same motivation given here for the foreground identity map. The new parts are therefore mainly the separation of identity from current world position and the use of a background XYZ map to put camera and object motion in one world frame. This matters because, with such a narrow delta, the paper needs strong evidence that these specific choices help, which the experiments do not provide (see point 2).

2. The experiments neither show a clear advantage nor test the main claims. On most metrics, GenCine is within a very small margin of Go-with-the-Track and Wan-Move, and on Object Interactions it ranks only third on R-PSNR. With 75 examples and no reported variance, these differences are not meaningful. The inputs also differ across methods (dense 2D trajectories for the baselines, sparse 3D handles plus the ground-truth camera path for GenCine), and the backbones differ. The central claim that 3D authoring resolves 2D ambiguity is supported only qualitatively, since the evaluation measures reconstruction of ground-truth videos. The identity map is never ablated, the background XYZ ablation is confounded by different camera settings, and LeviTor and Diffusion as Shader are not compared. The user study is close to chance against the strongest baselines and does not report the number of participants. This is consequential because the novel components are not shown to improve over existing 2D or depth-based controls.

3. Missing citations of closely related work. Several published works that address the same problem are neither cited nor compared:

Perception-as-Control (ICCV 2025) renders object parts and a world envelope as separate 3D-aware layers to jointly control camera and object motion from a single image, which closely matches the background and foreground maps here. I2V3D (ICCV 2025) lifts a single image into a 3D scene, lets users author camera and object motion in Blender, and uses the renders to guide video generation. This is very close to the artist workflow proposed in this paper. LeviTor (CVPR 2025) controls objects with sparse depth-aware control points and per-instance identities, and shows that the identity signal is critical, which is the same motivation given for the foreground identity map. ObjCtrl-2.5D, Uni3C and VidCRAFT3 also control object motion with depth-aware trajectories or combine camera and object control.

These works are central to the paper’s positioning, not peripheral. Without discussing and comparing against at least Perception-as-Control, I2V3D and LeviTor, it is difficult to judge what the paper adds beyond existing 3D-aware control methods.

[1] Wang, Hanlin, et al. "Levitor: 3d trajectory oriented image-to-video synthesis." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[2] Chen, Yingjie, et al. "Perception-as-control: Fine-grained controllable image animation with 3d-aware motion representation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[3] Zhang, Zhiyuan, Dongdong Chen, and Jing Liao. "I2v3d: Controllable image-to-video generation with 3d guidance." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Questions for authors

Related work. Please discuss how GenCine differs from Perception-as-Control, I2V3D and LeviTor, which address closely related problems but are not cited. Could you also compare against at least one of them where both methods apply, for example LeviTor on object-only motion with a static camera?

Evaluation of 2D ambiguity. The central claim is that 3D authoring resolves cases that 2D controls leave ambiguous. Could you evaluate on a small set of such cases (rotation direction, motion along the depth axis, camera versus object motion) and measure whether the generated motion follows the intended 3D direction?

Ablations of the key components. Could you rerun the background XYZ ablation in Fig. 6 with identical camera settings, and report the ablations on Object Interactions as well? If time permits, an ablation without the foreground identity map would also help.

Reliability of the comparisons. Since the margins over Go-with-the-Track and Wan-Move are very small, could you report the variance across seeds for Table 1, and the number of participants and trials in the user study?

6 comments
AuthorsOpenReview ↗

Response to Reviewer UN4f (1/3): novelty, related work, and the LeviTor comparison

We thank Reviewer UN4f for their thorough and constructive feedback. We are glad to see that they found our work to have "a clear motivation and a shared world frame for camera and object motion," that "the paper shows convincingly that 2D drags can be ambiguous," and that authoring both camera paths and object trajectories in the same 3D world frame "is a natural way to resolve this." We are also happy to find that the reviewer found that "joint camera and object control is a real need in practice," that "the qualitative examples (such as the bus turning during a camera pullback) show that the method can handle it," and that our "simple, practical conditioning design" keeps training cost "modest" while "the backbone's generative prior is preserved," which "makes the approach easy to adopt and extend to other pretrained video models." We further appreciate that the reviewer found the paper "clear and easy to follow," had "no substantive concerns about the paper's soundness," and summarized that the work "separates camera motion from object motion without physics simulators or category-specific priors, bringing the shot-composition workflow of classical virtual cinematography to generative video." We now answer the questions and concerns raised by the reviewer. We followed most of the suggestions; the additions are in Sec. 2, Table 1(b), Table 2, and Appendix A of the revised PDF.

W1 / W3 / Q1. Novelty and closely related work.

We agree that 3D authoring alone is not new. As discussed in our introduction and related work, it dates back to the Virtual Cinematographer (He et al., 1996). Our contribution is how user-authored 3D controls are aligned with the lifted scene and converted into a control representation that a pretrained video model can follow:

  • (i) current world-space XYZ of foreground and background points in one normalized frame;
  • (ii) a separate, time-invariant, handle-level identity map;
  • (iii) local rigid handles over selected regions that need no complete object geometry.

We now quantify (i) and (ii) with a matched identity ablation (see 2/3).

Sec. 2 of the revised PDF now cites these works, and App. A.3 and Table 4 compare them in detail:

  • Perception-as-Control controls part spheres. It conveys relative depth through sphere size, identifies parts by fixed colors, and renders camera motion as a world envelope. It does not provide world-space XYZ.
  • I2V3D customizes an image diffusion model to each input image with LoRA and generates 3D-guided keyframes, which it then interpolates. Part control relies on rigged meshes. GenCine generates the video in one stage without per-scene adaptation.
  • LeviTor controls clustered object-mask points with depth and per-instance labels. Its camera effects come from moving the background mask, not from an explicit 6-DoF camera trajectory. We agree it shares the motivation for an identity signal. GenCine keeps identity separate from current world XYZ and adds a background XYZ map, so camera and object motion share one world frame.
  • ObjCtrl-2.5D realizes depth-augmented 2D trajectories as camera poses without training. Uni3C combines point-cloud camera control with SMPL-X human motion. VidCRAFT3 controls camera, sparse object trajectories, and lighting. None of them uses local rigid handles with current world XYZ and persistent identities.

Comparison with LeviTor. We added LeviTor to the Object Interactions evaluation (Table 1(b) and App. Table 3; 37 DAVIS clips):

EPE↓PCK↑RPE-T$_{3D}$↓RPE-R_{3D}\ (°)↓Rot.↓Trans.↓
LeviTor73.150.3420.0921.935.032.41
GenCine7.860.9220.0361.403.741.05

LeviTor uses an SVD-XT backbone and has no explicit camera input. Its camera errors therefore partly reflect the interface, not only generation quality.

We evaluated LeviTor on the same Object Interactions clips as all other methods, rather than on the static-camera, object-only subset the reviewer suggested, so that its scores are directly comparable in Table 1(b). The object-motion metrics still isolate object motion: the 3D handle errors are measured in the shared world frame, where camera motion is factored out. Side-by-side videos against LeviTor and I2V3D on 15 scenes are at https://neuralcinema.github.io/#compare. They include fixed-camera, object-only scenes such as Goldfish and Sea turtle, which cover the reviewer's suggested setting qualitatively.

AuthorsOpenReview ↗

Response to Reviewer UN4f (2/3): testing the 3D-disambiguation claim and the ablations

Q2. Does generated motion follow the intended 3D direction?

We added two direct compliance measures (App. A.2), so the evaluation no longer relies only on reconstruction of ground-truth videos:

  • Image-space tracking. EPE/PCK compare tracks in the generated video with the authored controls projected through the target camera.
  • 3D handle motion. RPE-T_{3D} and RPE-R_{3D} measure, in the shared world frame, the adjacent-frame translation (including depth) and rotation of each handle after reconstruction and rigid fitting.

On Object Interactions, GenCine has the lowest errors on all four measures (EPE 7.86, PCK 0.922, RPE-T_{3D} 0.036, RPE-R_{3D} 1.40°).

Errors are larger for methods with coarse object-level or depth-aware 2D controls:

  • LeviTor: RPE-T_{3D} 0.092, RPE-R_{3D} 1.93°;
  • VerseCrafter: 0.046 and 1.56°.

The supplement also shows a qualitative example (Fig. 8): a teapot moves toward the camera while keeping its world orientation. The image-space baseline (Wan-Move) produces inconsistent geometry, while GenCine preserves the orientation.

We agree that a paired diagnostic set (rotation direction, depth-axis motion, and camera versus object motion) would test the ambiguity claim most directly; we have not built one. The per-handle 3D errors are our current direct evidence. They compare authored and generated relative poses in the world frame, so a rotation in the wrong direction or motion along the wrong depth direction incurs a large error even when its 2D projection looks similar. Camera and object errors are reported on the same videos, so motion attributed to the wrong source raises both.

Q3. Ablations of the key components.

  • Identity map. We trained a matched current-XYZ-only variant (Sec. 4.2, Table 2). It is worse than the full model on all nine metrics:

    R-LPIPS↓Subj.↑Rot.↓Trans.↓EPE↓PCK↑
    w/o Fid\mathbf{F}^{\mathrm{id}}0.2460.8784.200.45614.320.652
    Full0.1840.9023.480.3858.140.804

    Fig. 7 shows the qualitative effect: without identity, two crossing characters blend and fail to exchange positions.

  • Background XYZ and the camera difference in Fig. 6. In GenCine, the camera path is conveyed mainly by Fbg\mathbf{F}^{\mathrm{bg}}: static background points re-projected under the authored camera. The foreground maps cover only the handle footprints, so they cannot separate camera motion from object motion. The w/o-Fbg\mathbf{F}^{\mathrm{bg}} variant (foreground XYZ + identity) therefore has no channel for the camera condition. Even with the same authored inputs, its camera motion differs from the full model's. The camera difference in Fig. 6 is thus the effect of the removed component, not a setting that can be held fixed: with a static authored camera, the foreground-only result shows unwanted background motion. Sec. 4.2 now states this explicitly. The quantitative ablation on the same cases confirms this. Without Fbg\mathbf{F}^{\mathrm{bg}}, camera errors are the largest (Rot. 5.76, Trans. 0.586, vs. 3.48/0.385 for the full model), and EPE also increases (11.34 vs. 8.14). We acknowledge that the ablations are reported on Camera Shooting only.

AuthorsOpenReview ↗

Response to Reviewer UN4f (3/3): reliability, input/backbone differences, and the user study

W2 / Q4. Small margins, inputs, backbones, and variance.

Margins. On fidelity metrics, GenCine is on par with Go-with-the-Track and Wan-Move. For example, its Object Interactions R-PSNR (15.40) is within 0.06 dB of the best (15.46), and it has the lowest translation error and R-LPIPS in both categories (Table 1). It achieves this with far sparser input (below) and leads every direct control metric (Table 1(b)), which is where the representation's benefit is measured.

Inputs. ATI, Wan-Move, and Go-with-the-Track receive dense ground-truth 2D trajectories for both foreground and background. The background tracks also encode the ground-truth camera motion in image space. GenCine receives sparse 3D handles and a camera path derived from the same reference motion. Thus, GenCine uses substantially sparser inputs than these strong baselines. Importantly, dense ground-truth 2D trajectories are generally unavailable to users during practical inference and can only be provided in controlled evaluations. In contrast, GenCine's sparse 3D handles and camera paths can be directly authored by users, making its control interface more practical for real-world applications. App. A.4 now explicitly clarifies this difference in input requirements. We also provide side-by-side video comparisons using realistic user-authored inputs at https://neuralcinema.github.io/#compare.

Backbones.

  • GenCine: Wan2.1-I2V.
  • ATI, Wan-Move, SymphoMotion: Wan2.1-I2V.
  • VerseCrafter: a Wan-based pipeline.
  • Go-with-the-Track: Wan2.2-A14B.
  • LeviTor: SVD-XT.
  • I2V3D: SDXL and SVD.

Variance. Re-generating Table 1 for several seeds means re-running all seven methods, each built on a large video diffusion backbone, on all 75 examples. We are actively running them and try to complete before the rebuttal deadline.

User study. It was an A/B preference study with 10 video pairs. Each pair was generated by two methods from the same reference image and motion controls. Participants chose the video more consistent with both the reference image and the motion controls, or marked a tie. We collected 12 questionnaires, giving about 120 pairwise judgments (App. A.4).

  • 3D accuracy. GenCine has the lowest per-handle 3D errors of all eight methods: RPE-T_{3D} 0.036 vs. 0.040–0.096, and RPE-R_{3D} 1.40° vs. 1.45–2.02°. It also has the best EPE/PCK (Table 1(b)), despite sparser inputs than the dense-track baselines.
  • Qualitative comparisons. The side-by-side videos at https://neuralcinema.github.io/#compare cover 16 scenes, 7 with a fixed camera. They show differences that averaged fidelity metrics hide (see also Sec. 4, Fig. 5):
    • Living room: GenCine moves the tunnel with less distortion than Wan-Move and VerseCrafter.
    • Bus · turning: GenCine coordinates the turn with the camera pullback, while VerseCrafter keeps the bus's initial orientation.
    • Robotic arm: Go-with-the-Track moves only the gripper region and VerseCrafter adds extraneous boxes, whereas GenCine articulates the connected arm segments.
    • Goldfish: Go-with-the-Track moves the controlled fish while the other fish stay frozen, whereas GenCine keeps the uncontrolled fish swimming naturally.

We thank the reviewer again for the thorough review. The positioning discussion, identity ablation, direct 3D metrics, and LeviTor comparison were added in direct response to the reviewer's comments.

Reviewer_ZSnqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GenCine, a framework that jointly controls camera and object motion from a single image. The method projects image to a 3D point cloud, enables artists to define the camera motion paths and motion handles within a world coordinate system, and projects those paths and handles to the background XYZ map, the foreground XYZ map, and the foreground identity map. A trainable side branch and LoRA adapters condition a frozen Wan-based backbone on these maps. The method reports competitive results on 75 examples from DAVIS and SpatialVid, alongside artist-authored examples.

Critical strengths
  1. Useful methodological contribution by combining local 3D motion handles with shared world-space guidance for joint camera and object control.
  2. Method reports competitive results against several other existing methods. Its ablation study supports several architectural choices, including the use of foreground XYZ guidance.
Critical weaknesses
  1. Specific benefit of persistent foreground identities, which are central to the proposed representation, is not sufficiently isolated. The added value of the representation over related geometric-control approaches, particularly VerseCrafter and Diffusion as Shader, therefore remains unclear. Although the ablations support foreground XYZ guidance, they do not separately evaluate persistent identity guidance.
  2. Evaluation reports camera accuracy and visual fidelity without without direct measurement of compliance with trajectories and orientations of the target objects. This confirms successful implementation of the motion conditioning process, yet does not prove the accuracy of control over both the camera and object motions.
  3. Differences from the strongest baselines are small.
Questions for authors
  1. Perform an identity mapping ablation specifically for isolating the role played by constant foreground identity guidance.
  2. Document any errors in the motion of objects as indicators of deviation from the desired trajectory and orientation of the objects especially when both object motion and camera motion are involved.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZSnq (1/2): identity ablation and comparison with VerseCrafter/DaS

We thank Reviewer ZSnq for their constructive feedback. We are glad to see that they found our work a "useful methodological contribution by combining local 3D motion handles with shared world-space guidance for joint camera and object control." We are also happy to find that the reviewer found that the method "reports competitive results against several other existing methods" and that "its ablation study supports several architectural choices, including the use of foreground XYZ guidance." We now answer the questions and concerns raised by the reviewer. Both are addressed with new experiments in the revised PDF (Table 1(b) and Table 2).

W1 / Q1. Isolating the persistent foreground identity map.

We trained a separate current-XYZ-only variant (w/o Fid\mathbf{F}^{\mathrm{id}}). It keeps the background and foreground XYZ maps and removes the identity map from both the main and side branches. Training data, backbone, adaptation capacity, and training budget match the full model. Evaluation cases, camera paths, handle trajectories, prompts, and seed protocol are identical (Sec. 4.2, Table 2).

Camera ShootingR-LPIPS↓R-PSNR↑Subj.↑Rot.↓Trans.↓EPE↓PCK↑
w/o Fid\mathbf{F}^{\mathrm{id}}0.24615.380.8784.200.45614.320.652
Full0.18417.160.9023.480.3858.140.804

Removing identity degrades all nine metrics. The largest effect is on trajectory following: EPE rises by 76% and PCK drops by 0.15. The variant also has the lowest subject consistency of all ablations.

As Table 2 and Fig. 7 show, current XYZ alone is not enough. It tells the model where the controlled points are, but a moving point's XYZ color changes every frame. Without a persistent handle label, the model is less able to associate a controlled region across frames.

Fig. 7 shows two characters that move around each other and exchange positions. Without the identity map, they blend into one figure and end on their original sides. The full model keeps them distinct and completes the exchange.

W1. Added value over VerseCrafter and Diffusion as Shader (DaS).

  • VerseCrafter represents each object with a single Gaussian, enabling object-level motion control but not independent control of individual object parts. In contrast, GenCine supports part-level motion control, allowing users to independently manipulate different regions of an object, such as a person's arms or legs. In Table 1(b), GenCine substantially outperforms VerseCrafter in EPE (7.86 vs. 61.71) and PCK (0.922 vs. 0.287), while also achieving lower 3D translation error (RPE-T$_{3D}$: 0.036 vs. 0.046).
  • DaS targets a different use case. It is a general video "shader" conditioned on a dense 3D tracking video. Its applications include motion transfer from an existing video, mesh-to-video rendering, camera control, and object manipulation, which moves the 3D points of a segmented object. Its authors note that high-quality tracking videos currently come from animated meshes or existing videos, and that regions without tracking points may be out of control (DaS, Sec. 5). GenCine is designed for authoring new motion from a single image. The user moves a few selected regions with sparse handles together with the camera path, and the model generates the remaining motion. The representations also differ. DaS colors each track by its initial XYZ, so for a moving point the color identifies the track but no longer gives its current position. GenCine separates the two signals: XYZ maps carry current world positions, and the identity map carries a handle-level label.

Our ablations support this separation (Table 2; Fig. 7 shows the qualitative effect). Removing current foreground XYZ but keeping identity gives the worst tracking (EPE 18.63). Removing identity but keeping current XYZ gives EPE 14.32. Only the combination reaches 8.14.

AuthorsOpenReview ↗

Response to Reviewer ZSnq (2/2): object-motion errors under camera motion, and margins

W2 / Q2. Errors in object trajectory and orientation when both object and camera move.

We now measure compliance with the authored object motion directly, alongside camera accuracy on the same generated videos (Table 1(b), with full results in App. A.2, Table 3; 37 DAVIS Object Interactions clips).

  • Trajectory (image space). We project each authored control trajectory through the target camera and track the same points in the generated video with CoWTracker. EPE is the mean pixel error. PCK is the fraction of errors within 1% of the frame diagonal.
  • Trajectory and orientation (3D). We reconstruct each generated video with VGGT-Ω\Omega in the initial camera frame. A single scale is fitted to static background points. We then fit a rigid pose to each handle's tracked support and compare adjacent-frame relative poses with the authored ones. This gives a translation error normalized by handle size (RPE-T_{3D}) and a rotation error in degrees (RPE-R_{3D}). Because poses are compared in the shared world frame, camera motion is factored out of the object-motion error.
MethodRot.↓Trans.↓EPE↓PCK↑RPE-T$_{3D}$↓RPE-R_{3D}\ (°)↓
ATI3.971.2612.950.7700.0471.54
Wan-Move3.761.0819.500.6290.0401.46
VerseCrafter4.021.2461.710.2870.0461.56
Go-w-Track3.731.088.390.9140.0401.45
LeviTor5.032.4173.150.3420.0921.93
GenCine3.741.057.860.9220.0361.40

GenCine has the lowest EPE, the highest PCK, and the lowest 3D translation and rotation errors. It also has the lowest camera translation error and the second-lowest rotation error. Object and camera control therefore hold together.

The 3D scores are reconstruction-based estimates and inherit the accuracy of the depth, pose, and tracking estimators (App. A.2).

W3. Small differences from the strongest baselines.

GenCine achieves comparable fidelity to Go-with-the-Track and Wan-Move while offering more accurate 3D motion control with sparse, practical user inputs, rather than relying on dense ground-truth trajectories.

  1. Practical user inputs versus oracle trajectories. Go-with-the-Track and Wan-Move receive dense ground-truth 2D trajectories for both foreground and background, which are available in controlled evaluations but generally unavailable when users author new motion from a single image. Manually specifying every point's trajectory is impractical. In contrast, GenCine requires only sparse user-authored 3D handles and a camera path, enabling intuitive control over individual object parts and camera motion. Our side-by-side videos (https://neuralcinema.github.io/#compare) demonstrate GenCine using realistic user-authored inputs.

  2. Superior motion controllability. Despite using much sparser inputs, GenCine achieves the best results across all direct control metrics (Table 1(b)). Compared with VerseCrafter, SymphoMotion, and LeviTor, the EPE improvement is substantial (7.86 vs. 61.71–77.55).

  3. Advantages beyond quantitative metrics. Our side-by-side videos show differences not fully captured by standard metrics:

    • Robotic arm: Go-with-the-Track and ATI move only the gripper region, while GenCine produces coordinated motion across the arm.
    • Goldfish: Go-with-the-Track moves the controlled fish but leaves others frozen, whereas GenCine preserves natural motion in the surrounding fish.

We thank the reviewer again and hope these results address the concerns regarding practical controllability, identity isolation, and direct motion compliance.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer ZSnq,

Thank you once again for your time and constructive feedback on our paper. We have posted our response and supplementary experiments addressing your points. These include the new ablation that removes the foreground identity map while keeping current XYZ (Table 2), the comparison with VerseCrafter and Diffusion as Shader, and direct object-trajectory and orientation errors measured while the camera also moves (Table 1(b), App. A.2).

As we approach the end of the discussion period, please let us know if any part of our response requires further clarification or if there are remaining questions we can help resolve.

Best regards,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: identity ablation, direct 2D/3D motion metrics, and positioning

We thank all reviewers for their constructive feedback. We are glad to see that they found our work a "useful methodological contribution" (ZSnq) that addresses "a real need in practice" (UN4f) with "clear practical relevance" (MJ44). Reviewers also found "a clear motivation and a shared world frame for camera and object motion" and "a simple, practical conditioning design" (UN4f), a conditioning representation that is "well motivated," and local handles that "offer useful authoring flexibility" (MJ44). We are also happy that reviewers found that GenCine "reports competitive results" (ZSnq) and that the qualitative examples "show that the method can handle" joint camera and object control (UN4f). The shared concerns were: (1) the persistent identity map was not isolated; (2) object-motion compliance was not measured directly; (3) positioning relative to closely related 3D-aware controls; and (4) small margins and comparison reliability.

The revised PDF marks all changes in dark blue. The key new results are in the main paper (Table 1(b), Table 2, Sec. 2); Appendix A gives the protocols and further analyses.

1. Matched identity ablation (Sec. 4.2, Table 2; App. A.1, Fig. 7). We trained a current-XYZ-only variant (w/o Fid\mathbf{F}^{\mathrm{id}}) with the same data, backbone, adaptation capacity, and budget, removing the identity map from both branches. It is worse on all nine metrics:

Camera ShootingR-LPIPS↓Subj.↑Rot.↓Trans.↓EPE↓PCK↑
w/o Fbg\mathbf{F}^{\mathrm{bg}}0.2350.8895.760.58611.340.644
w/o Ffg\mathbf{F}^{\mathrm{fg}}0.2230.9123.550.39118.630.517
w/o Fid\mathbf{F}^{\mathrm{id}} (new)0.2460.8784.200.45614.320.652
Full0.1840.9023.480.3858.140.804

The maps play complementary roles. Removing background XYZ mainly increases camera error, removing foreground XYZ mainly increases tracking error, and removing identity degrades both tracking and identity retention.

2. Direct motion compliance (Table 1(b); App. A.2, Table 3). On the 37 DAVIS Object Interactions clips, we add two kinds of metric. EPE/PCK compare CoWTracker tracks in the generated video with the authored controls projected through the target camera. RPE-T/R$_{3D}$ measure per-handle translation and rotation errors after VGGT-Ω\Omega reconstruction and rigid fitting.

MethodRot.↓Trans.↓EPE↓PCK↑RPE-T$_{3D}$↓RPE-R_{3D}\ (°)↓
Wan2.15.142.5392.120.2170.0941.99
ATI3.971.2612.950.7700.0471.54
Wan-Move3.761.0819.500.6290.0401.46
VerseCrafter4.021.2461.710.2870.0461.56
SymphoMotion5.202.5877.550.3340.0962.02
Go-w-Track3.731.088.390.9140.0401.45
LeviTor (new)5.032.4173.150.3420.0921.93
GenCine3.741.057.860.9220.0361.40

3. Positioning (Sec. 2; App. A.3, Table 4). Sec. 2 now cites, and App. A.3 discusses, Perception-as-Control, I2V3D, LeviTor, DaS, FlexTraj, VerseCrafter, ObjCtrl-2.5D, Uni3C, and VidCRAFT3. LeviTor is added as a quantitative baseline.

4. Videos: GenCine follows the authored motion where the baselines fail. Side-by-side results for GenCine and seven baselines, including LeviTor and I2V3D, are at https://neuralcinema.github.io/#compare. They cover 16 scenes, with the input images and the authored 3D motion. The videos show recurring baseline failures that averaged metrics do not capture:

  • Only the controlled region moves.

    • Robotic arm: Go-with-the-Track and ATI move a gripper-like region while the rest of the arm stays fixed.
    • Goldfish: Go-with-the-Track moves the controlled fish, but the other fish stay frozen.

    GenCine articulates the connected arm segments, and the uncontrolled fish keep swimming naturally.

  • Distortion and hallucinated content. Wan-Move stretches the robot arm and distorts the tunnel in Living room; VerseCrafter adds extraneous boxes in Robotic arm.

  • Wrong 3D motion. In Bus · turning, VerseCrafter keeps the bus's initial orientation, while GenCine coordinates the turn with the camera pullback.

Comparable fidelity with far sparser input, and the best control. Go-with-the-Track and Wan-Move receive dense ground-truth 2D tracks for every foreground and background point. GenCine receives only a few 3D handles and a camera path, which is the input a user can realistically author. With this sparse input, GenCine is on par with them on fidelity: it has the lowest translation error and R-LPIPS in both categories (Table 1). It also leads every direct control metric (Table 1(b)).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses

Dear Reviewers,

Thank you again for your valuable feedback. We would like to kindly follow up on our rebuttal, which we submitted about 2 days ago. We would greatly appreciate it if you could take a look at our responses when you have a chance.

Please let us know if you have any further questions or concerns. We would be happy to provide additional clarification.

Thank you for your time and consideration!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.