acceptodds
Under review as a conference paper at ICLR 2027

4Director: Controlling Video World Models with Rigid 3D Geometry

Abstract

Precise control over camera and object motion is essential for professional video production. Existing methods control objects only coarsely, through image-plane cues that are ambiguous in depth and rotation or through 3D tracks and blobs that lack complete geometry and lose consistency across viewpoint changes. We introduce 4Director, a video world model conditioned on an explicit 4D scene representation: each object is reconstructed once from the input image as a canonical mesh and moved by one prescribed rigid transformation per frame. This representation provides an intuitive 3D control interface and prevents unobserved geometry from being regenerated independently in every frame. We render the controlled scene as a depth video and introduce a Motion Adapter that transforms this geometric scaffold into video while synthesizing view-consistent appearance, illumination, and non-rigid dynamics. For training, we construct RealCOD-Rigid, a new dataset of 20,774 clips annotated with rigid 3D scenes by our automatic pipeline. We further introduce Identity-Gated IoU (IG-IoU), which jointly evaluates adherence to prescribed object motion and preservation of object identity. Experiments demonstrate that 4Director consistently outperforms prior methods in visual quality and in camera and object control.

Primary area:
generative models
Keywords:
controllable video generation, video world models, camera and object motion control, image-to-video generation, 4D scene representation, geometry-conditioned video diffusion, motion control dataset

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fef3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper present 4Director that addresses joint camera and object motion control in image-to-video generation. It represents objects as complete canonical meshes with per-frame rigid transformations and renders the scene into depth videos to condition a pretrained video generator through a VACE-based Motion Adapter. The paper also introduces RealCOD-Rigid, containing 20774 automatically annotated clips, and Identity-Gated IoU (IG-IoU). Evaluation on 100 held-out clips, representation ablations, and a user study report improvements over four baselines in visual quality and motion control.

Critical strengths
  • The control representation is pratical and well motivated. Complete meshes and rigid transformations provide a direct way to specify object position, orientation, and occlusion alongside camera motion. The automatic annotation pipeline makes this representation usable for training at scale.

  • Consistent improvements supported by controlled ablations. IG-IoU improves from 54.8 for the strongest baseline to 60.4, alongside better visual-quality and camera-control scores. Under the same training recipe, removing surface geometry or rotation reduces IG-IoU to 50.3 or 53.0, supporting the value of the proposed representation.

Critical weaknesses
  • The rotation-control claim is not adequately quantified. IG-IoU measures silhouette overlap and identity, but can miss substantial orientation errors when silhouettes remain similar. Rotation examples are persuasive, but no direct object-orientation evaluation is reported. The Qwen-based identity gate also lacks reported validation against human judgments.

  • The reliability of the reported margins is insufficient. Quantitative evaluation uses 100 clips, and camera errors exclude cases where trajectory estimation fails without reporting per-method success rates. Consequently, methods may be compared on different subsets. Confidence intervals are also absent, making the smaller improvements difficult to assess.

Questions for authors

see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to W1 (Object orientation and identity gate validation)

Screenshots of the study form: https://anonymous-author-page.github.io/4Director/#rebuttal-fef3-study

We thank you for identifying these two gaps in the evaluation. We agree that IG-IoU can miss orientation errors when silhouettes remain similar. We added a direct orientation metric and a human check of the identity judge.

(1) Orientation metric. We use Orient Anything V2 (Wang et al., NeurIPS 2025) to estimate how far the object has turned about the vertical axis since the first frame. After compensating for camera motion, we compare the estimated object turn with the target turn at each of 15 later frames. We report the mean angular error and the percentage of frames with an error within 30° (Acc@30°). As a check of the estimator, its median error on rendered 180° turns with known rotations is 8.3°. The target turn differs between the two evaluations:

  • Authored trajectories (DAVIS-NewTraj). The target is the turn we prescribe. The error is the angular difference between the turn estimated from the generated video and the prescribed turn. We evaluate 59 cases, excluding the soccer ball. 4Director achieves a mean error of 22.0° versus 47.5° for the strongest baseline, SymphoMotion, and an Acc@30° of 81% versus 54% (Table fef3-R1). MotionCtrl and PaC lose the object in three and one cases, respectively.

Table fef3-R1: Object orientation on DAVIS-NewTraj. Errors are measured against the authored rotations. Parentheses indicate 95% confidence intervals.

MethodMean error (°) ↓Acc@30° (%) ↑
MotionCtrl81.7 (74.0, 90.6)25 (20, 30)
PaC50.0 (41.4, 57.7)53 (46, 61)
VerseCrafter58.3 (48.4, 66.2)40 (32, 50)
SymphoMotion4̲7̲.̲5̲ (40.8, 54.2)5̲4̲ (46, 62)
4Director22.0 (15.2, 30.7)81 (71, 89)
  • Test clips with GT videos. On RealCOD-Rigid-Test (original paper), the target is the object's turn in GT videos, estimated in the same way. The error is the angular difference between the turns estimated from the generated and GT videos. On the 100 clips, 4Director achieves a mean difference of 23.2° versus 30.3° for the strongest baseline, PaC, and an Acc@30° of 79% versus 70% (Table fef3-R2). MotionCtrl loses the object in one case.

Table fef3-R2: Object orientation on RealCOD-Rigid-Test (original paper). Differences are measured against rotations estimated from the ground-truth videos. Parentheses indicate 95% confidence intervals.

MethodMean difference (°) ↓Acc@30° (%) ↑
MotionCtrl72.9 (63.5, 82.2)37 (29, 44)
PaC3̲0̲.̲3̲ (24.9, 36.1)7̲0̲ (64, 76)
VerseCrafter35.0 (28.6, 41.9)66 (59, 72)
SymphoMotion32.2 (26.7, 38.1)68 (62, 74)
4Director23.2 (17.4, 29.4)79 (73, 85)

4Director has the lowest error and highest Acc@30° in both settings.

(2) Human validation of the identity gate. We asked 10 annotators to evaluate the same 50 frames from RealCOD-Rigid-Test (original paper), with one frame per clip. For each of the five generation methods, we sampled two frames from each of five judge-score ranges.

For each frame, every annotator viewed the real and generated crops side by side and answered "same object, not broken?" (Yes/No). The target object was outlined in the real crop; method names and judge verdicts were hidden. The majority vote of the 10 annotators determined the human label for each frame. One frame received five Yes and five No votes and was excluded, leaving 49 frames for comparison.

We then compared each judge's pass/fail decisions with these human labels. The original Qwen3-VL-2B judge achieves the highest weighted accuracy, 85%, compared with 67–80% for the other tested judges (Table fef3-R3).

Table fef3-R3: Agreement with human labels on 49 frames. Accuracy uses sampling weights to represent all judged frames in RealCOD-Rigid-Test (original paper).

JudgeAccuracy (%)
Qwen3-VL-2B (paper)85
Qwen2.5-VL-7B80
Qwen3.5-9B67
Qwen3-VL-32B70
AuthorsOpenReview ↗

Response to W2 (Reliability of the reported improvements)

We appreciate your emphasis on the reliability of the reported improvements. We address the two points in turn: camera-estimation failures and confidence intervals.

(1) Camera-estimation failures and different subsets. When the camera of a generated video cannot be reconstructed, no camera error can be computed. This affects only the camera errors: in Table 1, FID, FVD, CLIP-SIM, VBench-I2V, recognition rate and IG-IoU are computed on all 100 clips for every method, and IG-IoU scores a frame in which the object is lost as zero. For the camera errors, our protocol follows common practice in camera-controlled video generation and averages them over the successfully reconstructed videos, as do CamCo (Xu et al., arXiv 2024), CamI2V (Zheng et al., arXiv 2024) and RealCam-I2V (Li et al., ICCV 2025). We agree that the success rates should be reported, and we add two results for the 100 clips of RealCOD-Rigid-Test (original paper): the success rate of every method (Table fef3-R4) and a comparison on the same clips (Table fef3-R5).

Table fef3-R4: Camera-estimation success rate on RealCOD-Rigid-Test (original paper). A clip fails if MegaSaM pose estimation fails or the Sim(3) alignment to the target trajectory fails.

MethodSuccess rate (%) ↑
MotionCtrl25
PaC57
VerseCrafter80
SymphoMotion8̲6̲
4Director92

Because each method's mean error covers a different set of clips, Table fef3-R5 recomputes the errors of each baseline and 4Director on the clips where both succeed. 4Director has lower rotation and translation errors against every baseline.

Table fef3-R5: Camera errors on shared clips of RealCOD-Rigid-Test (original paper). For each baseline, the baseline and 4Director are evaluated on the clips where both succeed.

BaselineShared clipsRot. (°) ↓: baseline / 4DirectorTrans. ↓: baseline / 4Director
MotionCtrl225.05 / 2.540.232 / 0.119
PaC537.20 / 1.860.229 / 0.094
VerseCrafter776.17 / 3.240.171 / 0.109
SymphoMotion823.88 / 3.680.129 / 0.109

(2) Confidence intervals. Table fef3-R6 adds 95% confidence intervals to Table 1 of the paper. Comparing 4Director with the next-best method on the same clips, the gains in IG-IoU (+5.6 over VerseCrafter, 95% confidence interval +3.0 to +8.2) and in translation error (on the shared clips of Table fef3-R5) are significant, whereas the smaller margins in FID, FVD, CLIP-SIM, rotation error and recognition rate lie within the confidence intervals.

Table fef3-R6: Table 1 of the paper with 95% confidence intervals on RealCOD-Rigid-Test (original paper). Camera errors are averaged over each method's successful clips (Table fef3-R4).

MethodFID ↓FVD ↓CLIP ↑Rot. (°) ↓Trans. ↓Rec. ↑IG-IoU ↑
MotionCtrl115.6 (108.0, 123.2)1653.3 (1456.7, 1849.9)27.5 (26.8, 28.2)4.74 (2.99, 6.80)0.231 (0.194, 0.268)32.6 (27.2, 38.4)18.8 (15.6, 22.2)
PaC55.7 (51.8, 59.6)1454.6 (1301.8, 1607.4)30.2 (29.6, 30.8)7.00 (5.60, 8.60)0.228 (0.205, 0.251)80.5 (75.6, 85.2)40.8 (36.9, 44.8)
VerseCrafter55.0 (51.0, 59.0)484.7 (417.5, 552.0)3̲1̲.̲7̲ (31.1, 32.2)6.15 (4.65, 8.21)0.174 (0.150, 0.199)9̲4̲.̲8̲ (92.6, 96.7)5̲4̲.̲8̲ (51.8, 57.7)
SymphoMotion4̲5̲.̲8̲ (42.6, 49.0)4̲0̲5̲.̲4̲ (355.4, 455.3)30.9 (30.4, 31.5)3̲.̲8̲9̲ (2.46, 6.16)0̲.̲1̲3̲2̲ (0.112, 0.153)90.2 (87.1, 93.1)52.9 (48.9, 56.7)
4Director44.1 (40.8, 47.4)370.4 (312.8, 428.0)31.8 (31.3, 32.4)3.65 (2.05, 5.99)0.122 (0.099, 0.144)94.9 (92.8, 96.9)60.4 (56.6, 64.1)

Thanks for the rebuttal. The authors have addressed part of my concerns, I would like to maintain my rating as weak accept.

Thank you very much for taking the time to review our rebuttal. We truly appreciate your thoughtful feedback and your decision to maintain your positive assessment. If you have any further questions, please let us know.

Reviewer_RAGWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to do video editing by adding camera control and object editing. They lift the object and scene to point maps and mesh. Then, render it to depth map as the conditioning to video model and output the edited video. The authors evaluate on VBench-I2V and RealCOD25k. The contributions are 1. disentangle scene representation, 2. motion adapter, 3. and new dataset to do this task.

Critical strengths
  • The results in the supplementary look nice.
Critical weaknesses
  • There is no novelty. The technique from lifting 2d image to 3D for conditioning has done before.
  • Some result as in boat (object reotation) doesn't seem to not disentangle the camera control and object motions. The camera is fixed but the background is moving.
  • It is not that practical to edit the scene in 3D as it is time consuming to manually do this frame by frame. For example, adjusting the human doing pushups is cumbersome or the warrior jumping off the roof.
Questions for authors
  • Why do you need to reconstruct a 3D mesh at all? It seems like wasteful given that you just translate, rotate the object, which could have been done using 3D bounding box as in Mise-en-Video [1]. They can handle the rotation with just 3D box. I think the introduction line 93-98 is incorrect.
  • If there is occlusion in the selected object, Pixel3D is unlikely to work and thus, it will give incorrect depth conditioning but the authors show some cases that work. Do you have to do inpainting if the selected object is occluded? I believe Pixal3D won't perform well with missing parts.

[1] Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation

10 comments
AuthorsOpenReview ↗

Response to W1 (Novelty)

Thank you for prompting us to clarify the technical contribution. We agree that lifting an image to 3D for conditioning has been explored, and our paper discusses these works (Sec. 2, Fig. 2): 3D boxes and rigid trajectories, and proxies lifted from the input image, such as tracked 3D points (SymphoMotion), spheres (Perception-as-Control) and Gaussian blobs (VerseCrafter). Open problems remain: boxes carry no object shape, lifted proxies cover only the visible part of an object, so the sides revealed by a turn or a new view are left to the generator, and point proxies need per-point trajectories that are hard to author. Our contribution is a shared 3D scene with complete object meshes, individual rigid trajectories, and a controlled camera, together with a Motion Adapter trained to generate video from its depth renderings.

The goal is to let users direct a video by specifying simple paths and orientations for each actor and the camera (Sec. 1, Fig. 1). This design addresses two challenges:

  • Unseen parts. When the camera or an object moves far, sides absent from the input come into view. Boxes and partial proxies (3D points, Gaussian blobs) lack object surface geometry there (Fig. 2). We reconstruct every object once as a complete mesh, in one coordinate frame with the background and camera. The geometry of these newly visible sides can then be rendered consistently, together with occlusion and object permanence (Figs. 7–8).
  • Complex animation. Designing the motion of every limb is cumbersome. We ask only for a simple line or curve, represented by one rigid transformation per object and frame. Our Motion Adapter turns the rendered scene into video, and the generator supplies non-rigid motion. We train this mapping on RealCOD-Rigid, our dataset of 20,774 clips annotated with rigid 3D scenes.

We also introduce IG-IoU to evaluate spatial agreement together with object identity preservation.

AuthorsOpenReview ↗

Response to W2 (Background motion with a fixed camera)

Videos with the original and new prompts, and the baselines: https://anonymous-author-page.github.io/4Director/#rebuttal-boat

Thank you for looking closely at this example. We agree that the generated background drifts slightly even though the target camera is fixed. This drift is common in this task: in the same case, the backgrounds of MotionCtrl, SymphoMotion and VerseCrafter also move, and 4Director is the only method that completes the prescribed 180° turn of the boat.

  • Background motion despite a fixed target camera. The prescribed motion rotates the boat while keeping the camera still, so the shoreline should remain stationary. In the generated video, however, the shoreline drifts slightly.
  • Factors that may contribute to the drift. The scene contains open water and a low, uniform shoreline, with few distinctive background features to help maintain a stable view. The original prompt, inherited from the source clip, also describes the boat as "moving through calm blue water, leaving a wake." This wording may encourage additional motion beyond the prescribed boat rotation.
  • Result with an explicit static-camera prompt. We repeated generation after adding "The camera stays completely still." to the original prompt, keeping the input image, control trajectories, and random seed unchanged. The drift disappears and the shoreline stays still, while the boat still completes the prescribed 180° turn.
AuthorsOpenReview ↗

Response to W3 (Trajectory authoring effort)

The rooftop jump with every frame and with five keyframes: https://anonymous-author-page.github.io/4Director/#rebuttal-rooftop

Thank you for raising this practical concern; we should have made this clearer in the paper. 4Director does not require frame-by-frame editing:

  • A few keyframes are enough. The user places each object and the camera at a few key moments on a timeline, such as the take-off, peak, and landing of the warrior's jump. The frames in between are interpolated automatically (Appendix H.1, Fig. 10). With only five keyframes (start, take-off, peak, landing, and end), the warrior still runs, jumps off the roof, and lands.
  • No per-frame poses. For push-ups or a jump, users move the subject as a whole, and the model fills in the limb motion. They do not need to set the body's pose in each frame, avoiding the effort of animating every joint throughout the sequence.
Reviewer_V93FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

4Director enables camera and object-motion control for image-to-video generation by building an explicit rigid 4D scene. Each selected object is reconstructed as a canonical mesh, then placed with the background point cloud and camera in a shared coordinate frame and transformed rigidly at every frame. Depth renderings of that scene condition a pretrained video generator through a Motion Adapter, while the generator supplies appearance, illumination, and residual non-rigid motion. The paper further contributes RealCOD-Rigid, an automatically annotated dataset of 20,774 clips, and Identity-Gated IoU (IG-IoU), a metric that combines object placement with a gate for identity preservation. Against four baselines for joint camera and object control, the reported experiments improve visual quality, camera error, object control, and user ratings. Ablation experiments attribute these gains in part to rotation modeling and complete surface geometry.

Critical strengths
  1. Complete canonical meshes paired with one rigid transform per frame provide a well-motivated and useful control representation. Compared with 2D boxes, sparse point tracks, or Gaussian proxies, this design handles depth, orientation, and newly exposed surfaces more directly.

  2. Beyond the control signal, this work presents a coherent end-to-end system rather than an isolated conditioning component. It gives substantial implementation detail for scene construction, depth encoding, the Motion Adapter, the authoring interface, dataset creation, and the evaluation protocol.

  3. Evaluation spans complementary facets of the task. Alongside FID, FVD, and text alignment, the study measures camera-trajectory error, object control, VBench dimensions, and user ratings, with useful qualitative cases covering rotation, occlusion, and an object that exits and later re-enters the view.

  4. Targeted ablations clarify which design choices matter. Comparisons with 2D boxes, 3D boxes, and a non-rotating 3D mesh separate the contributions of depth, orientation, and full geometry. The LoRA variant also indicates that the improvement is not merely a consequence of the full adapter's parameter count.

Critical weaknesses
  1. External validity remains uncertain because the principal quantitative benchmark contains only 100 clips from RealCOD-25K, the same source collection used for training, and its target trajectories are recovered with components closely related to the annotation pipeline. The results therefore say little about different image domains, reconstruction failures, unusual camera motion, or trajectories authored independently from source videos. The 61 authored examples receive only qualitative evaluation.

  2. Reliability of IG-IoU is insufficiently established. The metric relies on propagated masks and a learned vision-language judgment of identity and structure, yet the reported results do not compare those decisions with manual annotations, provide agreement or calibration statistics, or examine sensitivity to the confidence threshold. Stronger validation is important because the largest object-control claim rests on this metric.

  3. Rigid-body control substantially restricts the method for people, animals, and other articulated or deformable subjects. Although the generator can introduce incidental non-rigid motion, users cannot specify it, and the presented failure case shows that depth conditioning may instead freeze an articulated subject in its initial pose.

Questions for authors
  1. How does the method perform on a benchmark collected from a different video source, and on quantitatively measurable trajectories authored independently rather than recovered from existing clips?

  2. What level of agreement does the identity gate achieve with manual judgments, and how sensitive are recognition rate and IG-IoU to the selected judge, prompt, mask-propagation method, and confidence threshold?

  3. What proportion of source clips and user-provided images fail during scene construction at inference time, and how do errors in mesh alignment, camera estimation, or depth propagate into the generated video?

  4. Could the authors provide more details for the user study and automatic metrics ?

7 comments
AuthorsOpenReview ↗

Response to W1/Q1 (Generalization and authored trajectories)

Examples of the authored trajectories: https://anonymous-author-page.github.io/4Director/#rebuttal-traj

We sincerely thank you for encouraging a broader evaluation of generalization. We added two quantitative evaluations on DAVIS (Pont-Tuset et al., arXiv 2017), a video object segmentation benchmark with pixel-level annotations, to test generalization beyond RealCOD-25K.

(1) A different video source. We evaluate DAVIS source sequences, using the first frame as input and the camera and object trajectories recovered from each source video as the target motion. The original videos and object masks provide evaluation references (Tables V93F-R1–V93F-R2). 4Director achieves an IG-IoU of 53.2, compared with 44.1 for the strongest baseline, and the lowest FID and FVD.

Table V93F-R1: DAVIS: visual quality, text alignment, and camera and object control under recovered source-video trajectories.

MethodFID ↓FVD ↓CLIP ↑Succ. (%) ↑Rot. (°) ↓Trans. ↓Rec. ↑IG-IoU ↑
MotionCtrl150.62297.729.442.49.810.30235.017.3
PaC103.62054.330.260.68.750.30653.822.9
VerseCrafter7̲9̲.̲8̲740.33̲1̲.̲6̲7̲8̲.̲8̲4.650̲.̲1̲8̲3̲8̲5̲.̲8̲4̲4̲.̲1̲
SymphoMotion81.57̲2̲0̲.̲2̲3̲1̲.̲6̲7̲8̲.̲8̲3̲.̲7̲6̲0̲.̲1̲8̲3̲78.839.9
4Director76.6650.132.381.82.580.14786.753.2

Table V93F-R2: DAVIS: VBench image-to-video quality scores for the cases in Table V93F-R1.

MethodSubject consist.Background consist.Motion smooth.Dynamic degreeAestheticImagingI2V subjectI2V background
MotionCtrl77.088.694.581.844.345.684.286.0
PaC87.890.195.290.948.959.790.591.0
VerseCrafter89.291.998.675.851.36̲6̲.̲7̲95.495.9
SymphoMotion88.590.89̲7̲.̲6̲8̲4̲.̲8̲49.864.19̲4̲.̲7̲9̲5̲.̲5̲
4Director8̲9̲.̲0̲9̲1̲.̲0̲9̲7̲.̲6̲81.85̲0̲.̲1̲69.694.494.9

(2) Independently authored trajectories. Using the same DAVIS first frames, we evaluate 61 independently authored trajectories, called DAVIS-NewTraj (Tables V93F-R3–V93F-R4). Their motion is specified independently of the source videos and covers nine types: 180° object turns with a static camera or a camera orbit of ±180°; camera orbits of ±120° or ±180° around a static object; and object translations of ±3 body widths with a following camera.

These trajectories have no corresponding ground-truth videos or object masks, so we do not report IG-IoU. We evaluate camera control against the prescribed camera paths and use recognition rate to measure object identity preservation: 4Director achieves 90.4%, compared with 86.7% for the strongest baseline.

Table V93F-R3: DAVIS-NewTraj: evaluation on 61 independently authored trajectories.

MethodFID ↓FVD ↓CLIP ↑Succ. (%) ↑Rot. (°) ↓Trans. ↓Recognition (%) ↑
MotionCtrl162.92230.526.723.035.510.36526.3
PaC150.42336.030.250.830.390.37272.0
VerseCrafter130.51468.83̲1̲.̲4̲70.528.530.2508̲6̲.̲7̲
SymphoMotion1̲2̲8̲.̲6̲1̲4̲6̲0̲.̲4̲30.77̲8̲.̲7̲9̲.̲2̲7̲0̲.̲0̲8̲5̲83.3
4Director127.21417.931.682.08.930.08290.4

Table V93F-R4: DAVIS-NewTraj: VBench image-to-video quality scores for the cases in Table V93F-R3.

MethodSubject consist.Background consist.Motion smooth.Dynamic degreeAestheticImagingI2V subjectI2V background
MotionCtrl68.684.592.788.541.047.682.585.2
PaC89.59̲2̲.̲1̲96.677.045.762.090.592.2
VerseCrafter89.592.098.782.04̲8̲.̲9̲6̲9̲.̲4̲96.496.8
SymphoMotion87.991.997.79̲0̲.̲2̲4̲8̲.̲9̲67.994.995.5
4Director8̲8̲.̲6̲92.29̲7̲.̲9̲91.849.171.39̲5̲.̲7̲9̲5̲.̲6̲
AuthorsOpenReview ↗

Response to W2/Q2 (Reliability of IG-IoU)

Screenshots of the study form: https://anonymous-author-page.github.io/4Director/#rebuttal-study

Thank you for highlighting the need to validate the identity gate. We added a human validation study and sensitivity tests to assess the reliability of IG-IoU.

(1) Agreement with human judgments. We asked 10 annotators to evaluate the same 50 frames from RealCOD-Rigid-Test (original paper), with one frame per clip. For each of the five generation methods, we sampled two frames from each of five judge-score ranges.

For each frame, every annotator viewed the real and generated crops side by side and answered "same object, not broken?" (Yes/No). The target object was outlined in the real crop; method names and judge verdicts were hidden. The majority vote of the 10 annotators determined the human label for each frame. One frame received five Yes and five No votes and was excluded, leaving 49 frames for comparison.

We then compared each judge's pass/fail decisions with these human labels. The original Qwen3-VL-2B judge achieves the highest weighted accuracy, 85%, compared with 67–80% for the other tested judges (Table V93F-R5).

Table V93F-R5: Agreement with human labels on 49 frames. Accuracy uses sampling weights to represent all judged frames in RealCOD-Rigid-Test (original paper).

JudgeAccuracy (%)
Qwen3-VL-2B (paper)85
Qwen2.5-VL-7B80
Qwen3.5-9B67
Qwen3-VL-32B70

(2) Sensitivity to evaluation settings. Using the same generated videos from RealCOD-Rigid-Test (original paper), we change one evaluation component at a time:

  • Judge: compare the original judge with three alternative vision–language models.
  • Prompt: reword the instructions while retaining the same five verdict labels, or swap the crop order to show the generated crop first.
  • Masks: change the segmentation model or propagation direction, keeping the first-frame prompt fixed. We then recompute mask overlap and repeat the identity check on crops defined by the new masks.
  • Threshold: test minimum passing scores of 0.3, 0.5, 0.7, and 0.9. A higher threshold makes the identity check stricter.

4Director achieves the highest IG-IoU in every tested setting, exceeding the strongest baseline by 5.4–8.3 points (Table V93F-R6).

Table V93F-R6: Sensitivity of IG-IoU on RealCOD-Rigid-Test (original paper). All settings use the same generated videos. The first row reports the paper's setting and scores; each subsequent row changes one evaluation component (–: unchanged).

JudgePromptMasksThresholdMotionCtrlPaCVerseCrafterSymphoMotion4Director
Qwen3-VL-2BpaperSAM3 bidirectional≥ 0.518.840.85̲4̲.̲8̲52.960.4
Qwen2.5-VL-7B–––11.336.84̲9̲.̲9̲45.955.7
Qwen3.5-9B–––7.226.337.43̲7̲.̲6̲45.9
Qwen3-VL-32B–––7.324.331.53̲2̲.̲3̲38.5
–paraphrased––33.144.85̲6̲.̲9̲56.162.3
–images swapped––32.844.55̲6̲.̲6̲56.062.0
––SAM3 forward only–18.840.85̲5̲.̲0̲53.160.7
––SAM2.1 bidirectional–18.240.65̲2̲.̲8̲51.958.2
–––≥ 0.318.940.95̲4̲.̲9̲53.060.5
–––≥ 0.715.439.15̲3̲.̲2̲51.359.6
–––≥ 0.912.636.14̲9̲.̲1̲47.756.8

4Director ranks first in recognition rate in eight of eleven settings (Table V93F-R7). In the remaining three (paraphrased prompt, swapped crops, and SAM2.1 masks), it ranks second but remains close to VerseCrafter, with gaps of 0.1–1.1 percentage points. Its IG-IoU remains 5.4 points higher in all three cases (Table V93F-R6).

Table V93F-R7: Recognition rate under different evaluation settings. Settings match Table V93F-R6.

JudgePromptMasksThresholdMotionCtrlPaCVerseCrafterSymphoMotion4Director
Qwen3-VL-2BpaperSAM3 bidirectional≥ 0.532.680.59̲4̲.̲8̲90.294.9
Qwen2.5-VL-7B–––15.265.98̲4̲.̲7̲74.885.8
Qwen3.5-9B–––8.543.16̲1̲.̲3̲57.266.8
Qwen3-VL-32B–––8.536.94̲8̲.̲9̲46.853.3
–paraphrased––79.998.1100.098.99̲9̲.̲7̲
–images swapped––78.397.199.098.69̲8̲.̲9̲
––SAM3 forward only–32.780.49̲5̲.̲1̲90.995.4
––SAM2.1 bidirectional–31.480.392.988.19̲1̲.̲8̲
–––≥ 0.332.880.89̲5̲.̲1̲90.595.3
–––≥ 0.724.974.19̲1̲.̲4̲86.292.9
–––≥ 0.918.464.48̲2̲.̲8̲77.786.8
AuthorsOpenReview ↗

Response to W3 (Limits of rigid-body control)

Videos of Appendix G and the pose-control runs: https://anonymous-author-page.github.io/4Director/#rebuttal-dance

We appreciate your careful assessment of the limitations of rigid-body control. Our rigid trajectories specify each subject's overall position and orientation, while the generator supplies local motion. This design aims to save ordinary users and artists time and effort: they move each subject as a whole instead of posing it in every frame. We agree that this does not directly specify joint poses or deformations.

(1) Articulated subjects. The paper shows a running person, a crawling baby, and a skier (Figs. 12–13), as well as dogs, a dancer, and turning animals (Figs. 14–15). These examples demonstrate that rigid controls can produce non-rigid motion, although the limb movements are generated rather than prescribed.

(2) The failure case and explicit pose control. We agree that this failure case reflects a limitation of rigid control. In Appendix G, the rigid mesh keeps the handstand of the input image and slides three body widths to the right with the camera following, so the dancer's depth is the same in every frame. The control thus asks for a dancer gliding sideways in a fixed handstand, and the model follows it rather than producing a plausible dance: a rigid trajectory cannot express the limb motion such a movement would need. To examine whether articulated motion can be prescribed through the same interface, we recovered the dancer's poses from the source video using GVHMR (Shen et al., SIGGRAPH Asia 2024) and represented them as an animated SMPL-X mesh (Pavlakos et al., CVPR 2019). We replaced the rigid mesh in the depth control while keeping the model, input image, and seed unchanged:

  • Original motion: the body follows the source video. The generated dancer follows the source dance moves instead of holding the handstand.
  • Original motion with a slide: the same body also slides to the right, with the camera following. With the 3-body-width slide of Appendix G, the moves hold at first, but the body deforms near the end. With a shorter, more realistic slide of 1.5 body widths, the dancer performs the source moves throughout.

These runs show that the limitation can be addressed with an animated mesh, which the depth-conditioning interface accepts without changes to the model. They also show what remains limited. The poses must come from an external source, such as a pose estimator or an artist's animation, which needs a reference video or more authoring work than a rigid trajectory; we therefore see an animated mesh as an option for when the generated motion is not enough, not as the default. In addition, the model follows the given motion rather than correcting it, so a less realistic motion, such as the long slide, still degrades the result. Without an animated mesh, limb motion is generated rather than prescribed, as in (1), and is not guaranteed. We will clarify this limitation in the revision.

Comments on the paper

2 comments

Dear Reviewers,

As the discussion phase is coming to an end, could you please take a moment to review the authors' responses and finalize your ratings?

If you have any remaining concerns or questions, please feel free to raise them with the authors before the discussion closes.

Thank you very much for your time and support!

Best regards,

Xiaojuan

AuthorsOpenReview ↗

Summary of Discussion for AC and Reviewers

Dear AC and Reviewers,

We sincerely thank all reviewers for their constructive feedback and engagement. As the discussion period closes, we provide a concise summary for convenience.

Strengths noted by reviewers

  • A well-motivated control representation: complete meshes with rigid transformations handle depth, orientation, and newly exposed surfaces (V93F, fef3).
  • A coherent end-to-end system with substantial implementation detail (V93F).
  • Clear presentation (RAGW, fef3), targeted ablations (V93F, fef3), and strong visual results (RAGW).

Main concerns and our responses

  • Generalization beyond the training source (V93F): New quantitative evaluation on DAVIS, a different video source: IG-IoU 53.2 vs. 44.1 for the strongest baseline, with the lowest FID and FVD. On 61 independently authored trajectories (DAVIS-NewTraj), 4Director achieves the highest recognition rate (90.4% vs. 86.7%) and the lowest camera errors.
  • Reliability of IG-IoU (V93F, fef3): A human validation study (10 annotators) shows 85% weighted agreement for our identity judge. In a sensitivity analysis over judges, prompts, mask models, and thresholds, 4Director achieves the highest IG-IoU in all 11 settings (margin +5.4 to +8.3).
  • Direct orientation evaluation (fef3): Using Orient Anything V2, 4Director's mean orientation error is 22.0° vs. 47.5° for the strongest baseline on authored trajectories, and 23.2° vs. 30.3° on the test set.
  • Statistical reliability (fef3): We report per-method camera-estimation success rates (4Director highest, 92%), comparisons on shared clips (lower rotation and translation errors than every baseline), and 95% confidence intervals. The IG-IoU gain is significant (+5.6, CI +3.0 to +8.2).
  • Novelty and meshes vs. 3D boxes (RAGW): A 3D box encodes pose but not surface geometry; our ablation shows complete meshes outperform 3D boxes (IG-IoU 60.4 vs. 50.3; FVD 370.4 vs. 556.0). Mise-en-Video appeared on arXiv after the ICLR submission deadline and is concurrent work; its video model is not released, so a direct comparison is not currently possible.
  • Authoring effort (RAGW): Users set a few keyframes rather than per-frame poses; five keyframes suffice for the rooftop jump. As RAGW observed, keyframes even yield cleaner results than the automatically estimated per-frame trajectory.
  • Also addressed: background drift with a fixed camera (removed by an explicit static-camera prompt; baselines also drift) (RAGW), and scene-construction failure rates (2.3% at single-image inference) (V93F).

Limitations we acknowledge

  • Rigid trajectories do not prescribe limb motion; we show that an animated mesh can be used through the same interface without changing the model (V93F).
  • Heavily occluded objects would require amodal completion before reconstruction (RAGW).

Should any concern remain unclear, we would be glad to address it in the remaining discussion time.

We thank the AC and reviewers again for their time and thoughtful feedback.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.