acceptodds
Under review as a conference paper at ICLR 2027

Learning Rotation-Aware Video Motion Control from Real-World Videos

Abstract

Motion control for video generation prescribes the translation and the rotation of the content of a clip. Training a model to follow an explicit rotation requires per-frame six-degree-of-freedom supervision that real footage does not provide. Prior work therefore relies on simulation or robotics data, but this introduces a domain shift for general motion editing. We present Swivel, a framework that learns 6-DoF video motion control from real-world video alone. It builds on an observation about 3D foundation models: from their outputs alone, the rigid parts of a clip can be proposed automatically and the 6-DoF motion of each part can be measured without any human marking, which turns ordinary footage into a large corpus of clips paired with 6-DoF trajectories. We then train a video diffusion model on this corpus for sparse but precise 6-DoF control, with rotating cubes in screen space as the condition signal, and the model supports both image-to-video generation and inpainting-based video editing. Results show that Swivel follows the 6-DoF motion signals more closely than prior motion controls and generates higher-quality videos at 720p than the baselines, and human evaluation shows a large preference for Swivel over the baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video generation, motion control, 6-DoF control, video editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JFzNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Swivel, a framework for sparse 6-DoF motion control in image-to-video generation and inpainting-based video editing. Its annotation pipeline uses pretrained models to recover scene geometry, cameras, human mesh trajectories, and generic 3D point tracks from real videos. It identifies local rigid anchors, estimates their translation and rotation, and filters unreliable estimates. These trajectories are rendered as colored cube videos and used to condition a Wan 2.2-based video diffusion model.

The resulting interface allows users to specify the position and orientation of individual subjects or their parts. The authors evaluate the method on held-out real videos through quantitative baseline comparisons, qualitative demonstrations, a human preference study, and an ablation of cube selection and density.

Critical strengths
  • Automatic 6-DoF supervision from real footage. Automatically extracting translation and rotation supervision from ordinary videos is a practical contribution, reducing reliance on synthetic motion data or specialized capture setups.

  • Sparse, explicit control over local parts. Users can specify the position and orientation of individual subjects or their parts across people, animals, and objects. The interface supports both generation and editing.

  • Favorable reported results. The method outperforms the evaluated baselines across the reported quantitative metrics, and the user study shows substantial preferences for its motion following, subject consistency, and overall quality.

  • Qualitative evidence of responsiveness to motion edits. Different controls applied to the same scene and prompt produce corresponding changes in motion, including reversed windmill rotation and altered clock-hand motion. These examples support the model’s responsiveness to user-authored controls. [Figure 5.]

Critical weaknesses

1. Limited isolation of the proposed method’s contribution

This is my main concern. The experiments show favorable performance of the complete system, but do not clearly isolate the proposed conditioning’s contribution. Swivel uses Wan 2.2 14B, VACE uses Wan2.1-VACE-14B, and the evaluated DaS and 3DTrajMaster implementations use CogVideoX-5B. Differences in backbone and training setup make it difficult to attribute the gains to the proposed components. The only ablation varies control selection and density, without testing the contribution of orientation information under otherwise matched conditions. [Section 4; Table 3; Appendix D.]

A position-only versus position-and-orientation comparison using the same checkpoint, data, and training procedure would help establish the incremental benefit of orientation within the system. This would not fully explain the gains over other methods, but would strengthen the evidence for this component. The reported system-level improvements and potential value of the annotation pipeline remain meaningful.

2. Limited direct assessment of rotation fidelity

Quantitative motion-adherence metrics assess projected anchor positions, without directly measuring orientation. Qualitative examples and the user study support rotation following, but its precision and reliability are less directly characterized. A focused evaluation of orientation accuracy, through a reliable rotation-specific measure or targeted human assessment, would strengthen the evidence. [Section 4; Figures 5-7; Appendices D and E.]

3. Limited comparative coverage leaves broader competitiveness uncertain

The quantitative comparison covers VACE and DaS for image-to-video generation and only VACE for inpainting; 3DTrajMaster appears in the qualitative evaluation and user study. These are relevant comparisons, but provide limited coverage. [Table 2; Section 4.]

Although the examples demonstrate useful rotation control, their visual quality and motion range are not consistently compelling to me. Some motions are modest, leaving competitiveness with strong 2D motion-control methods uncertain. This is a preliminary impression, not an established ranking.

A broader quantitative comparison on motions all methods can represent would better assess whether explicit orientation control comes with competitive visual quality and general motion adherence.

Minor Weaknesses and Reporting Issues

  • The related-work discussion should also cover [1], which uses a flexible reference-video representation for both 2D object manipulation and camera control. For camera control, it reprojects an estimated 3D point cloud along prescribed trajectories, connecting geometric motion specifications to image-space guidance

  • Presentation and clarity. The method section could be clearer, particularly regarding the motivation for the conditioning representation and its relationship to the pretrained model. The experimental narrative could better explain the distinct roles of annotation evaluation, positional adherence, and rotation following.

  • Data provenance is insufficiently documented. The paper describes curation, corpus size, and test-set composition, but does not identify the source datasets or sufficiently explain the training/test split. These details are needed to reproduce the pipeline and interpret the evaluation.

[1] Singer et al., Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising, ICLR 2026.

Questions for authors
  1. Contribution of orientation conditioning: How does Swivel compare with a position-only variant trained using the same starting checkpoint, data, and training procedure?

  2. Rotation fidelity: Can the authors evaluate orientation following separately from translation by varying the requested rotation direction, angle, or axis while keeping the initial image, prompt, and a suitable single anchor’s center trajectory fixed? A focused rigid-object subset, assessed with a reliable rotation-specific measure or targeted human judgments, could make this evaluation more diagnostic.

  3. Broader comparison: Could the authors include an additional strong motion-control baseline on shared tasks, particularly to assess visual quality and motion adherence beyond the current comparisons?

  4. Data sources and separation: Which public datasets supply the training and evaluation videos, and how is the held-out split constructed relative to the original source videos?

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/4]

Thank you for your time and effort in reviewing our paper. We appreciate that you find it valuable to scalably learn 6-DoF supervision from real-world videos, and that part-level motion control gives better precision. We address each question below.


W1: Limited isolation of the proposed method’s contribution

This is my main concern. The experiments show favorable performance of the complete system, but do not clearly isolate the proposed conditioning’s contribution. Swivel uses Wan 2.2 14B, VACE uses Wan2.1-VACE-14B, and the evaluated DaS and 3DTrajMaster implementations use CogVideoX-5B. Differences in backbone and training setup make it difficult to attribute the gains to the proposed components. The only ablation varies control selection and density, without testing the contribution of orientation information under otherwise matched conditions. [Section 4; Table 3; Appendix D.]

A position-only versus position-and-orientation comparison using the same checkpoint, data, and training procedure would help establish the incremental benefit of orientation within the system. This would not fully explain the gains over other methods, but would strengthen the evidence for this component. The reported system-level improvements and potential value of the annotation pipeline remain meaningful.

Q1: Contribution of orientation conditioning: How does Swivel compare with a position-only variant trained using the same starting checkpoint, data, and training procedure?

We thank the reviewer for raising this point. We answer in two parts below.

Controlled training comparison: position-only versus position-and-orientation. We agree that the matched comparison suggested by the reviewer is more convincing. Although training a MoE video model of about 28B parameters is costly, we train two models that differ only in the control signal. One is conditioned on our cubes, which carry position and orientation. The other is conditioned on a blob at the same position, which carries no orientation. Both start from the same public checkpoint, Wan2.2-Fun-A14B-Control, which has a high-noise and a low-noise expert of 14B each and about 28B in total. Both use the same data and training settings. Each model is trained for 6,250 iterations at 240p and then 1,000 iterations at 720p. These two models are trained separately for this comparison with a shorter schedule, and are not the model in Table 2.

To test how well each model controls object rotation, we generate videos from 10 first frames created with a commercial image model. In each case, the position of the subject is held fixed and only its rotation changes. We test two versions of the control. In the first, each subject is marked by one cube or one blob. This tests the extremely sparse case of a single control signal, which our method supports. In the second, we place two smaller cubes or blobs on the left and right parts of the subject, and they move rigidly with it. With two points, even a position-only signal carries some rotation through their relative motion. For each video, we check whether the generated subject follows the requested rotation. A video counts as correct only if the subject turns in the requested direction by about the requested angle. A video counts as a failure if the subject does not turn, turns too little or too much, or shows hallucinated motion. We judge each video by visual inspection, because whether the subject follows the requested rotation is clearly visible in these cases. Every video is shown on the page linked below together with its verdict, so each case can be checked directly.

VariantsOne signalTwo signals
Cube (position-and-orientation)10/1010/10
Blob (position-only)0/105/10

With two signals, a position-only control can already express some rotation, yet the blob model follows it in only 5 cases, while the cube model follows it in all 10. With a single signal, the blob model does not follow the rotation in any case, while the cube model still follows it in all 10. This shows that the orientation in our control signal is what lets the model follow the requested rotation. All generated videos are shown at this anonymous link.

AuthorsOpenReview ↗

Rebuttal by Authors [2/4]

Extended baselines. The reviewer points out that the baselines use different base models and that some of them are weaker than ours. We agree. It is infeasible for us to retrain every baseline method on our base model, so we add Wan-Move [1] and TTM [2], two recent open-source motion-control baselines built on stronger base models than CogVideoX-5B. TTM was also pointed out by the reviewer. We run TTM on Wan 2.2 14B, which Swivel also builds on, and Wan-Move is built on Wan 2.1 14B. We evaluate all methods on the held-out evaluation set of Table 2 and measure rotation error by running our annotation pipeline on the generated videos. The full results are in the table in our response to W3. Swivel has the lowest rotation error and the best visual quality, including against TTM, which builds on the same Wan 2.2 14B model.

[1] Chu et al., Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance, NeurIPS 2025.

[2] Singer et al., Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising, ICLR 2026.

We will add both comparisons to the revised draft.


W2: Limited direct assessment of rotation fidelity

Quantitative motion-adherence metrics assess projected anchor positions, without directly measuring orientation. Qualitative examples and the user study support rotation following, but its precision and reliability are less directly characterized. A focused evaluation of orientation accuracy, through a reliable rotation-specific measure or targeted human assessment, would strengthen the evidence. [Section 4; Figures 5-7; Appendices D and E.]

Q2: Rotation fidelity: Can the authors evaluate orientation following separately from translation by varying the requested rotation direction, angle, or axis while keeping the initial image, prompt, and a suitable single anchor’s center trajectory fixed? A focused rigid-object subset, assessed with a reliable rotation-specific measure or targeted human judgments, could make this evaluation more diagnostic.

We thank the reviewer for this suggestion. We agree that rotation fidelity should be assessed directly, and we answer in two parts below.

Rotation accuracy. Rotation can be measured directly with our own annotation pipeline. On the held-out evaluation set, we run the pipeline on both the real source videos and the videos generated by each method, and compare the recovered rotations. We include all baselines, together with the two added baselines Wan-Move [1] and TTM [2]. Rotation errors over the evaluation set are reported below. Swivel has the lowest rotation error of all methods.

Swivel (ours)VACEDaSWan-MoveTTM
Rotation error (°) ↓13.236.223.616.732.9

[1] Chu et al., Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance, NeurIPS 2025.

[2] Singer et al., Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising, ICLR 2026.

Rotation with position held fixed. In real videos, rotation and position usually change together, so changing only the rotation of a real clip often gives an implausible motion, and videos in which a subject turns while its position stays fixed are hard to find. To run the test the reviewer suggests, we therefore build our own cases. We generate 10 images with a commercial image model and use them as the first frames of our test samples. In each sample, we keep the cube at the same position and change only its rotation. We run our method and all I2V baselines on these 10 cases. Each baseline receives the same motion in its own control format. Wan-Move receives point tracks sampled on the cube and moved with it. TTM receives the subject cut out of the first frame and moved with the cube. DaS receives 3D points of the subject moved rigidly with the cube. VACE receives the 2D box of the subject. We check whether the generated subject follows the requested rotation. We report the number of correct cases for each method. A video counts as correct only if the subject turns in the requested direction by about the requested angle. A video counts as a failure if the subject does not turn, turns too little or too much, or shows hallucinated motion. We judge each video by visual inspection, because whether the subject follows the requested rotation is clearly visible in these cases. Every video is shown on the page linked below together with its verdict, so each case can be checked directly.

Swivel (ours)VACEDaSWan-MoveTTM
Follows the requested rotation ↑10/101/102/105/105/10

Swivel follows the requested rotation in all 10 cases. All generated videos are shown at this anonymous link.

We will add both evaluations to the revised draft.

AuthorsOpenReview ↗

Rebuttal by Authors [3/4]

W3: Limited comparative coverage leaves broader competitiveness uncertain

The quantitative comparison covers VACE and DaS for image-to-video generation and only VACE for inpainting; 3DTrajMaster appears in the qualitative evaluation and user study. These are relevant comparisons, but provide limited coverage. [Table 2; Section 4.]

Although the examples demonstrate useful rotation control, their visual quality and motion range are not consistently compelling to me. Some motions are modest, leaving competitiveness with strong 2D motion-control methods uncertain. This is a preliminary impression, not an established ranking.

A broader quantitative comparison on motions all methods can represent would better assess whether explicit orientation control comes with competitive visual quality and general motion adherence.

Q3: Broader comparison: Could the authors include an additional strong motion-control baseline on shared tasks, particularly to assess visual quality and motion adherence beyond the current comparisons?

We thank the reviewer for this suggestion. We agree that a broader comparison helps, and we answer in two parts below.

Coverage of baselines and extended experiments. We add two recent open-source motion-control methods, Wan-Move [1] and TTM [2]. We evaluate all methods on the held-out evaluation set of Table 2. All methods receive the same first frame and the same prompt, and follow the motion of the same source video, so the comparison is on shared tasks. We report rotation error measured by running our annotation pipeline on the generated videos, position error, and video quality metrics.

Rotation (°) ↓Position (px) ↓PSNR ↑SSIM ↑LPIPS ↓FVD ↓
Swivel (ours)13.214.621.80.8000.115263.9
VACE36.252.017.00.6430.253460.9
DaS23.629.220.80.7580.153353.0
Extended baselines
Wan-Move16.713.719.20.7150.148349.6
TTM32.944.018.70.7070.189432.8

Swivel has the lowest rotation error and the best visual quality. Its position error is comparable to that of Wan-Move and much lower than that of the other baselines. Wan-Move, TTM and DaS do not support inpainting, so VACE remains the baseline in that setting. 3DTrajMaster is text-to-video only, so we compare with it qualitatively and in the user study.

[1] Chu et al., Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance, NeurIPS 2025.

[2] Singer et al., Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising, ICLR 2026.

Visual quality and motion range. To test larger motions directly, we build dedicated rotation tests, where the position is held fixed and only the rotation changes, with turns of up to 90°. We report the results of these tests in our responses to W1 and W2, comparing cubes with a position-only blob in W1 and Swivel with all baselines in W2. For visual quality, Swivel is better than Wan-Move and TTM, two strong recent 2D motion-control methods, on all four quality metrics.

We will add these comparisons to the revised draft.

Reviewer_zbBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Swivel to control where objects or body parts move and how they turn in generated videos. It extracts 3D tracks or human meshes from 150,000 real video clips, estimates local rigid motion and filters unreliable labels. The resulting positions and rotations are rendered as colored cubes and used to fine-tune Wan 2.2. At inference, a user provides an image or video clip, text and edited cube trajectories. The model generates a video following these controls. Experiments report better position following and video quality than VACE and DaS, with strong preferences in the user study.

Critical strengths
  1. The method provides a common control format for people, animals and objects. It obtains local motion labels from ordinary videos and checks whether the tracked parts support a reliable rigid-motion estimate. This is a useful approach to collecting part-level controls at scale (Sections 3.1–3.2).

  2. The full system gives strong practical results. Image-to-video position error is 14.6 pixels, compared with 29.2 for DaS and 52.0 for VACE (Table 2). In the 30-person study, motion-following preference is 89.0% against VACE and 92.9% against DaS (Figure 7). The results support the usefulness of the control interface and trained model.

Critical weaknesses
  1. The contribution beyond LooseControlVideo needs clearer evidence. That method already uses oriented 3D boxes for rotation control and automatically labels real videos. A direct comparison would help show the benefit of Swivel’s automatic selection and verification of local rigid parts.
  2. The evaluation could show rotation control more directly. The position metric and user study provide useful evidence of motion following, but do not separately measure rotation accuracy. A test that changes orientation while keeping position fixed would help clarify how well the model follows the requested rotation.
Questions for authors
  • How accurately does the model follow a requested rotation when position is held fixed?
  • How does Swivel compare with LooseControlVideo in controlling the motion of individual parts?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/2]

We sincerely thank the reviewer for the constructive feedback and for the time and effort put into reviewing our submission. Thank you for recognizing our video control format, our autonomous pipeline for obtaining local motion labels from real-world videos, and the strong practical results of our full system. We would like to address the weaknesses and questions below:


W1: The contribution beyond LooseControlVideo needs clearer evidence. That method already uses oriented 3D boxes for rotation control and automatically labels real videos. A direct comparison would help show the benefit of Swivel’s automatic selection and verification of local rigid parts.

Q2: How does Swivel compare with LooseControlVideo in controlling the motion of individual parts?

We thank the reviewer for this question. We agree that the difference from LooseControlVideo should be made clearer, and we answer in two parts below.

Comparison with LooseControlVideo. We also wanted to compare with LooseControlVideo directly. However, it has not released code or model weights. The two methods also differ in what they label. LooseControlVideo detects and segments each object and fits one oriented 3D box to the whole object, so its training labels are object-level. Swivel finds local rigid parts on real videos and verifies them, so one subject carries several cubes, each with its own rotation. The closest open-source method with object-level 6-DoF control is 3DTrajMaster, which controls each subject with one rigid pose per frame, the same granularity as one oriented box per object. We already compare with it in our paper.

That said, to better address the concern, we explored a way to compare object-level and part-level control within our own model. LooseControlVideo moves each object with one oriented box, so we build this object-level control from our part cubes. In every frame, we fit one rigid motion to all part cubes of an object and move all of them with this single motion. The parts can then no longer move relative to each other, as with one box per object. One rigid motion per object cannot specify how its parts move. We test on scenes where the parts move differently from the whole object. For the clock, one rigid motion cannot give each hand its own rotation, and the generated hands spin erratically. For the baseball scene, the ball starts right in front of the batter. Object-level labels come from detecting and segmenting whole objects, and when two objects overlap like this, they can be merged into one object with a single box. One rigid motion then drags the batter away with the ball, and the body breaks apart. For fencing, one rigid motion per fencer leaves no stepping motion, so the fencer slides forward and the model hallucinates an extra leg. In contrast, our part-level cubes do not produce these failure cases. We show video comparisons at this anonymous link.

Anchor verification. To show the benefit of our anchor selection and verification, we run an ablation. Real videos have no rotation ground truth, so we use 120 synthetic human motion clips rendered in Unreal Engine, which provide exact mesh ground truth. We start from a random-anchor baseline and add our components one by one.

VariantRotation error (°) ↓
1. Baseline (random anchors, plain rotation fit)21.2
2. + Robust rotation fit16.9
3. + Motion-based anchor proposal19.5
4. + Anchor verification (full pipeline)15.6

The full pipeline has the lowest error. Row 3 is higher than row 2 because the motion-based proposal selects anchors that rotate more, and larger rotations give larger errors. Verification then removes the unreliable anchors and lowers the error from 19.5° to 15.6°.

AuthorsOpenReview ↗

Rebuttal by Authors [2/2]

W2: The evaluation could show rotation control more directly. The position metric and user study provide useful evidence of motion following, but do not separately measure rotation accuracy. A test that changes orientation while keeping position fixed would help clarify how well the model follows the requested rotation.

Q1: How accurately does the model follow a requested rotation when position is held fixed?

We thank the reviewer for this suggestion. We agree that rotation should be evaluated separately from position, and we add the evaluations below.

Rotation accuracy. On the held-out evaluation set, we run our rotation annotation pipeline on both the real source videos and the videos generated by our method and all baselines, to test whether the generated motion still follows the requested rotation. To cover more comparisons, we add two recent open-source motion-control methods, Wan-Move [1] and TTM [2]. Rotation errors over the evaluation set are reported below. Swivel has the lowest rotation error of all methods.

Swivel (ours)VACEDaSWan-MoveTTM
Rotation error (°) ↓13.236.223.616.732.9

[1] Chu et al., Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance, NeurIPS 2025.

[2] Singer et al., Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising, ICLR 2026.

Rotation with position held fixed. In real videos, rotation and position usually change together, so changing only the rotation of a real clip often gives an implausible motion, and videos in which a subject turns while its position stays fixed are hard to find. To run the test the reviewer suggests, we therefore build our own cases. We generate 10 images with a commercial image model and use them as the first frames of our test samples. In each sample, we keep the cube at the same position and change only its rotation. We run our method and all I2V baselines on these 10 cases. Each baseline receives the same motion in its own control format. Wan-Move receives point tracks sampled on the cube and moved with it. TTM receives the subject cut out of the first frame and moved with the cube. DaS receives 3D points of the subject moved rigidly with the cube. VACE receives the 2D box of the subject. We check whether the generated subject follows the requested rotation. We report the number of correct cases for each method. A video counts as correct only if the subject turns in the requested direction by about the requested angle. A video counts as a failure if the subject does not turn, turns too little or too much, or shows hallucinated motion. We judge each video by visual inspection, because whether the subject follows the requested rotation is clearly visible in these cases. Every video is shown on the page linked below together with its verdict, so each case can be checked directly.

Swivel (ours)VACEDaSWan-MoveTTM
Follows the requested rotation ↑10/101/102/105/105/10

Swivel follows the requested rotation in all 10 cases. All generated videos are shown at this anonymous link.

To isolate the effect of orientation in the control, we also train two models that differ only in the control representation, one with cubes and one with position-only blobs. On the same 10 cases, with two control signals on each subject, the cube model follows the requested rotation in all 10 cases and the blob model in 5. With a single signal, the cube model still follows it in all 10 cases and the blob model in none. The videos are shown at this anonymous link.

We will add these evaluations to the revised draft.

AuthorsOpenReview ↗

Rebuttal Follow-up by Authors

Dear Reviewer zbBM,

Thank you for helping us improve our work! As the discussion period is coming to an end, we would be grateful if you could update your assessment if you find our added experiments and discussion have addressed your concerns. Otherwise, we are happy to answer any further questions. Thanks!

Best,
Authors

Reviewer_wFTrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies video generation about 6-DoF motion control. Previous control signals either struggle to express rotation or rely on synthetic or robotics data, which introduces a domain gap. The authors propose Swivel, which learns 6-DoF control from real-world videos alone. Its core is an automatic annotation pipeline. The model supports both image2video generation and inpainting based editing. Experiments show that Swivel outperforms VACE and DaS in position-following accuracy, PSNR, LPIPS, and FVD, and is clearly preferred in a user study. For annotation, the pipeline is about 25 times faster than ProxyPose and produces smoother estimates.

Critical strengths
  1. Rotation is a blind spot of existing motion control methods, and the cube representation is intuitive, easy to edit by hand, and naturally encodes orientation.

  2. Using only real-world videos, the pipeline relies on 3D foundation models and a criterion to automatically propose anchors, and combines this with multiple verification checks to turn ordinary videos into 6-DoF annotated data.

Critical weaknesses
  1. The work focuses on rotation awareness, however the only quantitative control metric is position error. Rotation accuracy is assessed only through qualitative examples and the user study. The authors could easily rerun their own annotation pipeline on the generated videos to measure rotation error.

  2. It's a little unfair because VACE is given only a 2D bounding box, without any orientation information.

  3. This work doesn't have ablation on method of anchor and representation of cubes.

Questions for authors
  1. Could you please add some ablation studies about the anchor and representation of cubes? I think this way can strengthen the quality of this paper.

  2. Could you please offer a fair comparison with VACE?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/2]

We sincerely thank the reviewer for the positive feedback and for the time and effort put into reviewing our submission. Thank you for recognizing our choice of cubes as the motion control representation and our scalable way of obtaining rotation labels from real-world videos. We would like to address the weaknesses and questions below:


W1: The work focuses on rotation awareness, however the only quantitative control metric is position error. Rotation accuracy is assessed only through qualitative examples and the user study. The authors could easily rerun their own annotation pipeline on the generated videos to measure rotation error.

We thank the reviewer for pointing this out. We agree that position error alone does not show rotation accuracy. As suggested, we rerun our own annotation pipeline on the generated videos to measure it directly.

Setup. On the held-out evaluation set, we run our rotation annotation pipeline on both the real source videos and the videos generated by our method and all baselines, to test whether the generated motion still follows the requested rotation. As part of our response to W2, we also add two baselines, Wan-Move [1] and TTM [2].

Results. Rotation errors over the evaluation set are reported below. Swivel has the lowest rotation error of all methods. We will add this evaluation to the revised draft.

Swivel (ours)VACEDaSWan-MoveTTM
Rotation error (°) ↓13.236.223.616.732.9

[1] Chu et al., Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance, NeurIPS 2025.

[2] Singer et al., Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising, ICLR 2026.


W2: It's a little unfair because VACE is given only a 2D bounding box, without any orientation information.

Q2: Could you please offer a fair comparison with VACE?

We thank the reviewer for raising this point. We also wanted to compare with VACE on 3D rotation. However, the sparse motion control of VACE is a 2D box, which carries position but not orientation. VACE's other control modalities, such as depth and flow, are pixel-dense signals that are usually extracted from another video, so they are not suited as a sparse control signal like our cubes. Its pose control is template-based and applies only to human bodies. DaS in our paper already serves as a baseline with such dense signals. It is driven by dense 3D tracks of the source video, which carry more motion information than our cubes. The closest open-source method with explicit 3D orientation control is 3DTrajMaster, which we already compare with in our experiments. 3DTrajMaster is text-to-video only and cannot run on this image-to-video benchmark. This also motivates us to release our models for 3D motion control in video generation upon publication.

That said, to better address the concern, we further add two recent open-source motion-control methods, Wan-Move [1] and TTM [2], and evaluate all methods on the held-out evaluation set of Table 2.

Results. We report rotation error (measured as in our response to W1), position error and video quality metrics in the table below. Swivel achieves the best results on both rotation accuracy and visual quality metrics. Its position error is comparable to that of Wan-Move and much lower than that of the other baselines.

Rotation (°) ↓Position (px) ↓PSNR ↑SSIM ↑LPIPS ↓FVD ↓
Swivel (ours)13.214.621.80.8000.115263.9
VACE36.252.017.00.6430.253460.9
DaS23.629.220.80.7580.153353.0
Extended baselines
Wan-Move16.713.719.20.7150.148349.6
TTM32.944.018.70.7070.189432.8

[1] Chu et al., Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance, NeurIPS 2025.

[2] Singer et al., Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising, ICLR 2026.

AuthorsOpenReview ↗

Rebuttal by Authors [2/2]

W3: This work doesn't have ablation on method of anchor and representation of cubes.

Q1: Could you please add some ablation studies about the anchor and representation of cubes? I think this way can strengthen the quality of this paper.

We thank the reviewer for this suggestion. We add both ablations below and will include them in the revised draft.

Representation of cubes. The model sees the control through the video VAE, so a good representation should keep the orientation readable after VAE encoding. We render the conditioning cubes from the evaluation set in four representations: six-coloured cube (ours), three coloured axes, single-colour cube, and blob (position only). We pass them through the video VAE and recover the rotation from the decoded video.

Six-coloured cube (ours)Three coloured axesSingle-colour cubeBlob (position only)
Rotation error (°) ↓0.841.982.9229.04

The six-coloured cube preserves orientation best. A single-colour cube is ambiguous under symmetric rotations, which raises its error. A blob carries no orientation at all. This is why we choose the six-coloured cube as our control representation.

To further support this choice from the perspective of controllability in the generated videos, we train two models that differ only in the control representation, one with cubes and one with position-only blobs. We test them on 10 cases where the position is held fixed and only the rotation changes. With two control signals on each subject, the cube model follows the requested rotation in all 10 cases and the blob model in 5. With a single signal, the cube model still follows it in all 10 cases and the blob model in none. The videos are shown at this anonymous link.

Method of anchor. To test how much each step of our anchor method contributes, we compare the estimated rotations with ground truth. Real videos have no rotation ground truth, so we use 120 synthetic human motion clips rendered in Unreal Engine, which provide exact mesh ground truth. We start from a random-anchor baseline and add our components one by one.

VariantRotation error (°) ↓
1. Baseline (random anchors, plain rotation fit)21.2
2. + Robust rotation fit16.9
3. + Motion-based anchor proposal19.5
4. + Anchor verification (full pipeline)15.6

The full pipeline has the lowest error. Row 3 is higher than row 2 because the motion-based proposal selects anchors that rotate more, and larger rotations give larger errors. Verification then removes the unreliable anchors and lowers the error from 19.5° to 15.6°.

AuthorsOpenReview ↗

Rebuttal Follow-up by Authors

Dear Reviewer wFTr,

Thank you for helping us improve our work! As the discussion period is coming to an end, we would be grateful if you could update your assessment if you find our added experiments and discussion have addressed your concerns. Otherwise, we are happy to answer any further questions. Thanks!

Best,
Authors

Comments on the paper

3 comments
Area_Chair_2Bc7OpenReview ↗

Reminder for discussion

Dear Reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check all of your assigned papers. If the authors have answered questions or responded to critiques in reviews, please leave a comment for the authors!

Thank you for your contribution to the community!

Your AC

Area_Chair_2Bc7OpenReview ↗

Author-Reviewr Discussion Period is About to Finish

Dear reviewers,

Thank you for your reviews and constructive feedback.

The author-reviewer discussion period will be finishing soon. Please take a moment to review it and consider whether your concerns have been adequately addressed. It would be helpful if you could briefly indicate:

  • Whether the rebuttal addresses your main concerns;

  • Whether any significant issues remain unresolved;

  • Whether your overall recommendation has changed.

If appropriate, please also update your review and recommendation in the system.

Thank you for your time and valuable contributions.

Best regards,

AC

AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Dear Reviewers, Area Chairs, and Program Chairs,

Thank you for the time spent reviewing our paper and for the constructive feedback. We summarize our rebuttal below.

We appreciate that the reviewers recognized the contributions of this work:

  • A video model with part-level 6-DoF control and strong practical results (Reviewers zbBM, JFzN).
  • A practical and scalable pipeline that turns ordinary real-world videos into 6-DoF annotated data (Reviewers wFTr, zbBM, JFzN).
  • An intuitive and general control representation that is easy to edit and naturally encodes orientation (Reviewers wFTr, zbBM).

We address the questions raised by the reviewers as follows, and all new results and discussions are included in the revised paper.

  1. Direct measurement of rotation accuracy. We add a quantitative measure of rotation accuracy (Reviewers wFTr, zbBM, JFzN) and a test that separates rotation following from position following (Reviewers zbBM, JFzN).
    • Quantitative evaluation of rotation accuracy (Table 2). We run our annotation pipeline on the generated videos of every image-to-video method and compare the recovered rotations with those of the source videos. Swivel has the lowest rotation error of 13.2° versus 16.7° for the best baseline.
    • Isolating rotation from position (Table 6). We build 10 cases where the position is held fixed and only the rotation changes. Swivel follows the requested rotation in 10/10 cases, Wan-Move in 5/10, TTM in 5/10, DaS in 2/10 and VACE in 1/10.
  2. Extended ablations on design choices. We ablate the orientation in the control (Reviewer JFzN), the cube representation (Reviewer wFTr), and the anchor proposal and verification (Reviewer wFTr).
    • Cube versus position-only control (Table 7). We train two models with the same checkpoint, data and training procedure that differ only in the control. We test them on the cases of Table 6. With two control points per subject, the cube model follows the requested rotation in 10/10 cases and the position-only model in 5/10. With a single control point, the cube model still follows it in 10/10 cases and the position-only model in 0/10.
    • Control representation (Table 3). To ablate our choice of the cube as the control representation, we pass four representations (six-coloured cube, three coloured axes, single-colour cube, position-only blob) through the video VAE and recover the rotation. The six-coloured cube keeps orientation best at 0.84° error.
    • Anchor proposal and verification (Table 4). On synthetic data with exact ground truth, the rotation error drops from 21.2° for the baseline to 15.6° for the full pipeline.
  3. Extended baselines and comparisons. We add additional baselines for a broader comparison (Reviewer JFzN) and a comparison with object-level control (Reviewer zbBM).
    • Extended baselines (Table 2). We add two recent open-source motion-control methods, Wan-Move and TTM. Swivel has the lowest rotation error and the best visual quality of all methods. Its position error is comparable to that of Wan-Move and much lower than that of the other baselines.
    • Object-level versus part-level control (Appendix D). As LooseControlVideo has not released code or weights, we compare object-level and part-level control within our own model. Part-level control can specify how each part of an object moves, which one rigid motion per object cannot.
  4. Additional clarifications (Section 2, Section 4, Appendix E). As suggested by Reviewer JFzN, we also discuss TTM in related work, state the role of each experiment, and give the data sources and split.

We thank all reviewers and the Area Chair again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.