Learning Rotation-Aware Video Motion Control from Real-World Videos

Motion control for video generation prescribes the translation and the rotation of the content of a clip. Training a model to follow an explicit rotation requires per-frame six-degree-of-freedom supervision that real footage does not provide. Prior work therefore relies on simulation or robotics data, but this introduces a domain shift for general motion editing. We present Swivel, a framework that learns 6-DoF video motion control from real-world video alone. It builds on an observation about 3D foundation models: from their outputs alone, the rigid parts of a clip can be proposed automatically and the 6-DoF motion of each part can be measured without any human marking, which turns ordinary footage into a large corpus of clips paired with 6-DoF trajectories. We then train a video diffusion model on this corpus for sparse but precise 6-DoF control, with rotating cubes in screen space as the condition signal, and the model supports both image-to-video generation and inpainting-based video editing. Results show that Swivel follows the 6-DoF motion signals more closely than prior motion controls and generates higher-quality videos at 720p than the baselines, and human evaluation shows a large preference for Swivel over the baselines.