DancingThings: Procedural Data Generation for Optical Flow with Spline-Based Deformation

Existing optical flow methods do not perform well on non-rigid movement. We remedy this problem by introducing a procedural data generator focusing on non-rigid motion that can generate unlimited data for optical flow. Our generator takes a scene mesh as input and outputs an animated version of the scene where objects are deformed with a novel spline-based deformation method called SPD. SPD works by creating free-form deformation fields around the assets and animating them with 4D NURBS, where there is an extra time curve that controls how the control points move. We prove two theoretical results showing that our deformation method is expressive and general, making methods trained on our data robust to a wide range of real-world motion. We use our procedural generator to generate and render a sample-efficient optical flow dataset called DancingThings. To our knowledge, DancingThings is the first optical flow dataset that focuses on general non-rigid motion in contrast to previous works, which include only specific types of deformations such as human motion, cloth simulation, and soft-body dynamics. We show that optical flow methods trained on DancingThings achieve significantly higher accuracy on non-rigid motion compared to methods trained on standard datasets. In particular, DancingThings achieves higher accuracy on non-rigid benchmarks than FlyingChairs and TartanAir combined with 138x fewer image pairs. Furthermore, DancingThings surprisingly improves performance on rigid-motion benchmarks as well, even outperforming rigid training datasets. All data and code will be open-sourced.