acceptodds
Withdrawn from ICLR 2027

DancingThings: Procedural Data Generation for Optical Flow with Spline-Based Deformation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing optical flow methods do not perform well on non-rigid movement. We remedy this problem by introducing a procedural data generator focusing on non-rigid motion that can generate unlimited data for optical flow. Our generator takes a scene mesh as input and outputs an animated version of the scene where objects are deformed with a novel spline-based deformation method called SPD. SPD works by creating free-form deformation fields around the assets and animating them with 4D NURBS, where there is an extra time curve that controls how the control points move. We prove two theoretical results showing that our deformation method is expressive and general, making methods trained on our data robust to a wide range of real-world motion. We use our procedural generator to generate and render a sample-efficient optical flow dataset called DancingThings. To our knowledge, DancingThings is the first optical flow dataset that focuses on general non-rigid motion in contrast to previous works, which include only specific types of deformations such as human motion, cloth simulation, and soft-body dynamics. We show that optical flow methods trained on DancingThings achieve significantly higher accuracy on non-rigid motion compared to methods trained on standard datasets. In particular, DancingThings achieves higher accuracy on non-rigid benchmarks than FlyingChairs and TartanAir combined with 138x fewer image pairs. Furthermore, DancingThings surprisingly improves performance on rigid-motion benchmarks as well, even outperforming rigid training datasets. All data and code will be open-sourced.

Primary area:
datasets and benchmarks
Keywords:
optical flow, dataset, procedural generation, NURBS

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nNdZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors identify a gap in the existing datasets used for optical flow: they lack non-rigid motion. In order to address this, they propose a data generator based on spline deformations. They show that optical flow methods trained on their approach achieve better performance.

Critical strengths
  1. The motivation behind the work is clear. The gap that the authors identify is real and important.
  2. The paper is well written and easy to follow. The figures are helpful in understanding the paper.
  3. Ablation study (Table 6) is good and builds trust in the proposed method.
Critical weaknesses
  1. The authors claims that Dancing Things outperforms rigid-motion datasets on KITTI (Table-4). However, the published SEA-RAFT (Table 2 of [A]) results for the same setting are much better than the ones reported in the table and also better than the results achieved by the proposed method. Can the authors clarify ?
  2. [Line 412] The authors claim that “even though our dataset is significantly smaller in size, training on our dataset yields ….. across most of the metrics”. I have questions then about how the different models trained on these datasets compared? Were they evaluated after a fixed number of training steps, or were they evaluated at convergence ? If it was the latter, how was convergence determined ? I think this matter for interpreting the result: if step counts are matched, the advantage reflects data diversity rather than size.
  3. Ideally, the claim of “138x” should be supported with a scaling curve.
  4. The author claim that the proposed method can provide unlimited data [Line 015], yet the proposed dataset has a fixed size [Line 412]. A natural question arises if the authors have tried to increase the dataset size, and whether doing that improves the performance.

[A] Wang Y, Lipson L, Deng J. Sea-raft: Simple, efficient, accurate raft for optical flow. InEuropean Conference on Computer Vision 2024 Sep 29 (pp. 36-54). Cham: Springer Nature Switzerland.

Questions for authors

Please see the weakness section

Reviewer_taS6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DancingThings, a procedural data generation pipeline for non-rigid optical flow. The core technical contribution is a spline-based procedural deformation module (SPD) that deforms arbitrary meshes by embedding them in a 4D NURBS control lattice and sampling displacement coefficients from uniform distributions. The generated dataset consists of 2376 image pairs rendered with random object trajectories and superimposed SPD deformations. Experiments involving non-standard training and validation procedures demonstrate that training SEA-RAFT and WAFT flow models on this small dataset produces decent performance.

Critical strengths
  1. Addressing the lack of diverse non-rigid training data through procedural generation rather than physics simulation or manual animation is a valuable direction.
  2. The 4D NURBS formulation provides a clean mathematical framework for representing spatiotemporal deformations with controllable smoothness and infinite sampling capacity.
  3. The use of Infinigen as a scene backend offers strong potential for scaling to unlimited layouts and object categories.
Critical weaknesses
  1. The experimental section of this paper contains several major issues.
  1. The experimental protocol deviates from established optical flow evaluation standards in several critical ways. Standard zero-shot generalization evaluation requires training on FlyingChairs plus FlyingThings and validating on Sintel and KITTI. This paper deliberately omits FlyingThings from the training set in Tables 2 and 4, replacing it with TartanAir, which is not part of the standard optical flow training pipeline. When FlyingThings does appear in Table 3, the evaluation dataset is switched from Sintel and KITTI to DOT, which avoids direct comparison with the canonical C+T -> S/K generalization benchmark. The paper also fails to report results on the Sintel final pass for most configurations, leaving an incomplete picture of performance under realistic rendering conditions.
  2. Furthermore, the standard workflow includes fine-tuning the C+T pretrained model on Sintel, KITTI, and Spring training splits followed by online test submission. Given that DancingThings contains only 2376 pairs, it is unclear whether its benefits persist after subsequent in-domain fine-tuning, or whether the small data size leads to insufficient feature initialization that degrades final performance. This entire fine-tuning and online testing pipeline is absent from the paper.
  3. The model selection is not representative of the field. Only two recent large-capacity architectures (SEA-RAFT and WAFT) are tested. Many representative methods spanning different design philosophies are missing: lightweight iterative methods such as RAFT and GMA, transformer-based approaches including GMFlow and FlowFormer, diffusion-based models such as DDVM and FlowDiffuser, and multi-frame methods including MEMFOF and MegaFlow. Validating the dataset's effectiveness across diverse architectural families is necessary to establish that the observed gains are due to data quality rather than accidental compatibility with specific model designs.
  1. The experimental design conflates multiple variables. The comparison between DancingThings and existing datasets varies simultaneously in rendering engine, texture source, data size, deformation type, and scene diversity. It is impossible to attribute observed gains specifically to the non-rigid deformation prior. The KITTI improvement is particularly suspicious because it could arise from stronger occlusion supervision, Eevee's noise characteristics, different training schedules, or simply the regularization effect of smaller data, none of which are controlled. The paper lacks an ablation that isolates foreground-only deformation from full-scene deformation.

  2. Ground truth quality metrics are absent. The paper does not report self-intersection rates, minimum Jacobian determinants, normal consistency after deformation, or any measure of physical plausibility. Since control point displacements are sampled independently from a uniform distribution, a significant fraction of deformed meshes likely contain inverted elements or surface penetrations. If the training distribution contains many physically implausible configurations, the network may learn texture-matching heuristics rather than genuine motion understanding, which would explain both the strong performance on textured non-rigid benchmarks and the questionable generalization to real-world scenarios.

  3. Minor:Table 1 contains misleading formatting. Multiple entries in each column could reasonably be considered optimal depending on the criterion, yet only DancingThings cells are bolded. For instance, if fewer image pairs are preferable, several datasets (1593, 389, 400, and others) have fewer pairs than 2376 but are not bolded. This selective emphasis creates an impression of superiority that is not supported by the data itself.

Questions for authors

First, the authors should follow the standard zero-shot generalization experimental protocol from C+T to S/K to conduct experiments, and provide complete fine-tuning and online test submission results following the established C+T pretraining then in-domain finetune pipeline.

Second, the model selection should be expanded beyond SEA-RAFT and WAFT to include representative lightweight methods such as RAFT and GMA, transformer-based approaches like GMFlow and FlowFormer, and diffusion-based models such as DDVM and FlowDiffuser. Controlled ablations that isolate foreground-only deformation from full-scene deformation should be conducted, and quantitative ground truth quality metrics including self-intersection rates and minimum Jacobian determinants should be reported.

Finally, the misleading bold formatting in Table 1 should be corrected.

Reviewer_NX3oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a novel data generation method and DancingThings, a dataset for non-rigid motion. The method uses spline-based procedural generation with NURBS control points to construct four-dimensional space-time deformation fields. It offers useful guidance for the automated generation of large-scale datasets and has substantial practical value for the research community.

Critical strengths

1.The core observation that optical flow methods fail on non-rigid motion primarily due to lack of training data is compelling, and the result that a much smaller but more diverse dataset can outperform larger but more limited training sets is practically valuable. The 138x sample efficiency claim is striking and well-supported across multiple benchmarks.

2.The SPD method is elegant and general. Extending FFD with a time dimension to create space-time deformation fields is a natural idea, and the two approximation theorems provide useful theoretical grounding for the expressiveness of the approach. The comparison against Blender soft-body simulation in the ablation also shows that procedural deformations can outperform physically-based simulation for this task.

3.The evaluation is thorough, covering non-rigid benchmarks such as Spring Non-Rigid, Sintel, and DOT, as well as rigid benchmarks such as KITTI and Spring Rigid. The ablation study analyzes rendering engine, deformation amplitude, number of control points, and motion type, which helps readers understand the design tradeoffs.

Critical weaknesses

1.The theoretical results are approximation theorems in the supremum norm, which do not directly translate to optical flow accuracy or to the visual plausibility of the deformations.

2.The dataset is very small, with only 24 scenes and 2,376 image pairs. The small size also raises concerns about overfitting to the specific benchmark distributions.

Questions for authors

1.The comparison to other approaches for generating non-rigid training data, such as video warping, 2D deformation on still images, or learning-based deformation models, is limited. AutoFlow is mentioned briefly but a more systematic comparison with these alternative data generation strategies would be helpful.

2.The reason why DancingThings also improves performance on rigid motion benchmarks is not fully explained. Whether this stems from rigid motion components still present in the procedural data, or from improved general robustness of the model, is unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.