acceptodds
Withdrawn from ICLR 2027

Follow the Ball: Do Video Diffusion Models Control Where An Object Moves?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video diffusion models can generate realistic futures from short input videos, yet they remain unreliable at following instructions about how objects should move. Despite rapid progress in controllable video generation, it remains unclear whether these instructions actually guide the generated motion or whether models mainly extrapolate from the input video. We begin by asking how much a generated object’s motion changes when its instructed path changes. To separate instruction following from general future prediction, we consider two complementary tests: (1) response to intervention, where two generations share the same input video and randomness but receive different object paths, and (2) future similarity, where each generation is compared with the recorded continuation. Broadcast sports provide a natural testbed for this distinction because the movement of a ball or shuttlecock is central to what happens next. Using this setting, we introduce SportWorldBench, a controlled framework for evaluating object-motion control in video generation. Across changes to path labels, how paths are provided, model training, video resolution, and random samples, we find no reliable response to changed paths. Moreover, at higher resolution the models resemble the recorded future more closely than simply repeating the last input frame, yet this resemblance is nearly identical whether the instructed path matches the recording or is deliberately altered. These findings show that generating a plausible future does not imply following object-motion instructions and motivate methods that make such controls reliably influence generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video diffusion models, controllable video generation, object trajectory control, motion control, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cE5COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SportWorldBench to study the trajectory-conditioned generation capability of video diffusion models on sports data. A pretrained video diffusion model (VDM) is fine-tuned on per-frame heatmap conditioning that denotes trajectories of balls. Then the model is tested on two questions: (1) given a correct trajectory and a shifted trajectory, does the output vary accordingly? (2) compared to a static video that repeats the last context frame, is the generated video conditioned on the correct trajectory more consistent with the ground truth video? The experiments are conducted on 2B and 14B variants of a video diffusion model. The authors provide their findings.

Critical strengths

None

Critical weaknesses

The primary weakness is the lack of critical information in paper presentation. The name of the VDM is not mentioned, and the details of adding additional conditions are not provided adequately. Therefore, it is hard to understand the experimental setup and the implications of the findings. Even if the name of the VDM is provided, the paper lacks a comprehensive comparison across different VDMs to show the applicability of the proposed benchmark.

The second weakness is that the designed test may not reflect the property we aim to study. In question 1, the authors aim to study the trajectory-following capability. However, the output conditioned on a shifted trajectory may not follow this trajectory because the shifted trajectory is wrong and cannot guide the object movement. The current test cannot distinguish the model's failure in trajectory following from the low quality of the shifted trajectory. In question 2, using per-frame similarity (LPIPS) to measure the trajectory quality may not be appropriate. A better way is to directly measure trajectory similarity between the generated video and the GT video.

Considering the lack of critical information and the issues in the designed test, I recommend rejecting the paper.

Questions for authors

Please resolve the concerns in weakness section.

Reviewer_VoKqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SportWorldBench to evaluate whether video diffusion models follow object-motion commands rather than simply predict plausible futures. The authors compare generated videos under different trajectory commands while keeping other conditions fixed. Experiments across four sports and multiple model configurations show that changing trajectories has little effect on generated object motion, even when the models produce plausible future videos.

Critical strengths

The motivation is reasonable, as video generation models may produce plausible motion based on learned patterns rather than actually following control signals. The difficulty of distinguishing motion prediction from controllability is a meaningful evaluation problem.

Critical weaknesses

1. Limited model coverage and insufficient description of the evaluated models.

The evaluation is limited to two pre-adapted model configurations (2B and 14B), without comparisons to established trajectory-controlled video generation methods. Moreover, the paper does not clearly identify the pretrained backbones or describe how the initial checkpoints were adapted for spatial conditioning. This makes it difficult to assess the generality of the findings.

2. Potentially unrealistic trajectory interventions.

The benchmark shifts object trajectories horizontally by a fixed 134 pixels while keeping the input video and other spatial controls unchanged. Such interventions may conflict with observed motion or physical constraints. It is unclear whether the models would show similar failures under smaller, physically plausible trajectory variations. What motivated the choice of a fixed 134-pixel horizontal shift (approximately 15% of the image width)? Was this value determined through preliminary experiments?

3. Insufficient validation of the additional training.

The authors continue training the existing checkpoints for only 1,000 steps on 400 video clips. However, no convergence analysis or training-duration ablation is provided. It is therefore unclear whether the limited controllability reflects the models' capabilities or insufficient adaptation to the modified conditioning.

4. Low object detection coverage.

The evaluation relies heavily on automatic object detection, but joint detection coverage is low in several settings, particularly for basketball, where it sometimes falls below 5%. Although the authors provide missingness bounds, these are often too wide to rule out meaningful responses in unobserved frames. This limits the strength of the conclusions about trajectory controllability.

Questions for authors

See the weaknesses.

Reviewer_FnqYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SportWorldBench, a benchmark that separates action responsiveness from future prediction in video generation. Although models respond reliably to known translations, generated objects show no consistent response to changes in instructed paths across the tested settings. At native resolution, generations resemble the recorded future more than a repeated input frame, with similar scores under recorded and altered paths. A vehicle-transfer test shows the same pattern.

Critical strengths
  1. The paper examines an important problem: reliable action control is essential for controllable video generation.
  2. SportWorldBench offers a systematic way to test action following in a constrained setting, revealing a gap between plausible future prediction and responsiveness to action instructions.
Critical weaknesses
  1. The paper would benefit from clearer explanations of the base model and conditional model, especially how control signals are encoded and used.
  2. It is unclear whether the findings reflect a fundamental limitation of video diffusion models or the specific conditioning method and model capacity tested.
  3. More and stronger baselines would make the conclusions more convincing. An analysis of why action control fails would also provide useful guidance for future work.
Questions for authors
  1. How are the base and conditional models constructed, and how are action signals encoded? Does modifying the input channels provide an effective control signal, and how do you verify that the model uses it?
  2. Do the findings hold for stronger models, such as the Seedance series, with denser character and camera controls? Could stronger conditioning or model capacity resolve the observed limitations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.