acceptodds
Under review as a conference paper at ICLR 2027

What Controls Motion in Video Diffusion Models? A Mechanistic Study

Abstract

Motion is central to video generation models, yet what inside the model controls it remains largely unexplored. Some methods control object motion by training an added module on video data. Others require no training and edit the latent or the attention at inference time. These methods change the generated motion, but they do not show what inside the model produces the change. We present a mechanistic study of that question. We train a video diffusion transformer on synthetic videos, where the launch angle of a ball is controllable, and we read that angle in degrees from generated videos. This model allows us to control object motion more precisely than a pretrained model does. We present three findings. First, adding one shared vector at every token position, the standard form of activation steering, does not rotate the trajectory. Second, letting the added vector vary across positions does reach the request. Third, the control depends on a small group of conditioning-related attention heads, and holding them at their un-intervened outputs removes most of the rotation. The same pattern holds on the pretrained Wan2.1-1.3B. Together these experiments identify what controls motion inside video diffusion models.

Primary area:
interpretability and explainable AI
Keywords:
video diffusion models, mechanistic interpretability, activation steering, motion control, controllable video generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_drKTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies activation interventions for motion control in video diffusion transformers. A BallDiT model trained on synthetic projectile videos provides a controlled setting with pixel-level trajectory measurements. With model weights frozen, an additive activation field is optimized to match counterfactual target latents, typically using 24 fitting clips and 100 held-out evaluation clips. The paper compares shared and token-wise interventions, studies their decomposition and rank, and uses attention-head replacement and pinning to examine intervention dependence. Extensions include inverted synthetic clips in Wan2.1 and natural-prompt facing and color experiments.

Critical strengths
  1. The controlled experimental design is careful. Matched-noise counterfactuals, pixel-based motion measurements, held-out evaluation, image-quality tracking, and norm/objective/optimization controls help distinguish changes in the generated trajectory from changes in a probe readout. The main intervention is fitted jointly and evaluated on unseen fitting-set-excluded clips, rather than optimized independently for each test video.
  2. The analysis extends beyond intervention success to decomposition, rank, and attention-head replacement/pinning with matched random-head controls. The paper also reports important negative results, including position sensitivity, poor cross-pair transfer, and successful shared interventions when allowed to vary across layers and denoising steps. This transparency is valuable.
Critical weaknesses
  1. The central shared-vector versus token-wise comparison does not sufficiently separate intervention structure from capacity and parameter allocation. A shared 384-dimensional vector is compared with a 576-by-384 field with substantially greater flexibility. Its greater effectiveness is therefore not, by itself, a surprising mechanistic finding. Norm matching controls perturbation magnitude, not parameter count; decomposing an optimized full field does not independently resolve this issue either. I appreciate the rank and optimization studies, but low-rank fields still introduce learnable token-specific coefficients. More direct comparisons at similar parameter and optimization budgets are needed to establish what is specifically gained from token-wise structure. Moreover, the layer- and step-dependent shared intervention achieves approximately 86 degrees of rotation, restricting the negative result to the fixed-site intervention family. The optimized field establishes a way to control the model, but does not itself identify its endogenous motion representation, as the paper acknowledges.
  2. The evidence is concentrated in an overly restricted motion setting relative to the broader question of motion mechanisms in video diffusion models. BallDiT is trained on many videos, so this is not a single-video memorization objection; nevertheless, the core setting uses simple projectile motion, a clean background, and explicit low-dimensional motion conditioning. Held-out clips establish transfer within this setting, not across complex motion distributions. Using Wan does not fully address this limitation because the main quantitative motion experiments still use synthetic projectile clips. Natural-prompt facing and color changes do not establish that the findings hold for articulated dynamics, occlusion, interacting objects, or coupled camera and object motion. The reported position dependence and frequent cross-pair transfer failures further limit the generality. The large number of controlled experiments does not yet translate into sufficiently broad or non-obvious insight about real-world video motion.
Questions for authors
  1. Can you compare token-wise interventions against alternatives that allocate a similar parameter budget across layers and/or denoising steps, with comparable optimization effort and intervention magnitude? What evidence would distinguish a benefit of token-wise structure from capacity or optimization flexibility?
  2. Can the central structural and head-dependence findings be tested on at least one genuinely dynamic natural-video setting beyond projectile motion, facing direction, or color? A focused test involving articulated motion, occlusion, or simultaneous camera/object motion would be more informative than additional variants of the same toy setting; a full general-purpose motion controller is not required.
  3. Given the successful layer/step-dependent shared intervention and limited cross-pair transfer, what precise mechanistic conclusion generalizes beyond the chosen intervention parameterization and synthetic distribution? Please clarify the contribution that remains after these scope restrictions.
Reviewer_8aAoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies what intervention structure are necessary to change a spatially-varying motion attribute in a video diffusion transformer. They perform this study on a rather small and controlled setup where a DiT video model is trained on a synthetic dataset where launch angle is a ground-truth conditioning input. Once the model is trained, the authors provide a conditioning angle of theta but then try to change certain DiT attributes to be able to generate a video where the launch angle is theta + alpha. They measure how close they get to this target angle. They show that (i) a single vector shared across all token positions fails to rotate the trajectory, while a position-varying ("token-wise") field succeeds.

Critical strengths
  1. The paper presents a controlled experimental setting with ground-truth conditioning and a pixel-level readout in the attribute's own units (degrees). This synthetic setup makes it possible to do more detailed quantitative evaluations.
  2. Some interesting observations are presented. For example, when they decompose per-token vectors into a common vector and per-token residuals, the residuals carry to most important component. They show that they can select the different vector directions from 16 different directions.
  3. Partial validation on a 1.3B pretrained model (Wan2.1-1.3B) for both the split and the head-dependence.
Critical weaknesses
  • While the controlled setup is useful, it is not clear how the observations can generalize to more complex and real video DiTs. The model is trained on very short clips, and only a single motion attribute of launch angle is studied. Even for a rather small video spatial size, they find that 16 different rotation vectors are needed, this is almost the same as optimizing for a per-token residual vector.

  • The paper does not study the effect of different diffusion timesteps very rigorously even though it is known that the structure/layout and hence motion can be related to earlier denoising steps.

  • The title "What Controls Motion" and the abstract's "internal components that mediate motion control" could be read as identifying the motion representation. This risks overclaiming, as mentioned before it is not clear to me how the observations from this paper can generalize to practical motion editing for large scale video models.

  • Overall I find the paper difficult to follow and understand, I had to iterate several times. A more clear overview section which summarizes the goals and the setup would have been useful.

Questions for authors
  1. Have you tried injecting the token-wise field at only the first 10 denoising steps (or only the last 10)? Does the rotation drop, or is the field's effect distributed uniformly across steps?

  2. Is the position-locking an inherent property of the model's representation (it truly encodes "correction at pixel (i,j)"), or could a learned warping (e.g., align the field to the ball's detected centroid before injection) recover transfer? Have you tested even a simple translation of the field field to the new ball position?

  3. Wan generalization - Can you report results on a larger, more diverse set (e.g., different objects, backgrounds, and motion types)?

Reviewer_PPJuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how activation interventions control motion in video diffusion models. Using a diffusion transformer trained on synthetic projectile videos, the authors compare a vector shared across tokens with a token-wise additive field optimized to match counterfactual generations. At a fixed injection layer, the token-wise field successfully changes launch angles, while the shared vector has little effect. Decomposing a successful field shows that its position-dependent component retains nearly all of the rotation effect. Head replacement and pinning experiments further identify a small group of conditioning-related attention heads that mediate this effect.

Critical strengths

The controlled synthetic setup enables precise interventions and direct measurement of their effects on generated motion. The norm-matching and random-head controls strengthen the evidence for the importance of position-dependent interventions and the selected attention heads.

Critical weaknesses

The additional insight beyond prior work and its broader implications remain unclear. Although the experiments are carefully controlled, it is not sufficiently clear what previously unknown aspect of motion control they reveal. As discussed in the paper’s related work, prior studies have already examined position-dependent interventions, identified motion-related attention heads, and demonstrated discrepancies between the decodability of an attribute and its susceptibility to intervention. The shared/offset decomposition and head-pinning experiments connect these observations, but the paper could more clearly articulate what this connection adds to our understanding. In particular, the heads are selected based on their ability to change the launch angle through counterfactual replacement, so their involvement in the field-induced rotation is informative but not especially surprising. Furthermore, the implications for contemporary video diffusion models remain underdeveloped. Most evidence comes from a controlled projectile setting or inverted synthetic clips, while the natural-language examples concern facing direction rather than motion trajectories. It is therefore unclear what actionable guidance these findings provide for understanding or improving motion generation and control in more diverse settings.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.