- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a mechanistic interpretability study of "motion planning" in text-to-video diffusion models (Wan2.1-T2V-1.3B). It tries to formulate a pipeline: early motion planning -> candidate region competition -> spatial anchoring -> physically implausible trajectory -> RoPE modification. Using attention entropy, support quality, and convergence-speed metrics, the authors show that cross-attention maps form object trajectories within the first ~5 denoising steps, and by combining these metrics with attribution patching and zero ablation they identify a small subset of cross-attention heads that causally drive motion planning — notably, a visible trajectory pattern is shown to be insufficient for causal responsibility. In self-attention, they argue that RoPE-induced spatial attention decay ("spatial anchoring") lets early candidate regions lock in and suppress physically plausible but spatially distant candidates in adjacent frames, producing physics violations. Their fix scales the spatial RoPE frequencies (λ^(h/w) < 1) during the first 5 denoising steps, applied training-free or via LoRA fine-tuning with an early-timestep-focused sampler. Validation is on VideoPhy (human-evaluated SA/PC): the training-free variant raises PC from 29.45 to 39.94, and LoRA + modified RoPE reaches 41.98 PC vs. 35.57 (LoRA) and 37.90 (VideoREPA).
Critical strengths
- The causal head analysis. Showing that a clear trajectory pattern in a cross-attention head is neither necessary nor sufficient for driving motion planning — established with two independent methods (attribution patching adapted to flow matching, and zero ablation) — is a transferable methodological insight for interpretability work on generative models, and it disciplines a literature that often reads causality off attention maps.
- The analysis-to-intervention loop is closed. The spatial-anchoring diagnosis yields a minimal, cheap fix (one scaling factor, early steps only) that improves physical commonsense in both training-free and fine-tuned settings and composes additively with prompt refinement. Mechanism papers often stop at description; this one derives a working intervention from its own hypothesis.
Critical weaknesses
- The quantitative evidence rests on a fragile evaluation protocol. All VideoPhy results use a single generation seed (42), binary human judgments with no reported annotator count, agreement, or variance, and the LoRA checkpoint (800 steps) and test-time λ (0.70 vs. 0.75 in training; per-case tuning in the training-free setting) appear to have been selected by performance without a described validation set. Several gains in Table 1 are small relative to plausible noise at 343 prompts, so the central empirical claim cannot currently be assessed reliably.
- The mechanistic claim, i.e. the paper's core contribution, is not verified at the level of the mechanism. The evidence for spatial anchoring is correlational metrics plus one detailed failure anecdote (seed 20), and the paper never shows that modified RoPE actually changes candidate-region competition as predicted, nor rules out simpler explanations (e.g., generic attention smoothing) via controls such as temporal-axis scaling or all-step scaling. If the mechanism is wrong, the method is a useful trick attached to an incorrect explanation.
Questions for authors
Clarification requests (answers could change my assessment):
- Were the checkpoint at 800 steps and the λ values selected on a validation set disjoint from the VideoPhy test set? If so, please describe it.
- How many annotators produced the human SA/PC judgments in Table 1, what was their inter-annotator agreement, and were all baselines (including PR and VideoREPA) re-evaluated under the same protocol? What is the seed-to-seed variance of the Table 1 results?
- Does modified RoPE measurably alter candidate-region competition in the predicted direction (e.g., the winner–loser mutual-consistency gap, or the rate at which early-dominant physically reasonable regions are suppressed)?
- Control results: scaling the temporal RoPE axis (λ^f) instead of the spatial axes, and applying λ at all denoising steps rather than only the first 5.
8 comments
For Weakness 2: The causality between RoPE spatial anchoring and physics violations
In the revised manuscript, we have incorporated a new section, Section 6.4, specifically dedicated to discussing the causality between RoPE spatial anchoring and physics violations. Specifically,we provide two sets of experiments for verification: (1) We design a controlled experiment including the baseline (original model), aggravated spatial decay of RoPE (), and temporal adjustments of RoPE (). (2) We provide a post-hoc interpretability analysis for RoPE modification, verifying that our method indeed modulates self-attention by mitigating spatial decay, thereby leading to closer cross-frame attention within object regions.
Regarding (1), we evaluate on VideoPhy, and find that when the spatial decay is aggravated, the overall PC drops from 29.33% to 26.54%, performing far worse than our method. This bidirectional intervention rules out the possibility of incidental gains resulting from unidirectional modifications. Besides, when intervening in the temporal dimension of RoPE, the overall PC of the model decreases slightly by 1.03% on average, showing an insignificant effect on motion quality.
Regarding (2), we initially disregard semantic information and assume that the query and key tokens are identical (i.e., the self-correlation case). We theoretically derive the attention between two tokens and plot its decay curve with respect to positional distance. As shown in Figure 17(b), the decay rate is notably alleviated when applying .
Beyond this ideal setting, we empirically examine the shifts in self-attention during denoising before and after RoPE modification, incorporating both semantic and positional information. As shown in Figure 10 in Section 6.4, we visualize the self-attention from the object region in the initial frame to other frames. In the baseline, attention across frames remains locked onto the object region from the first frame. This causes candidate regions near the top to be assigned higher confidence in the latter half of the video, leading to the basketball hovering in the air. In contrast, with our method, the model not only attends to the identical region corresponding to the object's initial position during early generation (e.g., at denoising step 5), but also allocates a portion of attention to other regions. Consequently, following motion planning, the model is able to progressively shift attention toward the actual locations of the object in subsequent frames(e.g. the change in attention from step 7 to 20). Overall, our method enhances the exploration during motion planning by altering the attended regions in self-attention, thereby improving the physical commonsense of the model.
For Question 1: About the validation set
Yes. We utilized the 343 samples from the official VideoPhy training set hosted on Hugging Face as our validation set. Please refer to our response to Weakness 1 (Part II) for full details.
For Question 2: About human evaluation and multi-seed evaluation
We have provided more details in our reply to Weakness 1 (Part I). We sincerely hope those additional results and clarifications resolve your concerns.
For Question 3: Mechanistic analysis of RoPE modification
Yes, and we have provided details in our reply to Weakness 2.
For Question 4: Control results
In fact, both ablation studies suggested by the reviewer were explored during our experiments:
- Modifying along the temporal axis: We have discussed about this in our response to Weakness 2. Overall, intervening on the temporal RoPE axis introduces motion discontinuities or stuttering without yielding significant gains in physical commonsense.
- Applying across all denoising steps: We have already discussed about this in Appendix H.1.4. Overall, this configuration degrades generation quality, particularly hurting visual aesthetics, since the mid-late denoising stage mainly focuses on appearance generation and does not require any intervention on RoPE for better motion planning.
We have already investigated both phenomena empirically and will incorporate these detailed ablation analyses into the revised manuscript. We thank the reviewer for these constructive suggestions.
In conclusion, we sincerely hope our explanations and additional experiments have clarified your concerns. We would be deeply grateful if you would consider reassessing our work in light of these clarifications.
We hope this message finds you well. As the discussion period is nearing its end, we would like to ensure that we have addressed all your concerns with satisfaction. If there are any additional points or feedback you'd like us to consider, we are happy to be informed.
We'll appreciate it if you could reconsider our work with a new assessment. Looking forward to your reply~
Thank you for your valuable suggestions. We will make every effort to address your concerns.
For Weakness 1: Evaluation with more seeds & Details of the human evaluation
Part I (Evaluation details)
To ensure the reliability of the evaluation results, we conducted additional evaluations across four other random seeds (10, 20, 30, and 40) beyond seed 42. For the generations from each random seed, five evaluators with expertise in relevant research fields were invited to rate the outputs. We employed Fleiss' Kappa to assess inter-evaluator agreement. The resulting score of approximately 0.71 indicates Substantial Agreement, which validates the reliability of our evaluation.
The evaluation results are summarized below. The reported numbers represent the mean and standard deviation across all seeds, where the overall seed-to-seed variance is 1.03 %², the average variance for SA is 0.99 %², and that for PC is 1.07 %². Overall, averaging the evaluation results across multiple seeds effectively mitigates the evaluation noise mentioned by the reviewer. The final experimental findings remain highly consistent with those reported in the manuscript, demonstrating that our method retains a stable and robust advantage across different random seeds. We sincerely hope these additional experiments address your concerns.
Part II (About the validation set)
We did not cherry-pick the "best" results based on test-set performance, as tuning on the test set would violate standard evaluation integrity. Specifically, an official training set of VideoPhy is publicly available on Hugging Face. As explicitly stated in the VideoPhy paper, the entire benchmark is split equally into a training set and a test set, each containing 343 distinct prompts. Accordingly, we trained our method on WISA, utilized the 343 samples from the VideoPhy training set strictly as our validation set, and evaluated final performance exclusively on the unseen VideoPhy test set.