acceptodds
Under review as a conference paper at ICLR 2027

Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms

Abstract

Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the ”motion planning” process of text-to-video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the ”first shape, then details” finding, we combine cross-attention trajectory patterns with causal head contributions to identify a specific subset of attention heads driving motion planning. Further, our self-attention analysis shows that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated videos.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, video generation, physical commonsense

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bMTsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a mechanistic interpretability study of "motion planning" in text-to-video diffusion models (Wan2.1-T2V-1.3B). It tries to formulate a pipeline: early motion planning -> candidate region competition -> spatial anchoring -> physically implausible trajectory -> RoPE modification. Using attention entropy, support quality, and convergence-speed metrics, the authors show that cross-attention maps form object trajectories within the first ~5 denoising steps, and by combining these metrics with attribution patching and zero ablation they identify a small subset of cross-attention heads that causally drive motion planning — notably, a visible trajectory pattern is shown to be insufficient for causal responsibility. In self-attention, they argue that RoPE-induced spatial attention decay ("spatial anchoring") lets early candidate regions lock in and suppress physically plausible but spatially distant candidates in adjacent frames, producing physics violations. Their fix scales the spatial RoPE frequencies (λ^(h/w) < 1) during the first 5 denoising steps, applied training-free or via LoRA fine-tuning with an early-timestep-focused sampler. Validation is on VideoPhy (human-evaluated SA/PC): the training-free variant raises PC from 29.45 to 39.94, and LoRA + modified RoPE reaches 41.98 PC vs. 35.57 (LoRA) and 37.90 (VideoREPA).

Critical strengths
  1. The causal head analysis. Showing that a clear trajectory pattern in a cross-attention head is neither necessary nor sufficient for driving motion planning — established with two independent methods (attribution patching adapted to flow matching, and zero ablation) — is a transferable methodological insight for interpretability work on generative models, and it disciplines a literature that often reads causality off attention maps.
  2. The analysis-to-intervention loop is closed. The spatial-anchoring diagnosis yields a minimal, cheap fix (one scaling factor, early steps only) that improves physical commonsense in both training-free and fine-tuned settings and composes additively with prompt refinement. Mechanism papers often stop at description; this one derives a working intervention from its own hypothesis.
Critical weaknesses
  1. The quantitative evidence rests on a fragile evaluation protocol. All VideoPhy results use a single generation seed (42), binary human judgments with no reported annotator count, agreement, or variance, and the LoRA checkpoint (800 steps) and test-time λ (0.70 vs. 0.75 in training; per-case tuning in the training-free setting) appear to have been selected by performance without a described validation set. Several gains in Table 1 are small relative to plausible noise at 343 prompts, so the central empirical claim cannot currently be assessed reliably.
  2. The mechanistic claim, i.e. the paper's core contribution, is not verified at the level of the mechanism. The evidence for spatial anchoring is correlational metrics plus one detailed failure anecdote (seed 20), and the paper never shows that modified RoPE actually changes candidate-region competition as predicted, nor rules out simpler explanations (e.g., generic attention smoothing) via controls such as temporal-axis scaling or all-step scaling. If the mechanism is wrong, the method is a useful trick attached to an incorrect explanation.
Questions for authors

Clarification requests (answers could change my assessment):

  1. Were the checkpoint at 800 steps and the λ values selected on a validation set disjoint from the VideoPhy test set? If so, please describe it.
  2. How many annotators produced the human SA/PC judgments in Table 1, what was their inter-annotator agreement, and were all baselines (including PR and VideoREPA) re-evaluated under the same protocol? What is the seed-to-seed variance of the Table 1 results?
  3. Does modified RoPE measurably alter candidate-region competition in the predicted direction (e.g., the winner–loser mutual-consistency gap, or the rate at which early-dominant physically reasonable regions are suppressed)?
  4. Control results: scaling the temporal RoPE axis (λ^f) instead of the spatial axes, and applying λ at all denoising steps rather than only the first 5.
8 comments

Thank you for your valuable suggestions. We will make every effort to address your concerns.

For Weakness 1: Evaluation with more seeds & Details of the human evaluation

Part I (Evaluation details)

To ensure the reliability of the evaluation results, we conducted additional evaluations across four other random seeds (10, 20, 30, and 40) beyond seed 42. For the generations from each random seed, five evaluators with expertise in relevant research fields were invited to rate the outputs. We employed Fleiss' Kappa to assess inter-evaluator agreement. The resulting score of approximately 0.71 indicates Substantial Agreement, which validates the reliability of our evaluation.

The evaluation results are summarized below. The reported numbers represent the mean and standard deviation across all seeds, where the overall seed-to-seed variance is 1.03 %², the average variance for SA is 0.99 %², and that for PC is 1.07 %². Overall, averaging the evaluation results across multiple seeds effectively mitigates the evaluation noise mentioned by the reviewer. The final experimental findings remain highly consistent with those reported in the manuscript, demonstrating that our method retains a stable and robust advantage across different random seeds. We sincerely hope these additional experiments address your concerns.

Baseline: Wan2.1-T2V-1.3B, mRoPE: modified RoPE, PR: prompt refinement |Model|Overall SA|Overall PC|Solid-Solid SA|Solid-Solid PC|Solid-Fluid SA|Solid-Fluid PC|Fluid-Fluid SA|Fluid-Fluid PC| |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Baseline|53.53±0.33|29.33±0.33|47.18±0.00|24.65±0.00|55.89±1.04|31.23±0.97|63.64±2.23|36.36±1.82| |PR|80.35±0.33|44.20±0.33|69.30±0.63|28.45±0.59|87.13±0.41|52.88±0.46|90.91±1.82|61.82±1.29| |LoRA|55.69±0.21|35.57±0.21|47.89±0.00|28.17±0.00|60.27±0.62|39.04±0.65|63.64±1.29|45.45±2.87| |VideoREPA|56.44±0.33|37.78±0.33|48.59±0.00|30.28±0.00|61.37±0.97|41.51±0.97|63.64±1.82|47.27±1.29| |mRoPE|57.43±0.21|39.94±0.21|49.30±0.31|30.99±0.63|63.70±0.70|47.26±0.98|61.82±1.29|43.64±2.87| |mRoPE+PR|86.42±0.33|59.01±0.33|83.38±0.80|44.65±0.80|87.67±0.31|67.81±0.31|90.91±1.29|72.73±1.29| |LoRA+mRoPE|64.72±0.46|41.98±0.46|52.82±0.63|34.51±0.31|75.34±1.40|47.95±1.25|67.27±2.87|45.45±1.29| |LoRA+mRoPE+PR|87.40±0.24|62.56±0.33|84.37±0.50|49.72±0.87|87.67±0.31|71.23±0.31|94.55±1.29|72.73±2.87|

Part II (About the validation set)

We did not cherry-pick the "best" results based on test-set performance, as tuning on the test set would violate standard evaluation integrity. Specifically, an official training set of VideoPhy is publicly available on Hugging Face. As explicitly stated in the VideoPhy paper, the entire benchmark is split equally into a training set and a test set, each containing 343 distinct prompts. Accordingly, we trained our method on WISA, utilized the 343 samples from the VideoPhy training set strictly as our validation set, and evaluated final performance exclusively on the unseen VideoPhy test set.

For Weakness 2: The causality between RoPE spatial anchoring and physics violations

In the revised manuscript, we have incorporated a new section, Section 6.4, specifically dedicated to discussing the causality between RoPE spatial anchoring and physics violations. Specifically,we provide two sets of experiments for verification: (1) We design a controlled experiment including the baseline (original model), aggravated spatial decay of RoPE (λh/w=1.20\lambda^{h/w} = 1.20), and temporal adjustments of RoPE (λt=0.70,1.20\lambda^{t} = 0.70, 1.20). (2) We provide a post-hoc interpretability analysis for RoPE modification, verifying that our method indeed modulates self-attention by mitigating spatial decay, thereby leading to closer cross-frame attention within object regions.

Regarding (1), we evaluate on VideoPhy, and find that when the spatial decay is aggravated, the overall PC drops from 29.33% to 26.54%, performing far worse than our method. This bidirectional intervention rules out the possibility of incidental gains resulting from unidirectional modifications. Besides, when intervening in the temporal dimension of RoPE, the overall PC of the model decreases slightly by 1.03% on average, showing an insignificant effect on motion quality.

Regarding (2), we initially disregard semantic information and assume that the query and key tokens are identical (i.e., the self-correlation case). We theoretically derive the attention between two tokens and plot its decay curve with respect to positional distance. As shown in Figure 17(b), the decay rate is notably alleviated when applying λh/w\lambda^{h/w}.

Beyond this ideal setting, we empirically examine the shifts in self-attention during denoising before and after RoPE modification, incorporating both semantic and positional information. As shown in Figure 10 in Section 6.4, we visualize the self-attention from the object region in the initial frame to other frames. In the baseline, attention across frames remains locked onto the object region from the first frame. This causes candidate regions near the top to be assigned higher confidence in the latter half of the video, leading to the basketball hovering in the air. In contrast, with our method, the model not only attends to the identical region corresponding to the object's initial position during early generation (e.g., at denoising step 5), but also allocates a portion of attention to other regions. Consequently, following motion planning, the model is able to progressively shift attention toward the actual locations of the object in subsequent frames(e.g. the change in attention from step 7 to 20). Overall, our method enhances the exploration during motion planning by altering the attended regions in self-attention, thereby improving the physical commonsense of the model.


For Question 1: About the validation set

Yes. We utilized the 343 samples from the official VideoPhy training set hosted on Hugging Face as our validation set. Please refer to our response to Weakness 1 (Part II) for full details.

For Question 2: About human evaluation and multi-seed evaluation

We have provided more details in our reply to Weakness 1 (Part I). We sincerely hope those additional results and clarifications resolve your concerns.

For Question 3: Mechanistic analysis of RoPE modification

Yes, and we have provided details in our reply to Weakness 2.

For Question 4: Control results

In fact, both ablation studies suggested by the reviewer were explored during our experiments:

  1. Modifying λf\lambda^f along the temporal axis: We have discussed about this in our response to Weakness 2. Overall, intervening on the temporal RoPE axis introduces motion discontinuities or stuttering without yielding significant gains in physical commonsense.
  2. Applying λ\lambda across all denoising steps: We have already discussed about this in Appendix H.1.4. Overall, this configuration degrades generation quality, particularly hurting visual aesthetics, since the mid-late denoising stage mainly focuses on appearance generation and does not require any intervention on RoPE for better motion planning.

We have already investigated both phenomena empirically and will incorporate these detailed ablation analyses into the revised manuscript. We thank the reviewer for these constructive suggestions.


In conclusion, we sincerely hope our explanations and additional experiments have clarified your concerns. We would be deeply grateful if you would consider reassessing our work in light of these clarifications.

We hope this message finds you well. As the discussion period is nearing its end, we would like to ensure that we have addressed all your concerns with satisfaction. If there are any additional points or feedback you'd like us to consider, we are happy to be informed.

We'll appreciate it if you could reconsider our work with a new assessment. Looking forward to your reply~

Reviewer_bs9qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates why video models generate motion that violates physics, arguing that the spatial bias introduced by RoPE tends to lock objects into incorrect positions prematurely. The authors examine this phenomenon through attention analysis and intervention studies, and improve the generated results by adjusting the RoPE during the early denoising stage.

Critical strengths
  • The analysis does more than just display attention maps. The authors also checks whether they actually influence motion by removing the attention heads.
  • The proposed method is simple, and the results are clear: without fine-tuning, VideoPhy’s physical accuracy improved from 29.45% to 39.94%.
Critical weaknesses
  1. It has not yet been proven that RoPE causes physical errors through the mechanism described by the authors. The paper identifies spatial attention biases and finds that modifying RoPE improves results; however, this does not directly validate the causal chain that “erroneous positions stabilize first, followed by the suppression of originally reasonable motion trajectories.” Existing attention head ablations primarily demonstrate that certain heads influence motion but do not directly validate this suppression process. Furthermore, the decay curves in the appendix omit data for true Q/K values and cannot serve as a sufficient explanation for actual attention behavior.

  2. Core analysis metrics cannot clearly distinguish between motion planning, appearance generation, and physical correctness. The attention head contribution measures changes in the denoised prediction within an object region, which also includes shape and appearance; the “winner” of a candidate region is defined based on the position ultimately generated by the model—even if the final motion is incorrect, it is still considered the “winner.” Therefore, while these metrics can describe how the generated results are formed, they are insufficient to confirm the existence of a “correct physical solution” within the model or how such a solution is pruned.

Questions for authors
  • Is it possible to directly intervene in the attentional connections believed to cause “incorrect spatial anchoring” to verify whether reasonable trajectories are restored, and to include a random intervention as a control?
  • How can we confirm that the identified attentional heads are primarily responsible for motion rather than object appearance? Is it possible to add a control experiment that “modifies only appearance, not motion”?
  • Does the standard LoRA baseline use the same early-time-step sampling? We would like to compare the effects of modifying RoPE in isolation, while keeping all other training settings exactly the same.
  • The motion scenarios used for analysis are relatively simple. Is it possible to report the physical error of the trajectory in controlled scenarios,such as free-fall and collisions,while also reporting the magnitude of motion, in order to distinguish between improvements in physical accuracy and general improvements in motion?
5 comments

Thank you for your valuable suggestions. We will make every effort to address your concerns.

For Weakness 1: The causality between RoPE spatial anchoring and physics violations

In the revised manuscript, we have incorporated a new section, Section 6.4, specifically dedicated to discussing the causality between RoPE spatial anchoring and physics violations. Specifically,we provide two sets of experiments for verification: (1) We design a controlled experiment including the baseline (original model), aggravated spatial decay of RoPE (λh/w=1.20\lambda^{h/w} = 1.20), and temporal adjustments of RoPE (λt=0.70,1.20\lambda^{t} = 0.70, 1.20). (2) We provide a post-hoc interpretability analysis for RoPE modification, verifying that our method indeed modulates self-attention by mitigating spatial decay, thereby leading to closer cross-frame attention within object regions.

Regarding (1), we evaluate on VideoPhy, and find that when the spatial decay is aggravated, the overall PC drops from 29.33% to 26.54%, performing far worse than our method. This bidirectional intervention rules out the possibility of incidental gains resulting from unidirectional modifications. Besides, when intervening in the temporal dimension of RoPE, the overall PC of the model decreases slightly by 1.03% on average, showing an insignificant effect on motion quality.

Regarding (2), we initially disregard semantic information and assume that the query and key tokens are identical (i.e., the self-correlation case). We theoretically derive the attention between two tokens and plot its decay curve with respect to positional distance. As shown in Figure 17(b), the decay rate is notably alleviated when applying λh/w\lambda^{h/w}.

Beyond this ideal setting, we empirically examine the shifts in self-attention during denoising before and after RoPE modification, incorporating both semantic and positional information. As shown in Figure 10 in Section 6.4, we visualize the self-attention from the object region in the initial frame to other frames. In the baseline, attention across frames remains locked onto the object region from the first frame. This causes candidate regions near the top to be assigned higher confidence in the latter half of the video, leading to the basketball hovering in the air. In contrast, with our method, the model not only attends to the identical region corresponding to the object's initial position during early generation (e.g., at denoising step 5), but also allocates a portion of attention to other regions. Consequently, following motion planning, the model is able to progressively shift attention toward the actual locations of the object in subsequent frames(e.g. the change in attention from step 7 to 20). Overall, our method enhances the exploration during motion planning by altering the attended regions in self-attention, thereby improving the physical commonsense of the model.

For Weakness 2: About the metrics in our core analysis

There may be some slight misunderstandings about our paper, and we are glad to clarify them in detail below:

Metrics for attention head contribution

The concern raised by the reviewer was carefully considered in our original design. Specifically, when formulating the patching metric, our core objective was to decouple physics-related motion planning from appearance generation.

  1. Prior works (as cited in Related Work) have established that during the early denoising stage (the initial 10% of denoising steps), the model primarily focuses on low-frequency structural representations—such as motion and coarse spatial layout, rather than fine-grained appearance details. To further isolate spatial layout, we explicitly constrained the patching target to the object region. Consequently, this design guarantees that our patching metric isolates and highlights physics-related motion planning.
  2. Furthermore, the zero-ablation experiments in Section 4.2 provide an empirical, a posteriori validation of our patching findings. This guarantees that our patching metric truly separates motion planning from appearance generation, A more detailed formulation and explanation of this metric can be found in Appendix E.1.

About the definition of the “winner”

The term "winner" reflects the model's internal belief regarding where the object should be located during the denoising process. It does not denote the physically ground-truth object position. Accordingly, the winner-loser gap quantifies the model's confidence toward a specific candidate region. In our mechanistic interpretability framework, we do not presuppose that the model's generated content conforms to physical laws (if it already did, improving physical commonsense would be redundant). Instead, our goal is to introduce a self-attention-derived metric that faithfully mirrors the model's latent decision-making dynamics during motion planning. We hope this clarification dispels any ambiguity.

For Question: 1 Intervention on attention

Regarding the ablation approach suggested by the reviewer, we explored this in our preliminary experiments. We observed that directly zero-ablating these identified self-attention heads or performing random ablation drastically degrades overall video quality (particularly visual aesthetics) without yielding meaningful gains in physical commonsense. This suggests that these attention heads likely participate in generative aspects beyond pure motion. We will incorporate a detailed discussion of these empirical findings in the revised manuscript.

For Question 2: Attention heads that only modifiy appearence

The reviewer raises an insightful question. In practice, we found an exhaustive search for such heads unnecessary: our exploration of cross-attention heads revealed that, aside from the heads dedicated to motion planning (that are found via our patching metric), the vast majority predominantly modulate the appearance of the generated content.

Moreover, we observed that their influence evolves across denoising timesteps. Specifically, these heads exert minimal impact on appearance during the early denoising phase (first 10% of steps) and become more influential in the mid-to-late stages (as detailed in our response to Weakness 2). We will add a dedicated discussion on this temporal behavior in the revision. We thank the reviewer for this constructive suggestion.

For Question 3: About early timestep sampling

We took this factor into account during our evaluation. First, in the Table 1 presented in the paper, the LoRA baseline does not utilize early timestep sampling. Second, our empirical findings showed that applying early timestep sampling to the LoRA baseline fails to improve performance and instead harms generation quality. Therefore, to ensure a fair and rigorous comparison, we reported results against the strongest baseline.

For Question 4: Evaluation scenarios

We genuinely appreciate this suggestion and gave it thorough consideration. However, current open-source academic models, such as the Wan2.1 backbone used in our study, are fundamentally unable to support such fine-grained evaluation scenarios:

  1. Scenario Generation Capabilities: In our early exploratory runs, Wan2.1 consistently failed to generate realistic collision scenarios (whether a ball collides with a wall or two balls collide). The basketball free-fall setting analyzed in our work represents the most challenging physical phenomenon that the model could reliably synthesize for quantitative study.
  2. Fine-grained Physical Metrics: Conducting precise quantitative measurements on physical properties such as instantaneous velocity and acceleration remains beyond the representational limits of Wan2.1. In fact, generating physically rigorous trajectories of this fidelity remains an active challenge even for frontier proprietary models (e.g., Seedance 2.0).

Consequently, conducting such granular physical evaluations on current open-source foundations is practically infeasible, which also explains why established general benchmarks for physical commonsense in video generation do not incorporate them. A more exhaustive quantitative evaluation along these lines will likely require more capable open-source foundation models in the future. We kindly ask for the reviewer's understanding regarding these community-wide modeling bottlenecks.


In light of the above clarifications and additional experiments, we sincerely hope our response resolves your concerns. We would be deeply grateful if you would consider reassessing our work in your final evaluation.

We hope this message finds you well. As the discussion period is nearing its end, we would like to ensure that we have addressed all your concerns with satisfaction. If there are any additional points or feedback you'd like us to consider, we are happy to be informed.

We'll appreciate it if you could reconsider our work with a new assessment. Looking forward to your reply~

Reviewer_rrX7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why video diffusion models violate physics through an interpretability analysis of Wan2.1-T2V. The authors identify cross-attention heads responsible for motion planning and argue that RoPE’s spatial decay in self-attention causes physically implausible positions to lock in early. They propose scaling the spatial RoPE frequencies during early denoising steps, which improves physical commonsense on VideoPhy.

Critical strengths
  1. Systematic mechanistic analysis. By combining correlational metrics (convergence speed), causal attribution, and zero ablation, the paper convincingly distinguishes heads that merely display trajectory patterns from those that actually drive motion planning. Extending this to a candidate competition analysis in self-attention forms a coherent narrative.

  2. Simple and practical fix derived from the analysis. A single RoPE frequency scaling improves physical commonsense without additional modules, complements prompt refinement, and outperforms a LoRA-only baseline, partially disentangling the effect of the training data.

Critical weaknesses
  1. Limited and insufficiently detailed evaluation. Quantitative results are restricted to VideoPhy with a single seed and the 1.3B model, while the 14B results are only qualitative. Details of the human evaluation (number of evaluators, inter-rater agreement) are missing. Without video quality metrics (e.g., VBench) or motion magnitude measures, it is unclear whether scaling λ improves physical plausibility or simply increases motion magnitude or diversity.

  2. Weak causal link between RoPE spatial anchoring and physics violations. The paper does not re-analyze the competition dynamics after the modification to confirm that failures are resolved through the proposed mechanism. Moreover, the training-free variant requires per-seed tuning of λ (Fig. 22), which weakens the claim of addressing the root cause.

Questions for authors

Does the same phenomenon, and the effectiveness of the proposed fix, hold for other 3D RoPE-based models (e.g., Wan2.2, HunyuanVideo, CogVideoX), along the temporal RoPE axis, or with other positional encodings? In addition, does reducing λ introduce side effects in static scenes or small-motion cases, such as degraded object permanence?

4 comments

Thank you for your valuable suggestions. We address your concerns in detail below.

For Weakness 1: Limited and insufficiently detailed evaluation

We have made the following improvements:

Evaluation with more seeds

To ensure the reliability of our evaluation results, we conducted additional experiments across four distinct random seeds (10, 20, 30, and 40) in addition to seed 42. The detailed results are summarized below, where the values represent the mean and standard deviation across all the 5 seeds. Overall, these findings are consistent with the results reported in our initial submission, demonstrating that our method maintains robust and consistent advantages regardless of seed variations.

Baseline: Wan2.1-T2V-1.3B, mRoPE: modified RoPE, PR: prompt refinement |Model|Overall SA|Overall PC|Solid-Solid SA|Solid-Solid PC|Solid-Fluid SA|Solid-Fluid PC|Fluid-Fluid SA|Fluid-Fluid PC| |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Baseline|53.53±0.33|29.33±0.33|47.18±0.00|24.65±0.00|55.89±1.04|31.23±0.97|63.64±2.23|36.36±1.82| |PR|80.35±0.33|44.20±0.33|69.30±0.63|28.45±0.59|87.13±0.41|52.88±0.46|90.91±1.82|61.82±1.29| |LoRA|55.69±0.21|35.57±0.21|47.89±0.00|28.17±0.00|60.27±0.62|39.04±0.65|63.64±1.29|45.45±2.87| |VideoREPA|56.44±0.33|37.78±0.33|48.59±0.00|30.28±0.00|61.37±0.97|41.51±0.97|63.64±1.82|47.27±1.29| |mRoPE|57.43±0.21|39.94±0.21|49.30±0.31|30.99±0.63|63.70±0.70|47.26±0.98|61.82±1.29|43.64±2.87| |mRoPE+PR|86.42±0.33|59.01±0.33|83.38±0.80|44.65±0.80|87.67±0.31|67.81±0.31|90.91±1.29|72.73±1.29| |LoRA+mRoPE|64.72±0.46|41.98±0.46|52.82±0.63|34.51±0.31|75.34±1.40|47.95±1.25|67.27±2.87|45.45±1.29| |LoRA+mRoPE+PR|87.40±0.24|62.56±0.33|84.37±0.50|49.72±0.87|87.67±0.31|71.23±0.31|94.55±1.29|72.73±2.87|

Evaluation on the 14B Model

Due to computational resource constraints, we were only able to fine-tune the 1.3B model. For the 14B model, we conducted experiments in a training-free manner and demonstrated that a training-free approach alone suffices to improve the physical commonsense of the 14B model. We have made every effort to present the evaluation results for the 14B model in the paper, and we believe that fine-tuning the 14B model would yield even stronger results. We kindly ask for the reviewer's understanding regarding our resource limitations and hope this clarifies the validity of our reported evaluation results.

More details on the human evaluation

For the outputs generated under each random seed, we invited five evaluators with expertise in relevant research fields to score the results. We employed Fleiss' Kappa to assess inter-evaluator agreement. The resulting score of approximately 0.71 indicates Substantial Agreement, confirming the reliability and consistency of our evaluations.

More evaluation metrics

In fact, the video quality metrics were already considered during our evaluation. The VBench benchmark primarily spans two evaluation dimensions: video-condition consistency and video quality.

  • Video-condition consistency: The Semantic Alignment (SA) metric from VideoPhy that we adopted already covers the evaluation of video-condition consistency (including semantics and style).
  • Video quality: We detailed the evaluation protocol for this dimension in Appendix H.2. Specifically, while scoring the generated videos using the SA and PC metrics, five domain-expert annotators were also instructed to evaluate them in strict accordance with VBench's video quality criteria (including both temporal quality and frame-wise quality). Because we observed that our RoPE modification improves the model's physical commonsense without significantly affecting overall video quality (aside from a minor degradation in visual aesthetics when updating the MLP during LoRA fine-tuning), we did not elaborate on these detailed quality scores in the main text. This design choice is noted in Section 6.3 and Appendix H.1.2, where additional details can be found. We will add more details in the final version of our submission.

We hope these additional experimental details and clarifications address your concerns.

For Weakness 2: The causality between RoPE spatial anchoring and physics violations

Part I

In the revised manuscript, we have incorporated a new section, Section 6.4, specifically dedicated to discussing the causality between RoPE spatial anchoring and physics violations. Specifically,we provide two sets of experiments for verification:

  • (1) We design a controlled experiment including the baseline (original model), aggravated spatial decay of RoPE (λh/w=1.20\lambda^{h/w} = 1.20), and temporal adjustments of RoPE (λt=0.70,1.20\lambda^{t} = 0.70, 1.20).
  • (2) We provide a post-hoc interpretability analysis for RoPE modification, verifying that our method indeed modulates self-attention by mitigating spatial decay, thereby leading to closer cross-frame attention within object regions.

Regarding (1), we evaluate on VideoPhy, and find that when the spatial decay is aggravated, the overall PC drops from 29.33% to 26.54%, performing far worse than our method. This bidirectional intervention rules out the possibility of incidental gains resulting from unidirectional modifications. Besides, when intervening in the temporal dimension of RoPE, the overall PC of the model decreases slightly by 1.03% on average, showing an insignificant effect on motion quality.

Regarding (2), we initially disregard semantic information and assume that the query and key tokens are identical (i.e., the self-correlation case). We theoretically derive the attention between two tokens and plot its decay curve with respect to positional distance. As shown in Figure 17(b), the decay rate is notably alleviated when applying λh/w\lambda^{h/w}.

Beyond this ideal setting, we empirically examine the shifts in self-attention during denoising before and after RoPE modification, incorporating both semantic and positional information. As shown in Figure 10 in Section 6.2, we visualize the self-attention from the object region in the initial frame to other frames. In the baseline, attention across frames remains locked onto the object region from the first frame. This causes candidate regions near the top to be assigned higher confidence in the latter half of the video, leading to the basketball hovering in the air. In contrast, with our method, the model not only attends to the identical region corresponding to the object's initial position during early generation (e.g., at denoising step 5), but also allocates a portion of attention to other regions. Consequently, following motion planning, the model is able to progressively shift attention toward the actual locations of the object in subsequent frames(e.g. the change in attention from step 7 to 20). Overall, our method enhances the exploration during motion planning by altering the attended regions in self-attention, thereby improving the physical commonsense of the model.

Part II

Furthermore, it is worth clarifying that our statement in Fig. 22 ("the training-free variant requires per-seed tuning of λ\lambda”) was simply an honest acknowledgement of the limitations of one specific realization among our various RoPE modification variants. It does not imply that all of our RoPE modification methods share this drawback. More importantly, our alternative realization, the training-based variant, substantially mitigates physical violations in the model. Therefore, we believe our approach addresses one of the fundamental causes of physics violations in video generation models. We hope this clarification clears up any misunderstanding regarding our statements.


For Question 1: Effectiveness of our method on other Models

Yes. We also verified our approach on Wan2.2 and CogVideoX and observed consistent phenomena. Our method demonstrates significant improvements on these models as well. We will incorporate results across additional models into the final paper.

Regarding RoPE modification on the temporal axis, we addressed this in our response to Weakness 2. Overall, intervening on the temporal RoPE axis introduces motion discontinuities or stuttering without yielding significant gains in physical commonsense.

For Question 2: Potential Side Effects of λ\lambda

Regarding this concern, we have already provided discussions in Section 6.2 and Appendices H.1 & H.2 of our paper. In fact, our human evaluation on video quality was specifically designed to investigate this exact issue. Overall, reducing λ\lambda does not induce noticeable side effects on video quality (following the evaluation requirements of VBench). The reviewer could refer to the corresponding sections in the Appendix for further details.


Thank you again for your valuable time and insightful feedback. We sincerely hope our response helps address your concerns and would greatly appreciate your reconsideration of our submission.

We hope this message finds you well. As the discussion period is nearing its end, we would like to ensure that we have addressed all your concerns with satisfaction. If there are any additional points or feedback you'd like us to consider, we are happy to be informed.

We'll appreciate it if you could reconsider our work with a new assessment. Looking forward to your reply~

Thank you for your thoughtful rebuttal and additional experiments. My concerns are largely addressed so I will maintain my positive rating (weak accept).

Comments on the paper

1 comment
Area_Chair_ZgfXOpenReview ↗

Please engage in the discussion

Dear Reviewers,

Thank you for your reviews and contributions so far. The authors have now posted their response, and I would like to kindly remind you to review it and engage in the discussion where appropriate.

Please also consider whether the authors' clarifications address any of your concerns and update your assessment if warranted.

Thank you for your time and effort.

Best regards,

Your AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.