acceptodds
Under review as a conference paper at ICLR 2027

Towards Realistic Temporal Dynamics in Video Diffusion Models

Abstract

Text-to-video diffusion models have achieved remarkable progress in visual fidelity and semantic alignment, but still struggle to reproduce the temporal dynamics observed in real videos. We characterize this discrepancy along two complementary dimensions: motion magnitude, which measures how much temporal change occurs, and temporal pacing, which describes how that change unfolds over time. Across three video generators, we observe similar temporal miscalibration patterns, while the required calibration varies across models and individual samples. Based on these observations, we introduce Adaptive Temporal Coordinate Calibration, an inference-time method that reparameterizes the temporal RoPE coordinates of a pretrained video diffusion model. During denoising, we predict temporal statistics from intermediate representations and calibrate them toward real video dynamics. We evaluate our method on LTX-Video-13B, Wan2.1-T2V-1.3B, and CogVideoX-5B using held-out SSv2 and Panda-70M videos with external temporal benchmarks, including ChronoMagic-Bench and Ref4D-VideoBench. On the held-out dataset, our method reduces -Gap by 17.2% and Prog- by 19.0% on average across the three evaluated models, while largely preserving visual quality. The method requires no additional denoising steps and adds less than 0.7% overhead across the evaluated models.

Primary area:
generative models
Keywords:
Text-to-Video Generation, Video Diffusion Models, Temporal Dynamics, Temporal Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XC5wOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why videos generated by text-to-video diffusion models often show less realistic temporal dynamics than real videos. The authors characterize the discrepancy through two statistics: motion magnitude, which measures how much change occurs, and temporal pacing, which describes when the change occurs. Comparing caption-matched real and generated videos from SSv2 and Panda-70M, they observe that LTX-Video, Wan2.1, and CogVideoX all produce less motion than real videos, while the required correction differs across models and samples. Based on these observations, they propose Adaptive Temporal Coordinate Calibration, a training-free method that reparameterizes the temporal RoPE coordinates during denoising. At a chosen denoising step, the method predicts the final temporal statistics from intermediate hidden states, retrieves similar samples from calibration banks of real and generated videos, and adjusts a temporal scale and a pacing warp to move the statistics toward those of real videos. On held-out SSv2 and Panda-70M videos, the method reduces M-Gap by 17.2% and Prog-W1W_1 by 19.0% on average across the three models, with less than 0.7% inference overhead. It is also evaluated on ChronoMagic-Bench and Ref4D-VideoBench.

Critical strengths

S1. Well-motivated problem with a simple, training-free method

Generated videos often show less motion than real videos. The paper makes this measurable through caption-matched comparisons, which show under-motion in all three models, with generated-to-real ratios of 0.32, 0.55, and 0.33 (Fig. 2). It then defines temporal realism through two statistics, motion magnitude and pacing, and calibrates them by reparameterizing the temporal RoPE coordinates. The method needs no training, preserves the frame order, adds less than 0.7% inference overhead, and is applied to three different backbones.

Why it is consequential: The method can be applied directly to existing video models, which makes it practical. If the concerns in W1 and W2 are addressed, it would be a useful tool for improving temporal dynamics at inference time.

S2. Systematic analysis of the readability and controllability of temporal statistics

The paper shows that motion magnitude and pacing can be predicted from intermediate hidden states, and measures how the temporal coordinates change these statistics (App. C.4). These analyses are used to choose when and how to intervene, so each part of the method is connected to an observed problem. The appendix also covers a wide range of analyses, including alternative target constructions, local versus global targets, and hyperparameter sensitivity, and the limitations section openly discusses cases where global corrections are competitive.

Why it is consequential: This analysis gives insight into how temporal dynamics are represented in video diffusion transformers, which may be useful independently of the proposed calibration.

Critical weaknesses

W1. Unclear whether the increase in motion statistics reflects more realistic dynamics

The main claim is that calibration makes temporal dynamics more realistic. However, the calibration targets and the evaluation may not distinguish realistic motion from other temporal changes.

(a) Motion and pacing measures. The magnitude target MM is computed from pixel differences between frames rather than from motion itself. Pixel differences also increase with flickering, lighting changes, or local texture changes, so increasing MM toward the real value does not necessarily increase realistic motion. For pacing, c1c_1 summarizes the timing of motion with a single number. As a result, a realistic video whose motion is concentrated late, such as a throw at the end, and a failure case that stays static and then changes abruptly can have similar c1c_1 values.

(b) Evaluation of temporal quality. The supplementary videos suggest that the added motion is not always realistic. In several Wan2.1 examples, the calibrated videos show more flickering or local texture changes rather than more realistic motion. However, the paper does not report standard temporal quality metrics or a user study, and most of the reported metrics measure the same properties that the method calibrates.

Why it is consequential: If part of the increase in MM comes from flickering or local artifacts, or if the pacing correction changes videos whose timing is already appropriate, the improvements in Tables 1 and 2 may overstate the main claim. (See Q1–Q3.)

W2. Insufficient experimental evidence

(a) Baselines. The CogVideoX baseline videos in the supplementary material are nearly blank, which raises the question of whether the baseline was run properly. In addition, the method is compared with the base models alone, without any existing training-free method for improving motion.

(b) Mixed results. The improvements do not hold consistently. For every model, at least one metric worsens after calibration, especially on the external benchmarks (Tables 1 and 2).

(c) Benefit of per-sample calibration. The main contribution is adapting the correction to each video. However, compared with applying the same correction to every video, the adaptive version is better on ChronoMagic (0.576 vs. 0.808), similar on the held-out set (1.087 vs. 1.080), and worse on Ref4D (1.634 vs. 1.473) (Tables 13 and 18).

(d) Reliability of the targets and per-model design. The per-sample targets are built from caption-matched pairs of real and generated videos, and from neighbors retrieved for the current video. However, the gap within a pair may not always reflect a limitation of the generator. For example, real videos for "Pushing something from left to right" can show either a quick or a slow push, so a generated video with a slow push may appear to under-move simply because its paired real video is quick. It is also unclear whether the retrieved neighbors are relevant to the current video. In addition, several components are set separately for each model, and the offline cost of building the banks is not reported.

Why it is consequential: These comparisons are the main evidence for the method. With a possibly faulty baseline, no comparison with existing methods, and mixed results, it is unclear whether the calibration reliably improves over the base models, and whether the per-sample components and their cost are justified. (See Q4–Q9.)

Questions for authors

I find the problem setting valuable. The paper treats the magnitude and pacing of motion as concrete problems, and each part of the method is designed to address them, which I find meaningful. This is an important problem for video generation. The analysis is systematic, and the effort to connect each observed problem to a specific solution is clear throughout the paper. However, I have concerns about the experiments and the evaluation, and from the current results it is unclear whether the calibration works as intended. My questions below focus on these points, and the answers could affect my assessment.

[R] = additional result requested, [C] = clarification.

1. Realistic dynamics (W1)

  • Q1 [R/C] (a). The paper notes that MM captures "related but distinct" variation from optical-flow magnitude (App. C.2), and shows that the improvement holds when evaluated with residual flow. Why was a pixel-based measure chosen as the calibration target rather than a flow-based one, and would a flow-based target change the results?

  • Q2 [R/C] (a). A video in which something is thrown at the end and a video that stays static and then changes abruptly at the end can both have their motion concentrated late, and thus similar c1c_1 values. The former may be realistic, whereas the latter is the kind of failure that the introduction describes (L36). How does the method distinguish the two? Could the authors show such examples before and after calibration?

  • Q3 [R] (b). In the Wan2.1 examples, the added motion is not always realistic or related to the prompt.

    • For "Putting trivet underneath candle holder," the calibrated video flickers more than the baseline even while the scene is static. Later, a hand appears abruptly and lights the candle, which is already lit.
    • For the snowy-mountain prompt, the main change is larger and more frequent snowflakes.
    • For the explosion prompt, the main change is a bubbling texture on the fireball.

    Apart from per-frame Imaging Quality, the reported metrics measure motion magnitude and pacing, and two of them (MM-Gap and c1c_1-W1W_1) use the same statistics as the calibration. Could the authors report standard temporal quality metrics, such as VBench Subject Consistency, Background Consistency, Temporal Flickering, and Motion Smoothness, along with a text-alignment metric? A user study would also help.

2. Experimental evidence (W2)

  • Q4 [C] (a). App. A.3 states that the official inference configuration is used, but all four CogVideoX baseline videos are nearly blank. In my experience with the official CogVideoX-5B setting, such outputs are uncommon. What are the exact resolution, number of frames, and guidance scale? The SSv2 prompts are also short (e.g., "Putting a black pen on a surface"), while CogVideoX recommends prompt extension. Was prompt extension applied, and how often does the baseline produce nearly blank outputs?

  • Q5 [R] (W2a). The experiments compare the calibrated models with the base models alone, without any existing method for improving motion or temporal dynamics. How does the method compare with existing training-free approaches, such as FlowMo (cited in Sec. 2.2)? FlowMo has public code and is evaluated on Wan2.1-1.3B and CogVideoX-5B, two of the backbones used in this paper, so such a comparison seems feasible.

  • Q6 [C] (b). Several metrics worsen after calibration. Ref4D Prog-W1W_1 increases for LTX-Video (0.0210 → 0.0278) and Wan2.1 (0.0261 → 0.0341). For CogVideoX, Ref4D Motion decreases (69.85 → 68.83), and Flow-MM Gap increases on both Ref4D (0.3569 → 0.4332) and the held-out set (0.5845 → 0.5945). Imaging Quality decreases for Wan2.1 (0.6919 → 0.6650 in Table 1). Could the authors discuss these cases?

  • Q7 [C] (W2d). Both the analysis in Sec. 3 and the calibration banks rely on caption-matched pairs, where each real video is compared with a video generated from its caption. However, the paper does not describe how the captions were obtained. In the examples in Table 19, the SSv2 prompts are short action labels (e.g., "Putting trivet underneath candle holder") that do not specify the speed or extent of motion, and the Panda-70M prompts mainly describe the appearance of the scene. Could the authors describe how the captions were obtained, and discuss whether they are detailed enough to determine the expected motion? Since the paper notes "substantial sample-level variation" in these pairs (App. C.1), it would also be helpful to show how much MM varies across seeds of the same prompt.

  • Q8 [R] (W2d). The signature qq averages hidden-state changes over spatial positions and channels, so it may carry little information about the content. For example, if the generator produces nearly static videos for both "Throwing something" and "Pushing something from left to right" in SSv2, the two may have similar signatures and retrieve each other as neighbors. However, the corresponding real videos differ in motion, so the two would require different corrections. Could the authors show the retrieved neighbors for such example queries, and whether these videos actually receive different corrections (e.g., different values of α\alpha and γ\gamma)?

  • Q9 [C] (W2d). The intervention steps (Table 6), the bank size NBN_B and neighbor count KK (Table 9), and the ranges of α\alpha and γ\gamma (App. B.4) are set per model. How were they chosen, and how sensitive are the results to them? In addition, the paper reports less than 0.7% inference overhead (Table 5), but applying the method to a new model also requires generating videos for two banks and running response sweeps. What is this offline cost for each model? Since a single global correction performs comparably on some benchmarks (W2(c)), this cost is important for judging the practical benefit of the method.

Suggestions (not affecting the assessment)

  • Fig. 5 uses symbols such as αB\alpha_B, GnG_n, RnR_n, Δn\Delta_n, and ℓ′\ell' that are not defined in the text.
  • The distance used for Prog-W1W_1 (e.g., cosine or L2 between DINOv2 features) is not specified.
9 comments
AuthorsOpenReview ↗

Response to XC5w (1/4): Realistic Dynamics and Independent Validation (Q1–Q3)

We thank the reviewer for the detailed and constructive feedback. We address each question below.

Q1 [R/C] — Pixel-based vs. flow-based calibration targets

Our flow-based target ablation demonstrates that pixel-based calibration achieves nearly identical motion-alignment improvements to residual-flow-based calibration.

We evaluated 100 held-out LTX-Video generations (Stage I only), replacing the pixel-based magnitude target with camera-motion-compensated residual optical flow. Importantly, we re-estimated both the calibration-bank statistics and the control response for the alternative target.

The two approaches exhibit similar control sensitivities (0.533 vs. 0.537) and strongly correlated estimated corrections (r = 0.81).

More importantly, their generated videos achieve nearly identical results:

  • Flow-M Gap: 1.087 (pixel) vs. 1.086 (flow)
  • Residual-Flow Gap: 0.957 (pixel) vs. 0.957 (flow)
  • Prog-W1: 0.0414 (pixel) vs. 0.0418 (flow)

These results demonstrate that the computationally simpler pixel-based target is sufficient to achieve comparable motion alignment in the tested operating range, without requiring optical-flow estimation for calibration-bank construction.

The near-identical outcomes partly reflect bounded control parameters; we therefore interpret this result as practical equivalence within the evaluated setting, rather than equivalence of the underlying motion measures.

Revision: Appendix F.1, Table 22.

Q2 [R/C] — Realistic late actions vs. abrupt temporal changes

Our additional qualitative analysis demonstrates that temporal calibration can substantially reduce concentrated abrupt changes while preserving continuous late-stage actions.

We added two before-and-after examples: (i) a continuous pouring action occurring late and (ii) a nearly static scene followed by an abrupt hand movement (Fig. 10).

Both examples use the same correction (α = 1.20, γ = 0.15), yet exhibit different temporal responses.

For the continuous action, c1 remains nearly unchanged (−0.25 → −0.24), and the action remains continuous.

For the abrupt-change example, c1 shifts from −0.44 to −0.24, while the maximum single-step share of temporal change decreases substantially from 0.126 to 0.028 (−77.8%).

This shows that temporal-coordinate calibration can redistribute concentrated motion rather than simply increasing overall motion magnitude. We additionally introduce an abruptness diagnostic (Cmax) to characterize temporal discontinuities beyond c1.

While c1 is not a semantic action classifier, the examples demonstrate meaningful corrections to different temporal behaviors.

Revision: Appendix C.7, Fig. 10.

Q3 [R] — Temporal consistency and text alignment

We extended our evaluation beyond motion magnitude to directly examine temporal consistency, visual stability, and text alignment.

We additionally report VBench Subject Consistency, Background Consistency, Temporal Flickering, Motion Smoothness, and ViCLIP-based Overall Consistency across three backbones and three benchmarks.

For example, on Wan2.1 held-out videos:

  • Dynamic Degree: 58 → 71
  • Overall Consistency (text alignment): 21.75 → 22.21
  • Temporal Flickering: 98.47 → 97.65
  • Motion Smoothness: 99.04 → 98.45

These results show that the substantial increase in motion activity is accompanied by a small improvement in text alignment, although some temporal-consistency scores decrease modestly.

Thus, the expanded evaluation establishes a more complete picture of temporal calibration: motion alignment can improve without a corresponding loss of text alignment, while visual-stability trade-offs remain measurable and model-dependent.

Revision: Appendix F.7, Table 29.

AuthorsOpenReview ↗

Response to XC5w (2/4): Corrected CogVideoX Evaluation and FlowMo Comparison (Q4–Q5)

Q4 [C] — CogVideoX generation settings and nearly blank outputs

We have corrected the CogVideoX generation pipeline and regenerated both Baseline and Ours using the recommended inference configuration. The revised results confirm improvements in motion magnitude, temporal pacing, and benchmark-specific motion scores.

Our investigation identified deviations from the recommended pipeline. The original experiments used 720 × 480 resolution, 49 frames (8 FPS), guidance scale 6.0, 50 inference steps, and bfloat16 precision, but dynamic CFG was disabled and the original short dataset captions were used without prompt extension. The original pipeline also used the CogVideoX DDIM scheduler.

We corrected these settings by enabling dynamic CFG, adopting extended prompts, and using the DPM trailing scheduler. Both Baseline and Ours were regenerated under the same corrected configuration.

The updated results show:

Held-out evaluation:

  • M-Gap: 0.925 → 0.813 (−12.1%)
  • c1-W1: 0.0297 → 0.0206 (−30.6%)
  • Dynamic Degree: 0.830 → 0.880

Ref4D:

  • Motion: 67.46 → 67.81
  • Prog-W1: 0.0817 → 0.0750

ChronoMagic:

  • MT: 3.12 → 3.24
  • Flow-M Gap: 1.010 → 0.715 (−29.2%)

These results replace the earlier CogVideoX evaluation and provide evidence of calibration benefits under the corrected generation pipeline.

Some metrics remain model- and benchmark-dependent, including held-out Flow-M Gap (0.225 → 0.293). We report these results transparently in the revised manuscript.

Revision: Tables 1–2 and Sec. 5.1.

Q5 [R] — Direct comparison with FlowMo

Our new FlowMo comparison demonstrates that Ours achieves stronger held-out motion-magnitude alignment and better semantic progression on ChronoMagic, while also showing complementary benefits when combined with FlowMo.

We evaluated Baseline, FlowMo, Ours, and FlowMo + Ours on Wan2.1-T2V-1.3B using matched prompts and seeds.

On the held-out set, Ours outperforms FlowMo on:

  • M-Gap: 0.810 vs. 1.017 (20.4% lower)
  • Flow-M Gap: 1.251 vs. 1.366 (8.4% lower)
  • Dynamic Degree: 0.700 vs. 0.560

Combining FlowMo with Ours further reduces M-Gap to 0.801 and c1-W1 to 0.0227.

On ChronoMagic, Ours achieves better semantic progression (Prog-W1: 0.0587 vs. 0.0697), while FlowMo produces a lower Flow-M Gap (0.515 vs. 0.553).

On Ref4D, the combination achieves the lowest Flow-M Gap (1.550), demonstrating that the two approaches can be complementary.

These comparisons highlight the distinction between the methods: FlowMo targets motion coherence through inference-time guidance, whereas Ours directly calibrates motion magnitude and pacing toward real-video statistics.

Importantly, our method provides measurable improvements over an existing training-free alternative without additional denoising steps, strengthening the evidence for its practical contribution.

Revision: Appendix F.4, Table 26.

AuthorsOpenReview ↗

Response to XC5w (3/4): Understanding Metric Discrepancies and Caption Variability (Q6–Q7)

Q6 [C] — Why do some metrics worsen after calibration?

Our additional component-wise and real-video analyses identify concrete sources of disagreement between temporal calibration objectives and independent evaluation metrics. These findings clarify the behavior of the method beyond simply reporting improvements in targeted statistics.

(1) Ref4D semantic progression.

Although Ours improves motion alignment on Ref4D, Prog-W1 increases for LTX-Video (0.0210 → 0.0278) and Wan2.1 (0.0261 → 0.0341).

To investigate this discrepancy, we separately evaluate magnitude and pacing calibration on LTX-Video.

  • Baseline Prog-W1: 0.0212
  • Stage I only: 0.0191
  • Stage II only: 0.0500

The Stage-I improvement demonstrates that motion-magnitude calibration can also benefit semantic progression. In contrast, the Stage-II result identifies pacing calibration as a major source of the degradation in this setting.

This distinction is informative because our c1 statistic captures pixel-level temporal allocation, whereas Prog-W1 measures progression using DINOv2 semantic features. The discrepancy is consistent with a domain-dependent mismatch between pixel-based pacing targets and semantic progression.

(2) Wan2.1 Imaging Quality.

We additionally measure Imaging Quality on real videos at matched resolutions to contextualize the observed decrease.

Across SSv2, Panda-70M, and Ref4D, real-video scores are 58.12, 64.58, and 69.88, compared with baseline generation scores of 66.45, 71.99, and 72.88.

Ours obtains intermediate scores of 63.22, 71.12, and 72.24, respectively.

Thus, the decrease in Imaging Quality moves the measured score toward the corresponding real-video distribution. However, because Imaging Quality is not a direct perceptual-realism measure, this observation does not by itself establish improved visual quality.

(3) CogVideoX under corrected inference settings.

Following the configuration correction detailed in Q4, Ref4D Motion improves from 67.46 to 67.81, and Prog-W1 decreases from 0.0817 to 0.0750, while Flow-M Gap increases slightly from 0.466 to 0.504. On ChronoMagic, MT improves from 3.12 to 3.24, accompanied by increases in Dynamic Degree (0.54 → 0.58) and Imaging Quality (0.6118 → 0.6147), although Prog-W1 increases from 0.0154 to 0.0213.

Together, these results demonstrate improvements in motion dynamics and temporal progression under the corrected CogVideoX configuration, while highlighting that different temporal metrics capture complementary properties and may respond differently to calibration. We have incorporated this analysis into the revised discussion.

Revision: Appendix C.5, Table 13; updated Tables 1–2.

Q7 [C] — Caption sources and motion variability across seeds

Our three-seed analysis shows that systematic under-motion remains observable despite substantial variation in individual generated trajectories.

We use the original SSv2 and Panda-70M captions without manual speed annotations. These captions describe actions but do not uniquely specify their duration or motion extent.

To quantify the resulting uncertainty, we evaluated 100 held-out prompts with three generation seeds.

The within-prompt standard deviations of log motion magnitude are 0.68, 0.45, and 0.62 for LTX-Video, Wan2.1, and CogVideoX, respectively.

Nevertheless, the generated-to-real motion-magnitude ratios remain consistently below one across all evaluated seeds:

  • LTX-Video: 0.27–0.29
  • Wan2.1: 0.45–0.46
  • CogVideoX: 0.28–0.35

These results demonstrate that the observed under-motion tendency is not an artifact of a particular generation seed.

Our framework therefore uses caption-matched videos to estimate real-generated temporal distributions, rather than assuming that a caption uniquely determines the correct sample-level motion trajectory.

Revision: Appendix F.8, Table 30.

Reviewer_EWA2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for video temporal dynamics calibration during the inference stage, adjusting both how much movement there is and when it happens. The method first builds an offline calibration library between real videos and their corresponding generated videos. Then, during denoising, it reads the intermediate features of the current sample, uses the calibration library to determine the correction amount, and adjusts the generated result by scaling and non-linear transformation of the temporal RoPE coordinates. The authors also analyze the predictability and controllability at different denoising stages, using this to choose the intervention timing for each model. This method doesn’t require extra denoising steps or training. Experiments were done on multiple video models.

Critical strengths

The method completes time dynamics adjustment in a single generation process, without needing to update the generation model's weights or increasing the number of denoising steps. The authors verified it on three video generation models, showing that the method is somewhat applicable and useful.

The paper analyzed how predictable the final dynamics are at different denoising stages, and how much dynamic control is retained at these stages, using this to decide when to intervene. This analysis helps understand when time dynamics adjustment is appropriate and is a valuable practical insight beyond just designing the method.

Critical weaknesses

I think the contribution of this paper is rather incremental, mainly combining distribution statistics and calibrating the generation process by modifying RoPE. The current results are not enough to fully support the claims, and there’s also a lack of direct comparisons with related existing methods (eg. Video Analysis and Generation via a Semantic Progress Function). From the provided videos, I didn’t see clear enough improvements; some results mainly show faster processes, but speeding up doesn’t necessarily mean more realistic, and more convincing evidence is needed for that.

For example, scenes like snow falling, flowers blooming, or plants growing, in my opinion, don’t clearly show speed issues that need correcting. Also, as I understand it, especially for these older generation open-source video models, they tend to generate videos close to the training distribution and can’t necessarily control the video speed effectively just through prompts. The authors also only did experiments on wan1.3b rather than wan14b. So even if the prompt says time-lapse, it doesn’t directly prove that the model can follow this speed requirement, and you can’t judge that the accelerated results are more reasonable based on that.

I think a more convincing validation would be to select scenes that should naturally happen quickly, first showing that the video model generates them unreasonably slowly, and then showing that this method corrects the issue. The current examples don’t fully answer this question, and the main thing we see is processes that are originally slow being sped up. If the main effect is acceleration, then compared to uniformly adjusting RoPE (Table 14), it needs to be clearer what extra benefits this adaptive calibration actually brings.

Also, in the provided CogVideoX examples, the output from the native method is almost entirely solid color, which the authors need to explain further, including whether there are issues with generation settings or video presentation.

Besides, I’m also curious whether this issue is still significant in more updated and stronger video generation models. The current experiments aren’t enough to tell, and it’s unclear how much the observed phenomena reflect the limitations of the selected models and how much extra benefit this method can provide as the base model improves. If this issue is already much less pronounced in stronger models, then the applicability and practical value of this method need to be more clearly defined. Adding representative newer models like Seedance and running some inferences to see whether the issues described in the paper still exist in the more powerful models would be more helpful in answering this question.

Questions for authors

Please address the questions raised in Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to EWA2 (1/4): Contribution Beyond RoPE Manipulation

We thank the reviewer for the constructive feedback and for recognizing the value of our read–write analysis. Our additional experiments further strengthen the central contribution of our work: real-video-driven, sample-dependent calibration of motion magnitude and temporal pacing, guided by the read–write trade-off during denoising. In particular, we provide direct comparisons with ReTime and fixed temporal scaling to demonstrate the benefits of our framework beyond existing temporal-control approaches.

W1 — What distinguishes our contribution from existing temporal-control methods?

Our method outperforms ReTime under both SPF-based progression metrics and our independent temporal metrics.

We added a direct comparison on CogVideoX-5B using 100 held-out prompts (Appendix F.2, Tables 23–24). Compared with ReTime, Ours achieves:

  • SPF-Prog-W1: 0.0393 → 0.0246 (37.4% reduction)
  • Deviation W1: 0.0308 → 0.0217 (29.5% reduction)
  • c1-W1: 0.0522 → 0.0430 (17.6% reduction)
  • Prog-W1: 0.0560 → 0.0505 (9.8% reduction)

These improvements are particularly meaningful because they extend to SPF-based metrics, rather than being confined to the statistics optimized by our method.

The methodological distinction lies in how temporal corrections are determined. ReTime steers semantic progression toward a prescribed pacing profile, whereas our framework estimates real-video-aligned magnitude and pacing targets from intermediate denoising representations and translates them into sample-dependent temporal-coordinate corrections.

Furthermore, our read–write analysis identifies when temporal statistics become predictable and when temporal coordinates remain sufficiently controllable. This provides a principled basis for selecting intervention steps rather than applying a predefined temporal transformation throughout denoising.

Our contribution is therefore the analysis and closed-loop estimation-and-calibration framework, not the RoPE transformation alone.

Revision: Appendix F.2, Tables 23–24; Secs. 3–4.

W2 — Does adaptive calibration improve upon uniform temporal scaling?

Our fixed-parameter ablations demonstrate that sample-conditioned calibration can achieve substantially better temporal-progression alignment than applying a single correction across videos.

We added fixed-α and fixed-γ comparisons across all three backbones on 100 held-out prompts (seed 0). Each fixed baseline uses the model-wise mean estimated parameter, with the other control held at its identity value.

On LTX-Video, Ours achieves:

  • Prog-W1: 0.0291 vs. 0.0415 (fixed-α) and 0.0438 (fixed-γ)
  • Flow-M Gap: 1.269 vs. 1.293 (fixed-γ)

Notably, the Prog-W1 improvement over fixed-γ corresponds to a 33.6% reduction.

On CogVideoX, Ours also achieves lower Prog-W1 than both fixed baselines (0.0451 vs. 0.0460/0.0482).

These results demonstrate the value of conditioning temporal corrections on individual generation states, particularly for progression alignment. Global corrections remain competitive in some model–metric combinations, including Wan2.1 magnitude alignment, but they cannot directly represent different correction targets across samples.

Together, the ReTime comparison and fixed-parameter ablations provide empirical support for our core contribution: estimating and applying real-video-driven temporal corrections, rather than relying solely on prescribed or uniform temporal rescaling.

Revision: Appendix D.1, Tables 14–15; Sec. 5.2.

AuthorsOpenReview ↗

Response to EWA2 (2/4): Realistic Fast Actions and Motion-Intensive Evaluation

Our additional experiments demonstrate that temporal calibration is effective beyond slowly evolving natural phenomena. We provide a real-reference fast-action comparison and evaluate 128 motion-intensive VideoJAM-Bench prompts, where LTX-Video improves across all six reported metrics.

W3 — Can our method correct unnaturally slow actions rather than merely accelerate slow processes?

We added a qualitative example of rapid object manipulation in which calibration advances a delayed action toward the timing observed in a real-video reference.

The new figure compares Real, Baseline, Fixed-α, and Ours at matched timestamps, together with their framewise motion profiles (Appendix C.6, Fig. 9).

The baseline exhibits a delayed motion response relative to the real-video reference. In contrast, Ours produces earlier motion activity and changes the temporal distribution of the action, bringing its progression closer to the reference.

Importantly, the comparison also includes fixed-α scaling, which partially corrects the delay. This allows us to evaluate our method against uniform temporal acceleration rather than attributing all improvements to adaptive calibration.

This example directly addresses a motion-intensive action with an observable temporal reference, rather than relying on the assumption that accelerating snow, flowers, or plant growth necessarily improves realism.

Our objective is to correct measurable discrepancies in the timing and magnitude of generated motion, rather than to maximize motion speed indiscriminately.

Revision: Appendix C.6, Fig. 9.

W4 — Does the improvement extend to motion-intensive generation?

On VideoJAM-Bench, Ours improves all six reported metrics for LTX-Video, including substantial reductions in motion-magnitude and semantic-progression discrepancies.

We evaluated 128 motion-intensive prompts using seed 0 across LTX-Video, Wan2.1, and CogVideoX (Appendix F.5, Table 27).

For LTX-Video:

  • M-Gap: 0.525 → 0.309 (41.1% reduction)
  • Flow-M Gap: 0.730 → 0.516 (29.3% reduction)
  • c1-W1: 0.0371 → 0.0337
  • Prog-W1: 0.0456 → 0.0324 (28.9% reduction)
  • Dynamic Degree: 0.5234 → 0.5781
  • Imaging Quality: 0.5867 → 0.5930

Importantly, motion alignment, semantic progression, motion activity, and Imaging Quality improve simultaneously in this evaluation.

On Wan2.1 and CogVideoX, both Flow-M Gap and Prog-W1 also improve, providing additional evidence across backbones.

VideoJAM does not provide paired real-video references; therefore, its temporal discrepancies are evaluated against an independent held-out real-video distribution. These results establish distribution-level alignment rather than exact prompt-specific physical timing.

Together, the fast-action comparison and VideoJAM evaluation demonstrate that our method addresses temporal discrepancies in motion-intensive scenarios, extending substantially beyond the slow natural-process examples highlighted by the reviewer.

Revision: Appendix C.6, Fig. 9; Appendix F.5, Table 27.

AuthorsOpenReview ↗

Response to EWA2 (3/4): Corrected CogVideoX Evaluation

We have regenerated and reevaluated both Baseline and Ours using the recommended CogVideoX inference configuration. The revised experiments confirm improvements in motion magnitude, temporal pacing, and benchmark-specific motion scores, strengthening our evaluation on this backbone.

W5 — Investigation of CogVideoX generation quality

To investigate the nearly uniform-color generations highlighted by the reviewer, we carefully examined the original generation pipeline and identified configuration differences from the recommended CogVideoX inference procedure.

The original experiments used 720 × 480 resolution, 49 frames (8 FPS), guidance scale 6.0, 50 inference steps, and bfloat16 precision. However, dynamic CFG was disabled, short dataset captions were used without prompt extension, and the checkpoint's default DDIM scheduler was retained.

We corrected the pipeline by:

  • Enabling dynamic CFG
  • Applying prompt extension
  • Adopting the DPM trailing scheduler

Crucially, we regenerated both Baseline and Ours under the same corrected configuration and replaced the original CogVideoX results in the revised manuscript.

Updated results under the corrected configuration

Held-out evaluation:

  • M-Gap: 0.925 → 0.813 (12.1% reduction)
  • c1-W1: 0.0297 → 0.0206 (30.6% reduction)
  • Dynamic Degree: 0.830 → 0.880

Ref4D:

  • Motion: 67.46 → 67.81
  • Prog-W1: 0.0817 → 0.0750

ChronoMagic:

  • MT: 3.12 → 3.24
  • Flow-M Gap: 1.010 → 0.715 (29.2% reduction)

These results demonstrate that our calibration remains effective under the corrected CogVideoX inference pipeline, with improvements in both the directly targeted temporal statistics and selected independent evaluation metrics.

The response is not identical across all metrics; for example, held-out Flow-M Gap increases from 0.225 to 0.293. We report the complete results in the updated Tables 1–2.

Implications

The revised evaluation addresses the generation-quality concern while providing a more reliable assessment of temporal calibration on CogVideoX.

Importantly, the observed improvements are obtained by comparing Baseline and Ours under identical, corrected inference settings, rather than relying on the earlier problematic generations.

We appreciate the reviewer's observation, which prompted this additional verification and strengthened the experimental protocol.

Revision: Tables 1–2 and Sec. 5.1.

AuthorsOpenReview ↗

Response to EWA2 (4/4): Stronger-Model Validation on Wan2.1-14B

Our new Wan2.1-T2V-14B experiments demonstrate that temporal miscalibration persists in a substantially larger video generator and that our method effectively reduces motion-magnitude discrepancies: M-Gap decreases by 22.0% and Flow-M Gap by 12.8%.

W6 — Does temporal miscalibration persist in larger video models?

We additionally evaluated Wan2.1-T2V-14B, a substantially larger backbone from the same model family as Wan2.1-T2V-1.3B, directly addressing the reviewer's suggestion.

The evaluation uses 50 held-out prompts (25 SSv2 and 25 Panda-70M), seed 0, at 480p resolution.

On the pooled held-out set:

  • M-Gap: 1.198 → 0.934 (22.0% reduction)
  • Flow-M Gap: 1.442 → 1.258 (12.8% reduction)
  • Dynamic Degree: 0.500 → 0.520
  • Imaging Quality: 0.6651 → 0.6599

Both M-Gap and Flow-M Gap improve on SSv2 and Panda-70M individually, indicating that the magnitude-alignment gains are not confined to a single dataset.

The reduction in Flow-M Gap is particularly important: it independently validates the motion-magnitude improvement using optical flow, rather than relying exclusively on the pixel-based statistic used by our controller.

Implications for stronger video generators

These findings directly challenge the hypothesis that temporal miscalibration is merely a limitation of smaller, earlier-generation open-source video models.

Even at 14B scale, the baseline exhibits measurable motion-magnitude discrepancies relative to real videos. Our inference-time calibration substantially reduces these discrepancies without additional training or denoising steps.

Notably, the 22.0% M-Gap reduction demonstrates that the magnitude-calibration component remains effective when moving from the previously evaluated 1.3B-scale family member to a substantially larger backbone.

Temporal pacing remains a separate challenge: the pooled c1-W1 changes from 0.0274 to 0.0293 and Prog-W1 from 0.0492 to 0.0509. These results identify an opportunity for improving pacing calibration on larger models, while leaving the substantial magnitude-alignment improvements intact.

Taken together, the 14B evaluation establishes that our approach is not restricted to the smaller Wan2.1-1.3B backbone and provides direct evidence that real-video-driven motion calibration remains beneficial at substantially greater model scale.

Revision: Appendix F.3, Table 25.

Reviewer_MYtWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

From my understanding, the paper tries to argue that T2V diffusion models get temporal dynamics wrong in two separable ways: 1) motion magnitude, measured as fps-normalized mean absolute frame difference, 2) temporal pacing, c1, the first sine-mode coefficient of the cumulative-change curve.

The method rewrites temporal RoPE indices as τ(ℓ) = αℓ + γ(T−1)sin(πu). At hand-picked, model-specific steps, it computes a 30-d signature from three transformer blocks and retrieves K nearest neighbours from an offline bank of (signature, generated stat, real stat) triples. From these it builds a target: a weighted log-M shift for α, and conditional quantile transport for c1. It then inverts measured sensitivities (Jacobians or response curves) to obtain α and γ.

Using caption-matched videos from SSv2 and Panda-70M, they compare their method for LTX-Video-13B, Wan2.1-1.3B and CogVideoX-5B and come up with these finindngs:1)All three under-move, with generated-to-real ratios of 0.32, 0.55 and 0.33. 2)Pacing has almost no per-sample correspondence with the paired real video, and the direction of its distributional mismatch flips between datasets.3) There is a “read–write trade-off”: the final statistics become more predictable from hidden states later in denoising, while temporal-RoPE interventions lose effect.

Critical strengths
  1. this is a cheap method, model-agnostic, and evaluated on three backbones (ltx, wan, cogvideo). re-indexing of temporal RoPE plus a small KNN lookup seems negiligible overhead (table 5)
Critical weaknesses
  1. The only comparison is base model vs. base model + method. There is no fixed-α baseline (a single global temporal stretch with no prediction or retrieval) and no fixed-γ warp. There is also no comparison to existing inference-time motion or dynamics methods: FlowMo, which is cited but not compared, Spatiotemporal Skip Guidance, or simply raising the frame-rate/motion conditioning that some of these backbones expose.
  2. seems like the only setting where local retrieval helps is an 18-clip subset of ChronoMagic (Table 17).
  3. M is mean absolute pixel difference, which also rises with flicker, lighting change, texture noise and camera shake. The paper concedes in F.3 that for CogVideoX, larger α raises pixel difference while lowering RAFT flow, which is exactly the failure mode one would worry about.
  4. i couldnt find discussion why several numbers move the wrong way and are not mentioned in the text. for example, on Ref4D, Prog-W1 gets worse for LTX (0.0210 → 0.0278) and Wan (0.0261 → 0.0341). CogVideoX’s Ref4D Motion score drops (69.85 → 68.83) and its Flow-M Gap worsens on both Ref4D and the held-out set. Wan’s c1-W1 does not change at all (0.0200 → 0.0200), and Wan’s Imaging Quality drops by 2.7 points (0.6919 → 0.6650), which is hard to call “largely preserved”.
  5. no human preference study and no distribution-level video metric.
Questions for authors
  1. Metzer et al. (ReTime) is cited only as “analyzing nonlinear temporal progression”. In fact that paper retimes generation by warping the RoPE time embeddings so that semantic change unfolds at a constant rate, and it explicitly supports steering sequences toward arbitrary target pacing. That is essentially the γ control here, applied to the same problem. Why the difference is not stated directly, and why ReTime is not compared a baseline for the pacing half? also why videojam-bench is missing?
  2. For LTX-Video, the official pipeline takes a frame-rate input that, as far as I understand, rescales temporal positions. If so, how does α differ from simply changing that input?
6 comments
AuthorsOpenReview ↗

Response to MYtW (1/5): Fixed Calibration and Training-Free Baselines

We thank the reviewer for the detailed and constructive feedback and for recognizing the efficiency and broad applicability of our inference-time framework. Our new comparisons demonstrate that real-video-driven temporal calibration provides measurable benefits beyond fixed temporal scaling and existing training-free motion-control methods. In particular, we added fixed-parameter ablations across three video generators and direct comparisons with FlowMo, substantially strengthening the empirical validation of our approach.

W1(a) — Fixed-α\alpha and fixed-γ\gamma baselines

We evaluated fixed-α\alpha and fixed-γ\gamma baselines on 100 held-out prompts (seed 0) across LTX-Video, Wan2.1, and CogVideoX (Appendix D.1, Tables 14–15). Each fixed baseline applies the model-wise mean estimated parameter to all videos, with the other control held at its identity value.

On LTX-Video, Ours substantially improves temporal progression over fixed-γ\gamma:

  • Prog-W1: 0.0438 → 0.0291 (33.6% reduction)
  • Flow-M Gap: 1.293 → 1.269

On CogVideoX, Ours also achieves lower Prog-W1 than both fixed baselines (0.0451 vs. 0.0460/0.0482).

These results demonstrate that sample-conditioned calibration can improve temporal progression beyond applying the same correction to every video.

The relative effectiveness varies across models and objectives. For example, fixed-γ\gamma achieves lower magnitude gaps on Wan2.1. This is consistent with our finding that global corrections can be effective when the required adjustments are predominantly one-sided, whereas sample-conditioned calibration provides additional flexibility.

Revision: Appendix D.1, Tables 14–15.

W1(b) — Comparison with FlowMo

Ours achieves stronger held-out motion-magnitude alignment than FlowMo on Wan2.1-T2V-1.3B, and the two methods provide complementary benefits when combined.

We evaluated Baseline, FlowMo, Ours, and FlowMo + Ours under matched prompts and seed 0 (Appendix F.4, Table 26).

On the held-out set:

  • M-Gap: 1.017 (FlowMo) → 0.810 (Ours), −20.4%
  • Flow-M Gap: 1.366 → 1.251, −8.4%
  • Dynamic Degree: 0.560 → 0.700

Combining FlowMo with Ours further reduces M-Gap to 0.801 and c1-W1 to 0.0227.

On ChronoMagic, Ours achieves lower Prog-W1 than FlowMo (0.0587 vs. 0.0697). On Ref4D, their combination achieves the lowest Flow-M Gap (1.550).

These results support the complementary objectives of the two approaches: FlowMo modifies motion behavior through guidance, while Ours estimates sample-dependent motion and pacing corrections toward real-video statistics, without additional denoising steps.

Spatiotemporal Skip Guidance and other motion-conditioning strategies remain relevant alternatives; our direct training-free comparison is with FlowMo.

Revision: Appendix F.4, Table 26.

AuthorsOpenReview ↗

Response to MYtW (2/5): When Does Local Retrieval Help?

W2 — Is local retrieval useful only for the 18-clip ChronoMagic subset?

Our additional analysis demonstrates that the benefit of local retrieval extends beyond the 18-clip subset: across all 50 ChronoMagic clips, it reduces aggregate target-prediction error by 14.0%, and its advantage translates into a 28.7% reduction in Flow-M Gap after video generation.

Evidence 1 — Lower estimation error across the full evaluation set

The 18 clips highlighted in Table 20 are not the entire evaluation set. They correspond specifically to videos requiring non-positive motion corrections.

On these clips, local retrieval reduces target-prediction error from 2.125 (global) to 1.402 (local), a 34.0% reduction.

On the remaining 32 clips requiring increased motion, global targeting achieves lower error (0.663 vs. 0.808).

Crucially, when considering all 50 clips, the sample-count-weighted target-prediction error decreases from 1.189 (global) to 1.022 (local), a 14.0% reduction.

Thus, the 18-clip analysis identifies where local retrieval provides its strongest benefit, rather than representing the only evidence supporting the approach.

Evidence 2 — Improvements after actual video generation

The advantage is not confined to target estimation. We additionally compared global and local calibration after generating the corresponding videos.

On ChronoMagic, local targeting reduces Flow-M Gap from 0.808 to 0.576 (28.7%), demonstrating that improved target estimation can translate into better generated motion alignment.

The benefit is especially relevant for heterogeneous correction requirements, where some videos require motion increases while others require smaller or opposite corrections.

On Ref4D, approximately 93% of clips require increased motion, and a global target performs better (Flow-M Gap: 1.473 vs. 1.634). This contrast helps explain the operating conditions: a global correction can capture a predominantly one-directional shift, but cannot explicitly accommodate conflicting sample-level correction requirements.

Implications for our contribution

These findings support the central motivation for sample-conditioned calibration: temporal corrections are not necessarily shared across all videos.

Our framework uses intermediate denoising representations to estimate temporal statistics and determine model- and sample-dependent corrections. Local retrieval is one component of this framework, providing flexibility beyond applying a single dataset-level target to every sample.

The ChronoMagic results establish measurable benefits of this flexibility both in target estimation and in generated-video motion alignment.

Revision: Sec. 5.2 and Appendix D.5, Tables 19–21.

AuthorsOpenReview ↗

Response to MYtW (3/5): Independent Motion Validation and Metric Discrepancies

Our additional experiments provide independent validation of motion calibration and identify why certain temporal and visual metrics respond differently. These include a flow-based calibration-target ablation, expanded VBench evaluations, and component-wise analyses of magnitude and pacing corrections.

W3 — Does pixel-based M primarily capture flickering or appearance artifacts?

Replacing the pixel-based target with camera-motion-compensated residual optical flow produces nearly identical calibration outcomes on LTX-Video.

We performed a Stage-I target ablation on 100 held-out videos, re-estimating both calibration statistics and control responses using residual optical flow instead of pixel differences.

The two targets exhibit nearly identical control sensitivities (0.533 vs. 0.537) and strongly correlated estimated corrections (r = 0.81).

More importantly, the resulting generations achieve:

  • Flow-M Gap: 1.087 (pixel target) vs. 1.086 (flow target)
  • Residual-Flow Gap: 0.957 vs. 0.957
  • Prog-W1: 0.0414 vs. 0.0418

These results support the use of pixel-based M as an efficient calibration statistic: within the tested operating range, it achieves comparable alignment to a more computationally demanding flow-based target without requiring optical-flow estimation during inference.

The near-identical results partly reflect the bounded control range and do not imply that pixel differences are intrinsically equivalent to physical motion.

We further evaluate VBench Temporal Flickering, Motion Smoothness, Subject Consistency, Background Consistency, and ViCLIP-based text alignment to independently assess temporal stability and semantic consistency.

Revision: Appendix F.1 and F.7, Tables 22 and 29.

W4 — Why do certain evaluation metrics worsen?

Our new component-wise analysis separates the effects of magnitude calibration from those of pacing calibration, identifying a specific source of semantic-progression degradation.

Ref4D progression. Although the full method increases Prog-W1 on LTX-Video (0.0210 → 0.0278) and Wan2.1 (0.0261 → 0.0341), the LTX component analysis reveals an important distinction:

  • Baseline: 0.0212
  • Stage I only: 0.0191
  • Stage II only: 0.0500

Stage I improves semantic progression, whereas Stage II accounts for substantial degradation in this setting. This is consistent with the difference between pixel-based temporal allocation (c1) and DINOv2-based semantic progression (Prog-W1), particularly under domain shift.

Wan2.1 visual quality. We additionally evaluated real-video Imaging Quality at matched resolutions. On SSv2, Panda-70M, and Ref4D, Ours produces scores closer to the observed real-video scores than the corresponding baselines. This contextualizes the metric decrease, although proximity to the real-video score does not by itself establish better perceptual quality.

CogVideoX. After correcting the inference configuration and regenerating both methods, Ref4D Motion improves from 67.46 to 67.81, and ChronoMagic Flow-M Gap decreases from 1.010 to 0.715. Held-out and Ref4D Flow-M Gap remain less favorable (0.225 → 0.293 and 0.466 → 0.504), indicating domain-dependent responses.

Together, these analyses distinguish the behavior of our two calibration stages and explain why improvements in targeted temporal statistics need not coincide with improvements in every independent metric.

Revision: Appendix C.5, F.1, F.7; updated Tables 1–2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.