- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the degradation of motion dynamics during autoregressive video generation and proposes DynPrompt-Driven with two components. Temporal Prompt Recalibration (TPR) uses the existing visual KV cache to predict token-wise scaling weights for the text embeddings at each generation step. Motion-Guided Gradient Alignment (MGGA) computes gradient similarities between generated videos and a small library of high-motion reference videos using the fake score model, and uses these similarities to reweight per-sample DMD losses.
The method is implemented in a Wan2.1-1.3B self-forcing-style distillation pipeline. Experiments include short-video generation, 60-second adaptations of Rolling Forcing and Reward Forcing, component ablations, prompt-removal diagnostics, and a human study.
Critical strengths
-
Substantial motion improvements supported by controlled component ablations.
In the short-video ablation, dynamic degree increases from 66.38 to 73.61 with TPR, 71.28 with MGGA, and 83.88 with both. Appendix A states that the configuration without either module receives matched additional fine-tuning, addressing the possibility that the gains merely come from extra training. The 60-second component ablation similarly supports contributions from both modules. -
Evidence beyond a single short-video configuration.
The method improves 60-second dynamic degree from 27.00 to 44.35 on Rolling Forcing and from 62.71 to 73.57 on Reward Forcing. The prompt-removal intervention and human study provide complementary evidence. The paper reports individual quality dimensions and acknowledges modest trade-offs rather than showing only an aggregate score.
Critical weaknesses
-
The core method is not specified sufficiently for reliable reproduction.
TPR maps each text-to-cache attention row to a scalar using a learned linear projection, but the cache dimension can vary and the handling of this variation is not explained. The cache layers and heads used are also unspecified. For MGGA, Equation 6 describes sigmoid-based input masking, whereas Appendix C describes a splatted, min–max-normalized motion map without clearly relating the two. The projected-gradient dimensionality and aggregation across Fastfood blocks are also insufficiently specified. These details determine the implemented algorithm rather than being peripheral settings. -
The central interpretation extends beyond the strongest available evidence.
Dynamic degree measures increased motion but does not establish sustained execution of the actions requested by the prompt, particularly in the 60-second experiments. The paper acknowledges this limitation, but some conclusions still suggest improved prompt-motion alignment. In addition, a smaller absolute prompt-removal effect late in a rollout may partly reflect the lower baseline motion level, rather than uniquely identifying reduced prompt influence as the cause of degradation. MGGA’s additional analysis concerns prompt motion semantics, not the motion quality or usefulness of the generated samples. The results support motion enhancement, but the stronger mechanism and alignment interpretations remain less established.
Questions for authors
These questions primarily request clarification of existing implementation and results rather than new large-scale experiments.
-
TPR implementation:
Which cache layers and heads are used, and how is an attention row with a potentially varying number of cache tokens mapped to a scalar by a learned linear projection? Please specify the cache-size handling and the hardware/settings used to measure inference overhead. -
MGGA implementation:
How do the motion masks in Equation 6 and Appendix C relate to the actual implementation? Please specify the fake-model loss, parameter subset, gradient-projection dimension, aggregation across blocks, and whether the resulting sample weights are detached during the generator update. -
Evaluation protocol:
Please provide the prompt and seed counts for the 60-second comparisons and clarify the matched generation and adaptation settings. For the prompt-removal diagnostic, what are the sample count, baseline motion levels, and variability of the reported effects? -
Interpretation of the alignment signal:
Do existing training records show an association between MGGA alignment and measured motion in the generated videos, beyond the prompt-label association reported in Appendix C? If not, please scope the interpretation as an empirical weighting proxy and distinguish motion enhancement from prompt-specified action adherence.
10 comments
2 Unclear implementation of MGGA masks, gradients, and projections
Thank you for pointing out the insufficient clarity between Eq. (6) and the appendix. The specific definition of σ in Eq. (6), and its relationship to the motion-map construction in Appendix C, were not adequately explained. The actual implementation uses the motion map constructed from AllTracker trajectories in Appendix C: we first compute the L2 magnitude of displacement between adjacent time points, then splat at the trajectory coordinates in the destination frame, using weighted aggregation based on the product of visibility and confidence from the adjacent frames. The motion magnitude of the first frame is set to zero. The motion map is min–max normalized over all spatiotemporal positions within each video, with 10⁻⁶ added to the denominator, and then aligned to the latent grid through temporal linear interpolation and spatial bilinear interpolation. The actual implementation does not use sigmoid thresholding. We will make the main text and appendix consistent and clarify this construction.
The motion mask weights the prediction loss at each position, while the model input retains the complete video. Let z denote the clean latent, z_t the noisy latent, ε the sampled noise, and c the prompt condition. Under the current flow parameterization, the loss used to compute the alignment gradients is
Here, mean averages over all frames, channels, and spatial positions; it does not divide by the sum of the mask weights. This loss corresponds to L_fake used to extract gradients in the original Eq. (7), and is distinct from L_DMD used to update the generator. The reference and generated videos use the same fake model, share the attribution timestep and noise, and retain their respective prompts. The motion map is detached during this process. The original main text described multiplying the mask by the video input first, which did not accurately describe the actual computation. We will revise Eqs. (6)–(7) to the motion-weighted prediction loss above. The reference gradient is used only to compute alignment weights and does not directly update the fake model.
The gradient comparison uses the weights and bias of the fake model's final linear output head. During computation, the output-head features are retained and detached, and the gradients for this parameter subset are computed from the attribution loss's error signal with respect to the output, without retaining a full backward graph for the entire network. We will specify this subset in the appendix to avoid describing the computation as taking gradients with respect to all parameters individually.
The final projected vector has 512 dimensions. The current implementation partitions the gradient into 512-dimensional blocks, zero-padding the final block when needed. After applying a Fastfood transform to each block, the results of all blocks are summed into the same 512-dimensional vector, followed by frame-length normalization and L2 normalization. The 8 sets of random tables are reused cyclically according to the block index, and the projection random seed is fixed at 2026. Thus, “8 sets of tables” means neither dividing the gradient into only 8 blocks nor concatenating 8 projected vectors. We will add the definition of cross-block summation and clarify the meaning of T in the appendix accordingly.
Sample weights are computed from the cosine similarity between the projected gradients of generated and reference videos. In the current configuration, each GPU generates one training sample. After gathering the scores across GPUs, we first compute B×softmax(s/0.2), where B is the total number of samples being weighted. We then clip the weights at 0.5 and 2.0 and renormalize them so that their mean is approximately 1. The temperature of 0.2 here corresponds to an inverse temperature of 5; we will also correct the statement in the main text that the default inverse temperature is 1. The final weights after renormalization are not necessarily still within the clipping interval. The weights are detached before being multiplied by the corresponding per-sample DMD losses. Therefore, the generator does not backpropagate through the similarity to the reference gradient or the weight-computation path, and no second-order gradients along this path are required.
3 Insufficient details on the long-video protocol and prompt-removal statistics(1/2)
Thank you for reminding us to distinguish dynamics, text responsiveness, and action rationality. Below, we separately report the long-video evaluation, prompt-removal diagnostic, teacher-forcing comparison, and segment-wise motion rationality results, specifying which question each piece of evidence addresses.
For reproducibility, the generation configuration in the Rolling Forcing adaptation code uses BF16, 832×480 resolution, 16 FPS, and 3 latent frames per chunk. The denoising timestep list is [1000, 800, 600, 400, 200], with timestep shift=5. The rolling window contains 5 chunks, the clean-history KV cache has a capacity of 24 latent frames per layer, and TPR reads the most recent 3 frames from the end of the valid history. To generate a 60-second video, the code first generates 243 latent frames in complete chunks, decodes them into 969 frames, and then saves only the first 960 frames to ensure consistent output duration.
The training configuration for this adaptation uses AdamW, with a learning rate of 1.5×10⁻⁶ for the generator and TPR and 4×10⁻⁷ for the fake score model. Both use β=(0, 0.999), with a per-GPU batch size of 1. The fake score model is updated every iteration, while the generator and TPR are updated once every 5 iterations. The real score model, text encoder, VAE, and AllTracker remain frozen.
Within each backbone, the baseline and the model with our method start from the same pretrained checkpoint and receive additional training under the same budget: 600 outer iterations and a global batch size of 8. Apart from adding TPR and MGGA, both use the same training prompts, optimizers, and learning-rate settings. Here, 600 refers to outer training iterations, not 600 generator updates. The comparison uses a baseline with matched-budget continued training.
The 60-second comparison in Table 2 follows the long-video prompt settings of Reward Forcing, using the first 128 prompts from MovieGen. Rolling Forcing, Reward Forcing, and their respective configurations with our method use the same prompt list. Each prompt uses 5 inference seeds to generate 5 videos of 60 seconds, yielding 640 videos per method. Within the same backbone, the baseline and the model with our method use identical seeds and initial noise, with matched generation duration, resolution, frame rate, denoising steps, and cache settings. Evaluation follows the VBench-Long protocol in Appendix F: each video is divided into non-overlapping 2-second segments, and the six metrics are reported using the evaluator's aggregation procedure. Drift is computed as the sample standard deviation of the MUSIQ scores over the 30 segments of each video, then averaged across videos. Rolling Forcing's DD increases from 27.00 to 44.35, while MS changes from 98.94 to 98.76. These results show improved dynamics accompanied by a small change in smoothness; by themselves, they do not establish that all motion follows the actions specified in the prompt.
Regarding text influence, the prompt-removal experiment in Figure 1 removes the prompt at 5 seconds and 45 seconds, respectively, and compares the change in average motion intensity over the following 5 seconds. The early/late decreases for Rolling Forcing are 1.7/0.2, respectively, and become 1.9/0.8 with our method.
This diagnostic uses the first 128 MovieGen prompts, with 5 different inference seeds per prompt, yielding 640 paired samples for each method at each removal time. Each pair branches from the same history into continuations with the text retained or removed, sharing the subsequent sampling noise. The interventions at 5 seconds and 45 seconds each start from a history generated normally with the text retained. Motion intensity follows the definition used in Figure 1. The table below reports both the motion level with the text retained and the removal effect.
| Model | Removal time | Mean with text retained | Mean after text removal | Mean decrease (original paper value) |
|---|---|---|---|---|
| Rolling Forcing | 5 seconds | 4.30 | 2.60 | 1.70 |
| Rolling Forcing | 45 seconds | 1.60 | 1.40 | 0.20 |
| With our method | 5 seconds | 4.80 | 2.90 | 1.90 |
| With our method | 45 seconds | 2.90 | 2.10 | 0.80 |
As you pointed out, a lower initial motion level may limit the absolute decrease at later stages, and this explanation needs to be considered. We therefore interpret the existing prompt-removal results as indicating that a more pronounced text response is retained at later generation stages, and do not treat them alone as the sole proof of the cause of dynamics decay.
1 Insufficient details on the core TPR implementation and inference overhead
Thank you for requesting these implementation details. TPR adjusts text conditioning based on the visual content already generated. The key distinction is between the complete history cache and the fixed window actually read at each update. Taking the Rolling adaptation as an example, we use the V cache from Transformer block 9 (index 8), retain all 12 attention heads, and concatenate the 128-dimensional features from each head into 1,536 dimensions. Each time, we take the most recent 3 latent frames from the end of the valid history, corresponding to 4,680 visual tokens. TPR itself does not introduce an additional multi-head attention module.
The text passes through a two-layer projection of 4096→4096→4096, and the visual features through a two-layer projection of 1536→4096→4096. After RMSNorm, they are used to compute B×L×4680 text–visual correspondences. Each row associated with a text token always contains 4,680 elements, which are mapped to a scalar by a shared Linear(4680,1), followed by a gating function to reweight the text. Thus, although the total cache length can vary, the input dimension of the linear layer remains fixed, and the same parameters are shared across tokens and generation steps.
The features read here come from the original model's V cache and are then projected into the key features used for TPR matching; they are distinct from the original model's K cache. We will clarify this correspondence and the fixed-window extraction procedure before Eq. (4). Each update uses the original text embedding as its reference, and the visual cache is detached first. The first chunk uses the original text, and the update is skipped when the valid history is shorter than a complete chunk.
Regarding inference overhead, MGGA computes sample weights only during training. During inference, TPR uses the existing cache without adding generator denoising calls. The speed measurements in Table 1 of the paper use a single NVIDIA H100. We follow the official default single-video inference settings of Self Forcing, with a generation batch size of 1. The current inference code uses BF16, outputs at a resolution of 832×480, and decodes 21 latent frames into 81 frames, saved at 16 FPS for a duration of approximately 5 seconds. Each chunk contains 3 latent frames and uses 4 denoising steps, followed by an update to the clean-history cache after chunk generation.