- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Reflecting Optimization (SRO), a motion-enhancement method applicable to both text-to-video and image-to-video models. The paper argues that although temporal attention in video diffusion models helps preserve subject identity, appearance, and scene consistency through cross-frame information exchange, excessive cross-frame dependence can suppress temporal variation and lead to overly static videos. Before the actual sampling process, SRO optimizes the initial Gaussian noise to increase the proportion of within-frame attention in an early Transformer layer. The method is evaluated on several models, including Wan2.1-1.3B, Wan2.1-14B, CogVideoX-5B, and Wan2.1-I2V, using T2V-CompBench, Gemini-generated motion-intensive prompts, and VBench-I2V. The results show that SRO substantially improves motion-related metrics such as RAFT Flow, DEVIL, and DINO-Segm while introducing only modest additional inference time. However, some metrics related to text alignment, subject consistency, and temporal stability decline.
Critical strengths
-
The paper is clearly motivated, and the core observation, optimization objective, and constraint design are well aligned.
-
The proposed method demonstrates good generality and practical applicability. SRO does not require retraining the video generation model. Instead, it directly optimizes the initial noise before the actual sampling process, allowing the same framework to be applied to both text-to-video and image-to-video generation.
-
The improvements in motion dynamics are substantial, and the experiments include multiple forms of validation.
Critical weaknesses
- The existing motion metrics cannot adequately distinguish meaningful motion from unstable or distorted inter-frame variation, while several quality-related metrics consistently deteriorate. RAFT Flow, DINO-Segm, lower DINO similarity, and lower SSIM all reward greater inter-frame variation. However, such variation may result from correct subject motion, but it may also be caused by identity drift, structural deformation, camera shake, flickering, or physically implausible changes.
- The practical efficiency of the per-instance optimization procedure remains unclear. Images and scenes with different content may require different initial-noise perturbations. It is therefore unclear whether SRO must be executed separately for every input condition to identify an appropriate perturbation. Such per-instance optimization could affect the method’s practical runtime and scalability.
Questions for authors
-
Could the authors repeat the correlation analysis between the initial self-reflecting score and the motion dynamics of the final video across a larger and more diverse collection of prompts and motion categories?
-
Could the authors include compute-matched comparisons with best-of- initial-noise selection, random low-frequency perturbations, constrained noise optimization without the self-reflecting loss, and alternative initial-noise optimization objectives?
3 comments
Response to Reviewer F59N (2)
Q1. Could the authors repeat the correlation analysis between the initial self-reflecting score and the motion dynamics of the final video across a larger and more diverse collection of prompts and motion categories?
We find that the score-motion relationship in Figure 3 generalizes across various motion types. We repeated the analysis on a new set of prompts covering three distinct motion categories (foreground-only motion, fast background motion, and complex multi-object motion), and in all categories, seeds with a higher initial score consistently produce larger motion in the final video, including subject motion after compensating for camera movement.
To verify this, we generated videos with the standard sampler on Wan2.1-1.3B. Since the initial score at depends only on the initial noise, we first scored a large pool of random seeds and selected 100 seeds per prompt whose scores evenly span the observed range, without looking at the generated motion. We split the optical flow into subject and background regions using CLIPSeg masks, and additionally report camera-compensated subject motion, obtained by subtracting the global camera motion from the subject flow. We then compute the Spearman correlation between and the motion of the final video.
| Motion regime | ||||
|---|---|---|---|---|
| Foreground only | +0.56 | +0.41 | +0.64 | +0.54 |
| Fast background | +0.91 | +0.85 | +0.87 | +0.78 |
| Multi-object | +0.74 | +0.89 | +0.87 | +0.71 |
All correlations are positive and significant across categories, and remain strong after camera compensation, indicating that the score reflects genuine subject movement rather than camera motion. Moreover, in the foreground-only category, the score correlates more strongly with subject motion than with background motion, consistent with the motion specified by the prompt.
Q2. Could the authors include compute-matched comparisons with best-of-(N) initial-noise selection, random low-frequency perturbations, constrained noise optimization without the self-reflecting loss, and alternative initial-noise optimization objectives?
We conducted all the suggested comparisons, which further validate that SRO's gains stem from its self-reflecting objective, rather than from perturbing the noise or from additional computation.
On Wan2.1-1.3B, we compare SRO with the four alternatives suggested by the reviewer. All optimization-based variants use the same constraint operator (, ), iteration budget , learning rate, and sampler as SRO. Reversed loss minimizes instead of maximizing it. Feature-difference loss replaces with an attention-free motion proxy, maximizing the normalized difference between adjacent-frame features at the input of block . Best-of- selects the initial noise with the highest among candidates evaluated within the same runtime budget as SRO.
| Method | Flow | DEVIL | CLIP | VLM quality | Time | |
|---|---|---|---|---|---|---|
| Standard | 0.295 | 12.47 | 0.231 | 27.10 | 3.63 | — |
| Random low-frequency | 0.298 | 1.93 | 0.169 | 26.54 | 3.56 | 0 s |
| Reversed loss | 0.220 | 0.23 | 0.114 | 26.57 | 3.97 | 8.6 s |
| Feature-difference loss | 0.337 | 13.83 | 0.314 | 26.86 | 3.19 | 7.4 s |
| Best-of-N by S (N=70) | 0.378 | 29.35 | 0.410 | 26.84 | 3.13 | 8.0 s |
| SRO | 0.370 | 46.89 | 0.466 | 26.24 | 2.83 | 8.0 s |
Random low-frequency perturbations do not increase motion, and reversing the objective suppresses it, showing that the gains come from the direction of the objective rather than the perturbation itself. The attention-free feature-difference loss yields only marginal improvements.
Notably, Best-of- by also clearly improves motion, which confirms that is a meaningful motion indicator. Under the same budget, Best-of- better preserves quality, while SRO achieves substantially stronger motion at a quality cost consistent with the motion-quality trade-off (see W1 and our response to Reviewer sB6c, W2 & W3). We will include these results in the revision.
Thank you for your response.
Response to Reviewer F59N (1)
We thank Reviewer
F59Nfor the constructive comments. In response, we directly measured artifacts such as flicker, identity drift, and camera shake to verify that the increased motion is meaningful, clarified the cost of per-instance optimization, repeated the score-motion correlation analysis on new prompts covering diverse motion categories, and added compute-matched comparisons with alternative strategies, including Best-of-N selection and alternative optimization objectives. We respond to each point below.To address your concern, we directly measured each failure mode raised by the reviewer, and found that SRO's additional motion is not primarily explained by such artifacts: despite producing substantially stronger motion than MotionCFG, SRO exhibits fewer artifacts and remains close to the standard sampler.
On Wan2.1-1.3B, we measured flicker and deformation using flow-compensated warping error, identity drift using DINOv2 similarity to the first frame (separately for the CLIPSeg-masked subject and the entire frame), camera shake using temporal jitter in the optical-flow field, and structural integrity using a blind Qwen2.5-VL judge.
Compared to MotionCFG, SRO achieves lower warping error, better identity preservation, and higher structural integrity, while staying close to the standard sampler on all four measures. If the additional motion were mainly due to flicker or deformation, these artifact-specific metrics would degrade substantially, which is not the case.
Regarding the decrease in quality metrics, we show that it largely reflects an inherent motion-quality trade-off also present in standard generation and can be mitigated by the trust-region radius (see our response to Reviewer
sB6c, W2 & W3); moreover, human evaluators preferred SRO in visual quality over both baselines (Table 9 in our revised version).SRO is indeed run once per sample, as are other initial-noise optimization methods (e.g., InitNO [1], FreeInit [2], and CNO [3]). However, we argue that this does not limit its practicality, for two reasons: the optimization is lightweight, and it requires no per-input tuning.
First, SRO incurs only a modest overhead, since forward and backward passes are restricted to an early block (ℓ∗=2) without backpropagating through the denoising trajectory. As shown in Table 6 in our revised version, the added time is small relative to the full sampling time on all backbones, and is lower than that of FlowMo and FreeInit, which require several times the baseline runtime, as well as MotionCFG with its larger LLM. Second, we use the same hyperparameters (ε and K) for all inputs without any prompt or image-specific tuning, and this single setting consistently improves motion across all evaluated backbones and motion categories.
References
[1] InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization, CVPR 2024.
[2] FreeInit: Bridging Initialization Gap in Video Diffusion Models, ECCV 2024.
[3] Diverse Text-to-Image Generation via Contrastive Noise Optimization, ICLR 2026.