- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduce a camera location aware VFX generation technique given only single image as first frame. To support the task, the author introduces a new dataset, containing 10298 videos generated by inserting 19 special effect assets. The dataset is generated by lifting the scene into 3d and copy pasting the special effects into 3d locations. The authors then finetune a wan 2.2 with camera encoding and spatial anchor encoding through adaptive modulation on this dataset. It is shown that the technique with proposed adaptive modulation is most similar
Critical strengths
The paper did try to address an important problem, that existing VFX generation models often focus on videos with static camera poses, and generalizing to general footage with moving camera can be challenging and meaningful problem.
Critical weaknesses
- copy paste inserted VFX effects lacks visual realism. For example, interaction of visual effects with surroundings, such as light, water splashes, vapor from fire are all lacking. Even more concerning is if such effects can be generated by combined use of depth anything/SAM models and copy paste the effects into the video, why do we even bother training a video generative model to do that? It seems the motivation for the proposed task is not very clear.
- from line 245, the training data contains only 19 animated VFX assets. It remains unclear if the proposed method can generalize to wide variety of VFX that can be synthesized in real world. the author should provide qualitative examples and quantitative evaluations of out of domain visual effects that are very different from training data.
- The authors seem to claim joint adaptive fusion as contribution, but the technique seems to be very similar to adaptive norm of Tora: Trajectory-oriented Diffusion Transformer for Video Generation. The author could clarify the scope of contribution
- Given the main technical contribution is focused on camera control, the authors should compare with existing camera control techniques, such as alibaba-pai/Wan2.2-Fun-5B-Control-Camera checkpoint of the same Wan2.2 model, or Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation. Otherwise, the effectiveness of the camera control part of the story cannot be established.
- Evaluation is completely in-domain. Methods are trained and evaluated on VFXShot with no results on out-of-distribution generalization. For modern video generative model approaches, open domain generalization performance is critical.
Questions for authors
My biggest question is: for this simple copy paste type of VFX, why do we want to train a video diffusion model instead of simply adopting existing first frame based camera control method to generate a video, then simply copy pasting the VFX animation into the video with SAM/depth anything? Why is the proposed task/approach meaningful?