acceptodds
Withdrawn from ICLR 2027

From Composites to Control: Geometry-Grounded Supervision for Visual Effects Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent visual effects (VFX) generation methods primarily rely on large video generative models to synthesize both effects and their interactions with scenes. While this paradigm enables flexible creation, their reliance on uncertain generative priors makes effect fidelity and geometric controllability difficult to ensure, especially under camera or subject motion. We introduce effect-faithful geometry-grounded VFX generation, a formulation that decouples reliable VFX supervision from generative uncertainty. Instead of using a video generator to synthesize VFX videos, we composite curated VFX assets onto real-world footage using recovered camera motion, scene geometry, and human motion. This preserves natural video dynamics and asset-defined effect appearance while providing explicit geometric supervision. Based on this principle, we build VFXShot, a dataset of 10,298 videos with effect semantics, camera trajectories, and spatial anchor annotations. We further propose VFXDirector, a unified video diffusion framework that conditions on effect semantics, camera trajectories, and spatial attachment. Experiments demonstrate that effect-faithful geometry-grounded supervision substantially improves controllable VFX generation, enabling VFXDirector to achieve improved video quality, camera consistency, and spatial attachment. These results suggest that reliable compositional supervision provides an effective alternative to purely generative data construction for learning controllable visual effects.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual effects generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YNh1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduce a camera location aware VFX generation technique given only single image as first frame. To support the task, the author introduces a new dataset, containing 10298 videos generated by inserting 19 special effect assets. The dataset is generated by lifting the scene into 3d and copy pasting the special effects into 3d locations. The authors then finetune a wan 2.2 with camera encoding and spatial anchor encoding through adaptive modulation on this dataset. It is shown that the technique with proposed adaptive modulation is most similar

Critical strengths

The paper did try to address an important problem, that existing VFX generation models often focus on videos with static camera poses, and generalizing to general footage with moving camera can be challenging and meaningful problem.

Critical weaknesses
  1. copy paste inserted VFX effects lacks visual realism. For example, interaction of visual effects with surroundings, such as light, water splashes, vapor from fire are all lacking. Even more concerning is if such effects can be generated by combined use of depth anything/SAM models and copy paste the effects into the video, why do we even bother training a video generative model to do that? It seems the motivation for the proposed task is not very clear.
  2. from line 245, the training data contains only 19 animated VFX assets. It remains unclear if the proposed method can generalize to wide variety of VFX that can be synthesized in real world. the author should provide qualitative examples and quantitative evaluations of out of domain visual effects that are very different from training data.
  3. The authors seem to claim joint adaptive fusion as contribution, but the technique seems to be very similar to adaptive norm of Tora: Trajectory-oriented Diffusion Transformer for Video Generation. The author could clarify the scope of contribution
  4. Given the main technical contribution is focused on camera control, the authors should compare with existing camera control techniques, such as alibaba-pai/Wan2.2-Fun-5B-Control-Camera checkpoint of the same Wan2.2 model, or Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation. Otherwise, the effectiveness of the camera control part of the story cannot be established.
  5. Evaluation is completely in-domain. Methods are trained and evaluated on VFXShot with no results on out-of-distribution generalization. For modern video generative model approaches, open domain generalization performance is critical.
Questions for authors

My biggest question is: for this simple copy paste type of VFX, why do we want to train a video diffusion model instead of simply adopting existing first frame based camera control method to generate a video, then simply copy pasting the VFX animation into the video with SAM/depth anything? Why is the proposed task/approach meaningful?

Reviewer_oLu8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VFXShot, a geometry-guided dataset for visual effects generation, and VFXDirector, a video diffusion model conditioned on camera trajectories and first-frame effect anchors. Experiments, ablations, and a user study demonstrate improvements in visual quality and controllability over the evaluated baselines.

Critical strengths
  1. The dataset provides useful supervision for controllable VFX generation. VFXShot combines real footage with estimated geometry and human keypoints to produce paired examples with camera trajectories and effect-attachment annotations.
  2. The method offers a practical control interface with encouraging empirical support. Users specify a camera trajectory and an initial effect anchor, while the model generates the subsequent attachment behavior.
Critical weaknesses
  1. VFXDirector appears to offer limited architectural novelty. Camera-pose conditioning and adaptive feature modulation have clear precedents in ReCamMaster and Tora. Joint Adaptive Fusion extends these ideas by combining camera and anchor features before predicting scale and shift parameters.
  2. The experiments do not isolate the benefit of geometry-grounded supervision.
  3. Existing comparisons provide limited evidence of an advance over specialized VFX methods. Most baselines are general video generators, alongside a fine-tuned Wan model. Different control interfaces make direct comparison difficult, as the authors acknowledge.
  4. Reported improvements in attachment adherence require stronger validation. AAR increases by 0.77 percentage points over Wan fine-tuning and 0.81 points over additive fusion, without uncertainty estimates or quantified agreement between the VLM evaluator and independent human judgments.
  5. Generalization is evaluated within a fixed vocabulary of effects. Training and evaluation share the same 19 effect assets, although source videos are appropriately separated. The results therefore demonstrate transfer to new scenes using familiar effects. Broader claims about unseen effect appearances or effect–attachment combinations remain insufficiently supported.
  6. Important details of the camera-control evaluation remain unspecified. Appendix B.2 does not identify the pose estimator, scale and coordinate alignment procedures, or handling of failed estimates. T
Questions for authors
  1. What does Joint Adaptive Fusion contribute beyond established conditioning designs? Could you compare it with separate camera/anchor modulation or cross-attention under the same backbone, dataset, and training budget?
  2. How much of the improvement is attributable to geometry-aware data construction? A matched comparison with simpler compositing, using the same footage and effect assets, would help isolate the value of geometric grounding from that of domain-specific synthetic supervision.
  3. How are the training budget and control inputs matched between VFXDirector and Wan2.2-FT?
  4. How are camera trajectories recovered and aligned when computing CamMC, RotErr, and TransErr?
  5. Under which conditions do camera and anchor controls interfere with each other? Removing camera conditioning improves AAR, while additive fusion yields lower rotation error than the full model. Could you break down these results by camera-motion magnitude and anchoring mode to clarify when joint conditioning provides a consistent advantage?
Reviewer_5gXNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes obtaining VFX training data by deterministically compositing curated RGBA effect assets onto real footage, instead of generating both the scene and the effect with a generative model. Depth Anything 3, SAM3, YOLO-Pose and Qwen3-VL provide geometry, masks, keypoints and captions, and effects are placed either at a lifted scene point or on a tracked body keypoint. The resulting VFXShot dataset contains 10,298 videos (542 sources × 19 assets). VFXDirector fine-tunes Wan2.2-TI2V-5B with a camera encoder and an anchor encoder fused by AdaLN-style modulation, and reports the best FVD, camera errors and VLM-judged Anchor Adherence Rate (AAR), as well as a 30-participant user study.

Critical strengths

The approach to constructing the data is sensible and well executed. Real footage preserves natural scene, camera and human dynamics, the asset fixes the effect's appearance, and the camera and anchor labels come from the same deterministic process that produced the target, so supervision and video match exactly. The split groups all 19 effect variants of a source in one partition, and the AAR prompt, user-study interface and failure cases are provided.

The dataset is clearly useful. Fine-tuning Wan2.2 on VFXShot improves FVD from 549.17 to 222.08, CamMC from 30.99 to 22.66 and AAR from 56.75 to 64.78, and users prefer the full model on all four criteria. The inference interface is also practical, requiring only a first frame, a 2D point, a relation string, an effect description and a trajectory.

Critical weaknesses

The contribution of the control modules is marginal. Against Wan2.2-FT, the full model is close on every metric: AAR 65.55 vs 64.78, CamMC 20.62 vs 22.66 and FVD 197.40 vs 222.08, while Wan2.2-FT has the higher Dynamic Degree. The full model's RotErr (3.00) is essentially that of zero-shot Wan2.2 (3.05), and additive fusion is close to the proposed modulation.

The range of effects is very small. VFXShot is built from only 19 pre-selected RGBA assets, one per effect type, so each effect has a single fixed appearance and animation. The split is by source video, so all 19 effects appear in both training and test, and the 10,298 videos contain 542 sources but only 19 distinct effects. No experiment uses an effect outside these 19, so the results only show that seen effects can be placed in new scenes, not that the model generates VFX in general. The authors acknowledge this in the limitations, but it substantially narrows the contribution.

The ablations also show unexplained reversals. Removing the camera branch raises AAR to 67.46, and the w/o-anchor variant reaches AAR 56.70, essentially equal to zero-shot Wan2.2 and eight points below Wan2.2-FT (64.78). The paper does not explain why this variant anchors so much worse than Wan2.2-FT.

The evaluation does not test the paper's central claim. Apart from Wan2.2-FT, the baselines are general video generators, and VFX-specific methods are excluded because of interface differences. The claim that composites are better supervision than generative VFX data is never tested by training the same model on a generative VFX dataset. In addition, AAR is a binary Qwen3-VL judgment with no reported agreement with human labels. The data are billboard sprites with no relighting, shadows or occlusion in the human-anchored mode.

Questions for authors
  1. What is Wan2.2-FT conditioned on, and were the w/o-camera and w/o-anchor variants trained with the same schedule as the full model?
  2. Can the same recipe be trained on a generative VFX dataset and evaluated on both data sources?
  3. How does the model behave on effects outside the 19 training assets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.