acceptodds
Under review as a conference paper at ICLR 2027

HumanFX: Human-Centric Visual Effects Generation

Abstract

Visual effects (VFX) combine exaggerated changes in appearance with nonphysical dynamics. Generating these effects for human subjects requires preserving facial identity, body structure, and motion continuity as the effect develops. Multi-person scenes further require accurate effect assignment and coordinated interactions. We present HumanFX, an image-to-video framework for human-centric visual effects generation. First, identity-preserving supervised fine-tuning (SFT) strengthens facial conditioning through a dedicated reference branch. Face crops sampled from video frames provide detailed identity information across varied views and expressions. Second, human-centric reinforcement learning (RL) uses anatomical and motion rewards to improve body structure and temporal consistency, with supervised regularization to retain expressive effect dynamics. Finally, because high-quality multi-person effect data are scarce, we introduce vision-language model (VLM)-guided test-time optimization (TTO). A differentiable visual question-answering (VQA) reward guides updates to the initial noise, refining effect assignment and interaction within a small computational budget. Extensive experiments demonstrate improvements in identity preservation, human motion, effect alignment, and user preference, alongside efficient multi-person control.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation, Visual Effects Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VtZBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HumanFX, an image-to-video framework for human-centric visual effects generation that aims to preserve identity, anatomical structure, and motion continuity while improving effect assignment and interactions in multi-person scenes. Built on Wan2.2, the framework comprises three main components: identity-preserving supervised fine-tuning with a dedicated facial reference branch; reinforcement learning with anatomical and motion rewards, combined with supervised regularization to retain expressive effect dynamics; and VLM-guided test-time optimization, which constructs a differentiable visual question-answering objective to optimize the initial noise for more accurate effect assignment and character reactions in multi-person scenes.

Critical strengths
  1. The chosen task is interesting and holds practical value. The overall system design is well-targeted, and as an applied project, it is quite comprehensive. Making the dataset and model open-source for the community would likely enhance its impact.
  2. The visualizations are impressive, and the model itself appears to be of high quality.
Critical weaknesses
  1. The proposed VLM-guided test-time optimization is highly similar to UniRect-CoT [1] in both its underlying idea and methodology. Both construct differentiable semantic objectives from intermediate visual predictions and backpropagate gradients to the generation latents to improve instruction following. As such, the proposed approach appears to be an application of existing energy-based editing techniques, with limited methodological novelty. 2.The paper reads more like a technical report, primarily combining existing techniques rather than introducing substantial methodological innovations. The identity injection mechanism follows the common in-context conditioning paradigm and closely resembles Stand-In [2]. Likewise, the human-centric reward design and optimization rely largely on existing detectors, motion metrics, and DiffusionNFT. [1] UniRect-CoT: Enhancing Generation in Unified Multimodal Models via Reflective Rectification with Inherent Understanding, https://arxiv.org/abs/2604.13540 [2] Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation, https://arxiv.org/abs/2508.07901v4
Questions for authors
  1. See Weakness
  2. (Optional; will not affect the score) I am curious about the task addressed in this paper: what kind of results could be achieved using Seedance2.5 or Wan 3.0?
Reviewer_7WyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents HumanFX, an image-to-video framework for human-centric visual effects generation. It has three parts: identity-preserving supervised fine-tuning with a face-reference branch and asymmetric attention (Sec. 3.2, Eq. 1); human-centric RL with anatomical and motion rewards plus supervised regularization (Sec. 3.3, Eq. 2 and 3); and VLM-guided test-time optimization of the initial noise for multi-person effect assignment and interaction (Sec. 3.4, Eq. 4 and 5, Alg. 1). Experiments use a curated 100K single-person VFX corpus and evaluate on a 140-pair single-person benchmark, a 60-pair multi-person benchmark, ablations, and a 20-person preference study. HumanFX is competitive with the baselines it is compared against and achieves the best FaceSim, Dyn., Align., and user preference in Table 1, and Fig. 4 shows higher multi-person joint success under their TTO setup.

Critical strengths
  1. The problem is real and practical, and the system performs well on the metrics that matter most here. In Table 1, HumanFX achieves the best FaceSim, Dyn., Align., and user preference, which supports the claim that it improves human preservation and instruction following for VFX generation.
  2. The ablation on supervised regularization is strong and consequential. In Table 2, dropping the SFT loss raises the human rewards but sharply hurts Dyn. and Align., which is evidence that anatomy and motion rewards on their own push the model toward static or weakened effects.
Critical weaknesses
  1. The main experiments do not separate the contribution of the proposed modules from supervised fine-tuning on the curated corpus. Table 2 shows that plain Wan2.2 SFT already reaches Dyn. 77.6 and Align. 75.4, close to HumanFX (77.6 and 75.8). The paper does not establish how much of the reported gain comes from the identity branch and human-centric RL and how much comes from SFT alone.
  2. The evidence behind the identity, RL, and TTO claims is weaker than the paper implies. Single-person inference uses a synthesized alternate-view face reference from Qwen-Image-Edit (Sec. 4.1, App. D.1), which makes it hard to attribute the FaceSim gain. For RL, the clearest improvements in Table 2 are on Ra and Rm, and both are derived from the same criteria used in training; no independent anatomy-focused evaluation is reported. For TTO, the efficiency comparison is uneven, because TTO scores lightweight previews while best-of-N evaluates full generated videos (App. C.2).
Questions for authors
  1. App. E.3 says each user-study trial uses one "preselected" video from each method, while Sec. 4.1 says three outputs are generated per input. How were the displayed videos chosen?

  2. How much of the FaceSim gain remains if single-person inference uses only a face crop from the original input image, instead of the Qwen-Image-Edit synthesized alternate-view reference (App. D.1)?

  3. In Fig. 4, can you compare against a cost-matched best-of-N that also ranks lightweight previews instead of full generated videos? That would separate the effect of gradient guidance from the effect of cheaper preview scoring.

  4. App. B says rewards are scored only where the subject stays visible in human form, with applicability annotations that separate intended nonhuman transformations from anatomical defects. During online RL with 16 rollouts per input (Sec. 3.3), how are these valid human-form intervals determined automatically for newly generated rollouts?

Reviewer_4MiXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies human-centric visual effects generation from an input image and a textual instruction. The goal is to generate the requested effects while preserving facial identity, plausible human anatomy, and temporal continuity. HumanFX consists of three components:

  1. Identity-preserving supervised fine-tuning using facial references, a dedicated LoRA adapter, and asymmetric reference attention.

  2. Human-centric reward optimization combining anatomical artifact detection and motion reconstruction, with supervised regularization to retain expressive effects and motion.

  3. Test-time optimization for multi-person scenes, using differentiable visual question-answering feedback to update the initial noise and improve effect assignment and interactions.

The authors curate HumanFX-100K, a single-person training corpus, and evaluate single-person generation, multi-person control, human preference, and inference cost. The reported results show improvements in facial similarity, effect alignment, and user preference over the evaluated baselines, alongside a favorable success–cost trade-off for multi-person refinement.

Critical strengths

S1. The problem is practically relevant, and the design addresses distinct failure modes. Human-centric effects require identity preservation, plausible anatomy, coherent motion, and instruction following. The paper distinguishes anatomical artifacts, temporal discontinuities, and multi-person assignment errors, then introduces corresponding mechanisms. The overall motivation is clear.

S2. The experiments provide meaningful positive evidence. In Table 1, HumanFX improves FaceSim from 0.711 to 0.737 and effect alignment from 72.2 to 75.8 compared with MiniMax-H3. Its share of selections in the five-way preference study is 49.0%, compared with 39.6% for MiniMax-H3. The paper also reports metrics on which it does not lead, including visual quality, motion quality, and smoothness.

S3. The paper recognizes and investigates an important limitation of reward optimization. Anatomical and smoothness rewards can favor reduced movement. The authors address this through supervised regularization. Table 2 provides concrete supporting evidence: removing the supervised term increases human reward scores while reducing dynamic degree and effect alignment.

S4. The multi-person inference comparison has several useful controls. TTO and best-of-N share the generator and VQA reward, while a separate model evaluates final success. Runtime accounting includes intermediate prediction, VLM assessment, backward passes, and final generation, making the efficiency comparison practically relevant.

Critical weaknesses

W1. The applicability of human rewards is insufficiently specified for online rollouts. Section 3.3 and Appendix B exclude intentional disappearance and nonhuman transformations from human rewards and refer to applicability annotations. However, RL evaluates newly generated videos, whose transformation timing may differ from the training videos. It remains unclear how eligible human-visible intervals are determined for these rollouts and how videos with no eligible intervals are scored. This affects reproducibility and leaves open whether premature disappearance or transformation could avoid penalties.

W2. Independent evidence for the central human-quality claim is limited. Direct anatomical and motion scores mainly appear in the ablation study and use the reward functions optimized during training. The main cross-method comparison relies on general quality, smoothness, and identity metrics, which do not directly establish fewer anatomical defects. Qualitative examples are helpful, but a limited independent annotation study or another independent evaluation would strengthen this central claim. Appendix E.2 also states that motion-fitting coverage is recorded, but the corresponding results are not reported.

W3. The multi-person evaluation does not fully establish the combined human-preservation objective. The identity branch is disabled for multi-person inference, and the reported evaluation focuses on joint assignment and behavior success. It therefore remains unclear whether TTO preserves each person’s identity, anatomy, and overall visual quality. Moreover, instance-level control and spatial assignment of multiple effects have been studied previously. The contribution should be distinguished through human preservation and interaction semantics. Evaluating Omni-Effects in its prompt-guided mode does not establish an assignment advantage over its explicit spatial-control mode.

W4. Evaluation and statistical reporting limit the strength of some conclusions. FaceSim averages over valid face detections, but detection coverage and the treatment of intentional facial transformations are insufficiently specified. The human study uses one preselected video from three outputs per input without explaining the selection rule. For multi-person control, 94.4% versus 93.3% corresponds to approximately two additional successful videos out of 180, without uncertainty estimates. The evidence therefore supports TTO’s cost advantage more strongly than a reliable success-rate advantage over best-of-six.

Questions for authors

Q1. Reward applicability: How are human-visible intervals and intentional nonhuman transformations identified in online-generated videos? Does this require manual annotation or reuse of source-video boundaries? How are premature disappearance, unintended deformation, motion-fitting failure, and outputs with no eligible intervals handled?

Q2. Independent human-quality validation: Could the authors provide an independent assessment of anatomical defects and motion discontinuities on a fixed subset of the existing test outputs, together with motion-fitting coverage? This would clarify whether the reward improvements in Table 2 correspond to perceptible improvements in human quality.

Q3. Multi-person contribution: Could the authors report assignment success and behavior success separately, alongside identity-preservation or quality measurements before and after TTO? How does the approach compare with Omni-Effects’ spatial-control mode, and does the current baseline use its targeted assignment capability?

Q4. Evaluation reliability: Please clarify the video preselection rule in the human study, face-detection coverage, and the treatment of intentional facial transformations in FaceSim. Confidence intervals or paired analyses accounting for repeated outputs per input would help contextualize the results. Please also report the inference hardware, absolute runtime, and average number of performed updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.