acceptodds
Under review as a conference paper at ICLR 2027

HumANDiff: Articulated Noise Diffusion for Motion-Consistent Human Video Generation

Abstract

Despite tremendous recent progress in human video generation, generative video diffusion models still struggle to capture the dynamics and physics of human motions faithfully. In this paper, we propose a new framework for human video generation, HumANDiff, which enhances the human motion control with three key designs: 1) Articulated motion-consistent noise sampling that correlates the spatiotemporal distribution of latent noise and replaces the unstructured random Gaussian noise with 3D articulated noise sampled on the dense surface manifold of a statistical human body template. It inherits body topology priors for spatially and temporally consistent noise sampling. 2) Joint appearance-motion learning that enhances the standard training objective of video diffusion models by jointly predicting pixel appearances and corresponding physical motions from the articulated noises. It enables high-fidelity human video synthesis, e.g., capturing motion-dependent clothing wrinkles. 3) Geometric motion consistency learning that enforces physical motion consistency across frames via a novel geometric motion consistency loss defined in the articulated noise space. HumANDiff enables scalable controllable human video generation by fine-tuning video diffusion models with articulated noise sampling. Consequently, our method is agnostic to diffusion model design, and requires no modifications to the model architecture. During inference, HumANDiff enables image-to-video generation within a single framework, achieving intrinsic motion control without requiring additional motion modules. Extensive experiments demonstrate that our method achieves state-of-the-art performance in rendering motion-consistent, high-fidelity humans with diverse clothing styles.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Video Generation, Video Diffusion Models, Motion Control, Articulated Noise, Human Animation, Image-to-Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cv3dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a framework for motion-consistent human video generation that leverages articulated noise sampling to incorporate 3D human body structure into pretrained video diffusion models. It samples a shared noise texture on the UV manifold of a parametric human body model and maps it to individual frames according to the corresponding human motion. The framework further introduces two training strategies to improve motion fidelity and appearance consistency.

Critical strengths
  1. The paper represent motion-consistent noise on the UV manifold of a parametric 3D human body rather than relying on unstructured Gaussian noise or generic optical-flow-based warping. It provides a principled way to establish correspondences across frames and incorporate human body topology into the diffusion process.

  2. HumANDiff leverages the temporal modeling capabilities and visual priors of pretrained video diffusion models while introducing explicit 3D human structure through the noise representation. This design offers an alternative to training human-specific generative models from scratch or adding dedicated pose-control modules.

  3. The proposed JAML and GMCL objectives address different aspects of the problem. JAML encourages the denoised latent representation to retain information relevant to physical human motion, while GMCL explicitly enforces consistency in the canonical articulated noise space. The reported ablation results also show incremental improvements when these components are added.

Critical weaknesses
  1. The experimental evaluation is relatively limited in dataset diversity and task coverage. The main quantitative evaluation is conducted on UBC Fashion, which contains 600 monocular fashion videos, with 100 videos used for testing. While this benchmark is relevant to human animation, it does not sufficiently establish effectiveness across diverse human activities, body types, environments, and motion patterns. The out-of-domain experiments provide useful qualitative evidence, but broader quantitative evaluation on additional datasets and more challenging motions would be necessary to substantiate the generality of the proposed method.

  2. The paper motivates HumANDiff in terms of capturing human motion dynamics and physics, but the principal quantitative evaluation relies on image-level and video-level distributional metrics, such as LPIPS, FID, and FVD. These metrics do not directly measure adherence to the driving motion, 3D pose accuracy, joint trajectories, physical plausibility, or temporal consistency of individual body parts. Since the central contribution concerns articulated motion control, direct motion-specific metrics and a controlled evaluation of motion-following accuracy would provide evidence for the claimed improvements.

  3. GMCL maps predicted noise back to the canonical UV space through inverse warping and aggregation. However, the reliability of this operation may be affected by self-occlusion, incomplete surface visibility, inaccurate SMPL estimation, and ambiguities in the correspondence between the rendered surface and image-space features. Although the method applies masking in UV space, the paper does not fully establish how these factors affect the consistency loss or the resulting video quality. Further analysis of robustness to imperfect motion estimates and occluded body regions would be valuable.

  4. The "Human Animation Video Generation." section in related work lacks discussion of the following work [1,2,3]. The authors could enhance the analysis of the limitations of these works to make the motivation of this paper clearer.

[1] Comovi: Co-generation of 3d human motions and realistic videos

[2] MoSA: Motion-coherent human video generation via structure-appearance decoupling

[3] Generating Human Motion Videos using a Cascaded Text-to-Video Framework

Questions for authors

See weakness.

2 comments

Thank you for the review. We address the four concerns below.

1. Generalization beyond UBC Fashion

We agree that the UBC Fashion benchmark alone does not establish generality across diverse human activities. Appendices B.2 and B.3 also evaluate out-of-domain generalization, with quantitative results on novel DeepFashion subjects in Table 4 and qualitative results on DeepFashion and in-the-wild inputs in Figs. 4 and 5. On the novel test datasets, our method achieves a DINO-I score of 0.813 ± 0.041, compared with 0.693 ± 0.089 for UniAnimate. This supports appearance preservation outside the training distribution. We will make the scope of this evaluation clearer in the revised paper. Experiments on more varied activities, body types, and environments remain part of our future work.

2. Motion-specific evaluation

Appendix B.1, Table 2 reports a motion-centric evaluation on the UBC Fashion test set. MPJPE and PCK assess frame-wise pose accuracy, while trajectory and acceleration errors measure temporal agreement with the reference motion. Our method improves over GWTF on all the reported motion metrics:

MetricGWTFHumANDiff
MPJPE ↓0.03420.0274
[email protected] ↑0.8200.864
[email protected] ↑0.9320.948
Trajectory error ↓0.01360.0119
Acceleration error ↓0.01800.0133

These results complement the image- and video-quality metrics in the main text. We will give them greater prominence and clarify the evaluation protocol in the revision.

3. Visibility and imperfect SMPL estimates in GMCL

We agree that inaccurate SMPL estimates can affect the consistency constraint. Several aspects of our formulation address incomplete observations and reduce reliance on individual frames.

Supervision in noise space. GMCL compares the predicted and sampled noise in canonical UV space (Eq. 5). The correspondence is given by the rendered UV mapping used to construct the articulated noise, so the loss does not require matching appearance features across frames. This avoids an additional source of correspondence ambiguity, while retaining the dependence on the estimated geometry.

Visibility masking. We compute the UV-space loss only over observed surface regions. Unobserved texels are excluded from the constraint, so incomplete surface visibility does not introduce a reconstruction penalty for regions without valid observations.

Multi-frame fusion. The back-projected noise predictions are averaged in UV space before comparison with the shared noise texture. Each observed surface location can therefore receive supervision from multiple frames. This reduces dependence on a single frame and can lessen the influence of isolated frame-level errors.

Motion Noise Degradation. During training, we mix the warped noise with independent Gaussian noise using a degradation level γ sampled in [0, 1] (Sec. 3.3, Eq. 2). This design improves robustness to imperfect motion conditioning by exposing the model to different strengths of the geometric signal during training. Table 6 and Fig. 10 show how γ affects motion adherence and visual quality, motivating our choice of γ = 0.1 at inference.

The GMCL ablation in Table 5 and Fig. 9 also shows its benefit: adding GMCL to the model with JAML reduces LPIPS from 0.065 to 0.057 and L1 error from 0.0371 to 0.0364. These results support the use of GMCL in practice. We will make the visibility handling, multi-frame fusion, and noise degradation clearer in the revised paper.

4. Related work

We will add the three suggested works to the revised related-work section and discuss their motion representations and conditioning mechanisms:

  • CoMoVi jointly generates motion and RGB video through two coupled diffusion branches. Its motion branch serves the broader task of motion–video co-generation. HumANDiff focuses on video generation with a given driving motion and introduces the motion condition through shared surface noise.
  • MoSA uses sparse skeletal guidance, Human-Aware Dynamic Control modules, and a dense tracking loss. Its design addresses the limited expressiveness of sparse skeleton conditions through learned feature-level control. HumANDiff instead makes dense surface correspondences explicit in the noise representation, without adding a motion-control module to the inference backbone.
  • CAMEO combines text-to-motion generation, camera-view selection, and video generation conditioned on rendered SMPL sequences through a ControlNet-style branch. It addresses how to connect motion generation with video synthesis; our work explores how a supplied motion can guide synthesis through articulated noise while retaining the backbone's inference architecture.

In addition, to quantify pose accuracy beyond the appearance metrics, we additionally evaluate normalized mean 2D joint error (NME) on the test datasets:

VariantNME (% of GT person height) ↓
JAML comparison1.657
GMCL comparison1.220
Ours (Full)0.929

We use the same pretrained RTMPose-m estimator to extract 12 body joints from generated and corresponding GT images. NME is the mean Euclidean joint distance normalized by the GT person's bounding-box height, expressed as a percentage.

For the matched comparisons, Full reduces NME from 1.657% to 1.146% on the JAML test results and from 1.220% to 0.712% on the GMCL test results, corresponding to relative reductions of 30.8% and 41.7%. These measurements show improved pose agreement on the evaluated examples. They use image-estimated reference joints and provide frame-level evidence.

Reviewer_Qqt3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses human video generation through three key designs: (1) utilizing a human body mesh to warp consistent noise sampling, so that the projection of the noise remains consistent when body moves or viewpoint changes, (2) incorporating a motion prediction decoder to enhance the generation process, similar to what VideoJAM does, and (3) implementing a geometric consistency loss to ensure motion consistency across frames. The paper looks overall clear but I didn't check with the implementation details.

Critical strengths

I believe that the current submission does not meet the standards for acceptance. Although there are some novel or clever implementations compared to previous works, the overall quality and methodology do not impress me. It bases on established noise-noise-warping line of work while shows unsatisfying video results. The way of adding control to human-video generation does not seem particularly compelling.

Critical weaknesses
  1. The quality of human generation is significantly lower than that of current video systems that already achieved. Evidence shows that video models can now generate authentic human renderings using open-source models, such as Wan-animate [1] and SCAIL-2 [2], not to mention commercial models. The results presented in the supplemental video fall far short of expectations.

  2. The design is overly complex, comprising three or more components that may be difficult to adopt in large scale training and inference deployment. For example, SMPL human body motion is still unreliable to obtain, letting alone generation, and may contribute little to the complex clothing dynamics as the paper claimed. The comparison methods used are outdated: DreamPose is an image diffusion model, and Go-With-The-Flow has not been trained on human data, making the comparisons unfair. There are already many straightforward conditioning mechanisms that have produced impressive results in human generation, while this paper currently fails to compare or compete.

[1] https://humanaigc.github.io/wan-animate/
[2] https://github.com/zai-org/SCAIL-2

Questions for authors
  1. Human data are quite abundant in internet. Do you think by scaling data and computes, is it still important to use any noise warping technique during training and inference? Existing models have achieved strong motion and temporal consistency through straightforward conditioning methods (clay rendering, keypoint [1] or even video2video [2]), without utilizing any structured noise mechanisms. If the argument is that articulated noise enhances data efficiency, then it is a valid and intriguing claim; however, this should be stated explicitly and backed up with a study on data or compute scaling.

  2. Please also consider comparing to Wan-Animate [1] and SCAIL [2] as they also built upon Wan, while the results seem to be much impressive than the proposed method.

[1] https://humanaigc.github.io/wan-animate/
[2] https://github.com/zai-org/SCAIL-2

1 comment

Our contribution is pose-adapter-free human motion control in pretrained video diffusion models. HumANDiff samples shared noise on the canonical SMPL surface and maps it to each frame using the driving poses. Corresponding body locations thus share random variables across frames, making articulated correspondence part of the noise representation itself. This is the distinction from generic flow-based noise warping that we study in this paper.

The review places considerable weight on whether our videos match the visual quality of the latest animation systems. Visual quality matters, but it is not the sole criterion for evaluating a new conditioning method. The ICLR reviewer guidelines explicitly recognize that a paper can make a valuable contribution without achieving SOTA. We will make our performance claims specific to the benchmarks and settings evaluated.

We already compare with a Wan-based pose-adapter method in Figure 4 and Table 4. On out-of-domain subjects, the results against the Wan2.1-based UniAnimate baseline are:

MethodDINO-I ↑DINO-TC ↑
UniAnimate0.693 ± 0.0890.943 ± 0.022
HumANDiff0.813 ± 0.0410.940 ± 0.017

Our method preserves subject appearance better on this evaluation, while the adjacent-frame feature-consistency scores are close. Figure 4 also compares the methods under novel poses and viewpoints. This Wan-based comparison is missing from the review's discussion of our baselines. DreamPose is included as an established UBC Fashion baseline, and GWTF is a relevant comparison for noise-based control. We agree that their training settings need to be stated clearly, particularly because sharing a backbone does not equalize training data. We will also discuss Wan-Animate and SCAIL-2 in the revised related work.

Regarding complexity, JAML and GMCL add no inference modules. The motion decoder is used during training and discarded at inference; GMCL is a training loss. Motion control at inference comes from constructing the articulated noise, with no additional learned pose adapter. The framework is also model-agnostic: we have implemented and validated it on both CogVideoX-I2V-5B and Wan2.1-I2V-14B, as reported in Sec. 3.5, Appendix C, and Figure 12. SMPL processing and noise construction do have preprocessing costs, which we will distinguish from diffusion sampling.

SMPL provides body-surface correspondences, while the video model generates clothing appearance and motion-dependent details. We do not use SMPL as a cloth simulator. Driving motions can be taken from motion libraries such as AMASS, as described in Sec. 3.4. Fitting errors and clothing that moves substantially away from the body surface can reduce the accuracy of this guidance; we will clarify these limitations.

On scaling, our experiments show that articulated noise can provide motion control without a pose adapter in the evaluated fine-tuning setting. The success of other conditioning methods does not establish that this approach is redundant. A claim of superior data efficiency would require a controlled scaling study, and we have not established such a claim. We will revise the training-efficiency discussion accordingly.

Reviewer_KYtDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HumANDiff, which samples shared Gaussian noise on the SMPL UV manifold and warps it into each frame according to the driving motion, thereby replacing the standard Gaussian noise of a pretrained video diffusion model. Joint appearance-motion learning and geometric motion consistency learning are added during training, while no extra motion module is required at inference. Embedding motion correspondence directly in the diffusion noise is appealing, and the comparison with optical-flow-based warping is well motivated. However, several implementation details remain unclear, and the experiments do not yet convincingly support the claimed benefits of JAML, GMCL, and fine-grained motion controllability.

Critical strengths

Defining shared stochastic variables on the SMPL UV manifold provides explicit correspondence across articulated human poses and is better aligned with non-rigid human motion than generic optical-flow-based noise warping. This is a conceptually simple way to inject human-structure priors into a pretrained video diffusion model.

Critical weaknesses

Refer to the questions.

Questions for authors
  1. The specific 3D VAE is not identified. Section 3.1 represents both the input video and latent as having f frames, suggesting no temporal compression, whereas the appendix states that CogVideoX uses temporal downsampling by a factor of four. The authors should specify the encoder and reconcile the spatial and temporal dimensions in the two sections.
  2. Section 3.2 adds warped articulated noise to the human region and independent Gaussian noise to the background, but it does not explain how the two are combined. The authors should define the foreground mask, boundary and occlusion handling, and the final composition operation.
  3. The paper should clarify whether the motion decoder is pretrained or jointly trained from scratch, and precisely define the motion representation m, including its channels, value range, visibility convention, and resolution. The current ablation shows only marginal metric gains and does not include motion-centric metrics. Additional quantitative and qualitative results are needed to justify this module.
  4. GMCL is presented as a motion-consistency constraint, but Figure 9 mainly shows improvements in appearance, facial identity, and hand details. The authors should provide motion-specific evidence, such as pose trajectories, temporal correspondences, or acceleration errors.
  5. The current results mainly show visual quality and temporal consistency rather than controllability. The authors should disentangle identity from motion by applying the same driving motion to different identities, and visualize the corresponding SMPL UV motion sequence for direct comparison with the generated motion.
1 comment

Thank you for your constructive review and for recognizing the novelty and contribution of our approach. We clarify the implementation details and provide additional pose-based evidence below.

1. The 3D VAE and temporal compression

We use the pretrained 3D causal VAE of CogVideoX-I2V-5B, with both the encoder and decoder frozen during fine-tuning. Figure 2 correctly distinguishes the input video length F from the latent length f. The use of f for both quantities in Sec. 3.1 is a typographical error, which we will correct.

The VAE compresses the spatial dimensions by 8 and the temporal dimension by approximately 4, accounting for the first frame separately. Specifically, F = 4k + 1 input frames yield f = k + 1 latent frames. For example, the 49 × 480 × 720 × 3 input therefore corresponds to a 13 × 60 × 90 × 16 latent tensor, consistent with Appendix A.

2. Foreground–background noise composition

The UV noise and background Gaussian noise are sampled independently and combined using the human foreground mask. Let M_i denote the binary mask obtained from the visible SMPL rendering in frame i. A pixel belongs to the foreground when the rendered body provides a valid UV correspondence. The composed noise is

=M_i\odot\mathcal W_i(\epsilon_{\mathrm{UV}}) +(1-M_i)\odot\eta_i,$$ where η is independent of the shared UV noise texture. Thus, M_i selects articulated noise inside the rendered silhouette and independent Gaussian noise outside it. **Occlusion handling follows SMPL visibility:** only the visible surface contributes a UV correspondence in each frame. A newly visible surface queries the same shared UV texture, preserving its canonical noise assignment. Unobserved UV regions are masked out when computing GMCL. As described in Appendix A, the image-space noise is subsequently downsampled to the latent resolution using nearest-neighbor interpolation in space and time. We will include the composition formula and these mask conventions in the revised paper. **3–4. Motion decoder, motion representation, and motion-specific ablations** The motion decoder is **trained from scratch jointly with the DiT**. Its target m is a **two-channel UV coordinate map**, with U and V in **[0,1]** on the visible body surface. The map is spatially aligned with the video and has the **same spatial resolution**, e.g., 480 × 720 in our CogVideoX experiments. The foreground mask identifies valid UV pixels. The decoder provides auxiliary motion supervision during training and is **discarded at inference**. To quantify pose accuracy beyond the appearance metrics in Table 5, we additionally evaluate **normalized mean 2D joint error (NME)** on the test datasets: | Variant | NME (% of GT person height) ↓ | |---|---:| | JAML comparison | 1.657 | | GMCL comparison | 1.220 | | **Ours (Full)** | **0.929** | We use the same pretrained RTMPose-m estimator to extract **12 body joints** from generated and corresponding GT images. NME is the mean Euclidean joint distance normalized by the GT person's bounding-box height, expressed as a percentage. For the **matched comparisons**, Full reduces NME from **1.657% to 1.146% on the JAML test results** and from **1.220% to 0.712% on the GMCL test results**, corresponding to relative reductions of **30.8% and 41.7%**. These measurements show improved pose agreement on the evaluated examples. They use image-estimated reference joints and provide frame-level evidence. For **temporal motion fidelity**, Appendix B.1, Table 2 evaluates the complete model on UBC Fashion. Relative to GWTF, trajectory error decreases from **0.0136 to 0.0119**, and acceleration error decreases from **0.0180 to 0.0133**. The selected-frame ablations and this video-level evaluation address complementary aspects of motion fidelity. We will make these results and their evaluation protocols more prominent in the revision. **5. Controllability across identities** We will add **controlled visualizations using the same driving motion across different reference identities** in the revised version. These visualizations will show the reference images, the corresponding **SMPL UV motion sequences**, and the generated frames at matching time steps, enabling direct comparison between the motion input and generated poses. The driving joint-pose sequence and timing will be fixed across identities, while body shape will be retargeted to each reference person following Sec. 3.4. This will make the roles of the reference appearance and driving motion explicit and directly address the requested cross-identity comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.