- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a framework for motion-consistent human video generation that leverages articulated noise sampling to incorporate 3D human body structure into pretrained video diffusion models. It samples a shared noise texture on the UV manifold of a parametric human body model and maps it to individual frames according to the corresponding human motion. The framework further introduces two training strategies to improve motion fidelity and appearance consistency.
Critical strengths
-
The paper represent motion-consistent noise on the UV manifold of a parametric 3D human body rather than relying on unstructured Gaussian noise or generic optical-flow-based warping. It provides a principled way to establish correspondences across frames and incorporate human body topology into the diffusion process.
-
HumANDiff leverages the temporal modeling capabilities and visual priors of pretrained video diffusion models while introducing explicit 3D human structure through the noise representation. This design offers an alternative to training human-specific generative models from scratch or adding dedicated pose-control modules.
-
The proposed JAML and GMCL objectives address different aspects of the problem. JAML encourages the denoised latent representation to retain information relevant to physical human motion, while GMCL explicitly enforces consistency in the canonical articulated noise space. The reported ablation results also show incremental improvements when these components are added.
Critical weaknesses
-
The experimental evaluation is relatively limited in dataset diversity and task coverage. The main quantitative evaluation is conducted on UBC Fashion, which contains 600 monocular fashion videos, with 100 videos used for testing. While this benchmark is relevant to human animation, it does not sufficiently establish effectiveness across diverse human activities, body types, environments, and motion patterns. The out-of-domain experiments provide useful qualitative evidence, but broader quantitative evaluation on additional datasets and more challenging motions would be necessary to substantiate the generality of the proposed method.
-
The paper motivates HumANDiff in terms of capturing human motion dynamics and physics, but the principal quantitative evaluation relies on image-level and video-level distributional metrics, such as LPIPS, FID, and FVD. These metrics do not directly measure adherence to the driving motion, 3D pose accuracy, joint trajectories, physical plausibility, or temporal consistency of individual body parts. Since the central contribution concerns articulated motion control, direct motion-specific metrics and a controlled evaluation of motion-following accuracy would provide evidence for the claimed improvements.
-
GMCL maps predicted noise back to the canonical UV space through inverse warping and aggregation. However, the reliability of this operation may be affected by self-occlusion, incomplete surface visibility, inaccurate SMPL estimation, and ambiguities in the correspondence between the rendered surface and image-space features. Although the method applies masking in UV space, the paper does not fully establish how these factors affect the consistency loss or the resulting video quality. Further analysis of robustness to imperfect motion estimates and occluded body regions would be valuable.
-
The "Human Animation Video Generation." section in related work lacks discussion of the following work [1,2,3]. The authors could enhance the analysis of the limitations of these works to make the motivation of this paper clearer.
[1] Comovi: Co-generation of 3d human motions and realistic videos
[2] MoSA: Motion-coherent human video generation via structure-appearance decoupling
[3] Generating Human Motion Videos using a Cascaded Text-to-Video Framework
Questions for authors
See weakness.
2 comments
In addition, to quantify pose accuracy beyond the appearance metrics, we additionally evaluate normalized mean 2D joint error (NME) on the test datasets:
| Variant | NME (% of GT person height) ↓ |
|---|---|
| JAML comparison | 1.657 |
| GMCL comparison | 1.220 |
| Ours (Full) | 0.929 |
We use the same pretrained RTMPose-m estimator to extract 12 body joints from generated and corresponding GT images. NME is the mean Euclidean joint distance normalized by the GT person's bounding-box height, expressed as a percentage.
For the matched comparisons, Full reduces NME from 1.657% to 1.146% on the JAML test results and from 1.220% to 0.712% on the GMCL test results, corresponding to relative reductions of 30.8% and 41.7%. These measurements show improved pose agreement on the evaluated examples. They use image-estimated reference joints and provide frame-level evidence.
Reply
Thank you for the review. We address the four concerns below.
1. Generalization beyond UBC Fashion
We agree that the UBC Fashion benchmark alone does not establish generality across diverse human activities. Appendices B.2 and B.3 also evaluate out-of-domain generalization, with quantitative results on novel DeepFashion subjects in Table 4 and qualitative results on DeepFashion and in-the-wild inputs in Figs. 4 and 5. On the novel test datasets, our method achieves a DINO-I score of 0.813 ± 0.041, compared with 0.693 ± 0.089 for UniAnimate. This supports appearance preservation outside the training distribution. We will make the scope of this evaluation clearer in the revised paper. Experiments on more varied activities, body types, and environments remain part of our future work.
2. Motion-specific evaluation
Appendix B.1, Table 2 reports a motion-centric evaluation on the UBC Fashion test set. MPJPE and PCK assess frame-wise pose accuracy, while trajectory and acceleration errors measure temporal agreement with the reference motion. Our method improves over GWTF on all the reported motion metrics:
These results complement the image- and video-quality metrics in the main text. We will give them greater prominence and clarify the evaluation protocol in the revision.
3. Visibility and imperfect SMPL estimates in GMCL
We agree that inaccurate SMPL estimates can affect the consistency constraint. Several aspects of our formulation address incomplete observations and reduce reliance on individual frames.
Supervision in noise space. GMCL compares the predicted and sampled noise in canonical UV space (Eq. 5). The correspondence is given by the rendered UV mapping used to construct the articulated noise, so the loss does not require matching appearance features across frames. This avoids an additional source of correspondence ambiguity, while retaining the dependence on the estimated geometry.
Visibility masking. We compute the UV-space loss only over observed surface regions. Unobserved texels are excluded from the constraint, so incomplete surface visibility does not introduce a reconstruction penalty for regions without valid observations.
Multi-frame fusion. The back-projected noise predictions are averaged in UV space before comparison with the shared noise texture. Each observed surface location can therefore receive supervision from multiple frames. This reduces dependence on a single frame and can lessen the influence of isolated frame-level errors.
Motion Noise Degradation. During training, we mix the warped noise with independent Gaussian noise using a degradation level γ sampled in [0, 1] (Sec. 3.3, Eq. 2). This design improves robustness to imperfect motion conditioning by exposing the model to different strengths of the geometric signal during training. Table 6 and Fig. 10 show how γ affects motion adherence and visual quality, motivating our choice of γ = 0.1 at inference.
The GMCL ablation in Table 5 and Fig. 9 also shows its benefit: adding GMCL to the model with JAML reduces LPIPS from 0.065 to 0.057 and L1 error from 0.0371 to 0.0364. These results support the use of GMCL in practice. We will make the visibility handling, multi-frame fusion, and noise degradation clearer in the revised paper.
4. Related work
We will add the three suggested works to the revised related-work section and discuss their motion representations and conditioning mechanisms: