- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a bootstrapping-based motion optimization framework for generating videos of complex human movement. It begins by employing a human motion recovery model to extract SMPL-X motion from a rough video, then identifies reliable "anchor" segments based on physical metrics—such as foot sliding, ground penetration, root acceleration, and jerk—while using a text- and keyframe-conditioned motion model to complete or refine unreliable segments. Subsequently, the method renders 3D human mesh videos and scene depth videos under a unified camera trajectory. These are then encoded into a video latent space, concatenated with noisy video latents, and fed into a video generation model fine-tuned via LoRA.
Critical strengths
-
Unlike methods that rely on external motion videos or predefined action sequences, ReMoVe directly optimizes the motion produced by the base generator itself. This reduces the need for users to provide precise motion references and helps preserve the inherent semantic connections between the motion and the scene.
-
Rather than discarding the original video entirely, this approach retains reliable motion segments as anchors, regenerating only the unreliable intervals. This strategy of selective correction makes it easier to preserve the original intended motion compared to generating the motion from scratch.
-
The paper reports results across various metrics, including VBench, VBench-2.0, FVD, FID-VID, SSIM, LPIPS, PSNR, and camera trajectory error, and presents experiments on mesh/depth-conditioned generation, motion-conditioned generation, and camera control. The appendix provides a detailed explanation of the motion reliability metrics, thresholds, and the process for selecting anchor points.
Critical weaknesses
-
During training, the facial region within the mesh condition is copied directly from the corresponding frame of the target video, thereby incorporating the ground-truth identity and appearance information for the frame being generated. In contrast, during inference, only a static FLAME texture can be reconstructed from a single portrait image. The model may rely on frame-specific ground-truth facial information, which is unavailable at inference time, during training; this makes it difficult to attribute the observed identity preservation to the proposed method itself. The authors should consider constructing training conditions using single-image-based facial textures or, at the very least, provide experimental results that account for the elimination of this information leakage.
-
The current experiments primarily compare mesh-only, depth-only, and a combination of both. However, they lack comparisons against several baselines or ablations, such as: raw GVHMR motion, motion generated directly via Kimodo, a version without the physical reliability ranking, or a version excluding long-sequence generation and cropping. Consequently, the reported performance gains cannot be definitively attributed to the selective retention and optimization of the base model’s own motion, as opposed to being primarily driven by the mesh/depth controls or the external motion generator.
-
There is a lack of direct evaluation regarding motion preservation and accuracy. Metrics from VBench, such as Motion Smoothness, Dynamic Degree, and Human Anatomy, fail to assess whether the optimized motion remains faithful to the intent of the original video, nor do they confirm that reliable anchor points are effectively preserved. The paper should report metrics such as joint trajectory error, foot sliding, and motion completion rate; the inclusion of human evaluation should also be considered.
-
The "MOTION-CONTROLLED VIDEO GENERATION" section in related work lacks discussion of the following work [1,2,3]. The authors could enhance the analysis of the limitations of these works to make the motivation of this paper clearer.
[1] Comovi: Co-generation of 3d human motions and realistic videos
[2] MoSA: Motion-coherent human video generation via structure-appearance decoupling
[3] Generating Human Motion Videos using a Cascaded Text-to-Video Framework
Questions for authors
See weakness.
6 comments
2. Motion Refinement and Reliability-Adaptive Anchor Selection
Thank you for this suggestion. To isolate the contribution of motion refinement and reliability-adaptive anchor selection, we conducted additional controlled ablations by changing only the motion source while keeping mesh/depth conditioning, the video generator, and inference settings unchanged.
Specifically, we compare three variants:
- Raw GVHMR: Directly uses motion reconstructed from the coarse video without refinement.
- Fixed-5 + Kimodo: Generates motion conditioned on the first five GVHMR frames and the text prompt, without reliability-adaptive anchor selection.
- Ours: Adaptively selects reliable motion anchors based on physical plausibility before refining the motion with Kimodo.
[Revised manuscript: Table 1, Lines 386–387, Page 8; Sec. 4.3, Lines 461–473, Page 9]
Our full method achieves a Human Anatomy score of 92.66, compared with 79.39 for Raw GVHMR and 66.19 for Fixed-5 + Kimodo, representing improvements of 13.27 and 26.47 percentage points, respectively. It also achieves the highest Subject Consistency, Motion Smoothness, Background Consistency, and Temporal Flickering scores among these variants.
These improvements cannot be attributed solely to mesh/depth conditioning, since the conditioning pipeline remains unchanged across all variants.
Our motivation is not simply to replace the original motion with a more plausible sequence. The coarse video may encode valuable motion cues reflecting the video generator's understanding of the intended action, scene context, and human-environment interactions. ReMoVe therefore selectively preserves reliable motion segments instead of relying solely on Kimodo to regenerate the motion.
Regarding long-sequence generation and cropping, we initially experimented with directly generating 81-frame motion sequences using Kimodo. However, we observed that when the specified duration was insufficient for complex actions, Kimodo frequently generated incomplete movements or failed to initiate the intended action.
We therefore adopted longer motion generation (up to 5 seconds), followed by trimming and standardization to the target video length. This strategy provides additional temporal flexibility for complex actions to unfold and was motivated by our preliminary observations of incomplete action execution under shorter generation horizons.
3. Direct Evaluation of Motion Preservation and Accuracy
We agree that VBench metrics alone cannot directly establish whether reliable motion anchors are preserved or whether the refined motion is physically plausible. We therefore conducted three additional evaluations on 152 motion sequences, covering anchor preservation, physical motion quality, and human-rated action completion.
(1) Anchor Preservation.
We directly compare refined motion with the original GVHMR motion at selected anchor frames using joint-position error (MPJPE) and the proportion of anchors preserved within a 1 cm threshold. All sequences follow the same 81-frame evaluation protocol.
| Reliability Tier | Anchor Error ↓ (mm) | Preservation@1cm ↑ |
|---|---|---|
| High (47) | 0.0298 | 100% |
| Medium (54) | 0.1902 | 100% |
| Low (51) | 0.1748 | 100% |
Across 7,793 anchor frames, all joint-position errors are below 1 cm. These results confirm that Kimodo preserves the selected reliable motion anchors with high accuracy while allowing unreliable motion segments to be refined.
(2) Physical Motion Quality.
We further compare the original GVHMR motion and the refined Kimodo motion using the same physical metrics: foot sliding (FS), ground penetration (GP), root acceleration, and root jerk. All metrics are lower-is-better.
| Tier | FS ↓ (m/s) | GP ↓ (mm) | Accel. ↓ (m/s²) | Jerk ↓ (m/s³) |
|---|---|---|---|---|
| All (152) | 0.375 → 0.253 | 0.476 → 0.298 | 8.392 → 6.529 | 323.58 → 215.30 |
| High (47) | 0.105 → 0.105 | 0.102 → 0.102 | 4.785 → 4.785 | 191.70 → 191.70 |
| Medium (54) | 0.435 → 0.434 | 0.490 → 0.497 | 9.598 → 8.575 | 353.29 → 297.38 |
| Low (51) | 0.560 → 0.199 | 0.805 → 0.268 | 10.438 → 5.969 | 413.66 → 150.15 |
Overall, motion refinement reduces foot sliding by 32.4%, ground penetration by 37.3%, root acceleration by 22.2%, and root jerk by 33.5%.
Importantly, the improvements are reliability-dependent. High-reliability motion remains nearly unchanged, consistent with preserving all frames as anchors. Medium-reliability motion exhibits reduced acceleration and jerk, although foot sliding remains nearly unchanged and ground penetration slightly increases. The largest improvements occur in low-reliability motion, where foot sliding and root jerk decrease by 64.5% and 63.7%, respectively.
These observations are consistent with our objective of preserving reliable motion while selectively refining unreliable segments.
For transparency, foot-contact frames are identified using a geometric height-based criterion rather than manually verified contact annotations.
(3) Human Evaluation of Action Completion.
We additionally conducted a human evaluation of generated videos using a three-point scoring scale (1–3), assessing how completely the generated human motion follows the text prompt.
| Method | Action Completion Score ↑ |
|---|---|
| Raw GVHMR | 2.2 |
| Ours | 2.7 |
Our method achieves a higher average action-completion score (2.7 vs. 2.2). We observe that directly using motion recovered from the coarse video can produce satisfactory results for simple actions or loosely specified prompts. However, complex actions involving multiple ordered stages more frequently exhibit incomplete execution or abrupt motion transitions.
These results provide additional evidence that ReMoVe improves action completion while maintaining reliable motion anchors and reducing physical motion artifacts.
We thank the reviewer for the constructive comments and suggestions. We have conducted additional experiments to evaluate facial identity preservation, isolate the contribution of motion refinement, and directly assess motion preservation and physical plausibility. We have also expanded the discussion of related work in the revised manuscript. Our responses are provided below.
1. Facial Texture and Training–Inference Discrepancy
Thank you for raising this concern. During training, we use frame-specific facial pixels to construct the mesh conditions, primarily to avoid the substantial cost of generating individual head portraits for the large-scale training dataset. The facial regions are localized and cropped using ground-truth human motion, which introduces spatial alignment errors rather than providing perfectly aligned facial textures.
At inference time, ReMoVe does not access ground-truth facial information from future frames. Instead, it uses a static FLAME texture reconstructed from a single head portrait derived from the initial frame.
To evaluate the contribution of facial texturing, we conducted an additional inference-time ablation on 152 paired samples (304 generated videos) using the same trained video generator without retraining. The two variants differ only in their facial appearance conditions:
All other conditions, including human motion, depth, camera settings, and video generation parameters, remain unchanged. We use InsightFace (SCRFD for detection and ArcFace for identity embeddings) to evaluate facial identity consistency against the original video's first frame.
FaceSim-First is the macro-averaged cosine similarity between detected faces in the generated video and the original first-frame face, excluding frame 0 to avoid self-comparison bias. Face Detection Rate is the macro-averaged proportion of frames with successfully detected faces.
Our facial texturing improves identity similarity from 0.1935 to 0.2280 (+17.8% relative) and face detection rate from 86.57% to 89.03%. These results support the effectiveness of single-portrait-based facial texturing at inference time without accessing future ground-truth facial information.
We acknowledge that this inference-time ablation does not fully eliminate the training–inference discrepancy introduced by frame-specific facial conditions during training. We will clarify this distinction and its limitations in the revised manuscript.