acceptodds
Under review as a conference paper at ICLR 2027

ReMoVe: Refining a Video Model's Own Motion for Coherent Human Video Generation

12 views

Abstract

Video generation models have become increasingly capable of synthesizing realistic human motion, yet they often lose track of the very structure that makes the motion plausible. As generated human motion becomes more complex, videos can suffer from progressive body deformation and appearance drift across frames. Existing motion-controlled methods use explicit representations to stabilize generation, but typically rely on pre-defined motion sequences as references. This raises a fundamental question: if the base generator already produces a plausible coarse motion, why replace it with an external one? **ReMoVe** answers this question by treating the coarse output as a motion hypothesis rather than a final result. ReMoVe recovers and refines the underlying motion, then renders the refined motion as 3D mesh and transfers facial appearance onto it, thereby improving motion accuracy and identity consistency. We further derive depth from the same 3D scene and camera trajectory to impose scene-level spatial consistency beyond the mesh.By jointly conditioning video generation models on mesh and depth, ReMoVe synthesizes high-quality motion videos with controllable camera trajectories and enhanced spatio-temporal consistency. Extensive experiments demonstrate that ReMoVe improves human structural stability, appearance consistency, temporal coherence, and motion-following fidelity, while retaining the base generator's motion prior and offering accurate camera control.

Primary area:
generative models
Keywords:
controllable video generation, human video generation, motion control, camera control, identity consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NXMXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a bootstrapping-based motion optimization framework for generating videos of complex human movement. It begins by employing a human motion recovery model to extract SMPL-X motion from a rough video, then identifies reliable "anchor" segments based on physical metrics—such as foot sliding, ground penetration, root acceleration, and jerk—while using a text- and keyframe-conditioned motion model to complete or refine unreliable segments. Subsequently, the method renders 3D human mesh videos and scene depth videos under a unified camera trajectory. These are then encoded into a video latent space, concatenated with noisy video latents, and fed into a video generation model fine-tuned via LoRA.

Critical strengths
  1. Unlike methods that rely on external motion videos or predefined action sequences, ReMoVe directly optimizes the motion produced by the base generator itself. This reduces the need for users to provide precise motion references and helps preserve the inherent semantic connections between the motion and the scene.

  2. Rather than discarding the original video entirely, this approach retains reliable motion segments as anchors, regenerating only the unreliable intervals. This strategy of selective correction makes it easier to preserve the original intended motion compared to generating the motion from scratch.

  3. The paper reports results across various metrics, including VBench, VBench-2.0, FVD, FID-VID, SSIM, LPIPS, PSNR, and camera trajectory error, and presents experiments on mesh/depth-conditioned generation, motion-conditioned generation, and camera control. The appendix provides a detailed explanation of the motion reliability metrics, thresholds, and the process for selecting anchor points.

Critical weaknesses
  1. During training, the facial region within the mesh condition is copied directly from the corresponding frame of the target video, thereby incorporating the ground-truth identity and appearance information for the frame being generated. In contrast, during inference, only a static FLAME texture can be reconstructed from a single portrait image. The model may rely on frame-specific ground-truth facial information, which is unavailable at inference time, during training; this makes it difficult to attribute the observed identity preservation to the proposed method itself. The authors should consider constructing training conditions using single-image-based facial textures or, at the very least, provide experimental results that account for the elimination of this information leakage.

  2. The current experiments primarily compare mesh-only, depth-only, and a combination of both. However, they lack comparisons against several baselines or ablations, such as: raw GVHMR motion, motion generated directly via Kimodo, a version without the physical reliability ranking, or a version excluding long-sequence generation and cropping. Consequently, the reported performance gains cannot be definitively attributed to the selective retention and optimization of the base model’s own motion, as opposed to being primarily driven by the mesh/depth controls or the external motion generator.

  3. There is a lack of direct evaluation regarding motion preservation and accuracy. Metrics from VBench, such as Motion Smoothness, Dynamic Degree, and Human Anatomy, fail to assess whether the optimized motion remains faithful to the intent of the original video, nor do they confirm that reliable anchor points are effectively preserved. The paper should report metrics such as joint trajectory error, foot sliding, and motion completion rate; the inclusion of human evaluation should also be considered.

  4. The "MOTION-CONTROLLED VIDEO GENERATION" section in related work lacks discussion of the following work [1,2,3]. The authors could enhance the analysis of the limitations of these works to make the motivation of this paper clearer.

[1] Comovi: Co-generation of 3d human motions and realistic videos

[2] MoSA: Motion-coherent human video generation via structure-appearance decoupling

[3] Generating Human Motion Videos using a Cascaded Text-to-Video Framework

Questions for authors

See weakness.

6 comments

We thank the reviewer for the constructive comments and suggestions. We have conducted additional experiments to evaluate facial identity preservation, isolate the contribution of motion refinement, and directly assess motion preservation and physical plausibility. We have also expanded the discussion of related work in the revised manuscript. Our responses are provided below.

1. Facial Texture and Training–Inference Discrepancy

Thank you for raising this concern. During training, we use frame-specific facial pixels to construct the mesh conditions, primarily to avoid the substantial cost of generating individual head portraits for the large-scale training dataset. The facial regions are localized and cropped using ground-truth human motion, which introduces spatial alignment errors rather than providing perfectly aligned facial textures.

At inference time, ReMoVe does not access ground-truth facial information from future frames. Instead, it uses a static FLAME texture reconstructed from a single head portrait derived from the initial frame.

To evaluate the contribution of facial texturing, we conducted an additional inference-time ablation on 152 paired samples (304 generated videos) using the same trained video generator without retraining. The two variants differ only in their facial appearance conditions:

  • w/o Face Texture: The mesh face is rendered with a uniform skin color, without identity-specific facial texture.
  • Ours: The mesh face uses the FLAME texture reconstructed from a single reference portrait.

All other conditions, including human motion, depth, camera settings, and video generation parameters, remain unchanged. We use InsightFace (SCRFD for detection and ArcFace for identity embeddings) to evaluate facial identity consistency against the original video's first frame.

MethodFaceSim-First ↑Face Detection Rate ↑
w/o Face Texture0.193586.57%
Ours0.228089.03%

FaceSim-First is the macro-averaged cosine similarity between detected faces in the generated video and the original first-frame face, excluding frame 0 to avoid self-comparison bias. Face Detection Rate is the macro-averaged proportion of frames with successfully detected faces.

Our facial texturing improves identity similarity from 0.1935 to 0.2280 (+17.8% relative) and face detection rate from 86.57% to 89.03%. These results support the effectiveness of single-portrait-based facial texturing at inference time without accessing future ground-truth facial information.

We acknowledge that this inference-time ablation does not fully eliminate the training–inference discrepancy introduced by frame-specific facial conditions during training. We will clarify this distinction and its limitations in the revised manuscript.

2. Motion Refinement and Reliability-Adaptive Anchor Selection

Thank you for this suggestion. To isolate the contribution of motion refinement and reliability-adaptive anchor selection, we conducted additional controlled ablations by changing only the motion source while keeping mesh/depth conditioning, the video generator, and inference settings unchanged.

Specifically, we compare three variants:

  • Raw GVHMR: Directly uses motion reconstructed from the coarse video without refinement.
  • Fixed-5 + Kimodo: Generates motion conditioned on the first five GVHMR frames and the text prompt, without reliability-adaptive anchor selection.
  • Ours: Adaptively selects reliable motion anchors based on physical plausibility before refining the motion with Kimodo.

[Revised manuscript: Table 1, Lines 386–387, Page 8; Sec. 4.3, Lines 461–473, Page 9]

Our full method achieves a Human Anatomy score of 92.66, compared with 79.39 for Raw GVHMR and 66.19 for Fixed-5 + Kimodo, representing improvements of 13.27 and 26.47 percentage points, respectively. It also achieves the highest Subject Consistency, Motion Smoothness, Background Consistency, and Temporal Flickering scores among these variants.

These improvements cannot be attributed solely to mesh/depth conditioning, since the conditioning pipeline remains unchanged across all variants.

Our motivation is not simply to replace the original motion with a more plausible sequence. The coarse video may encode valuable motion cues reflecting the video generator's understanding of the intended action, scene context, and human-environment interactions. ReMoVe therefore selectively preserves reliable motion segments instead of relying solely on Kimodo to regenerate the motion.

Regarding long-sequence generation and cropping, we initially experimented with directly generating 81-frame motion sequences using Kimodo. However, we observed that when the specified duration was insufficient for complex actions, Kimodo frequently generated incomplete movements or failed to initiate the intended action.

We therefore adopted longer motion generation (up to 5 seconds), followed by trimming and standardization to the target video length. This strategy provides additional temporal flexibility for complex actions to unfold and was motivated by our preliminary observations of incomplete action execution under shorter generation horizons.

3. Direct Evaluation of Motion Preservation and Accuracy

We agree that VBench metrics alone cannot directly establish whether reliable motion anchors are preserved or whether the refined motion is physically plausible. We therefore conducted three additional evaluations on 152 motion sequences, covering anchor preservation, physical motion quality, and human-rated action completion.

(1) Anchor Preservation.

We directly compare refined motion with the original GVHMR motion at selected anchor frames using joint-position error (MPJPE) and the proportion of anchors preserved within a 1 cm threshold. All sequences follow the same 81-frame evaluation protocol.

Reliability TierAnchor Error ↓ (mm)Preservation@1cm ↑
High (47)0.0298100%
Medium (54)0.1902100%
Low (51)0.1748100%

Across 7,793 anchor frames, all joint-position errors are below 1 cm. These results confirm that Kimodo preserves the selected reliable motion anchors with high accuracy while allowing unreliable motion segments to be refined.

(2) Physical Motion Quality.

We further compare the original GVHMR motion and the refined Kimodo motion using the same physical metrics: foot sliding (FS), ground penetration (GP), root acceleration, and root jerk. All metrics are lower-is-better.

TierFS ↓ (m/s)GP ↓ (mm)Accel. ↓ (m/s²)Jerk ↓ (m/s³)
All (152)0.375 → 0.2530.476 → 0.2988.392 → 6.529323.58 → 215.30
High (47)0.105 → 0.1050.102 → 0.1024.785 → 4.785191.70 → 191.70
Medium (54)0.435 → 0.4340.490 → 0.4979.598 → 8.575353.29 → 297.38
Low (51)0.560 → 0.1990.805 → 0.26810.438 → 5.969413.66 → 150.15

Overall, motion refinement reduces foot sliding by 32.4%, ground penetration by 37.3%, root acceleration by 22.2%, and root jerk by 33.5%.

Importantly, the improvements are reliability-dependent. High-reliability motion remains nearly unchanged, consistent with preserving all frames as anchors. Medium-reliability motion exhibits reduced acceleration and jerk, although foot sliding remains nearly unchanged and ground penetration slightly increases. The largest improvements occur in low-reliability motion, where foot sliding and root jerk decrease by 64.5% and 63.7%, respectively.

These observations are consistent with our objective of preserving reliable motion while selectively refining unreliable segments.

For transparency, foot-contact frames are identified using a geometric height-based criterion rather than manually verified contact annotations.

(3) Human Evaluation of Action Completion.

We additionally conducted a human evaluation of generated videos using a three-point scoring scale (1–3), assessing how completely the generated human motion follows the text prompt.

MethodAction Completion Score ↑
Raw GVHMR2.2
Ours2.7

Our method achieves a higher average action-completion score (2.7 vs. 2.2). We observe that directly using motion recovered from the coarse video can produce satisfactory results for simple actions or loosely specified prompts. However, complex actions involving multiple ordered stages more frequently exhibit incomplete execution or abrupt motion transitions.

These results provide additional evidence that ReMoVe improves action completion while maintaining reliable motion anchors and reducing physical motion artifacts.

Reviewer_ZjPKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReMoVe enhances human structural stability, identity consistency, and motion control by recovering and refining the motion produced by the video generator itself. The pipeline first generates a coarse video, from which human motion is recovered using GVHMR; reliable anchors are then selected via physical-plausibility heuristics, and the motion is completed with Kimodo. Guided by the recovered motion, a human mesh with reference facial texture and a scene depth map are rendered under a shared camera trajectory. Their VAE latents are concatenated with the noisy video latent, and the base Wan2.2-TI2V-5B model is adapted through an expanded input layer and LoRA. The model is trained on 24k Motion-X++ samples and evaluated on 152 held-out examples, with additional ground-truth-motion, mesh/depth, and camera-control ablations provided in the appendix. ReMoVe outperforms the base Wan model across several structural and temporal consistency metrics as well as FVD.

Critical strengths
  1. Geometrically aligned human and scene conditioning provides a coherent and practically useful system design. Section 3 combines motion refinement, identity-aware mesh rendering, and scene depth into consistent control signals under a common camera trajectory. Section 3.5 reuses the video backbone through a relatively simple latent-concatenation and LoRA interface. Figure 7 illustrates the benefit of depth conditioning for background preservation. This offers a concrete approach to improving human and scene consistency jointly.
  2. The reported improvements over the base model span multiple quality dimensions. In Table 1, the full model improves Subject Consistency from 91.05 to 93.26, Human Anatomy from 90.88 to 92.66, and Temporal Flickering from 97.28 to 99.27, while reducing FVD-16 from 91.41 to 69.87 relative to Wan-TI2V-5B. Together with the qualitative examples, these results support the practical promise of the system. The camera-control results are mixed and do not constitute a primary argument for acceptance.
Critical weaknesses
  1. The central contribution of refining and preserving the generator’s own motion is not isolated experimentally, and the advance over the closest work remains unclear. Section 2.1 acknowledges ReVision as a closely related approach, but provides no experimental comparison; Uni3C also studies geometrically aligned human and camera control. It is therefore important to establish the additional value of reliability-based anchor selection, facial texturing, and the proposed conditioning combination. However, Section 4.3 only ablates mesh/depth conditioning, without comparing unrefined GVHMR motion, direct Kimodo generation, or fixed-anchor strategies. In particular, Appendix A.3 assigns 51/152 examples to the low-reliability group and retains only their first five frames. Subsequent motion could consequently be determined largely by Kimodo and the text prompt. This does not negate the practical benefit of avoiding user-supplied motion references, but it does not establish preservation and selective correction of the coarse video’s specific motion. Because this concerns the title and central contribution rather than an auxiliary detail, it is my main reservation. Controlled comparisons under the same generation/rendering protocol, together with motion-preservation and completion results by reliability group, would substantially strengthen attribution.

  2. The evaluation does not sufficiently distinguish correct motion refinement from quality gains associated with reduced dynamics, and key protocol details remain ambiguous. In Table 1, Dynamic Degree decreases from 66.45 for base Wan to 40.79 for the full model, and is 11.18 in the GT-motion setting. These values alone do not demonstrate excessive motion suppression. Nevertheless, without action-completion measurements, direct pose-following errors, or dedicated facial-identity metrics, improved smoothness and consistency are insufficient to establish the main claims. Moreover, it is unclear how the five-second, 30-fps motion synthesis and trimming in Appendix A.3 map to the default 81-frame output and its playback duration. Appendix A.5 also does not fully specify the sources of facial and background conditions in the GT-motion evaluation, particularly given that training copies facial pixels from each target frame whereas inference uses a static texture reconstructed from one portrait. These uncertainties affect the interpretation of motion, identity, and reconstruction results.

  3. The camera-trajectory evaluation has limited baseline coverage, and the reported results do not establish a clear advantage. The paper does not compare against relevant methods such as ReCamMaster, Uni3C, MoCam, and Vista4D, nor does it discuss their compatibility with the evaluation setting or explain their omission. Uni3C is particularly relevant because it is already cited and addresses joint human-motion and camera control. Its PCDController also supports camera-controlled generation from a single reference image without requiring a complete source video. The authors should therefore explicitly explain why it is not included in the camera-control evaluation. For joint control, human-motion conditions and their sources should additionally be matched.

    Moreover, on the translational trajectories in Table 2, ReMoVe performs worse than TrajectoryCrafter on ATE, RPE, TransErr, and CamMC. Its TransErr and CamMC are 0.2123 and 0.2132, compared with 0.1119 and 0.1133 for TrajectoryCrafter. Even accounting for TrajectoryCrafter’s access to the complete source video, ReMoVe also performs worse than RealCam-I2V, which operates in an image-to-video setting, on ATE, RPE, TransErr, and CamMC: 0.0584 vs. 0.0509, 0.0341 vs. 0.0279, 0.2123 vs. 0.1466, and 0.2132 vs. 0.1511, respectively (lower is better for all four metrics). Differences in available visual information therefore do not sufficiently explain the camera-control performance gap.

    Table 3 shows improvements over RealCam-I2V on rotational/orbit trajectories, but RealCam-I2V is the only baseline evaluated in that table. The authors explain that EPiC and TrajectoryCrafter lack an equivalent per-frame control interface under their evaluation setup. However, it remains unclear how this interface limitation prevents a fair comparison and whether evaluation could instead be conducted through trajectory conversion or on trajectories supported by all methods. Taken together, the current evidence does not sufficiently establish competitive camera control. Please further justify the baseline selection and interface limitations, and provide relevant comparisons under comparable input and control protocols, or qualify the claims accordingly.

Questions for authors
  1. Contribution validation: With the video generator and rendering conditions held fixed, how does adaptive anchoring compare with unrefined GVHMR motion, Kimodo conditioned directly on text/initial pose, and a fixed first-five-frame anchor strategy? Are there results by low/medium/high reliability group showing which specific motion information is retained and whether the action is completed? This evidence would directly affect my assessment of the central contribution.
  2. Inference-time camera assumption: Section 3.3 states that temporal-median background reconstruction assumes a largely static camera and excludes training clips with substantial camera motion. Section 3.4 applies the same procedure to coarse generated videos. How is this assumption enforced or checked at inference time? If a coarse video contains camera motion, is the background aligned before temporal aggregation, or is the video filtered or regenerated? Please clarify how such cases are handled in the reported camera-control experiments.
  3. Visibility requirements and recovery reliability: How does the pipeline handle partial-body framing, truncation, or prolonged occlusion in the coarse video? Appendix A.3 assesses anchor reliability using foot-contact and root-motion statistics, but does not specify whether joint visibility or image-based confidence is considered. Consequently, plausible motion inferred for unobserved joints might still be retained as an anchor. Does the evaluated setting require sufficient body and foot visibility, and are such cases filtered? Please clarify how anchor selection and body–scene scale alignment are handled when reliable motion or ground-contact information is unavailable.
  4. Camera-control baselines and interpretation: Why are relevant methods such as ReCamMaster, Uni3C, MoCam, and Vista4D omitted, particularly Uni3C/PCDController, which does not require a complete source video? How does the lack of an equivalent per-frame interface in EPiC and TrajectoryCrafter prevent fair comparison, and could trajectory conversion or commonly supported trajectories enable evaluation? Given the worse ATE, RPE, TransErr, and CamMC than both TrajectoryCrafter and RealCam-I2V in Table 2, and the comparison with only RealCam-I2V in Table 3, what scope of camera-control claims is supported?
  5. Appendix A.2 specifies 30 clips with five camera trajectories per clip, which should yield 150 test instances per method, whereas the paper reports 152. Please clarify the actual evaluation-set size and the number of instances contributing to each metric in Tables 2 and 3, including any additional samples, missing outputs, or excluded failures.
8 comments

We thank the reviewer for the detailed and constructive assessment. We agree that our central claim requires direct evidence of motion preservation and selective refinement, rather than improvements in video smoothness alone. We have added controlled ablations, reliability-stratified motion evaluations, facial-texture ablations, and additional camera-control comparisons. To avoid repetition, we address the overlapping weaknesses and questions together below.

1. Contribution Validation: Motion Refinement and Adaptive Anchoring

Addresses Critical Weakness 1 and Question 1.

Controlled motion-source ablations. We compare three variants using the same trained video generator, rendering pipeline, conditioning representations, and video-generation settings: Raw GVHMR, which directly uses recovered motion without refinement; Fixed-5 + Kimodo, which uses the first five recovered frames and the text prompt without reliability-adaptive selection; and ReMoVe, which adaptively retains motion anchors according to physical-plausibility criteria. The mesh and depth sequences are re-rendered from each variant's motion; their construction procedure, rather than their motion-dependent pixel values, is held fixed.

We chose Fixed-5 + Kimodo to preserve the initial pose and short motion context needed for consistency with the input image while allowing the subsequent motion to be generated primarily from the prompt. This is a minimally anchored baseline, not a separate text-only or single-initial-pose experiment.

[Revised manuscript: Table 1, Lines 386–387, Page 8; Sec. 4.3, Lines 461–473, Page 9]

MethodHA ↑FVD-16 ↓FID-VID ↓
Raw GVHMR79.3965.3234.89
Fixed-5 + Kimodo66.1972.5038.00
ReMoVe92.6669.8733.00

ReMoVe improves anatomical quality and FID-VID relative to both variants, although Raw GVHMR obtains a lower FVD-16. These results support the contribution of the motion-preparation strategy under a common downstream generation pipeline, but do not by themselves establish motion preservation. We therefore provide the following direct evaluations.

Reliability-stratified anchor preservation. We compare the recovered and refined 3D motion at selected anchor frames using MPJPE and the proportion of anchor frames with MPJPE below 1 cm.

Reliability TierRetained MotionAnchor MPJPE ↓ (mm)Preservation@1cm ↑
High (47)All frames0.0298100%
Medium (54)Selected reliable segments0.1902100%
Low (51)First five frames0.1748100%

All 7,793 anchor frames satisfy the 1 cm MPJPE criterion under the 81-frame protocol. This verifies preservation of the selected motion constraints, not the correctness of the underlying recovered poses.

For the low-reliability group, we agree with the reviewer that subsequent motion is largely determined by Kimodo and the prompt. This is intentional: retaining severely corrupted later states would propagate the original errors. Our claim is therefore reliability-adaptive preservation, not preservation of the entire original trajectory in every case. High-reliability sequences retain the full motion, medium-reliability sequences retain selected segments, and low-reliability sequences retain the initial motion context while prioritizing plausible completion.

Physical motion quality. We evaluate recovered and refined motion using the same body representation and physical metrics. Entries below show Raw GVHMR → Refined Motion, averaged equally across videos.

TierFoot Sliding ↓ (m/s)Ground Penetration ↓ (mm)Root Acceleration ↓ (m/s²)Root Jerk ↓ (m/s³)
All (152)0.3747 → 0.25330.476 → 0.2988.392 → 6.529323.579 → 215.302
High (47)0.1047 → 0.10470.102 → 0.1024.785 → 4.785191.702 → 191.702
Medium (54)0.4346 → 0.43400.490 → 0.4979.598 → 8.575353.286 → 297.379
Low (51)0.5602 → 0.19900.805 → 0.26810.438 → 5.969413.658 → 150.146

Overall, foot sliding, ground penetration, root acceleration, and root jerk decrease by 32.4%, 37.3%, 22.2%, and 33.5%, respectively. High-reliability motion remains nearly unchanged. Medium-reliability motion improves in acceleration and jerk, while sliding remains similar and penetration slightly increases. The largest improvements occur in the low-reliability group.

These are geometric physical-plausibility diagnostics, not errors against ground-truth motion. Contact is estimated using a foot-height criterion, rather than visually verified contact labels.

Human-rated action completion. We additionally evaluated generated videos on a three-point scale measuring completion of the prompted action. ReMoVe obtains 2.7, compared with 2.2 for videos generated using Raw GVHMR motion. Qualitatively, the latter can perform adequately for simple actions but more often exhibits incomplete execution or abrupt transitions for complex, ordered actions.

[Human-evaluation protocol: insert the number of evaluated video pairs, number of assessors, and definitions of scores 1–3.]

The completion score is currently an overall result, not a reliability-stratified completion rate. Together, the experiments provide complementary evidence of anchor retention, physical refinement, and improved action completion without treating these as interchangeable measurements.

2. Facial Identity, Dynamics, and Conditioning Contributions

Addresses Critical Weaknesses 1–2.

Facial-texture ablation. We evaluated 152 paired samples (304 generated videos) using the same trained generator. The variants differ only in whether the mesh face contains a single-portrait-based FLAME texture or uniform skin color. Motion, depth, camera settings, and video-generation parameters remain unchanged.

Using SCRFD for detection and ArcFace for identity embeddings, we compare generated faces with the first-frame face reference. We exclude frame 0 from identity similarity to avoid self-comparison and report video-level macro averages.

MethodFaceSim-First ↑Face Detection Rate ↑
w/o Face Texture0.193586.57%
ReMoVe0.228089.03%

Frame-specific target facial pixels are used during training for scalable condition construction. At inference, despite receiving only a static FLAME texture without future target-video pixels, ReMoVe successfully generalizes to this novel input, improving identity similarity from 0.1935 to 0.2280 compared to the texture-free baseline. This confirms that the model has robustly learned to extract and preserve identity cues from our explicitly rendered conditioning signal. We will explicitly acknowledge the training-inference discrepancy as a limitation in the revised manuscript.

Dynamics and pose following. We agree that lower Dynamic Degree is not sufficient evidence of correct refinement, just as higher Dynamic Degree does not establish successful action execution. The action-completion assessment above complements the physical metrics, which could otherwise favor reduced motion.

We also distinguish anchor preservation from final-video pose following: our anchor MPJPE evaluates the 3D motion refiner, not poses recovered from the final generated RGB video. SSIM, LPIPS, and PSNR evaluate reconstruction fidelity and are not substitutes for that missing pose-following measurement.

Mesh/depth complementarity. Mesh provides articulated structure and identity-aware appearance, while depth provides scene geometry and viewpoint constraints. During method development, our qualitative observations favored their combination for overall stability. Quantitatively, the combined model obtains the highest HA, SC, MS, and TF among the three conditioning variants, whereas mesh-only obtains the lowest FVD-16 and depth-only the highest IQ and BC. We therefore describe complementary benefits and trade-offs, rather than uniform superiority across all metrics.

Reviewer_Qaj9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses human motion video generation through a refinement pipeline consisting of: (1) human motion recovery from videos and a following motion refinement; (2) face identification and aligned mesh reconstruction; and (3) scene-depth video generation. Providing the additional generated conditions, the resulting video generation pipeline is improved for the quality on human motions.

Critical strengths

The conditioning design is reasonable. Rendered meshes provide explicit human motion and facial appearance, while depth provides spatial information. Intuitively, accurately constructed and aligned conditions should help improve generation quality.

Critical weaknesses
  1. The applicability of the method is limited.
The background reconstruction procedure assumes a largely static camera (Lines 240–241), limiting its applicability to scenarios with relatively static environments, as illustrated by the examples in the paper. However, substantial camera movement and changes in the environment are common in real-world motion videos, especially sports footage. This limitation deserves further discussion and evaluation.

  2. The experimental conclusions are not sufficiently convincing.
Table 1 provides quantitative comparisons with base generators but omits many related works such as Uni3C [1], which the paper explicitly cites. The omitted methods also support joint camera and human-motion control. The paper should include the comparison under matched evaluation settings with more related methods. It would be helpful to clearly outline the unique advantages over this method.

  3. The central motion-refinement contribution is not adequately isolated.
The ablation study is limited to mesh-only, depth-only, and combined conditioning, without independently evaluating motion refinement or adaptive anchor selection. The effects of the depth and mesh conditions also remain unclear: their combination improves some metrics but performs worse on others. These trade-offs require clearer analysis to establish the contribution of each component and the benefit of combining them.

[1] Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation

Questions for authors

None.

4 comments

We thank the reviewer for the constructive comments. We have conducted additional experiments and revised the manuscript to clarify the applicability, comparative performance, and individual contributions of ReMoVe. We address the three concerns separately.

1. Applicability under Camera Motion

We would like to clarify that the largely static-camera assumption in Sec. 3.3 primarily concerns training data construction, rather than a strict requirement at inference time.

During training, we exclude videos with substantial camera motion because, after removing the human subject using SAM3, reconstructing the missing background through temporal median aggregation becomes unreliable when corresponding background pixels are not spatially aligned across frames. Importantly, this design also benefits the learning process by decoupling complex human articulation from camera movement, allowing the model to focus on learning 3D human structure, motion priors, and human–scene interactions, rather than entangling these representations with complex camera-induced appearance changes.

At inference time, ReMoVe can accommodate camera motion in the coarse video, as demonstrated by the additional qualitative examples in Fig. 8 (Page 21 Lines 1101–1113). Furthermore, ReMoVe supports explicit user-specified camera control, allowing users to precisely define target camera trajectories by transforming the reconstructed scene geometry and human mesh.

2. Comparison with Uni3C

Our original baseline selection prioritized publicly available methods from 2025–2026 with approximately 5B-parameter backbones, to maintain comparable model capacity and reproducibility. We agree that Uni3C is relevant to joint human-motion and camera control and have therefore added it to our evaluation.

Specifically, we evaluate Uni3C and ReMoVe on the same 30 samples under five camera trajectories, yielding 150 test instances. Uni3C uses a Wan2.1-I2V-14B backbone with ground-truth human motion from Motion-X++, whereas ReMoVe uses a smaller Wan2.2-TI2V-5B backbone and refines motion recovered from its own coarse generation, without externally supplied motion sequences.

The additional results are summarized below (VBench scores, %):

MethodAQ↑BC↑DDHA↑IQ↑MS↑SC↑TF↑
Uni3C (14B, GT-motion)48.1393.8129.3391.9864.8299.4095.1798.15
ReMoVe (5B)48.7991.7070.6689.2768.1399.1593.0696.81

Despite using a substantially smaller backbone and no externally supplied ground-truth motion, ReMoVe achieves competitive overall video quality, with higher Aesthetic Quality and Image Quality. Uni3C performs better on Background Consistency, Human Anatomy, and several temporal consistency metrics. Its point-cloud-based scene guidance may contribute to stronger background consistency.

The lower Dynamic Degree of Uni3C (29.33 vs. 70.66) is largely associated with its use of ground-truth motion from Motion-X++, which exhibits relatively limited motion dynamics in our evaluation. Importantly, Uni3C relies on externally provided human-motion sequences and camera trajectories, whereas ReMoVe recovers, preserves, and refines the video generator's own motion without requiring external motion inputs.

3. Motion Refinement and Conditioning Ablations

We agree that isolating motion refinement is important for establishing the central contribution of ReMoVe. We have therefore conducted additional controlled ablations by changing only the motion source, while keeping mesh/depth rendering, the video generator, and generation settings unchanged.

We compare three variants:

  • Raw GVHMR: Directly uses motion recovered from the coarse video without refinement.
  • Fixed-5 + Kimodo: Uses only the first five GVHMR frames as fixed anchors, without reliability-adaptive selection.
  • ReMoVe: Selectively preserves reliable motion anchors and refines unreliable portions using Kimodo.

[Revised manuscript: Table 1, Lines 386–387, Page 8; Sec. 4.3, Lines 461–473, Page 9]

As reported in the revised Table 1, ReMoVe achieves a Human Anatomy score of 92.66, compared with 79.39 for Raw GVHMR and 66.19 for Fixed-5 + Kimodo, corresponding to improvements of 13.27 and 26.47 percentage points, respectively. It also achieves the highest Subject Consistency, Motion Smoothness, Background Consistency, and Temporal Flickering scores among these variants.

Rather than relying solely on Kimodo, ReMoVe selectively preserves reliable motion cues from the coarse video, which may reflect the video generator's understanding of the intended action and human-environment interactions. These results support the value of reliability-adaptive refinement, rather than directly retaining unrefined motion or regenerating motion with minimal anchoring.

Regarding mesh and depth conditioning, the two signals provide complementary but partially overlapping constraints. Mesh primarily constrains articulated human structure and identity-aware appearance, whereas depth provides scene-level geometry and viewpoint consistency. Although their combination does not achieve the best score on every metric, our qualitative observations during method development consistently showed that combining both conditions produces the most stable and coherent results, while removing either condition introduces noticeable artifacts in human structure or scene consistency. This observation is further supported by the highest Human Anatomy (92.66), Subject Consistency (93.26), Motion Smoothness (99.50), and Temporal Flickering (99.27) scores among the three conditioning variants.

Dear Reviewer, thank you again for your time and constructive feedback. As the discussion period is ending soon, we wanted to gently follow up to see if our rebuttal and revised manuscript (highlighted in blue) have addressed your concerns. We would greatly appreciate any further thoughts you may have before the deadline.

Comments on the paper

1 comment
Area_Chair_NcrEOpenReview ↗

Author-Reviewer Discussion Period is About to Finish

Dear reviewers,

Thank you for your reviews and constructive feedback.

The author-reviewer discussion period will be finishing soon. Please take a moment to review it and consider whether your concerns have been adequately addressed. It would be helpful if you could briefly indicate:

  • Whether the rebuttal addresses your main concerns;

  • Whether any significant issues remain unresolved;

  • Whether your overall recommendation has changed.

If appropriate, please also update your review and recommendation in the system.

Thank you for your time and valuable contributions.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.