acceptodds
Withdrawn from ICLR 2027

Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models have shown remarkable success in video generation. However, whether such models are truly aware of the 3D structure underlying visual observations, rather than simply reproducing plausible 2D projections, remains an open question. In this work, we investigate this question through human motion control, a task that requires precise modelling of 3D human geometry, motion, camera viewpoint, and scene context. Unlike prior methods that rely on rendered 2D motion guidance videos, we propose a render-free framework that conditions video generation directly on compressed 3D human mesh tokens. This representation preserves full 3D geometric information while enabling a unified token-based generation pipeline that processes video tokens jointly with motion tokens in a DiT-based architecture. This design requires the model to reason jointly about appearance, 3D structure, and camera viewpoint during video generation. Experimental results demonstrate strong performance on human motion control benchmarks, while reducing artifacts induced by view-dependent 2D guidance and trajectory-pose mismatches during editing. These findings suggest that video diffusion models, when equipped with mesh tokenization, can better capture complex 3D human structures and their interactions with the surrounding environment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Diffusion Model, Human Motion Control, 3D Awareness, Mesh Tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_hZXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a 3D-aware video diffusion model for human motion-conditioned video generation, using 3D human motion as a condition without relying on rendered 2D motion guidance. The authors tokenize 3D human pose with a VQ-VAE, separately encode the global trajectory, and combine them into motion tokens. These motion tokens are then injected into a pre-trained DiT-based video diffusion model through frame-wise cross-attention. The authors evaluate the proposed method on human motion control benchmarks, showing comparable or improved video quality compared to existing rendering-conditioned motion control video generation methods.

Critical strengths
  1. The direct use of tokenized 3D human motion conditioning for video motion control is well-motivated. The proposed render-free 3D motion representation provides 3D motion information to the video generation model and avoids the view-dependent bias of rendering-based approaches.

  2. The proposed method shows competitive video quality across the evaluated benchmarks. The authors provide useful ablations on input trajectory, body pose conditioning, and motion representation, which help justify their proposed design choices.

Critical weaknesses

1. Limited comparison with closely related prior work

The proposed render-free 3D motion tokenization is not sufficiently distinguished from closely related prior work, particularly MTVCrafter (ICLR 2026). It seems that MTVCrafter similarly tokenizes 4D human motion and uses the motion tokens to condition a video DiT for controllable video generation. Since direct 3D motion token conditioning is presented as a core contribution, a clear discussion of technical differences from MTVCrafter, with an empirical comparison, would be important for assessing the novelty of the proposed method.

[1] Ding et al., "MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation," ICLR 2026.

2. Insufficient evidence for 3D awareness

The core motivation of this paper is that render-free, direct 3D motion conditioning enables the video diffusion model to jointly reason about 3D motion and camera viewpoint, yielding more robust generation under viewpoint changes. However, the reported metrics primarily evaluate video quality rather than cross-view or 3D consistency. An experiment that generates the same underlying 3D motion under different, preferably unseen, camera viewpoints and measures cross-view consistency would provide more direct evidence for the claimed 3D awareness and robustness to viewpoint changes.

3. Limited analysis of 3D motion estimator dependency The render-free input formulation removes the need to rasterize 3D motion into 2D guidance, but it instead relies on 3D human motion estimator and human motion VQ-VAE to obtain the conditioning signal. This introduces an additional source of error that is not sufficiently analyzed. The authors argue that the proposed method is robust to imperfect combinations of trajectory and body pose, but it remains unclear how much error in the estimated 3D motion the model can tolerate. A quantitative analysis under different levels of motion estimation error would help clarify the robustness of the proposed conditioning pipeline.

Questions for authors
  • Does the decomposition of global trajectory and body pose support independent control? For example, can the same body pose sequence be combined with different global trajectories, or vice versa, while maintaining plausible generation quality?
  • The paper uses a pre-trained VQ-VAE mesh tokenization. Could the authors provide an ablation on the codebook size to clarify whether the performance is sensitive to the codebook capacity?
  • The figure ordering should be consistent. Figure 4 is currently presented before Figures 2 and 3.
Reviewer_KrN7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the 3D awareness of video diffusion models through the task of human motion control and proposes MeshToken, a render-free framework that directly conditions video generation on 3D human mesh tokens instead of rendered 2D motion guidance. The method decomposes human motion into body pose and global trajectory, compresses canonical human meshes into compact tokens using a pretrained VQ-VAE, and integrates the resulting motion representations into a frozen Wan-2.1 DiT backbone through per-frame cross-attention. Experiments on Trajectory100 and RealisDance-Val demonstrate competitive video generation quality and motion controllability, with improvements in temporal consistency and robustness to motion editing. Ablation studies further examine the contributions of trajectory and pose representations, mesh tokenization, and temporal alignment. However, while the results support the effectiveness of the proposed motion conditioning framework, they provide limited direct evidence for the paper's broader claim of 3D awareness.

Critical strengths

None

Critical weaknesses

Insufficient evidence to support the central claim of 3D awareness. A central motivation of this paper is to investigate whether video diffusion models possess genuine 3D awareness rather than merely learning 2D visual patterns. However, the experiments primarily evaluate video generation quality (e.g., PSNR, SSIM, FVD, and VBench) and motion controllability (e.g., translation and rotation errors), which do not directly measure 3D understanding. In particular, improved generation quality when conditioned on 3D mesh tokens may simply reflect the benefits of richer geometric input, rather than demonstrating that the video diffusion model has developed meaningful 3D spatial understanding. Furthermore, the claimed robustness to viewpoint changes is largely supported by qualitative examples, without systematic quantitative evaluation. More direct evidence, such as cross-view geometric consistency or generalization to unseen camera viewpoints, is needed to substantiate the paper's central claim.

Questions for authors

Q1. Direct Evidence of 3D Awareness. The paper aims to investigate whether video diffusion models possess genuine 3D awareness. However, the current evaluation primarily focuses on video quality and motion controllability. Can the authors provide more direct evidence of 3D understanding, such as quantitative cross-view geometric consistency or generalization to unseen camera viewpoints? How can the observed improvements be distinguished from simply benefiting from richer 3D conditioning signals?

Q2. Limited Improvement in Motion Controllability. In Table 1, MeshToken achieves only a marginal improvement in translation error over RealisDance-DiT (1.706 to 1.697), while rotation error becomes slightly worse (0.167 to 0.173). Given that accurate 3D motion control is a central motivation, how do the authors explain these limited gains? Could they provide more direct 3D motion fidelity metrics, such as joint position error, foot sliding, or ground penetration, to substantiate the claimed advantages?

Reviewer_43FTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether video diffusion models are 3D-aware and uses human motion control as a test bed, proposing MeshToken, a render-free alternative to rendered 2D motion guidance. Each frame's canonical SMPL mesh is compressed by a pretrained mesh VQ-VAE (from VQ-HPS) into 54 pose tokens, combined with world- and camera-space trajectory embeddings, and injected into a frozen Wan-2.1 I2V (14B) DiT through new per-latent-frame cross-attention layers trained on an internal ~300K-clip dataset. On Trajectory100, RealisDance-Val, and a motion-editing set, the method is on par with or slightly better than the same-backbone rendering-based baseline RealisDance-DiT (e.g., FVD 695.6 vs. 758.1, translation error 1.697 vs. 1.706 m), from which the authors conclude that video diffusion models can capture richer 3D structure when given suitable 3D motion representations.

Critical strengths
  1. A simple, clean render-free conditioning interface. Per-frame mesh tokens plus world- and camera-space trajectory embeddings condition a large frozen video DiT directly. This removes the rendering step and makes trajectory and body pose separately editable. The design-choice ablations are reasonably complete: trajectory spaces, per-condition CFG, the mesh tokenizer vs. an MLP vs. raw SMPL parameters, per-frame vs. global cross-attention, and a frozen vs. unfrozen backbone.

  2. Evidence that render-free conditioning is viable at scale (720p, 97 frames). Against the same-backbone rendered baseline RealisDance-DiT, MeshToken matches trajectory control, improves FVD (695.6 vs. 758.1), and shows qualitative gains under viewpoint variation and occlusion (Figs. 2–3).

Critical weaknesses
  1. The 3D-awareness claim is never tested. No experiment probes 3D awareness; the evidence is limited to appearance/VBench metrics and global trajectory error. The condition is also not view-independent, since camera-space translation and rotation are inputs (Eq. 3). Freezing the backbone does not isolate its 3D understanding either: the added cross-attention layers hold roughly 4B trainable parameters by my estimate (40 × 4 × 5120²), enough to learn the 3D-to-2D mapping themselves.

  2. The advantage over rendered guidance is not established. No controlled ablation replaces the mesh tokens with rendered SMPL maps under the same data and backbone, so the external comparison confounds the representation with a ~300K-clip internal dataset. Against the one comparable baseline, RealisDance-DiT, trajectory error is on par (1.697 vs. 1.706 m) and the other gains are small, with no variance reported. Body-pose fidelity, which is what the mesh tokens encode, is never measured. The editing "robustness" in Table 3 may partly reflect weaker adherence to the edited motion: MeshToken has the lowest Dynamic Degree on Walking, and the text concedes that the baseline follows the edits more rigidly.

Questions for authors
  1. Can you report a controlled comparison with the same training data, backbone, and budget, where the mesh tokens are replaced by rendered SMPL guidance (e.g., depth/normal maps as in RealisDance-DiT)? This is the most direct test of the paper's central claim.

  2. Can you quantify body-pose fidelity and edit adherence for MeshToken and RealisDance-DiT? For example: MPJPE/PA-MPJPE of GVHMR-re-estimated body poses (or 2D keypoint error) against the input or edited motion, plus foot-skating and floating metrics.

  3. What evidence supports 3D awareness beyond controllability? For example: generate the same world-space motion under several camera trajectories, using the world-space trajectory only, and measure the 2D reprojection error of the generated human against the projected SMPL mesh.

Reviewer_f5xbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores 3D understanding of video generation models using a render-free framework for human motion controlled video generation. The method canonicalizes SMPL meshes, encodes them using a pretrained VQ-VAE for mesh, and then combines the resulting pose tokens with embeddings of translation and orientation in world and camera coordinates. The paper adds a newly trained cross-attention modules which injects those tokens into a frozen Wan-2.1 backbone. The method gets competitive results on Trajectory100 and RealisDance-Val.

Critical strengths
  1. Paper makes a methodological contribution to human motion-controlled video generation by conditioning on 3D mesh tokens through a render-free framework.
  2. The method shows effective motion conditioning and competitive results against existing methods. Ablation study also supports several architectural choices.
Critical weaknesses
  1. Evaluation does not directly substantiate the claims of improved 3D reasoning and robustness during motion editing. Improvements in visual quality and global motion control demonstrate effective conditioning, but stronger controlled experiments are needed to establish improved 3D reasoning. Similarly, the motion-editing evaluation reports improved smoothness and visual consistency without measuring adherence to the edited poses and trajectories.
  2. Novelty over closely related render-free motion-token methods, particularly MTVCraft, is not sufficiently established. The paper also does not demonstrate the specific benefit of mesh tokens over joint-based representations.
Questions for authors
  1. Controlled evaluation of generated videos with different viewpoints could support the claim of improved 3D reasoning. Could you provide those evaluations?
  2. I think pose and trajectory errors for the motion-editing evaluation could be reported to better establish adherence to the edited controls.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.