acceptodds
Withdrawn from ICLR 2027

Image Diffusion Models Are Good 3D Human Motion Generators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogeneous skeletal layouts while preserving motion-specific kinematic structure. To this end, we organize root-centered joint trajectories into topology-informed joint–time images and model global root dynamics through a complementary latent stream. A motion autoencoder conditions local-motion decoding on the root latent, while a text-conditioned Scalable Interpolant Transformer (SiT) jointly generates both latent streams, with supervision on decoded 3D joint positions further constraining generation in motion space. In addition to dataset-specific models, we train a single motion autoencoder on HumanML3D and KIT-ML, followed by a single text-conditioned SiT shared across both datasets, preserving their respective skeletal definitions without skeleton retargeting to a common joint layout. Experiments on HumanML3D and KIT-ML show that our dataset-specific models achieve state-of-the-art performance, while shared training yields further gains on both benchmarks, demonstrating the benefits of shared modeling across heterogeneous skeletal layouts.

Primary area:
generative models
Keywords:
Text-to-motion generation, Diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2BJROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an image-compatible framework for text-conditioned 3D human motion generation across datasets with different skeletal layouts. It separates motion into global root dynamics and root-centered joint trajectories, organizing the latter into fixed-resolution joint–time images based on kinematic topology. A two-stream autoencoder encodes both motion components, with root-conditioned image decoding. A text-conditioned Scalable Interpolant Transformer (SiT) jointly generates their latents through flow matching, with additional supervision on decoded 3D joint positions. The framework also enables shared training on HumanML3D and KIT-ML using dataset conditioning and dataset-specific motion recovery, without skeleton retargeting or joint padding. Experiments compare dataset-specific and shared models against existing methods, alongside component ablations.

Critical strengths
  1. The paper presents a coherent framework that incorporates motion-specific structures into image-based generation through topology-informed joint ordering, root-conditioned decoding, and 3D position supervision. Ablation studies support several key components, demonstrating the effectiveness of adapting image-based architectures to structured human motion.
  2. The method achieves strong results on HumanML3D and KIT-ML, with the shared model obtaining the best reported performance across all six metrics on both datasets. Its improvements over dataset-specific models, without requiring skeleton retargeting, demonstrate the practical value of shared modeling across skeletal layouts.
Critical weaknesses
  1. The contribution of the image-compatible representation is not sufficiently isolated. The paper lacks controlled comparisons with alternative motion representations and an ablation on topology-informed joint ordering. Therefore, it remains unclear how much of the performance improvement comes from the proposed representation rather than other design choices, weakening the evidence for its central methodological contribution.
  2. The methodological novelty relative to prior work is not fully established. Motion Patches (CVPR 2024) explored image-compatible motion representations, while SALAD (CVPR 2025) combined skeleton-temporal latent modeling with diffusion-based motion generation. Although this paper introduces distinct components and shared modeling, its advances over these related approaches are not clearly distinguished, making its specific novelty difficult to assess.
Questions for authors
  1. Can the authors provide evidence isolating the benefits of the image-compatible representation from other architectural and training choices? In particular, how does it compare with non-image motion representations under controlled settings, and how sensitive is performance to topology-informed joint ordering?
  2. How does the proposed framework differ from Motion Patches (CVPR 2024) and SALAD (CVPR 2025) in terms of its core methodological contributions? Clarifying these distinctions would help assess the novelty and significance of the proposed approach.
Reviewer_BJfoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reshapes local joint trajectories into joint-time images and combines them with root-motion latents and a SiT generator for text-to-motion generation. It evaluates dataset-specific and shared models on HumanML3D and KIT-ML, reporting strong benchmark results.

Critical strengths
  1. The framework is reasonably complete, jointly modeling local pose and root motion through separate latent streams.

  2. Sharing one model across HumanML3D and KIT-ML is potentially useful in practice.

Critical weaknesses
  1. The cross-skeleton motivation is not fully convincing. The two datasets can be mapped using standard joint mapping or retargeting, while the proposed model still relies on dataset-specific ordering, normalization, recovery, and dataset conditioning.

  2. The unique benefit of the image representation is not demonstrated. Skeleton-image representations have been explored previously, and the paper lacks parameter-matched motion-native baselines and key ablations.

Questions for authors
  1. What happens if the image latent is replaced with a motion-native latent while keeping SiT, parameter count, training budget, and losses fixed?

  2. How much improvement comes independently from ImageNet initialization and decoded joint-position supervision?

  3. Why is there no comparison with, or discussion of the task differences from, video-assisted methods such as VIMoGen?

Reviewer_e2GUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates using image diffusion model component, such as pretrained image autoencoder for motion tokenization and image diffusion model architecture, i.e. SiT, for the task of text-to-motion generation. It focus on representation design, converting 3D human motion into image-like dimension, so most of the module from image diffusion can be reused without complicated modifications. It converts motion into root trajectory and joint-time image from root-centric joint trajectories. Quantitative experiments shows that the proposed method achieve better performance on HumanML3D benchmark.

Critical strengths
  1. Converting 3D motion into image-like representation is an interesting idea.
  2. The propose methods achieve better performance compared with the baseline on two benchmark. It also show promising performance with a unified model that can handle slightly different skeleton offsets.
Critical weaknesses
  1. Using image model architecture for 3D motion modeling is not completely new. MARDM used SiT for the same text2motion task.
  2. The main contribution of this paper is the image compatible motion representation. Some closely related work [1] should be discussed and compared. One key design is the topology-informed order, however, there's insufficient evidence to support the advance of this design.
  3. Lack of video results and comparisons. The submission material contains only few static qualitative examples, making perceptual quality difficult to access and the claim SOTA performance unconvincing.
  4. In Appendix B.1, the root feature has been compress to 16 temporal positions before learned encoding. Given that data in HumanML3D normally are of 196 frames, the compressed representation will lose information and cannot be fully recovered.
  5. Similarly, Sections 3.1–3.2 resize the joint–time array to 256x256, then resize the decoded image back. Resizing back is generally not the inverse of resizing forward. It's unclear how the frame rate, motion length are determined given this simple resize operation. Please check the questions below.

[1] Yu, Qing, Mikihiro Tanaka, and Kent Fujiwara. "Exploring vision transformers for 3d human motion-language models with motion patches." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Questions for authors

How is output length determined during text-only generation? Is ground-truth length used during evaluation, and is length explicitly conditioned on during training? Since recovery resamples local poses and per-frame root increments, how are root displacement and heading increments adjusted to maintain consistent global motion and articulation when the requested length changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.