acceptodds
Under review as a conference paper at ICLR 2027

CAT2M: Camera-Aware Text to Motion Generation

Abstract

Recent world models have achieved notable progress in synthesizing realistic videos with controllable camera movements; however, achieving fine-grained human motion control under dynamic viewpoints remains a significant challenge. Meanwhile, existing text-to-motion methods typically assume fixed camera settings, which limits their applicability in world model settings where human motion should be consistent with the text description under changing camera viewpoints. To address this gap, we introduce a new task: camera-aware text-to-motion generation, where human motion is generated based on both text descriptions and given camera trajectories. To facilitate this task, we construct the CAT-Motion dataset, a large-scale dataset with over 100K samples of paired human motion and camera trajectories, each annotated with rich captions including viewpoint-specific motion descriptions. Based on this dataset, we propose CAT2M, a unified framework for motion generation with text and camera trajectory conditions. A shared coordinate frame anchored to the first camera preserves human-camera relative geometry, while temporal camera cross-attention conditions generation on the changing viewpoint. We further introduce differentiable screen-space supervision through a cumulative training scheme that retains diffusion and 3D joint objectives, constraining the projected appearance alongside the underlying 3D motion. Extensive experiments demonstrate that our approach outperforms existing baselines, showing strong ability in producing human motions that are semantically accurate and geometrically consistent under camera constraints and text descriptions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camera-aware Motion Generation, Text-to-Motion Generation, Human Motion Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XA97OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on camera-aware text-to-motion generation, aiming to generate articulated 3D human motions that align with textual descriptions under prescribed camera trajectories. Essentially, the task seeks to control how human motions are perceived from specific camera viewpoints. To address this task, the authors propose a unified framework for motion generation conditioned on both text and camera trajectories. They also construct CAT-Motion, a dataset containing over 100K paired human motion and camera trajectory samples. Experimental results demonstrate that the proposed method outperforms existing baselines, generating human motions with accurate semantic alignment and geometric consistency.

Critical strengths
  1. The paper addresses a meaningful and practical task by enabling simultaneous control over camera viewpoints and human motion dynamics through textual prompts.
  2. The authors identify an important limitation of existing datasets, which typically focus either on text-motion pairs or camera trajectories, while largely lacking textual annotations that describe the spatial relationships between humans and cameras. The proposed CAT-Motion dataset fills this gap and provides a valuable resource for future research.
  3. The paper provides comprehensive experiments to validate the effectiveness of the proposed method. Additionally, the paper is well-organized, with a clear and easy-to-follow presentation.
Critical weaknesses
  1. The multimodal evaluator used to compute Bi-R@k, MMDist, and FD_motion is trained by the authors on CAT-Motion, and its motion-side representation is the camera-gauged D147 representation. This means that the evaluated models and the evaluator share the same representational priors. However, several baselines use different representations, such as Pulp199 for MoMask-Cam and PulpMotion-Cond, and their own VAE representation for MotionGPT3-Cam. What justifies directly using this evaluator to compare methods based on different motion representations?
  2. I have some concerns about the practical use cases of this task and the inference process. Specifically, what inputs are required at inference time? Does the model require a textual description, a camera-to-world pose sequence, and camera intrinsics? Requiring users to provide a camera-to-world pose sequence themselves seems impractical in many real-world scenarios.
  3. As shown in Tables 1 and 9, MotionGPT3-Cam slightly outperforms MLD-Cam on Bi-R@1, while MLD-Cam achieves a higher LLM Score than MotionGPT3-Cam. However, the human pairwise comparison results show that CAT2M wins against MotionGPT3-Cam by a substantially larger margin than it does against MLD-Cam. This discrepancy across evaluation metrics seems to warrant further discussion.
Questions for authors
  1. The user study appears relatively small in scale, with two raters each assessing 15 pairs per comparator (60 pairs per rater in total). This limited sample size seems insufficient to strongly support the claim in the conclusion that “Human evaluation further supports” the effectiveness of the proposed method.
3 comments

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

Thank you for the responses and clarifications. Most of my concerns have been addressed. The relevant clarifications and any necessary changes could be incorporated into the revised manuscript in a future revision. At this stage, I will maintain my original rating.

Thank you for the response! We will revise our paper accordingly.

Reviewer_sqMdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces camera-aware text-to-motion generation, where a model generates 3D human motion conditioned on a textual description and a prescribed camera trajectory. It contributes CAT-Motion and a generation framework combining a shared camera-referenced coordinate system, temporal camera cross-attention, and geometric supervision. The task is well motivated: performing the correct action in 3D does not necessarily produce the intended screen-space movement, framing, or visibility. The proposed dataset and method address this distinction directly.

Critical strengths
  1. The task formulation captures an important limitation of conventional text-to-motion generation, particularly for viewpoint-dependent descriptions.

  2. CAT-Motion supplies structured descriptions of entry, in-frame motion, and exit, providing useful supervision beyond action labels.

  3. Table 1 reports improvements across all evaluated metrics. ProjE decreases from 3.285 for MLD-Cam to 1.120 for CAT2M.

  4. Component ablations distinguish coordinate normalization, camera conditioning, and geometric supervision.

  5. The appendix provides additional positive support through source-level dataset splitting, detailed baseline adaptations, and human evaluation.

Critical weaknesses
  1. The evaluation is concentrated on the constructed camera distribution. The dataset deliberately emphasizes entry and exit, with 91% of samples containing one or both events. The main experiments do not establish whether the method generalizes to substantially different camera trajectories, intrinsics, or continuously visible subjects. This limits the breadth of the camera-awareness claim.

  2. Reference-based geometric metrics do not fully characterize conditional generation. The task is formulated as sampling from a conditional motion distribution, but ProjE measures agreement with a particular reference motion, while FDframe measures distributional framing agreement. Multiple motions could satisfy the same description and camera condition. The evaluation would be stronger if it distinguished reference similarity from satisfaction of the requested constraints and preservation of diversity.

  3. The contributions of the components differ substantially in magnitude. Gauge raises Bi-R@1 from 12.94% to 27.90%, whereas adding screen supervision to the 3D-supervised model increases it from 29.27% to 29.80%, while substantially improving ProjE (Table 2). The evidence therefore supports a particularly strong contribution from the coordinate system and a more targeted geometric benefit from screen supervision.

  4. The main quantitative tables do not establish the stability of smaller improvements. Tables 1,2 report point estimates without repeated-run uncertainty. This matters especially for the smaller differences between the full model and its strongest ablations.

Questions for authors
  1. Does performance remain strong on held-out camera transformation families or substantially different camera intrinsics?

  2. Can the authors report constraint-satisfaction and diversity measurements alongside reference-based projection error?

  3. Are the smaller gains over Gauge-Add and the strongest loss ablation consistent across repeated training runs?

  4. How much advantage remains when the strongest baseline receives comparable coordinate normalization, geometric supervision, and training resources?

3 comments

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

Thanks for the additional experiments. The stronger baseline comparison clarifies the contributions. I agree that screen-space supervision mainly improves geometric accuracy.

My main concern remains generalization to unseen camera trajectories, which is still not evaluated. Overall, I think the rebuttal addresses several of my concerns, and I'll keep my rating.

Thank you for your follow-up! We agree that generalization to unseen camera trajectories is an important direction and will investigate it in our future work.

Reviewer_U2XBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAT2M generates 3D human motion from a text description and a prescribed camera trajectory. The paper builds CAT-Motion from PulpMotion with augmented cameras and viewpoint-dependent captions. The model uses a shared first-camera coordinate frame, temporal camera cross attention and latent diffusion. Training combines 3D joint supervision with projected-joint and depth losses. Experiments compare five adapted baselines and ablate the representation, camera conditioning and losses.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The paper addresses a meaningful and practical open challenge by introducing a new task: camera-aware text-to-motion generation.
  3. An accompanying dataset is constructed, which may interest researchers and support further research in this area.
  4. The experimental results demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. The proposed screen-space supervision appears similar to and incremental over PulpMotion, which already uses projected human joints to enforce human–camera framing consistency. The paper does not clearly establish what methodological advantage this design offers over the prior approach.
  2. Key design choices lack isolated ablations, such as the two-stage training schedule and the separate projection and depth-barrier losses. Hyperparameter sensitivity, such as to the depth-barrier margin and temperature, is also not reported.
  3. Training time, inference latency, and computational cost comparisons with baselines are not reported, making the practical efficiency difficult to assess.
Questions for authors
  1. How often are joints, frames and sequences excluded from ProjE for each method? Would the ranking remain similar with explicit penalties for invalid predictions and separate entry/exit timing and visibility scores?

  2. With cameras fixed, does the evaluator prefer correct captions to captions with wrong actions or reversed screen directions? Please also show whether the generator responds appropriately to these controlled caption changes.

1 comment

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.