acceptodds
Under review as a conference paper at ICLR 2027

LLMDT: LLM-Based Dual-Transformer for Scene-Aware Text-Driven 3D Human Motion Generation

Abstract

Text-driven 3D human motion generation has a wide range of applications across various domains, while scene-aware generation requires joint reasoning about language, objects, navigation, and human-scene interactions. However, existing methods often struggle to separate global scene reasoning from local motion generation, resulting in inaccurate object grounding, trajectories, and human-scene interactions. We propose LLMDT, a hierarchical framework that addresses spatial reasoning, scene consistency, and motion quality in scene-aware human motion generation. We introduce LLM-based spatial reasoning and object-centric scene understanding to establish structured relationships between human actions and scene elements to improve spatial accuracy. We develop a trajectory-aware generation strategy that integrates global navigation with local scene context to improve scene consistency and generate feasible motion trajectories in complex environments. We further propose a pose-aware motion generation strategy with hierarchical residual representations to improve motion quality by generating realistic and temporally coherent human motions. Experiments on HUMANISE and HumanML3D demonstrate improvements over existing methods in spatial accuracy, scene consistency, and motion quality.

Primary area:
generative models
Keywords:
text-to-motion, 3d human motion generation, large language models, scene-aware motion generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GVbvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes LLMDT for scene-aware text-driven 3D human motion generation. A pretrained LLM (ChatGPT) parses the instruction into action, target/anchor objects and spatial relations. The target object is picked from a scene graph with a grounding score, and an object-centric 8×8×8 voxel representation is built around it. A trajectory diffusion Transformer predicts a path (x,y,θ)(x,y,\theta), and a ``trajectory sensor'' samples local voxel observations along it. Finally a pose-aware dual Transformer (a masked Transformer plus a residual Transformer over RVQ tokens) generates SMPL-X motion. LLMDT reports better goal distance, action accuracy and FID than prior work on HUMANISE, competitive results on HumanML3D, and qualitative results on PROX, with an ablation on HUMANISE.

Critical strengths
  • The problem is relevant, and the design is sensible: separating grounding, global navigation, and local pose synthesis, with local scene features sampled along the predicted path, is an intuitive way to improve scene consistency.
  • The HUMANISE results are good on goal distance (0.053 vs.\ 0.081) and action accuracy (0.986 vs.\ 0.966), and the ablation shows each module helping.
Critical weaknesses
  • Limited novelty relative to prior work, with weak baselines and ambiguous fairness. The LLM grounding plus object-centric representation plus trajectory-then-pose route follows Cen et al.\ (CVPR'24), and the pose generator closely resembles MoMask. The new parts are the trajectory sensor and swapping in the dual Transformer, but the ablation does not separate them from the stronger backbone. It is also unclear what the scene input is: if object instances and labels come from ground-truth annotations while baselines use raw point clouds, the comparison isn't apples-to-apples. In addition, Table 1 lists HSI-GPT, MLLM planning, UniHM and TSTMotion, and SceMoS is cited as close work, but none of them appear in Table 2.
  • The experimental claims are partly overstated. Many HUMANISE gains are tiny or saturated (scene-consistent rate 99.95 vs.\ 99.91, quality score 5.941 vs.\ 5.932), and contact and diversity are not the best. On HumanML3D, which has no scene, the text says LLMDT achieves the best FID, but in Table 3 DT3DPE's FID (0.049) appears lower than LLMDT's (0.058). MoMask and T2M-GPT are cited but missing from that table, and it is unclear how the LLM and scene modules are used there at all. PROX is qualitative only, and the ablation has no standard deviations.
Questions for authors
  • What are the scene inputs for LLMDT and for each baseline (GT instance segmentation/labels versus detected objects)? Are all baselines given the same information?
  • How does the model run on HumanML3D, and can you include MoMask and T2M-GPT? Is FID in Table 3 correctly marked?
  • Could you compare with SceMoS, HSI-GPT, UniHM and the other Table 1 methods, and report inference time (LLM plus two Transformers)?
  • How much of the gain comes from the trajectory sensor versus the MoMask-style Transformer?
6 comments
AuthorsOpenReview ↗

W-1: The novelty relative to Cen et al. and MoMask is limited, and the fairness of the comparisons is unclear. It is also uncertain whether LLMDT and the baselines receive equivalent scene information. Several methods listed in Table 1, including HSI-GPT, UniHM, TSTMotion, and SceMoS, are not included in Table 2.

R-1: We thank the reviewer for highlighting the need to clarify LLMDT's methodological contributions, the fairness of the comparisons, and the coverage of the evaluated baselines. Cen et al. incorporate local scene sensing alongside LLM-based target grounding, object-centric scene representation, and trajectory planning. We recognize the overlap between these components and LLMDT and do not claim them as individually novel. The main distinction is LLMDT's integration of global trajectory guidance, trajectory-conditioned local geometric observations, and explicit articulated pose generation within a unified framework. Its pose-aware dual Transformer models root translation, global root orientation, and individual joint rotations. The masked Transformer predicts coarse motion structure, while the residual Transformer progressively refines articulated poses through hierarchical residual vector quantization (RVQ). Unlike MoMask, which focuses on general text-driven motion generation using hierarchical masked and residual Transformers, LLMDT conditions motion generation on grounded objects, planned trajectories, joint rotations, and local scene observations sampled along the predicted path. Thus, our contribution integrates scene-conditioned navigation and geometric reasoning with hierarchical articulated motion synthesis, rather than introducing LLM grounding or masked/residual Transformers independently.

Table 2. Performance comparison on the HUMANISE dataset. Best results are in bold, and second-best results are underlined.

MethodsVenueScene-conditionalAction-conditionalPure Motion Quality
Goal Dist. ↓Contact ↑Scene-consistent ↑Accuracy ↑Diversity ↑Multimodality ↑FID ↓Quality Score ↑
HUMANISENeurIPS0.32279.5499.860.8894.3182.7211.1265.472
MDMICLR0̲.̲0̲8̲1̲94.6199.710.8984.1871.9640̲.̲0̲9̲1̲5.271
Afford-MotionCVPR0.13994.1899.840.9134.3912.4670.5924.821
Cen et al.CVPR0.13789.219̲9̲.̲9̲1̲0.9304.8212.3650.3915.281
Song et al.AAAI0.37696.8298.770̲.̲9̲6̲6̲4.9312.4720.1735̲.̲9̲3̲2̲
Ours-0.0539̲6̲.̲4̲8̲99.950.9864̲.̲8̲2̲4̲2̲.̲5̲6̲6̲0.0795.941

Table 3. Performance comparison on the HumanML3D dataset. Best results are in bold, and second-best results are underlined.

MethodsVenueR-PrecisionFID ↓MM DIST ↓Diversity ↑MultiModality ↑
Top-1 ↑Top-2 ↑Top-3 ↑
MLDCVPR0.4850.6780.7770.482-9.7482̲.̲4̲3̲2̲
MDMICLR--0.6120.5485.5989.5812.805
T2M-GPTCVPR0.4930.6780.7760.1423.124-1.828
ReMoDiffuseICCV0.5110.7040.7920.1082.9719.0451.806
MoMaskCVPR0.5220.7120.8080.0462.961-1.238
DT3DPECAD0̲.̲5̲2̲7̲0̲.̲7̲2̲1̲0̲.̲8̲1̲1̲0̲.̲0̲4̲9̲2̲.̲9̲6̲8̲9.6891.249
Ours-0.5350.7280.8230.0582.9599̲.̲7̲0̲1̲1.257

To clarify evaluation fairness, we have specified the scene-input configurations and evaluation protocols in the revised manuscript. LLMDT uses structured semantic reasoning and object-centric voxel representations, while scene inputs for other methods follow their respective configurations, including annotated object instances and semantic labels, detected objects, and raw scene geometry where applicable. We do not assume that all methods receive identical representations. Instead, we distinguish their actual input requirements and use the same test instances, coordinate conventions, motion reconstruction procedures, and metric implementations wherever applicable. On HUMANISE, Table 2 compares scene-conditioned performance using Goal Distance, Contact, Scene-consistent Rate, Action Accuracy, Diversity, Multimodality, FID, and Quality Score. LLMDT achieves a Goal Distance of 0.053, a scene-consistent rate of 99.95%, an Action Accuracy of 0.986, an FID of 0.079, and a Quality Score of 5.941. These results provide evidence of its performance under the reported evaluation protocol, although they do not independently isolate the contribution of each architectural component. On HumanML3D, Table 3 reports standard text-to-motion metrics, including R-Precision, FID, MM Distance, Diversity, and Multimodality, without scene-specific geometric conditioning. LLMDT achieves R-Precision scores of 0.535, 0.728, and 0.823 at Top-1, Top-2, and Top-3, respectively, and an MM Distance of 2.959. (Added and highlighted in the Main Document: Pages 7 and 8)

AuthorsOpenReview ↗

W-2: The claims are overstated given marginal HUMANISE gains and weaker contact and diversity results. On HumanML3D, DT3DPE achieves a lower FID than LLMDT. MoMask and T2M-GPT are missing from Table 3, module roles are unclear, PROX evaluation is qualitative, and ablation results lack standard deviations.

R-2: We thank the reviewer for highlighting these issues. We have revised the experimental discussion. Table 2 presents experimental results on the HUMANISE dataset. We acknowledge that some gains are modest, particularly the scene-consistent rate (99.95% vs. 99.91%) and quality score (5.941 vs. .932). We therefore describe these results as incremental improvements rather than substantial gains.

Table 2. Performance comparison on the HUMANISE dataset. Best results are in bold, and second-best results are underlined.

MethodsVenueGoal Dist. ↓Contact ↑Scene-consistent ↑Accuracy ↑Diversity ↑Multimodality ↑FID ↓Quality Score ↑
HUMANISENeurIPS0.322 ± 0.00479.54 ± 0.00699.86 ± 0.0030.889 ± 0.0034.318 ± 0.0512.721 ± 0.0391.126 ± 0.0465.472 ± 0.045
MDMICLR0̲.̲0̲8̲1̲ ± 0̲.̲0̲0̲1̲94.61 ± 0.00699.71 ± 0.0040.898 ± 0.0054.187 ± 0.1101.964 ± 0.1060̲.̲0̲9̲1̲ ± 0̲.̲1̲0̲8̲5.271 ± 0.098
Afford-MotionCVPR0.139 ± 0.00394.18 ± 0.00599.84 ± 0.0040.913 ± 0.0074.391 ± 0.1092.467 ± 0.1010.592 ± 0.1174.821 ± 0.107
Cen et al.CVPR0.137 ± 0.00289.21 ± 0.0039̲9̲.̲9̲1̲ ± 0̲.̲0̲0̲6̲0.930 ± 0.0054.821 ± 0.1162.365 ± 0.0990.391 ± 0.1115.281 ± 0.102
Song et al.AAAI0.376 ± 0.00396.82 ± 0.00598.77 ± 0.0060̲.̲9̲6̲6̲ ± 0̲.̲0̲0̲7̲4.931 ± 0.1182.472 ± 0.1030.173 ± 0.1155̲.̲9̲3̲2̲ ± 0̲.̲1̲0̲9̲
Ours–0.053 ± 0.0019̲6̲.̲4̲8̲ ± 0̲.̲0̲0̲3̲99.95 ± 0.0040.986 ± 0.0054̲.̲8̲2̲4̲ ± 0̲.̲1̲1̲1̲2̲.̲5̲6̲6̲ ± 0̲.̲1̲0̲0̲0.079 ± 0.1065.941 ± 0.112

Table 3 presents experimental results on the HumanML3D dataset. We have corrected the HumanML3D discussion to clarify that LLMDT does not achieve the best FID. DT3DPE and MoMask obtain lower FID scores of 0.049 and 0.046, respectively, compared with 0.058 for LLMDT. Instead, LLMDT achieves the best Top-1, Top-2, and Top-3 R-Precision and MM DIST. We retain MoMask and T2M-GPT in Table 3 for completeness. Since HumanML3D does not provide scene conditioning, this experiment evaluates standard text-to-motion generation and does not establish the contribution of the LLM-based grounding or scene-sensing modules.

Table 3. Performance comparison on the HumanML3D dataset. Best results are in bold, and second-best results are underlined.

MethodsVenueR-Precision Top-1 ↑R-Precision Top-2 ↑R-Precision Top-3 ↑FID ↓MM DIST ↓Diversity ↑MultiModality ↑
MLDCVPR0.485 ± 0.0020.678 ± 0.0030.777 ± 0.0040.482 ± 0.012–9.748 ± 0.0952̲.̲4̲3̲2̲ ± 0̲.̲0̲6̲9̲
MDMICLR––0.612 ± 0.0050.548 ± 0.0395.598 ± 0.0339.581 ± 0.0932.805 ± 0.070
T2M-GPTCVPR0.493 ± 0.0030.678 ± 0.0020.776 ± 0.0020.142 ± 0.0053.124 ± 0.009–1.828 ± 0.048
ReMoDiffuseICCV0.511 ± 0.0020.704 ± 0.0040.792 ± 0.0070.108 ± 0.0092.971 ± 0.0149.045 ± 0.0701.806 ± 0.036
MoMaskCVPR0.522 ± 0.0020.712 ± 0.0020.808 ± 0.0020.046 ± 0.0022.961 ± 0.008–1.238 ± 0.040
DT3DPECAD0̲.̲5̲2̲7̲ ± 0̲.̲0̲0̲5̲0̲.̲7̲2̲1̲ ± 0̲.̲0̲0̲7̲0̲.̲8̲1̲1̲ ± 0̲.̲0̲0̲6̲0̲.̲0̲4̲9̲ ± 0̲.̲0̲0̲6̲2̲.̲9̲6̲8̲ ± 0̲.̲0̲0̲3̲9.689 ± 0.0931.249 ± 0.051
Ours–0.535 ± 0.0060.728 ± 0.0060.823 ± 0.0070.058 ± 0.0072.959 ± 0.0039̲.̲7̲0̲1̲ ± 0̲.̲0̲9̲1̲1.257 ± 0.053

We clarify that the PROX results provide qualitative results to show how our model performs on unseen scenes. Table 4 presents mean values of ablation results with standard deviations for all evaluated metrics. We have clarified this reporting in the revised discussion and use the ablation results to assess the contributions of individual components. (Added and highlighted in the Main Document: Pages 8, 9, and 10)

Table 4. Ablation study of the proposed LLMDT framework on the HUMANISE dataset.

MethodGoal Dist. ↓Accuracy ↑Scene-consistent ↑FID ↓Quality ↑
w/o LLM Semantic Reasoning0.112 ± 0.0030.947 ± 0.00699.79 ± 0.0120.231 ± 0.0085.624 ± 0.094
w/o Object-Centric Scene Rep.0.097 ± 0.0020.958 ± 0.00499.83 ± 0.0090.184 ± 0.0065.701 ± 0.087
w/o Trajectory Transformer0.088 ± 0.0030.972 ± 0.00599.71 ± 0.0150.146 ± 0.0075.786 ± 0.102
w/o Trajectory Sensor0.072 ± 0.0020.978 ± 0.00399.81 ± 0.0080.119 ± 0.0055.852 ± 0.096
w/o Pose-Aware Dual Transformer0.069 ± 0.0020.975 ± 0.00499.88 ± 0.0060.127 ± 0.0065.831 ± 0.091
LLMDT (Full Model)0.053 ± 0.0010.986 ± 0.00599.95 ± 0.0040.079 ± 0.1065.941 ± 0.112
AuthorsOpenReview ↗

Q-1: What are the scene inputs for LLMDT and for each baseline (GT instance segmentation/labels versus detected objects)? Are all baselines given the same information?

A-1: We thank the reviewer for raising this concern. We have clarified that LLMDT uses object-centric voxel representations constructed from scene geometry, together with LLM-based semantic reasoning for target-object grounding and trajectory-conditioned local scene sensing. All methods are evaluated on the same test instances with consistent coordinate conventions and evaluation protocols. However, scene inputs follow each method's original configuration, and we do not assume that all baselines use identical object annotations or detection information. HumanML3D experiments follow the standard text-to-motion protocol without scene conditioning. (Added and highlighted in the Main Document: Pages 7 and 8)

Reviewer_YCH8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLMDT, a hierarchical framework for scene-aware text-driven 3D human motion generation. The method integrates LLM-based spatial reasoning, object-centric scene representations, trajectory diffusion, and a pose-aware dual-Transformer with residual vector quantization (RVQ). Experiments on HUMANISE and HumanML3D report improvements in spatial accuracy and motion generation quality. While the problem is relevant, I have significant concerns regarding the methodological novelty, technical clarity, and experimental validation. In particular, several components closely resemble existing methods, and the reported improvements are not sufficiently supported by the qualitative results.

Critical strengths
  1. The paper addresses an important problem in scene-aware human motion generation, where language understanding, object grounding, trajectory planning, and human-scene interactions must be considered jointly.

  2. The overall framework is reasonably structured, and the separation between global trajectory planning and local motion generation is intuitively reasonable.

  3. The paper provides quantitative comparisons on multiple datasets, along with component-wise ablation studies.

Critical weaknesses
  1. Limited methodological novelty. The overall framework closely resembles Cen et al. (CVPR 2024), which already incorporates LLM-based object grounding, object-centric scene representations, trajectory diffusion, and trajectory-conditioned local scene sensing. In particular, the Trajectory Sensor is explicitly introduced in that work, whereas Table 1 incorrectly indicates that Cen et al. lacks local scene sensing. Furthermore, the proposed masked and residual Transformers with hierarchical RVQ are conceptually similar to MoMask (CVPR 2024). The paper does not clearly establish what constitutes a fundamentally new technical contribution beyond combining existing components.

  2. Insufficient methodological details and experimental justification. Several important components, including the grounding score, Grounding Transformer, geometric feature encoder, and scene-conditioned motion generation, are insufficiently specified. The paper also lacks a controlled comparison demonstrating improvements over a straightforward combination of Cen et al.'s scene-aware framework and a MoMask-style motion generator. Moreover, the evaluation protocol and the use of scene-dependent components on HumanML3D require clarification. These issues make it difficult to assess the effectiveness and reproducibility of the proposed approach.

  3. Limited evidence of motion quality and physical plausibility. Despite the reported quantitative improvements, the qualitative results and supplementary video show noticeable artifacts, including human-scene interpenetration and foot sliding. This raises concerns about the claimed scene consistency and motion realism. Several reported improvements are also relatively small, and the scene-consistency scores are already close to saturation. Furthermore, the claim that LLMDT achieves the best FID on HumanML3D is inconsistent with Table 3, where DT3DPE achieves a lower FID (0.049 vs. 0.058). More rigorous evaluation of physical plausibility, contact consistency, and temporal stability is needed.

Questions for authors
  1. What are the specific technical differences between LLMDT and Cen et al. (CVPR 2024), particularly regarding object grounding, scene representation, and trajectory-conditioned local scene sensing? Why is local scene sensing marked as absent for Cen et al. in Table 1?

  2. How does the proposed pose-aware dual-Transformer differ from the masked and residual Transformer architecture of MoMask? Can the authors provide a controlled comparison against a Cen et al. framework equipped with a MoMask-style motion generator?

  3. How are physical plausibility and scene consistency evaluated? Can the authors provide additional quantitative evaluations of penetration, foot sliding, and contact stability, and clarify the evaluation protocols used for the reported baseline comparisons?

6 comments
AuthorsOpenReview ↗

W-1: Limited methodological novelty.

R-1: We thank the reviewer for this comment. We have revised the methodological discussion to clarify the technical distinctions between LLMDT, Cen et al. (CVPR 2024), and MoMask (Guo et al., 2024). We acknowledge that Cen et al. already incorporate LLM-based target grounding, object-centric scene representation, trajectory planning, and local scene sensing. Similarly, LLMDT shares the hierarchical masked and residual token-generation principle used by MoMask. We therefore do not present these individual components as independent novelties.

The main difference of LLMDT lies in its integration of trajectory-conditioned geometric reasoning with explicit articulated pose generation. The trajectory Transformer predicts a scene-conditioned path toward the grounded target, and a pelvis-centered sensor extracts local geometric observations along this path. These observations, together with the planned trajectory, semantic features, and object-centric scene representation, condition the pose-aware dual Transformer. Unlike a general text-to-motion generation pipeline, this design explicitly connects global navigation and local scene geometry with the prediction and refinement of root translation, root orientation, and individual joint rotations. The masked Transformer predicts coarse motion structure, while the residual Transformer progressively refines articulated poses through hierarchical RVQ representations.

AuthorsOpenReview ↗

W-2: Insufficient methodological details and experimental justification.

R-2: We thank the reviewer for highlighting the need for greater methodological clarity and reproducibility. We have expanded the methodology to specify the object-grounding procedure, geometric scene representation, trajectory-conditioned local sensing, and pose-generation pipeline. In particular, the grounding score in Equation~2 combines semantic confidence, language-object similarity, and scene-graph relational consistency to select the target object.

\begin{equation} \mathcal{J}(o_i)= \alpha P(o_i|\mathcal{T}) +\beta\psi(e_T,e_{o_i}) +\gamma\phi(G,e_R,o_i), \end{equation}

where P(oi∣T)P(o_i|\mathcal{T}) denotes the semantic confidence of candidate oio_i, eTe_T and eoie_{o_i} are language and object embeddings, ψ(⋅)\psi(\cdot) measures language-object similarity, and ϕ(⋅)\phi(\cdot) evaluates relational consistency with the scene graph. The coefficients α\alpha, β\beta, and γ\gamma balance the three terms. The grounding score combines semantic confidence, language-object similarity, and scene-graph consistency to assess each candidate using complementary semantic and geometric cues. The candidate with the highest score is selected as the target object o∗o^*, which guides object-centric scene representation and subsequent trajectory planning.

The geometric feature encoder integrates scene-level voxel features and anchor-sensor features, including occupancy, surface normals, and spatial coordinates. We also clarify how the trajectory Transformer predicts a path conditioned on language semantics and object-centric geometry, how local observations are sampled along the predicted trajectory, and how these observations condition the pose-aware dual Transformer.

The revised description further specifies the articulated motion representation, including root translation, global root orientation, and individual joint rotations. The masked Transformer predicts coarse motion tokens, while the residual Transformer progressively refines the motion representation using hierarchical RVQ. This formulation clarifies the relationship between global trajectory guidance, local scene geometry, and fine-grained pose generation. The current methodology describes these operations but does not identify a separate module explicitly named the ``Grounding Transformer''; we therefore clarify the grounding operations actually used in LLMDT rather than introduce an unsupported architectural component.

We have also clarified the distinction between the two evaluation settings. HUMANISE supports scene-conditioned generation using scene geometry, target-object grounding, trajectory planning, and trajectory-conditioned local sensing. HumanML3D follows the standard text-to-motion evaluation protocol using paired text-motion data and does not require scene geometry. Scene-dependent evaluation is therefore not directly applicable to HumanML3D without additional scene assignments and geometric inputs. We distinguish general text-to-motion evaluation from scene-conditioned evaluation to avoid implying that HumanML3D results measure scene-grounding or human-scene interaction performance. (Added and highlighted in the Main Document: Pages 4, 5, 6, 7, and 8)

AuthorsOpenReview ↗

W-3: Limited evidence of motion quality and physical plausibility

R-3: We thank the reviewer for emphasizing the importance of physical plausibility beyond conventional motion-generation metrics. To address this concern, we have added a quantitative physical-plausibility evaluation on the HUMANISE in the supplementary material, including penetration ratio, mean penetration depth, foot-sliding velocity, and stable-contact ratio. These metrics complement the existing scene-consistency and motion-quality measures by directly assessing geometric violations, foot-contact stability, and human-scene interactions.

As reported in Table 1 in the supplementary material, LLMDT achieves a penetration ratio of 0.72% and a mean penetration depth of 0.79 cm, compared with 0.76% and 0.84 cm for Song et al. (Song et al., 2026). LLMDT also reduces foot-sliding velocity from 2.91 to 2.79~cm/s and increases the stable-contact ratio from 95.72% to 95.94%. These results indicate moderate improvements in geometric compatibility and contact stability relative to the strongest reported baseline. However, the remaining penetration and foot sliding show that LLMDT does not completely eliminate physical artifacts.

Table 1. Physical plausibility evaluation on the HUMANISE dataset. Lower penetration ratio, penetration depth, and foot-sliding velocity indicate better physical plausibility, whereas a higher stable-contact ratio indicates more stable human-scene interactions.

MethodsPenetration Ratio (%) ↓Penetration Depth (cm) ↓Foot Sliding (cm/s) ↓Stable Contact (%) ↑
HUMANISE1.841.724.8391.26
MDM1.271.313.9693.84
Afford-Motion1.081.143.4294.61
Cen et al.0.930.983.1795.08
Strong Baseline0.810.893.0295.51
Song et al.0.760.842.9195.72
Ours0.720.792.7995.94

We have also added the corresponding metric definitions and evaluation protocol to the supplementary material to improve reproducibility. We interpret these results alongside the qualitative examples rather than relying solely on scene-consistency scores. Although the additional metrics quantify penetration, foot sliding, and contact stability, they do not independently establish temporal consistency. We therefore acknowledge temporal consistency as a distinct evaluation aspect that is not directly quantified by the four metrics reported in Table~1. (Added and highlighted in the Supplementary Material: Pages 2 and 3)

Reviewer_VZUcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LLMDT generates human motion from text in 3D scenes. It uses an LLM and a scene graph to ground the target object, predicts a trajectory from object-centric geometry, and samples local scene features along that trajectory. A masked Transformer and a residual Transformer then generate hierarchical motion tokens, which are decoded into body motion. The paper reports results on HUMANISE and HumanML3D, with qualitative examples on PROX.

Critical strengths
  1. The paper is well writen and easy to follow.

  2. The experimental results show the efficacy of the proposed method.

Critical weaknesses
  1. The method relies on an LLM for semantic reasoning and object grounding. Hallucinations or incorrect spatial reasoning could lead to the wrong target and affect the generated motion. It is unclear how the system handles these errors.

  2. Comparisons across different LLMs are missing, despite the method’s heavy reliance on LLM-based reasoning. It is therefore unclear how sensitive the results are to the choice of LLM.

  3. The method is very similar to prior work [1, 2], and the new contribution is unclear. Table 1 also incorrectly states that [1] lacks local scene sensing. Comparisons under the same setup would help show what the proposed changes actually add.

  4. The evaluation needs clearer explanations of the HUMANISE metrics and the HumanML3D setup. MoMask [2] is also missing from the comparisons. The claim of the best FID contradicts Table 3, where DT3DPE has a lower FID.

[1] Cen et al. “Generating Human Motion in 3D Scenes from Text Descriptions.” CVPR, 2024.

[2] Guo et al. “MoMask: Generative Masked Modeling of 3D Human Motions.” CVPR, 2024.

Questions for authors

Please see weaknesses.

5 comments
AuthorsOpenReview ↗

W-1: The method relies on an LLM for semantic reasoning and object grounding. Hallucinations or incorrect spatial reasoning could lead to the wrong target and affect the generated motion. It is unclear how the system handles these errors.

R-1: We thank the reviewer for highlighting the reliability of LLM-based semantic reasoning and object grounding. We have expanded the subsection ``LLM-Based Semantic Reasoning and Object Grounding'' to clarify how LLMDT mitigates potential errors in language understanding and spatial reasoning. Specifically, the LLM extracts the action, target, and anchor objects, and spatial relations from the instruction, while detected scene objects are organized into a scene graph representing object instances and their spatial relationships. As defined in Equation~1, LLMDT selects the target object o∗o^* by maximizing the grounding score over candidate objects:

\begin{equation} o^* = \arg\max_{o_i\in V}\mathcal{J}(o_i), \end{equation} The grounding score in Equation~2 combines semantic confidence, language-object similarity, and scene-graph relational consistency:

\begin{equation} \mathcal{J}(o_i)= \alpha P(o_i|\mathcal{T}) +\beta\psi(e_T,e_{o_i}) +\gamma\phi(G,e_R,o_i), \end{equation} where P(oi∣T)P(o_i|\mathcal{T}) denotes the semantic confidence of candidate object oio_i, eTe_T and eoie_{o_i} represent the language and object embeddings, ψ(⋅)\psi(\cdot) measures language-object similarity, and ϕ(⋅)\phi(\cdot) evaluates relational consistency with the scene graph. The coefficients α\alpha, β\beta, and γ\gamma balance these three terms. By integrating complementary semantic and geometric cues, LLMDT reduces reliance on the LLM's prediction alone and helps identify target objects consistent with both the instruction and the observed scene. The selected object determines the object-centric scene representation and guides subsequent trajectory planning and motion generation. Although this mechanism helps mitigate errors caused by ambiguous object references and inconsistent spatial descriptions, incorrect grounding may still affect downstream generation. We have clarified this limitation in the revised manuscript. (Added and highlighted in the Main Document: Pages 4 and 5)

AuthorsOpenReview ↗

W-2: Comparisons across different LLMs are missing, despite the method's heavy reliance on LLM-based reasoning. It is therefore unclear how sensitive the results are to the choice of LLM.

R-2: We thank the reviewer for raising this concern regarding the dependence of LLMDT on LLM-based reasoning. We have clarified the role of the LLM in the proposed framework and its interaction with the subsequent geometric reasoning and motion-generation components. Specifically, the LLM extracts structured semantic information, including the action, target and anchor objects, and spatial relations, while the grounding procedure combines semantic confidence, language-object similarity, and scene-graph relational consistency to select the target object. This design reduces reliance on the LLM's predictions alone by incorporating complementary semantic and geometric cues. We also evaluate LLMDT against existing methods, including Cen et al. (Cen et al., 2024) on HUMANISE (Table 2) and T2M-GPT (Zhang et al., 2023a) on HumanML3D (Table 3), to assess its overall performance relative to established approaches. (Added and highlighted in the Main Document: Page 8)

Table 2. Performance comparison on the HUMANISE dataset. Best results are in bold, and second-best results are underlined.

MethodsVenueGoal Dist. ↓Contact ↑Scene-consistent ↑Accuracy ↑Diversity ↑Multimodality ↑FID ↓Quality Score ↑
HUMANISENeurIPS0.322 ± 0.00479.54 ± 0.00699.86 ± 0.0030.889 ± 0.0034.318 ± 0.0512.721 ± 0.0391.126 ± 0.0465.472 ± 0.045
MDMICLR0̲.̲0̲8̲1̲ ± 0̲.̲0̲0̲1̲94.61 ± 0.00699.71 ± 0.0040.898 ± 0.0054.187 ± 0.1101.964 ± 0.1060̲.̲0̲9̲1̲ ± 0̲.̲1̲0̲8̲5.271 ± 0.098
Afford-MotionCVPR0.139 ± 0.00394.18 ± 0.00599.84 ± 0.0040.913 ± 0.0074.391 ± 0.1092.467 ± 0.1010.592 ± 0.1174.821 ± 0.107
Cen et al.CVPR0.137 ± 0.00289.21 ± 0.0039̲9̲.̲9̲1̲ ± 0̲.̲0̲0̲6̲0.930 ± 0.0054.821 ± 0.1162.365 ± 0.0990.391 ± 0.1115.281 ± 0.102
Song et al.AAAI0.376 ± 0.00396.82 ± 0.00598.77 ± 0.0060̲.̲9̲6̲6̲ ± 0̲.̲0̲0̲7̲4.931 ± 0.1182.472 ± 0.1030.173 ± 0.1155̲.̲9̲3̲2̲ ± 0̲.̲1̲0̲9̲
Ours–0.053 ± 0.0019̲6̲.̲4̲8̲ ± 0̲.̲0̲0̲3̲99.95 ± 0.0040.986 ± 0.0054̲.̲8̲2̲4̲ ± 0̲.̲1̲1̲1̲2̲.̲5̲6̲6̲ ± 0̲.̲1̲0̲0̲0.079 ± 0.1065.941 ± 0.112

Table 3. Performance comparison on the HumanML3D dataset. Best results are in bold, and second-best results are underlined.

MethodsVenueR-Precision Top-1 ↑R-Precision Top-2 ↑R-Precision Top-3 ↑FID ↓MM DIST ↓Diversity ↑MultiModality ↑
MLDCVPR0.485 ± 0.0020.678 ± 0.0030.777 ± 0.0040.482 ± 0.012–9.748 ± 0.0952̲.̲4̲3̲2̲ ± 0̲.̲0̲6̲9̲
MDMICLR––0.612 ± 0.0050.548 ± 0.0395.598 ± 0.0339.581 ± 0.0932.805 ± 0.070
T2M-GPTCVPR0.493 ± 0.0030.678 ± 0.0020.776 ± 0.0020.142 ± 0.0053.124 ± 0.009–1.828 ± 0.048
ReMoDiffuseICCV0.511 ± 0.0020.704 ± 0.0040.792 ± 0.0070.108 ± 0.0092.971 ± 0.0149.045 ± 0.0701.806 ± 0.036
MoMaskCVPR0.522 ± 0.0020.712 ± 0.0020.808 ± 0.0020.046 ± 0.0022.961 ± 0.008–1.238 ± 0.040
DT3DPECAD0̲.̲5̲2̲7̲ ± 0̲.̲0̲0̲5̲0̲.̲7̲2̲1̲ ± 0̲.̲0̲0̲7̲0̲.̲8̲1̲1̲ ± 0̲.̲0̲0̲6̲0̲.̲0̲4̲9̲ ± 0̲.̲0̲0̲6̲2̲.̲9̲6̲8̲ ± 0̲.̲0̲0̲3̲9.689 ± 0.0931.249 ± 0.051
Ours–0.535 ± 0.0060.728 ± 0.0060.823 ± 0.0070.058 ± 0.0072.959 ± 0.0039̲.̲7̲0̲1̲ ± 0̲.̲0̲9̲1̲1.257 ± 0.053
AuthorsOpenReview ↗

W-3: The method is very similar to prior work [1, 2], and the new contribution is unclear. Table 1 also incorrectly states that [1] lacks local scene sensing. Comparisons under the same setup would help show what the proposed changes actually add.

R-3: We thank the reviewer for highlighting the need to clarify the methodological distinctions from Cen et al. (Cen et al., 2024) and MoMask (Guo et al., 2024), and for identifying the inaccurate comparison in Table~1. We have corrected the table to acknowledge that Cen et al. incorporate local scene sensing alongside LLM-based target grounding, object-centric scene representation, and trajectory planning.

Table 1. Comparison of LLMDT with existing representative scene-aware and text-driven human motion generation methods.

MethodLLM/MLLM ReasoningObject GroundingTrajectory PlanningLocal Scene SensingHierarchical Pose GenerationRVQ
HUMANISE--✓--------
Afford-Motion--✓--------
Cen et al.✓✓✓✓----
IntentMotion--✓✓------
HSI-GPT✓✓--------
SceMoS--✓✓--✓--
MLLM Planning✓✓✓------
TSTMotion--✓--------
UniHM--✓----✓--
LLMDT (Ours)✓✓✓✓✓✓

We agree that these components overlap with parts of LLMDT and avoid presenting them individually as new contributions. The main distinction lies in how LLMDT integrates global navigation, trajectory-conditioned geometric observations, and articulated pose generation within a unified framework. Compared with Cen et al., LLMDT uses trajectory-conditioned local observations sampled along the predicted path to provide frame-wise geometric context to the pose-generation network. More importantly, its pose-aware dual Transformer explicitly models root translation, global root orientation, and individual joint rotations. The masked Transformer predicts coarse motion structure, while the residual Transformer progressively refines articulated poses through hierarchical residual vector quantization (RVQ). This design connects global trajectory guidance and local scene geometry with coarse-to-fine joint-pose estimation.

MoMask also employs masked and residual Transformers with hierarchical motion-token representations, but its primary focus is general text-driven motion generation. LLMDT adapts a related hierarchical generation principle to scene-aware synthesis by conditioning pose generation on the grounded object, planned trajectory, and trajectory-conditioned local scene observations. Thus, the contribution is not the independent introduction of LLM grounding or masked/residual motion modeling, but their integration with explicit global navigation and local geometric conditioning for articulated motion generation in 3D scenes. We have revised Table~1 to correct the description of (Cen et al., 2024). (Added and highlighted in the Main Document: Page 3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.