- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes LLMDT for scene-aware text-driven 3D human motion generation. A pretrained LLM (ChatGPT) parses the instruction into action, target/anchor objects and spatial relations. The target object is picked from a scene graph with a grounding score, and an object-centric 8×8×8 voxel representation is built around it. A trajectory diffusion Transformer predicts a path , and a ``trajectory sensor'' samples local voxel observations along it. Finally a pose-aware dual Transformer (a masked Transformer plus a residual Transformer over RVQ tokens) generates SMPL-X motion. LLMDT reports better goal distance, action accuracy and FID than prior work on HUMANISE, competitive results on HumanML3D, and qualitative results on PROX, with an ablation on HUMANISE.
Critical strengths
- The problem is relevant, and the design is sensible: separating grounding, global navigation, and local pose synthesis, with local scene features sampled along the predicted path, is an intuitive way to improve scene consistency.
- The HUMANISE results are good on goal distance (0.053 vs.\ 0.081) and action accuracy (0.986 vs.\ 0.966), and the ablation shows each module helping.
Critical weaknesses
- Limited novelty relative to prior work, with weak baselines and ambiguous fairness. The LLM grounding plus object-centric representation plus trajectory-then-pose route follows Cen et al.\ (CVPR'24), and the pose generator closely resembles MoMask. The new parts are the trajectory sensor and swapping in the dual Transformer, but the ablation does not separate them from the stronger backbone. It is also unclear what the scene input is: if object instances and labels come from ground-truth annotations while baselines use raw point clouds, the comparison isn't apples-to-apples. In addition, Table 1 lists HSI-GPT, MLLM planning, UniHM and TSTMotion, and SceMoS is cited as close work, but none of them appear in Table 2.
- The experimental claims are partly overstated. Many HUMANISE gains are tiny or saturated (scene-consistent rate 99.95 vs.\ 99.91, quality score 5.941 vs.\ 5.932), and contact and diversity are not the best. On HumanML3D, which has no scene, the text says LLMDT achieves the best FID, but in Table 3 DT3DPE's FID (0.049) appears lower than LLMDT's (0.058). MoMask and T2M-GPT are cited but missing from that table, and it is unclear how the LLM and scene modules are used there at all. PROX is qualitative only, and the ablation has no standard deviations.
Questions for authors
- What are the scene inputs for LLMDT and for each baseline (GT instance segmentation/labels versus detected objects)? Are all baselines given the same information?
- How does the model run on HumanML3D, and can you include MoMask and T2M-GPT? Is FID in Table 3 correctly marked?
- Could you compare with SceMoS, HSI-GPT, UniHM and the other Table 1 methods, and report inference time (LLM plus two Transformers)?
- How much of the gain comes from the trajectory sensor versus the MoMask-style Transformer?
6 comments
W-2: The claims are overstated given marginal HUMANISE gains and weaker contact and diversity results. On HumanML3D, DT3DPE achieves a lower FID than LLMDT. MoMask and T2M-GPT are missing from Table 3, module roles are unclear, PROX evaluation is qualitative, and ablation results lack standard deviations.
R-2: We thank the reviewer for highlighting these issues. We have revised the experimental discussion. Table 2 presents experimental results on the HUMANISE dataset. We acknowledge that some gains are modest, particularly the scene-consistent rate (99.95% vs. 99.91%) and quality score (5.941 vs. .932). We therefore describe these results as incremental improvements rather than substantial gains.
Table 2. Performance comparison on the HUMANISE dataset. Best results are in bold, and second-best results are underlined.
| Methods | Venue | Goal Dist. ↓ | Contact ↑ | Scene-consistent ↑ | Accuracy ↑ | Diversity ↑ | Multimodality ↑ | FID ↓ | Quality Score ↑ |
|---|---|---|---|---|---|---|---|---|---|
| HUMANISE | NeurIPS | 0.322 ± 0.004 | 79.54 ± 0.006 | 99.86 ± 0.003 | 0.889 ± 0.003 | 4.318 ± 0.051 | 2.721 ± 0.039 | 1.126 ± 0.046 | 5.472 ± 0.045 |
| MDM | ICLR | 0̲.̲0̲8̲1̲ ± 0̲.̲0̲0̲1̲ | 94.61 ± 0.006 | 99.71 ± 0.004 | 0.898 ± 0.005 | 4.187 ± 0.110 | 1.964 ± 0.106 | 0̲.̲0̲9̲1̲ ± 0̲.̲1̲0̲8̲ | 5.271 ± 0.098 |
| Afford-Motion | CVPR | 0.139 ± 0.003 | 94.18 ± 0.005 | 99.84 ± 0.004 | 0.913 ± 0.007 | 4.391 ± 0.109 | 2.467 ± 0.101 | 0.592 ± 0.117 | 4.821 ± 0.107 |
| Cen et al. | CVPR | 0.137 ± 0.002 | 89.21 ± 0.003 | 9̲9̲.̲9̲1̲ ± 0̲.̲0̲0̲6̲ | 0.930 ± 0.005 | 4.821 ± 0.116 | 2.365 ± 0.099 | 0.391 ± 0.111 | 5.281 ± 0.102 |
| Song et al. | AAAI | 0.376 ± 0.003 | 96.82 ± 0.005 | 98.77 ± 0.006 | 0̲.̲9̲6̲6̲ ± 0̲.̲0̲0̲7̲ | 4.931 ± 0.118 | 2.472 ± 0.103 | 0.173 ± 0.115 | 5̲.̲9̲3̲2̲ ± 0̲.̲1̲0̲9̲ |
| Ours | – | 0.053 ± 0.001 | 9̲6̲.̲4̲8̲ ± 0̲.̲0̲0̲3̲ | 99.95 ± 0.004 | 0.986 ± 0.005 | 4̲.̲8̲2̲4̲ ± 0̲.̲1̲1̲1̲ | 2̲.̲5̲6̲6̲ ± 0̲.̲1̲0̲0̲ | 0.079 ± 0.106 | 5.941 ± 0.112 |
Table 3 presents experimental results on the HumanML3D dataset. We have corrected the HumanML3D discussion to clarify that LLMDT does not achieve the best FID. DT3DPE and MoMask obtain lower FID scores of 0.049 and 0.046, respectively, compared with 0.058 for LLMDT. Instead, LLMDT achieves the best Top-1, Top-2, and Top-3 R-Precision and MM DIST. We retain MoMask and T2M-GPT in Table 3 for completeness. Since HumanML3D does not provide scene conditioning, this experiment evaluates standard text-to-motion generation and does not establish the contribution of the LLM-based grounding or scene-sensing modules.
Table 3. Performance comparison on the HumanML3D dataset. Best results are in bold, and second-best results are underlined.
| Methods | Venue | R-Precision Top-1 ↑ | R-Precision Top-2 ↑ | R-Precision Top-3 ↑ | FID ↓ | MM DIST ↓ | Diversity ↑ | MultiModality ↑ |
|---|---|---|---|---|---|---|---|---|
| MLD | CVPR | 0.485 ± 0.002 | 0.678 ± 0.003 | 0.777 ± 0.004 | 0.482 ± 0.012 | – | 9.748 ± 0.095 | 2̲.̲4̲3̲2̲ ± 0̲.̲0̲6̲9̲ |
| MDM | ICLR | – | – | 0.612 ± 0.005 | 0.548 ± 0.039 | 5.598 ± 0.033 | 9.581 ± 0.093 | 2.805 ± 0.070 |
| T2M-GPT | CVPR | 0.493 ± 0.003 | 0.678 ± 0.002 | 0.776 ± 0.002 | 0.142 ± 0.005 | 3.124 ± 0.009 | – | 1.828 ± 0.048 |
| ReMoDiffuse | ICCV | 0.511 ± 0.002 | 0.704 ± 0.004 | 0.792 ± 0.007 | 0.108 ± 0.009 | 2.971 ± 0.014 | 9.045 ± 0.070 | 1.806 ± 0.036 |
| MoMask | CVPR | 0.522 ± 0.002 | 0.712 ± 0.002 | 0.808 ± 0.002 | 0.046 ± 0.002 | 2.961 ± 0.008 | – | 1.238 ± 0.040 |
| DT3DPE | CAD | 0̲.̲5̲2̲7̲ ± 0̲.̲0̲0̲5̲ | 0̲.̲7̲2̲1̲ ± 0̲.̲0̲0̲7̲ | 0̲.̲8̲1̲1̲ ± 0̲.̲0̲0̲6̲ | 0̲.̲0̲4̲9̲ ± 0̲.̲0̲0̲6̲ | 2̲.̲9̲6̲8̲ ± 0̲.̲0̲0̲3̲ | 9.689 ± 0.093 | 1.249 ± 0.051 |
| Ours | – | 0.535 ± 0.006 | 0.728 ± 0.006 | 0.823 ± 0.007 | 0.058 ± 0.007 | 2.959 ± 0.003 | 9̲.̲7̲0̲1̲ ± 0̲.̲0̲9̲1̲ | 1.257 ± 0.053 |
We clarify that the PROX results provide qualitative results to show how our model performs on unseen scenes. Table 4 presents mean values of ablation results with standard deviations for all evaluated metrics. We have clarified this reporting in the revised discussion and use the ablation results to assess the contributions of individual components. (Added and highlighted in the Main Document: Pages 8, 9, and 10)
Table 4. Ablation study of the proposed LLMDT framework on the HUMANISE dataset.
| Method | Goal Dist. ↓ | Accuracy ↑ | Scene-consistent ↑ | FID ↓ | Quality ↑ |
|---|---|---|---|---|---|
| w/o LLM Semantic Reasoning | 0.112 ± 0.003 | 0.947 ± 0.006 | 99.79 ± 0.012 | 0.231 ± 0.008 | 5.624 ± 0.094 |
| w/o Object-Centric Scene Rep. | 0.097 ± 0.002 | 0.958 ± 0.004 | 99.83 ± 0.009 | 0.184 ± 0.006 | 5.701 ± 0.087 |
| w/o Trajectory Transformer | 0.088 ± 0.003 | 0.972 ± 0.005 | 99.71 ± 0.015 | 0.146 ± 0.007 | 5.786 ± 0.102 |
| w/o Trajectory Sensor | 0.072 ± 0.002 | 0.978 ± 0.003 | 99.81 ± 0.008 | 0.119 ± 0.005 | 5.852 ± 0.096 |
| w/o Pose-Aware Dual Transformer | 0.069 ± 0.002 | 0.975 ± 0.004 | 99.88 ± 0.006 | 0.127 ± 0.006 | 5.831 ± 0.091 |
| LLMDT (Full Model) | 0.053 ± 0.001 | 0.986 ± 0.005 | 99.95 ± 0.004 | 0.079 ± 0.106 | 5.941 ± 0.112 |
Q-1: What are the scene inputs for LLMDT and for each baseline (GT instance segmentation/labels versus detected objects)? Are all baselines given the same information?
A-1: We thank the reviewer for raising this concern. We have clarified that LLMDT uses object-centric voxel representations constructed from scene geometry, together with LLM-based semantic reasoning for target-object grounding and trajectory-conditioned local scene sensing. All methods are evaluated on the same test instances with consistent coordinate conventions and evaluation protocols. However, scene inputs follow each method's original configuration, and we do not assume that all baselines use identical object annotations or detection information. HumanML3D experiments follow the standard text-to-motion protocol without scene conditioning. (Added and highlighted in the Main Document: Pages 7 and 8)
W-1: The novelty relative to Cen et al. and MoMask is limited, and the fairness of the comparisons is unclear. It is also uncertain whether LLMDT and the baselines receive equivalent scene information. Several methods listed in Table 1, including HSI-GPT, UniHM, TSTMotion, and SceMoS, are not included in Table 2.
R-1: We thank the reviewer for highlighting the need to clarify LLMDT's methodological contributions, the fairness of the comparisons, and the coverage of the evaluated baselines. Cen et al. incorporate local scene sensing alongside LLM-based target grounding, object-centric scene representation, and trajectory planning. We recognize the overlap between these components and LLMDT and do not claim them as individually novel. The main distinction is LLMDT's integration of global trajectory guidance, trajectory-conditioned local geometric observations, and explicit articulated pose generation within a unified framework. Its pose-aware dual Transformer models root translation, global root orientation, and individual joint rotations. The masked Transformer predicts coarse motion structure, while the residual Transformer progressively refines articulated poses through hierarchical residual vector quantization (RVQ). Unlike MoMask, which focuses on general text-driven motion generation using hierarchical masked and residual Transformers, LLMDT conditions motion generation on grounded objects, planned trajectories, joint rotations, and local scene observations sampled along the predicted path. Thus, our contribution integrates scene-conditioned navigation and geometric reasoning with hierarchical articulated motion synthesis, rather than introducing LLM grounding or masked/residual Transformers independently.
Table 2. Performance comparison on the HUMANISE dataset. Best results are in bold, and second-best results are underlined.
Table 3. Performance comparison on the HumanML3D dataset. Best results are in bold, and second-best results are underlined.
To clarify evaluation fairness, we have specified the scene-input configurations and evaluation protocols in the revised manuscript. LLMDT uses structured semantic reasoning and object-centric voxel representations, while scene inputs for other methods follow their respective configurations, including annotated object instances and semantic labels, detected objects, and raw scene geometry where applicable. We do not assume that all methods receive identical representations. Instead, we distinguish their actual input requirements and use the same test instances, coordinate conventions, motion reconstruction procedures, and metric implementations wherever applicable. On HUMANISE, Table 2 compares scene-conditioned performance using Goal Distance, Contact, Scene-consistent Rate, Action Accuracy, Diversity, Multimodality, FID, and Quality Score. LLMDT achieves a Goal Distance of 0.053, a scene-consistent rate of 99.95%, an Action Accuracy of 0.986, an FID of 0.079, and a Quality Score of 5.941. These results provide evidence of its performance under the reported evaluation protocol, although they do not independently isolate the contribution of each architectural component. On HumanML3D, Table 3 reports standard text-to-motion metrics, including R-Precision, FID, MM Distance, Diversity, and Multimodality, without scene-specific geometric conditioning. LLMDT achieves R-Precision scores of 0.535, 0.728, and 0.823 at Top-1, Top-2, and Top-3, respectively, and an MM Distance of 2.959. (Added and highlighted in the Main Document: Pages 7 and 8)