- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses text-based human-human interaction generation. It proposes to use LLMs to decompose the input textual description into phase-based descriptions as high-level motion planning, and to use a fine-tuned large motion generation model, HY-Motion 1.0, with LoRA adaptation for "duet" motion generation. A motion representation named "social structure" is proposed, and the authors provide an automatically established HHI dataset with fine-grained motion-aligned annotations.
Critical strengths
- The idea that LLMs can handle high-level planning but cannot perform low-level execution such as trajectory generation, and that an additional model (motion generator) is therefore needed for execution, has been supported by many prior works. I agree with this view. The overall approach therefore seems workable.
- The authors provide a video demo, which helps reviewers understand the contribution.
Critical weaknesses
Overall, the method lacks details, which raises many questions and concerns. Before discussing these details, I would like to comment on novelty. I find the method relatively incremental. Segment/Parse the input description into event-based/phase-based atomics is a common way for long-term motion generation tasks. And LoRA adaptation is also known for motion stylization. However, I can see that the authors are trying to strengthen the concept of "Think with LLM, Move with Motion Skill," which is a positive sign. I now describe my specific questions and concerns.
Regarding Section 3.1, I question the argument that LLMs cannot perform low-level execution. I understand that Section 3.1 aims to explore the boundary of LLMs for low-level execution by using an LLM to generate low-level trajectories. However, I do not understand how the LLM performs this generation. The current version lacks details. Is a fine-tuned LLM for motion generation used as the test model, or is an LLM such as Qwen 3.5 directly called to output SMPL parameters for each frame? From Appendix C, I infer that it is the latter, that is, the LLM directly produces SMPL parameters. However, Figure 3 mentions HY-Motion without a specific citation, and it lacks a detailed experimental procedure and implementation details. This is not convincing and lacks an appropriate description and citation.
Regarding Section 3.2, I have concerns about the structure planning. From Equation 5, the input to the LLM includes not only the natural language description for this phase segment but also the phase label and "motion facts". The motion facts are numerical values. Considering that LLMs are not sensitive to numbers, I am not confident about directly feeding continuous numerical values, rather than discretized labels, as motion facts. I even suspect that the LLM cannot complete this task because the input numbers are too complex. To increase my confidence in the proposed method, I would like to see examples, including inputs and outputs. In addition, the descriptions of the attributes in the motion facts are too brief and lack corresponding details. How are they computed? I tried to search the appendix, but the appendix appears to be a rephrasing of the main text and still lacks details.
Section 3.2 introduces a core concept, phase. The idea of phase is very close to the idea of phrase in ECCV‘24 [1]. I suggest that the authors provide an appropriate citation and discussion. In addition, the related work on using LLMs for motion planning, in my view, misses several relevant papers, including [2, 3, 4]. Adding these papers could improve the completeness of the paper.
- [1] Liu X, et al. Bridging the gap between human motion and action semantics via kinematic phrases. ECCV 2024.
- [2] Wang W, et al. Sims: Simulating stylized human-scene interactions with retrieval-augmented script generation. ICCV 2025.
- [3] Jiang J, et al. LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference. CVPR 2026.
- [4] Lim D, et al. Event-driven storytelling with multiple lifelike humans in a 3d scene. ICCV 2025.
I am also concerned about method details. Section 3.3 mentions that the ego-relative representation includes "motion dynamics". I would like to know how the motion dynamics are represented. If this is not clearly explained, reproducibility will be low and the overall credit of the paper will be reduced.
The paper states that at inference time, the method rolls out two-person motion synchronously phase by phase. I would like to know how the motion execution time is obtained. It appears that the LLM motion plan does not include a time option (see Table 5). If there is no execution or duration information, how can HY-Motion 1.0 generate synchronized interaction motion? In this way, I question the soundness of the proposed method.
Regarding experimental results, I appreciate the video demo. However, the motion clips are short, the motions are simple, and some are not physically plausible, such as severe foot sliding. This suggests that the method was tested on a small dataset and that the motion complexity is insufficient. The paper may overclaim. To my understanding, HY-Motion 1.0 is supposed to output physically plausible motion, and it should not produce the situation shown in the left subfigure of Figure 1 (it shows that HY cannot understand the input text). Therefore, I am concerned that the proposed method is a negative optimization, and that the LoRA adaptor degrades the capability of the originally powerful motion generator.
In addition, I would like to know the results when ground truth is used as input. This is a necessary step to verify whether the evaluator is trained correctly and is always reported in motion generation tasks.
Minors: There are some issues with citation formatting. The authors should pay attention to using different LaTeX commands, such as \citep and \citet.
Questions for authors
Please see "Critical Weaknesses"