acceptodds
Withdrawn from ICLR 2027

Social Structure Matters in 3D Human-Human Interaction Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although text-to-motion generation has achieved strong progress in synthesizing realistic single-person motions from language, extending it to text-driven 3D human-human interaction (HHI) remains non-trivial, as HHI requires modeling the underlying **social structure** that governs phase progression, actor roles, and inter-actor coordination. In this paper, we formulate HHI generation as a social structure modeling and grounding problem: the model must first infer how an interaction unfolds and how the two actors coordinate their roles, and then realize this structure as continuous, physically plausible, and partner-aware 3D motion. To study how such structure should be modeled, we first examine the capability boundary of large language models (LLMs) for HHI generation. Our analysis shows that LLMs can think by recovering phase decompositions and partner-aware roles, but cannot directly move, as they fail to generate dynamic, physically plausible, and interaction-aware motion. This motivates our planner-executor paradigm, **Think with LLM, Move with Motion Skill.** The LLM planner converts implicit interaction semantics into motion-aligned social supervision by decomposing interactions into phases, assigning partner-aware actor roles, and aligning them with motion sequences. The motion executor then grounds the planned social structure into coordinated two-person motion by adapting a pretrained solo motion model with LoRA, previous-phase self-conditioning, and ego-relative partner conditioning. Together, our Solo-to-Social framework bridges social organization and motion realization, producing 3D HHI with improved phase consistency, role alignment, and partner-aware coordination.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, embodied systems, AI for creatives

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GNvxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses text-based human-human interaction generation. It proposes to use LLMs to decompose the input textual description into phase-based descriptions as high-level motion planning, and to use a fine-tuned large motion generation model, HY-Motion 1.0, with LoRA adaptation for "duet" motion generation. A motion representation named "social structure" is proposed, and the authors provide an automatically established HHI dataset with fine-grained motion-aligned annotations.

Critical strengths
  • The idea that LLMs can handle high-level planning but cannot perform low-level execution such as trajectory generation, and that an additional model (motion generator) is therefore needed for execution, has been supported by many prior works. I agree with this view. The overall approach therefore seems workable.
  • The authors provide a video demo, which helps reviewers understand the contribution.
Critical weaknesses

Overall, the method lacks details, which raises many questions and concerns. Before discussing these details, I would like to comment on novelty. I find the method relatively incremental. Segment/Parse the input description into event-based/phase-based atomics is a common way for long-term motion generation tasks. And LoRA adaptation is also known for motion stylization. However, I can see that the authors are trying to strengthen the concept of "Think with LLM, Move with Motion Skill," which is a positive sign. I now describe my specific questions and concerns.

Regarding Section 3.1, I question the argument that LLMs cannot perform low-level execution. I understand that Section 3.1 aims to explore the boundary of LLMs for low-level execution by using an LLM to generate low-level trajectories. However, I do not understand how the LLM performs this generation. The current version lacks details. Is a fine-tuned LLM for motion generation used as the test model, or is an LLM such as Qwen 3.5 directly called to output SMPL parameters for each frame? From Appendix C, I infer that it is the latter, that is, the LLM directly produces SMPL parameters. However, Figure 3 mentions HY-Motion without a specific citation, and it lacks a detailed experimental procedure and implementation details. This is not convincing and lacks an appropriate description and citation.

Regarding Section 3.2, I have concerns about the structure planning. From Equation 5, the input to the LLM includes not only the natural language description for this phase segment but also the phase label and "motion facts". The motion facts are numerical values. Considering that LLMs are not sensitive to numbers, I am not confident about directly feeding continuous numerical values, rather than discretized labels, as motion facts. I even suspect that the LLM cannot complete this task because the input numbers are too complex. To increase my confidence in the proposed method, I would like to see examples, including inputs and outputs. In addition, the descriptions of the attributes in the motion facts are too brief and lack corresponding details. How are they computed? I tried to search the appendix, but the appendix appears to be a rephrasing of the main text and still lacks details.

Section 3.2 introduces a core concept, phase. The idea of phase is very close to the idea of phrase in ECCV‘24 [1]. I suggest that the authors provide an appropriate citation and discussion. In addition, the related work on using LLMs for motion planning, in my view, misses several relevant papers, including [2, 3, 4]. Adding these papers could improve the completeness of the paper.

  • [1] Liu X, et al. Bridging the gap between human motion and action semantics via kinematic phrases. ECCV 2024.
  • [2] Wang W, et al. Sims: Simulating stylized human-scene interactions with retrieval-augmented script generation. ICCV 2025.
  • [3] Jiang J, et al. LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference. CVPR 2026.
  • [4] Lim D, et al. Event-driven storytelling with multiple lifelike humans in a 3d scene. ICCV 2025.

I am also concerned about method details. Section 3.3 mentions that the ego-relative representation includes "motion dynamics". I would like to know how the motion dynamics are represented. If this is not clearly explained, reproducibility will be low and the overall credit of the paper will be reduced.

The paper states that at inference time, the method rolls out two-person motion synchronously phase by phase. I would like to know how the motion execution time is obtained. It appears that the LLM motion plan does not include a time option (see Table 5). If there is no execution or duration information, how can HY-Motion 1.0 generate synchronized interaction motion? In this way, I question the soundness of the proposed method.

Regarding experimental results, I appreciate the video demo. However, the motion clips are short, the motions are simple, and some are not physically plausible, such as severe foot sliding. This suggests that the method was tested on a small dataset and that the motion complexity is insufficient. The paper may overclaim. To my understanding, HY-Motion 1.0 is supposed to output physically plausible motion, and it should not produce the situation shown in the left subfigure of Figure 1 (it shows that HY cannot understand the input text). Therefore, I am concerned that the proposed method is a negative optimization, and that the LoRA adaptor degrades the capability of the originally powerful motion generator.

In addition, I would like to know the results when ground truth is used as input. This is a necessary step to verify whether the evaluator is trained correctly and is always reported in motion generation tasks.

Minors: There are some issues with citation formatting. The authors should pay attention to using different LaTeX commands, such as \citep and \citet.

Questions for authors

Please see "Critical Weaknesses"

Reviewer_3tKEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies text-driven generation of two-person 3D human interactions. It proposes an explicit intermediate representation of “social structure,” comprising ordered interaction phases and phase-specific, partner-aware descriptions for both actors. During training-data annotation, phase labels and geometric facts extracted from observed motions constrain an LLM to produce descriptions aligned with the corresponding motion segments. At inference, the planner predicts this structure from the global interaction prompt alone. A Solo-to-Social executor adapts a pretrained HY-Motion backbone using LoRA, previous-phase self-conditioning, and ego-relative partner conditioning.

The paper evaluates the framework on sequences drawn from InterHuman and InterX, reporting improvements in phase-level text-motion retrieval, distributional fidelity, and geometric quality measures. Validation includes component ablations, qualitative comparisons, a user study, and an appendix comparison between motion-informed and text-only plans. Its intended conceptual contribution is that explicit temporal organization and complementary actor roles provide a useful interface between language planning and coordinated motion generation.

Critical strengths

1. A concrete and potentially reusable approach to motion-grounded interaction supervision.

The paper translates the intuition that interactions require temporally organized, complementary behaviors into a specific annotation and conditioning representation. This distinction is consequential because it concerns the quality of the supervision connecting language and motion, and could remain relevant when the underlying language or motion model changes.

Although the exact ablation construction requires clarification, this is suggestive evidence that grounding the annotations benefits motion quality beyond linguistic enrichment alone. This is the paper’s strongest candidate for a lasting insight.

2. A practical demonstration of transferring existing motion capabilities into interaction generation.

The framework builds on a pretrained solo motion model and includes a relevant existing-tools baseline, Social Planning + HYM1. The adapted executor improves over this baseline across the reported metrics, and the qualitative comparisons and human evaluation provide additional support for improved interaction quality. Appendix H further shows that the executor remains useful when driven by text-only plans, addressing an important difference between training annotation and deployment.

Together, these results support the practical value of the approach as a way to reuse existing motion skills. They provide a credible foundation for a stronger investigation of which semantic and geometric information is required for this transfer.

Critical weaknesses

1. The claimed planning-execution capability boundary is broader than the evidence, and the principal bottleneck remains unclear.

I consider an agentic approach a promising direction for human-human interaction generation, particularly when paired interaction data are limited. However, the current experiments do not provide a sufficiently clear diagnosis of why existing tools fail or which part of the proposed system limits performance. The capability analysis in Section 3.1 evaluates one Qwen configuration for direct SMPL-based motion generation against a specialized motion generator. This establishes a limitation of the tested model and interface; it does not establish a general boundary under which LLMs can plan but cannot execute motion. The observed failure could depend on model capability, motion representation, prompting, or the execution interface.

Appendix H provides a useful comparison between text-only and motion-informed reference plans using the same executor, demonstrating that planning quality affects downstream generation. However, this comparison does not distinguish incorrect phase organization, incorrect actor roles, and failure to execute an otherwise valid plan. Similarly, removing a component in Table 2 demonstrates its usefulness, but does not identify which component offers the greatest remaining opportunity for improvement. Consequently, the paper supports the proposed system more clearly than it supports a general explanation of the planning and execution difficulties in interaction generation.

  1. The distinct benefit of the proposed interaction representation is insufficiently established.

Coordination, actor-specific semantics, and language-guided motion execution have substantial precedents. A particularly relevant omitted work is InterControl [1], which already uses an LLM to generate individual motion descriptions and timed joint-level interaction constraints, realized through a pretrained solo motion model. The potentially important distinction here is the use of motion-aligned phases and complementary actor descriptions as supervision. However, the experiments do not sufficiently establish which limitation of existing interaction plans this representation resolves, or distinguish its benefits from richer annotations, temporal segmentation, and executor adaptation.

I acknowledge that Planning + HYM1 and the same-backbone ablations provide evidence beyond simply adopting a strong pretrained model. Nevertheless, cross-method comparisons also involve different motion priors and pretraining data, while the planning ablation does not clearly hold annotation detail and generation granularity fixed. This leaves the contribution of the representation itself incompletely isolated.

The evidence for precise interaction grounding also requires clarification. Although the full-body motion representation is defined, the contents and temporal extent of the partner-conditioning state are not explicit. The additional relational losses constrain root distance and facing, while the Contact metric measures the frequency of penetration-free proximity rather than whether the intended body parts interact at the appropriate times. The user study provides useful complementary evidence, but does not isolate which representational choice improves these behaviors. In addition, the phase-level retrieval protocol does not sufficiently explain how descriptions, segmentation, and alignment are made comparable across methods. These issues limit my confidence that the reported gains demonstrate the specific interaction capability attributed to the proposed representation.

References

[1] Wang, Z., Wang, J., Li, Y., Lin, D., & Dai, B. (2024). Intercontrol: Zero-shot human interaction generation by controlling every joint. Advances in Neural Information Processing Systems, 37, 105397-105424.

Questions for authors

The following clarifications and targeted analyses could materially affect my assessment. My main interest is in understanding what the proposed representation contributes and which failures it resolves. Although resolving them might requires substantial experiments and change in paper writing, I encourage the authors to address them as much as they could during the rebuttal period.

1. What information is available to the executor throughout inference?

Please specify the exact contents and temporal extent of the partner state (sk−1s_{k-1}) in Eqs. (8)/(22), including whether it contains root quantities, full-body joint features, or both. How are the initial poses, relative positions, and orientations of the two actors obtained? What conditions are used for the first phase, and how are phase durations determined and synchronized?

2. Which limitation of existing interaction representations does the proposed representation address?

In particular, how does the motion-aligned phase-and-role representation improve upon InterControl’s timed interaction plans? Could the authors provide a controlled comparison or targeted analysis that isolates this difference? Please also clarify what remains fixed in “w/o social planning,” including training segments, actor-description detail, executor adaptation, and generation lengths.

3. Where do the remaining failures originate, and how dependent is the diagnosis on the chosen models?

Building on Appendix H, can the authors distinguish failures caused by incorrect plans from failures to realize valid plans? Do the conclusions persist with another capable LLM or motion-generator backbone when the other component is held fixed? If such evidence is unavailable, which claims should be restricted to the tested model and interface?

4. How is phase-level retrieval made comparable across methods?

Are all generated motions evaluated against the same reference phase descriptions, or against descriptions produced by their respective planners? How are whole-sequence baseline outputs segmented and aligned when generated phase counts or boundaries differ from the references?

5. What evidence establishes correct body-part contacts and their timing?

Appendix A defines Contact as a proximity-based frame fraction, and Appendix H acknowledges that a higher score does not establish more accurate contact timing. What evidence shows that the intended joint pairs interact at the appropriate stages, and that separation is maintained during release or noncontact phases?

6. Can the approach generalize to multi-person?

If so, what adaptation shall be made then?

Reviewer_v6FBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames text-driven 3D human–human interaction generation as explicit “social structure” modeling: an LLM decomposes a global prompt into phases and complementary actor roles, then a phase-wise motion executor adapts HY-Motion with LoRA fine tuning. Experiments on InterHuman and InterX report improved semantic alignment and physical metrics over the baselines.

Critical strengths
  • Useful structural formulation. Explicitly representing phase progression and actor-specific roles addresses a real limitation of monolithic HHI generation (lines 44–53, 253–287). This is a sensible abstraction for interactions with asymmetric or changing roles, despite the lack of timining control.

  • Promising, parameter-efficient results. The method reports the best mean result on all five metrics (lines 419–425) using only approximately 8M trainable parameters (lines 373–377). The ablation studies are also extensive to understand the components.

Critical weaknesses
  • The planner lacks temporal grounding, leaving inference underspecified. The social plan contains only an ordered phase type and actor-specific descriptions (lines 275–282). Neither the plan representation nor the inference prompt specifies phase duration or transition time (lines 1097–1115), which I belive is important to actually use the phase decomposition. How can a phase be valid without time information? Nevertheless, Eq. 7 assumes a duration T_{k−1} to locate the end of the preceding phase and initialize the next one (lines 312–317). During annotation, durations come from ground-truth motion segments and are mapped to duration buckets (lines 983–986), but the paper does not explain how these durations are obtained during text-only inference. This is a critical component to understand how the phase works and if this decomposition is general and LLM results are reliable.

  • fig 2: phase decomposition analysis is unclear. I don’t quite get the logic of how this proves that LLM can do phase reasoning well? what is the ground truth? and what this plot shows? “Qwen3.5 can infer asymmetric actor roles with high cross-phase consistency”, what does cross-phase consistency mean here? With reference to what?

  • The motion quality in supp video does not seem to be very good. Supp video has lots of penetrations, e.g. Solo_to_Soical_Demo_b.html, this makes it suspicious that if the claimed phase decomposition can actually work. also the submitted supp videos contain very short motion clips, it does not really demonstrate the claimed phase of approach, contact, relase phases. One would expect demos of longer motions.

  • presentation problem: overall it is hard to follow this paper with lots of missing details. e.g. fig 2 setup and how does that prove LLM is able to reason phases well. and fig. 4 is hard to digest and it is unclear how T_k is obtained in eq 7 and eq 21.

  • The related-work discussion omits relevant temporally structured motion literature, including BABEL, FlowMDM, UniMotion, and FrankenMotion. Although these works primarily address single-person motion, they establish frame-aligned language annotations, temporally localized text conditioning, and sequential motion composition. They are important for positioning the proposed phase representation and for clarifying why the present planner predicts only an ordered phase list without explicit durations or transition times.

additional references:

  • FrankenMotion: Part-level Human Motion Generation and Composition. arXiv, 2026.

  • BABEL: Bodies, Action and Behavior With English Labels. CVPR, 2021.

  • Seamless Human Motion Composition with Blended Positional Encodings. CVPR, 2024. (FlowMDM)

  • UniMotion: Unifying 3D Human Motion Synthesis and Understanding. 3DV, 2025.

Questions for authors
  • How is each phase duration selected during inference? Is it fixed, sampled, predicted, or taken from the reference motion? Do any results use ground-truth sequence lengths, phase boundaries, or per-phase durations?

  • In Fig. 2, what are the reference labels and exact definitions of phase correctness, role accuracy, “role stability,” and “cross-phase consistency”? What dataset size and embedding model are used?

  • Why was LoRA selected over full fine-tuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.