acceptodds
Withdrawn from ICLR 2027

SocialDiT: Social-Aware Diffusion Transformer for Human-Human Interaction Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Two people performing the same action move differently depending on who they are and how well they know each other, yet text-driven dyadic motion generators condition on the action description alone. We study text-driven dyadic interaction generation additionally conditioned on per-actor Big Five personality and pair-level familiarity. We introduce SocialDiT, a flow-matching diffusion transformer that feeds the caption through text cross-attention and the social attributes through a Social-Aware Cross-person Attention (SACA) module, which modulates the attention between the two participants. We show that on Inter-X, where every subject pair carries a single fixed label, probes that predict these attributes from motion are confounded by pair identity. We therefore test whether the social input is used and controllable with interventions, a controlled familiarity probe, evaluator-free interaction geometry, and a user study. On a subject-disjoint split, SocialDiT reduces FID from to relative to the strongest socially conditioned baseline. It responds to interventions on its social input and to controlled changes in familiarity, reproduces interpersonal distances more closely than the baselines, and is preferred in a user study, including on matching the target familiarity. A controlled comparison of injection sites further shows that routing social attributes through cross-person attention best reproduces interpersonal geometry, suggesting that social context is most naturally injected where the two participants interact.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Human Interaction Generation, Human Motion Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t9C3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a latent flow-matching based diffusion transformer model SocialDiT to generate paired human motion sequence conditioned on the text prompts, the personality of two person and the paired familiarity. They modify the cross-attention modules by including gating mechanism to adaptively inject the personality information and familiarity information for motion sequence generation, which is referred as Social-awareness cross-attention in the paper. They train and test their model on Inter-X dataset, proving their advantages through multiple perspectives.

Critical strengths
  1. The paper clearly describe their objective and their methods, making it easy to understand.
  2. The proposed social-awareness cross-attention mechanism is intuitive and well-motivated.
  3. The proposed method demonstrates clear improvements over existing baselines on the Inter-X dataset.
Critical weaknesses
  1. The evaluation is relatively limited, as both training and testing are conducted on a single Inter-X dataset. The analysis is also insufficient to fully demonstrate the effectiveness and generalizability of the proposed method.
  2. The rationality of the task setting requires further justification. It would be great to provide empirical evidence demonstrating that Big Five personality traits and paired familiarity significantly influence human interaction motion patterns.
  3. The improvement seems to be from the combination of flow-matching losses, gating mechanism and the data augmentation, which looks too engineering and lacks deep analysis to provide new insights to this field.
  4. The generalization to unseen action categories is not discussed.
Questions for authors
  1. What is the number of parameters for your SocialDiT in comaprison with InterGen and InterMask. How do you initialize SocialDiT?
  2. Figure 4 is too small to distinguish the difference between your methods with other methods.
Reviewer_iU6KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses text-driven two-person motion generation, where an action description does not specify how personality and familiarity should shape the interaction, by conditioning cross-person attention on these social attributes.

Critical strengths
  1. Separating action descriptions from social attributes allows the same action to be generated with different interpersonal styles, which is useful for character animation.

  2. The analysis of identity-confounded personality evaluation explains how recognizing a participant pair can be mistaken for recovering personality, providing a useful warning for evaluating social control.

Critical weaknesses
  1. Social conditioning and feature modulation already appear in the cited work, but the comparison of injection sites does not establish that placing modulation in cross-person attention expresses social relationships more accurately, leaving the benefit of the principal modeling change unclear.

  2. Overall generation quality remains largely unchanged when social modulation is removed, so the headline quality advantage cannot establish the value of the social module and must be assessed separately from social-attribute control.

  3. The personality metric predicts the average traits of both participants, which is unchanged when their trait assignments are swapped, so it cannot verify whether a specified personality is expressed by the intended participant.

Questions for authors

No additional questions.

Reviewer_1CoAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SocialDiT for text-driven two-person motion generation conditioned on per-person Big Five traits and pair-level familiarity. Social attributes are injected into cross-person attention through the SACA module. The paper evaluates realism, social-input interventions, familiarity control, interpersonal geometry, and human preference on a subject-disjoint Inter-X split.

Critical strengths
  1. Careful treatment of evaluation confounding. The paper identifies that attribute probes can memorize pair identity and therefore introduces a subject-disjoint split and several complementary evaluations.

  2. Strong quantitative performance. SocialDiT improves FID from 1.90 to 1.03 over the strongest socially conditioned baseline and achieves the best interpersonal-distance metrics. The injection-site ablation and user study further support the proposed design.

Critical weaknesses
  1. Personality control is not semantically established. Each pair has only one fixed personality tuple, and the group-wise personality probe is negative even for real motion. The controlled experiment and user study validate familiarity, but not whether changing an individual Big Five trait produces the intended personality expression. Thus, the evidence mainly establishes condition dependence rather than correct personality control.

  2. Physical artifacts remain visible and insufficiently evaluated. Close-contact visualizations contain apparent body-overlap/contact ambiguities, while Table 2 reports substantially larger ground penetration, floating, and skating errors than real motion. SocialDiT also lacks an explicit physical constraint and does not report inter-person body-collision or contact-accuracy metrics. Therefore, good interpersonal-distance statistics do not guarantee physically plausible interactions.

Questions for authors
  1. Is there any controlled quantitative or human evaluation showing that editing individual Big Five traits produces the intended perceived personality while holding the caption, familiarity, and sampling noise fixed?

  2. Can the authors report inter-person penetration and intended-contact accuracy, particularly for hugging, kicking, or other close-contact actions?

  3. The supplement states that the default guidance weights were selected on the test split. What performance is obtained when these weights are selected exclusively using the validation split?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.