acceptodds
Under review as a conference paper at ICLR 2027

Generative Interactions: Weaving Multiparty Human Motion with Bilevel Latent Dynamics

Abstract

Human social behaviour is not a collection of independent motions, but a jointly organised process in which group dynamics and individual variation continuously shape one another. Yet existing social motion models often prioritise plausible trajectories while leaving interaction state implicit, limiting their ability to transfer across groups, tasks, and partial-observation regimes. To address this gap, we introduce Bilevel Representations for Agent Interaction Dynamics (BRAID ), a hierarchical sequential latent-variable model for generative multi-person interaction. BRAID explicitly formulates social motion generation as a meta-transfer learning problem: shared interaction priors are learned across datasets and adapted through arbitrary context sets of observed people and joints. The model represents each scene through a group-level latent state that captures shared interaction dynamics and person-level latent states that capture individual behaviour conditioned on the evolving group context. This modelling choice enables coherent generation under full, sparse, or partial observations while exposing compact social-state vectors that can serve as an interface for downstream embodied-agent systems. We evaluate BRAID under a unified SMPL-based representation on social forecasting, tracking and in-filling, and response generation, using metrics that assess not only reconstruction accuracy but also realism, diversity, temporal alignment, and interpersonal coordination. We further analyse the hierarchical latent space, showing that it captures separable group- and individual-level structure.

Primary area:
generative models
Keywords:
generative modelling, sequential, time series, latent, hierarchy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k82SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes BRAID (Bilevel Representations for Agent Interaction Dynamics), a hierarchical sequential latent-variable model for generative multi-person interaction, framed as meta-transfer learning over arbitrary context sets. The social state is factored into a group latent that captures shared interaction dynamics and person latents that encode individual behaviour conditioned on the evolving group context. Scenes are represented in a unified SMPL-X parameterisation, and training combines a temporal Neural Process formulation with a precision-weighted variational posterior and a frozen pretrained motion VAE decoder, so that forecasting, tracking/in-filling, and response generation differ only in which joints and people are observed. Results across five social-motion datasets show that BRAID achieves competitive performance in generative multi-person interaction. The latent analysis further indicates that the group latent captures activity-level structure, while the person latents preserve individual identity across different groups.

Critical strengths

++ Under partial observation, the proposed method remains sufficiently stable and robust, outperforming the compared baselines on the majority of metrics.

++ The embedding analyses provide solid evidence that the bilevel latents are well structured and informative.

++ Code is provided, which facilitates reproducibility.

Critical weaknesses

-- In social forecasting, the SDTW and Cr. SDTW metrics are consistently worse than those of the compared methods. The authors should clarify whether this phenomenon indicates limitations of the approach in certain respects.

-- Although the IP metric appears favourable in the reported results, the provided video samples show that the generated multi-person interactions are rather simple and involve almost no physical contact, which may also account for the low interpenetration rate. The paper lacks experimental validation on complex, contact-rich multi-person interaction generation.

-- The experiments appear to be restricted to scenes with a fixed number of participants. Can the method handle scenarios where the number of people changes dynamically over the course of an interaction, such as participants joining or leaving?

-- A closely related work, Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation (ICLR 2026), is neither cited and discussed nor experimentally compared in the paper.

-- The reference list contains over 30 arXiv citations, many of which have been formally published at conferences or journals. These should be replaced with their official versions.

Questions for authors

See above.

Reviewer_sziJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes BRAID, a Neural Process-based multi-layer latent variable model for multi-person generation. It maintains a group latent and a person latent for each person in the interaction and uses a separately trained, frozen motion VAE to decode each latent into an SMPL-X motion sequence. BRAID was trained under forecasting, partner inpainting, and missing-joints setups, and can generate 2- to 5-person interactions under full and partial observations. The latent vectors are semantically meaningful, with ZgZ^g clustered according to different types of group activity, and ZindZ^{ind} able to identify the same individual across different groups. The method achieved SOTA performance in many metrics under the social forecasting, partner inpainting, and partner/diadic forecasting setups.

Critical strengths

The model treats forecasting, partner inpainting, and missing joints/person prediction as a unified problem and is able to predict 2- to 5- person interactions under arbitrary observed information. This provides a clean formulation and is a contribution to the multi-person motion modeling community.

Critical weaknesses

Novelty.

The method is a direct addition to the prior work MAGNet (Maluleke et al. 2025) and adds two capabilities: (1) a time-varying group-person latent state and (2) per-joint conditioning. The modeling ingredients: sequential latents (Singh et al., 2019), ladder VAE posterior (Sønderby et al., 2016; Kingma et al., 2016), and frozen motion VAE (Holden et al., 2016; Chen et al., 2023; Maluleke et al., 2025), are already established by prior works. Overall, the contributions are reasonable but incremental, and their value rest on evaluation results that shows these additions matter.

Performance.

The model does not outperform in key interaction metrics that the authors deem to be important (p.7 L363–366, "..prioritise interaction-level structure..."), namely, ΔRR\Delta RR, ΔDET\Delta DET, SDTWSDTW, and Cr.SDTWCr. SDTW.

  • In Table 2 full, for N=2/3/4, BRAID is worse in both SDTWSDTW and Cr.SDTWCr.SDTW metrics compared to MAGNet, the main baseline and the prior work that the paper builds upon. For N=5, the ΔDET\Delta DET, SDTWSDTW, Cr.SDTWCr. SDTW and FDFD is even worse than RS/NN, which are training-free methods that simply replays the closest data from the dataset.
  • In Table 2 partial, for all N (N=2/3/4/5), BRAID is worse in both SDTWSDTW and Cr.SDTWCr.SDTW metrics compared to RoHM. The authors need to explain why BRAID is worse in interaction metrics than a single-person motion reconstruction method.
  • In Table 3 partial, BRAID is worse in Cr.SDTWCr.SDTW metrics compared to RoHM in both datasets.
  • In Table 4 partial, BRAID is worse in both SDTWSDTW and Cr.SDTWCr.SDTW metrics compared to MAGNet.
Questions for authors
  1. What's the joint masking ratio, number of observed frames, number of predicted frames in Table 2-4? Without these, the numbers in the main experimental tables cannot be interpreted.
  2. In Table 2, why do the SDTW and Cr.SDTW of RoHM perform better than BRAID in ALL partial settings? Is joint information in the forecasting window also given to RoHM? How was RoHM, a single-person motion reconstruction method, able to predict future motion?
  3. In Table 3, what joint information is supplied to the model for the target person? For RoHM again, are some joints for the target person given to RoHM? If not, how does RoHM predict without any joint information of the target person? If this is true, then the task becomes reconstructing the target person given partial joint observation, not "partner inpainting".
  4. In Table 4, please clarify the experimental settings; why is BRAID only reported under the Partial setting, but compared with full observability settings baselines (RS, NN, R2R, MAGNet)? Why is a partial-observability setting comparable to full-setting results, and can you report the full-setting results (same config as MAGNet in Table 4) for BRAID?
  5. Can the authors report the confidence interval and number of seeds for reproducibility?
  6. What's the setup of MAGNet, RoHM, Social Processes, Duolando, R2R? Did the authors use the original weights, or retrained them? Are they converted to the same set of SMPL-X joint representations?
  7. How does the authors address SMPL-X information leakage? It is likely that ZindZ^{ind} carries personal information encoded in the SMPL-X body shape vector β\beta, which encodes personal information and remains constant across different groups.
1 comment

Thank you for the author's detailed response. I do not have additional questions regarding questions 1 to 3.

However, the author's response regarding the performance gap is not convincing enough. SDTW is still 5.195, significantly behind MAGNet's 1.560; MAGNet’s DIV of 0.52–0.71, although an improvement upon MAGNet, is still largely trailing against 65–90 for a random real clip. In addition, the author never explained in detail how the MAGNet baseline was run. So it's hard to compare these numbers with the correct context.

In addition, in question 6, the authors did not directly answer my question regarding MAGNet's setup. In question 7, the authors have confirmed, instead of resolving, my concern regarding body shape leaking identity. Body shape alone retrieves identity about 100% of the time, and the person latent only retrieves it 41.4% of the time, so the latent adds nothing beyond shape for identity.

Overall, I would like to keep my rating of weak rejection.

Reviewer_DR6XOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BRAID, a hierarchical sequential latent-variable framework for generative multi-person human interaction modeling. The key idea is to explicitly separate the interaction representation into a group-level latent state and multiple person-level latent states. The group latent is designed to capture shared interaction dynamics, such as coordination, interaction phase, and spatial relationships, while the person latents model individual behaviors conditioned on the group context. By representing observed people and joints as arbitrary context sets, the same framework can be applied to multiple tasks, including long-term motion forecasting, missing-joint completion, partner inpainting, response generation, and multi-person future prediction. The method is trained in a variational framework and further incorporates a pretrained temporal motion VAE to improve local motion quality. Experiments are conducted on several social-motion datasets covering conversation, dance, boxing, and multi-person interactions. The authors evaluate both motion reconstruction and generative quality using a broad range of metrics, and additionally analyze whether the proposed group and person latents capture different levels of interaction structure.

Critical strengths
  1. The paper studies an interesting and relatively underexplored problem of learning an explicit social-state representation for multi-person interaction, instead of treating interaction generation only as trajectory prediction.
  2. The hierarchical design of group-level and person-level latent variables is intuitive and well motivated. It provides a clear mechanism for modeling shared interaction dynamics together with individual behavioral variation.
  3. The proposed arbitrary-context formulation is flexible and allows a single model to handle several tasks, including forecasting, inpainting, missing-person completion, and partial-observation settings, without introducing separate task-specific architectures.
  4. The experimental evaluation is relatively comprehensive. In addition to conventional reconstruction metrics, the paper considers diversity, distributional quality, physical artifacts, temporal alignment, and cross-person coordination, which are important for evaluating stochastic multi-person interaction generation.
  5. The latent-space analyses and ablation experiments are useful. The results provide evidence that the group and person latents capture different aspects of the interaction, and the cross-group person retrieval and latent-switching experiments make the learned representations more interpretable than a purely implicit generative representation.
Critical weaknesses
  1. The proposed “group dynamics” are not sufficiently validated. The paper assumes that multi-person interaction can be represented by a group-level latent variable and person-level latent variables, but this decomposition is learned without explicit supervision. The current clustering, retrieval, and latent-space analyses only show that the latent variables contain different statistical information, rather than proving that the group latent actually represents meaningful interaction dynamics. Similar works such as DHVAE [1] have also explicitly modeled interaction-related latent variables. Stronger evidence, such as controlled latent intervention, interaction-level supervision, or semantic evaluation, is needed to justify this central claim.

  2. The current evaluation cannot adequately verify the claimed social dynamics. Metrics such as MPJPE, FD, diversity, CRQA, and cross-person Soft-DTW mainly measure motion similarity, distributional properties, or temporal correlation, but they cannot determine whether the semantic roles and causal relationships within an interaction are correct. For example, in boxing, these metrics cannot distinguish a meaningful attack-response interaction from two participants simultaneously performing similar punching motions. This problem is visible in the supplementary videos, where some participants appear to punch simultaneously in a stiff and poorly coordinated manner. A user study or stronger interaction-semantic evaluation is necessary to support the claim of improved social dynamics.

  3. The contribution of the proposed bilevel hierarchy is not sufficiently isolated from the complicated training pipeline. Different datasets and experiments use different versions of the model: some employ the pretrained motion VAE and multiple continuation stages, while several multi-person experiments and hierarchy ablations are conducted using only the base BRAID model. Therefore, it is difficult to determine whether the reported improvements come from the group/person latent decomposition itself or from the motion VAE, posterior dropout, code supervision, VAE-informed posterior, and other training techniques. A consistent full-pipeline ablation is needed.

  4. The training description is incomplete. The manuscript still contains the placeholder “[give the interval or mask distribution]” when describing the temporal in-filling training strategy. Since arbitrary context masking is one of the central components of BRAID, the exact masking protocol is important for reproducibility and for understanding the reported performance. This section must be completed and carefully checked.

  5. The practical motivation for the partial-observation and missing-joint settings is not sufficiently convincing. The experiments simulate partial observations by masking clean SMPL-X joints or entire participants after motion reconstruction. However, realistic partial observation in vision applications typically originates from occlusion, tracking failure, or uncertainty in monocular pose estimation. Artificially masking clean SMPL-X motion bypasses much of the actual perception challenge. The paper should better justify the intended application scenario or evaluate on more realistic noisy or incomplete observations.

  6. The qualitative results are not sufficiently convincing, and visual comparisons with competing methods are largely missing. The supplementary videos contain noticeable foot sliding, jitter, global drift, and weak or absent physical contact in several interaction examples. These artifacts directly affect the perceived quality and coherence of human interaction. Side-by-side qualitative comparisons with strong baselines would be important for demonstrating whether BRAID actually improves interaction quality rather than only selected numerical metrics.

  7. The claimed capability for general multi-party interaction is not sufficiently demonstrated qualitatively. Although the paper reports quantitative results for up to five participants, the supplementary demonstrations are dominated by dyadic interactions and only contain limited multi-person examples. The three-person examples generally involve small motion ranges and unclear interaction semantics, while convincing four- or five-person coordinated interactions are largely absent. More challenging polyadic examples are needed to support the claimed scalability to multi-party social motion.

  8. The claimed transferability and downstream usefulness of the learned “social state” are stronger than what is currently demonstrated. The cross-domain experiments mainly show that some coarse formation information can be retained when conditioning on motion from a new domain, while activity-specific interaction dynamics still degrade substantially. Moreover, the paper repeatedly suggests that the learned group latent could be useful for planners, policies, memory systems, or embodied agents, but no downstream experiment is provided. If reusable social-state representation is a major contribution, its utility should be demonstrated on an actual downstream task rather than inferred solely from latent-space visualization.

  9. Some paper references are not updated, such as Human-X [2], which is an ICCV highlight. Please have the author revise this.

[1] Geng et al., “Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation,” ICLR 2026. [2] Ji et al., “ Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis,” ICCV 2025.

Questions for authors

I would like to ask the author the following questions regarding the weaknesses:

  1. For the concept of group dynamics, how do you supervise it, and how do you justify its efficacy?
  2. Regarding the poor visualization, do you add smoothness to it? Or is it because of the extensive compression by the HVAE?
  3. The work seems not completed, is there any other experiments or statements needs to be updated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.