acceptodds
Under review as a conference paper at ICLR 2027

DyaChain: Strategizing Concurrent, Holistic and Dyadic Motion Generation via Progressive Interaction Chain

Abstract

Generating conversational motion is fundamental to creating lifelike virtual humans for immersive communication and human–computer interaction. While existing methods enable reactive individual movement synthesis, they overlook the **concurrent** and **holistic** behaviors of both speakers in an interactive manner. Recent datasets also rarely provide aligned annotations at both the interlocutor and interaction levels. To address these limitations, we introduce **HOWDY**, a dyadic conversation dataset that augments synchronized speech and holistic 3D motion from both participants with fine-grained body-part motion descriptions and hierarchical interaction annotations. Additionally, we propose **DyaChain**, a holistic motion generation framework to concurrently model dyadic speakers by strategizing a **Progressive Interaction Chain** approach. To ensure spatiotemporal coherence between the two concurrently evolving motion streams while preserving asynchronous action with response dynamics, the chain progresses through three levels: conversational states, dyadic relations, and motion events. Here, we first derive temporally aligned conversational states to delineate speaking and response periods, and then reason dyadic relations between interlocutors. Conditioned on these cues, we design a Temporal-Relational Interactor **TRI** module to capture cross-individual body-part dependencies and asynchronous responses using causal counterpart attention and learnable response delays. Moreover, we retrieve static posture descriptions from the HOWDY and then infer part-aligned dynamic motion events, motivating a newly proposed Part-Aligned Semantic Routing **PSR** module. PSR decomposes holistic motion into semantically coherent body regions and injects static and dynamic motion semantics into the corresponding motion representations to guide region-specific dynamics. Extensive experiments demonstrate that **DyaChain** outperforms SOTA methods for dyadic motion generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
gesture generation, dyadic interaction, human motion modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_76ndOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HOWDY, a new two-person conversation dataset that contains synchronized speech, two-person motion, and annotations for body-part motion and interaction. It additionally introduces DyaChain, a full-body (body + hands + face) motion generation method from synchronized speech and a paired transcript. Dyachain reasons about a progressive interaction chain: dyadic relations, motion events, and conversational states. The paper further proposes Temporal-Relational Interactor and Part-Aligned Semantic Routing, which gate interaction features and inject motion events + conversational state information into the generation process. DyaChain sets the new SOTA in the HOWDY dataset across gesture, face, and holistic interaction generation metrics.

Critical strengths
  1. The HOWDY dataset pairs full-body movements of the two interactants with body part-level motion annotations, frame-level conversational states, and relation attributes; Its rich and hierarchical annotations position it uniquely in the speech-interaction dataset realm, which is a sound contribution to the community if open-sourced.
  2. DyaChain is the first method that generates full-body interaction (body + hand + face) from speech and text, and its evaluation is extensive and shows strong empirical results.
Critical weaknesses

Unfair comparison and unclear gains: DyaChain makes use of motion transcripts, LLM-labeled motion events, and dataset-retrieved motion descriptions, which the baseline methods cannot leverage. When removing text input (w/o PSR in Table 2, P-FHD 10.920 vs Co3 10.055) or using random text (P-FHD 10.255 vs Co3 10.055), the P-FHD metric is worse than the Co3 baseline, and the P-FGD is similar or worse. Combined with the absence of confidence intervals, the fact that the proposed Dyachain is only tested on the in-house HOWDY dataset and a shot-level data split (the same video might appear in both the training and test sets), the SOTA metrics are not backed up by significant evidence.

Questions for authors
  1. What input configurations are the baselines evaluated under? Can the transcript, relation attributes, and part-level motion annotation be provided to ConvoFusion and DyaDiT (which are able to receive text input) and evaluated again? "w/o PSR" in Table 2 is inferior to GestureLSM and Co3 Gesture.
  2. Since train/test/validation is split at the shot level, does the same pair of persons appear in both train and test sets?
  3. Can you repeat the experiments in Tables 1, 2, and 3 and report variance and confidence intervals?
  4. Learnable response delays are claimed but not analyzed. Can the authors provide analysis to support this claim? For example, show the distribution of the listener's response latency in the generated interactions, and visualize this along with the learned BtB_t?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors of W1 (Part 1) and Q1

We thank Reviewer 76nd for your thoughtful review and constructive feedback. We are glad that you recognized that HOWDY's rich, hierarchical annotations "position it uniquely in the speech-interaction dataset realm", and found DyaChain's evaluation extensive with strong empirical results. To fulfill HOWDY's value to the community, we confirm that we will publicly release the full dataset and codebase (with the core code of DyaChain shared at https://anonymous.4open.science/r/DyaChain-FA5A/). Below, we address your concerns and questions point by point.

W1 (Part 1) & Q1: Baseline input configurations, text-augmented baselines, and "w/o PSR" vs. "w/o Text"

R1: Thanks for raising these important questions. We clarify the baseline input configurations, evaluate text-augmented baselines, and distinguish w/o PSR from w/o Text point by point as below.

a) For fair comparisons, as described for Table 1 (Section 5.2, Lines 409–414), all methods retain their original audio encoders, and speech transcripts are also provided to baselines that originally take them as input (SemGes, GestureLSM, ConvoFusion, and Beyond Talking).

b) Following your suggestion, to examine whether our gains mainly stem from extra semantic inputs, we newly added Table 3 (Section 5.2, Lines 435–447), augmenting ConvoFusion and DyaDiT under the same text protocol as DyaChain—trained with dataset annotations and evaluated with the identical inferred Progressive Interaction Chain descriptions encoded via frozen FLAN-T5-base and injected through each baseline's original conditioning mechanism (cross-attention for both methods), denoted as ConvoFusion$^{+}$ and DyaDiT$^{+}.Asshownbelow(with95. As shown below (with 95% CIs over 10 sampling runs), these inferred descriptions consistently improve P-FGD, P-FED, and P-FHD for both baselines, confirming the transferable benefit of our progressive interaction reasoning. Still, under identical semantic inputs, DyaChain clearly outperforms ConvoFusion^{+}$ and DyaDiT$^{+}$, underscoring the effectiveness of PSR and TRI.

MethodsP-FGD ↓\downarrowDiv →\rightarrowP-FED ↓\downarrowP-FHD ↓\downarrow
ConvoFusion1.194$^{\pm0.010}$133.859$^{\pm0.590}$6.592$^{\pm0.089}$11.746$^{\pm0.113}$
ConvoFusion$^{+}$1.083$^{\pm0.026}$137.931$^{\pm0.740}$5.558$^{\pm0.059}$10.355$^{\pm0.094}$
DyaDiT1.211$^{\pm0.026}$150.448$^{\pm0.923}$7.956$^{\pm0.097}$11.192$^{\pm0.128}$
DyaDiT$^{+}$1.116$^{\pm0.023}$157.995$^{\pm0.524}$7.113$^{\pm0.029}$10.108$^{\pm0.045}$
DyaChain (w/o Text)1.063$^{\pm0.027}$137.856$^{\pm0.514}$5.074$^{\pm0.092}$9.758$^{\pm0.180}$
DyaChain (Ours)0.824$^{\pm0.021}$140.964±0.251^{\pm0.251}4.009±0.057^{\pm0.057}7.482^{\pm0.066}\

c) In our original ablation table (now Table 4, Lines 453–472), w/o PSR removes PSR altogether—not only dropping text injection (GRCA), but also replacing part-specific motion encoders and body-part identity embeddings (Table 6) with a single shared motion encoder. Collapsing heterogeneous upper-body, finger, and facial parameters into a shared encoder may impair both motion representation and subsequent cross-person interaction modeling in TRI, causing w/o PSR to underperform GestureLSM and Co$^{3}$Gesture on P-FHD. To isolate the effect of removing motion descriptions alone, we newly added DyaChain (w/o Text) in Tables 3 and 4, which removes GRCA while retaining part-specific encoding and identity embeddings. As shown below and compared with Table 1, without motion descriptions, DyaChain (w/o Text) already achieves lower P-FGD, P-FED, and P-FHD than GestureLSM, Co3^{3}Gesture, and all other unaugmented baselines. Meanwhile, in w/ Random text (Table 5, formerly Table 3), randomly mismatched static posture priors may conflict with the current conversation, causing the model to generate poses that deviate further from the ground-truth distribution than DyaChain (w/o Text).

MethodsP-FGD ↓\downarrowDiv →\rightarrowP-FED ↓\downarrowP-FHD ↓\downarrow
w/o PSR1.135$^{\pm0.033}$139.082$^{\pm0.548}$6.027$^{\pm0.136}$10.858$^{\pm0.166}$
DyaChain (w/o Text)1.063$^{\pm0.027}$137.856$^{\pm0.514}$5.074$^{\pm0.092}$9.758$^{\pm0.180}$
DyaChain (Full)0.824$^{\pm0.021}$140.964±0.251^{\pm0.251}4.009±0.057^{\pm0.057}7.482^{\pm0.066}\
AuthorsOpenReview ↗

Rebuttal by Authors of W1 (Part 2) and Q2

R2: Thanks for raising this concern. We clarify our shot-level split protocol on HOWDY and report a newly added zero-shot cross-dataset evaluation on the Seamless Interaction dataset [A] point by point as below.

a) Yes, the same pair of interlocutors may appear in both the training and test sets under our shot-level split. Nevertheless, all 6-second clips segmented from the same continuous shot are strictly assigned to a single split to prevent temporal-window overlap (Appendix A.1.3, Lines 925–930), and when partitioning the test set, we also avoid including test clips that are temporally adjacent to training clips. Moreover, all baselines are trained and evaluated on the identical split, and DyaChain takes only audio and transcripts without speaker-identity conditioning, so the comparison remains fair.

b) Regarding your concerns, we newly conducted a zero-shot cross-dataset evaluation on the Seamless Interaction dataset [A] to examine whether DyaChain's improvements carry over to an existing dyadic benchmark with unseen pairs of interlocutors. Following our data protocol, we synchronized the dyadic audio, word-level transcripts, and motion sequences from batch 0000 of the naturalistic test split (naturalistic/test/0000) of Seamless Interaction into 6-second (90-frame) clips at 15 FPS, yielding 7,013 valid dyadic clips. All compared baselines and DyaChain (trained on HOWDY) were directly evaluated across all 7,013 clips without any target-domain fine-tuning. During inference, DyaChain inferred its Progressive Interaction Chain conditions from Seamless Interaction's audio and transcripts while retrieving static posture priors from the HOWDY training set, without requiring any target-domain annotations.

c) Since Seamless Interaction provides SMPL-H body and hand poses without 50D FLAME expression parameters, facial and FLAME-dependent holistic metrics (FED, MSE, and FHD) are not applicable; we therefore report all four upper-body and hand gesture metrics (P-FGD, S-FGD, BC, and Div) across 10 sampling runs. As shown in the table below, DyaChain consistently outperforms all compared baselines across every metric under this zero-shot setting, confirming strong generalization to unseen dyadic conversations. We have added this experiment and discussion in the revised manuscript (Appendix A.4.1, Lines 1099–1116, Table 7).

MethodsP-FGD ↓\downarrowS-FGD ↓\downarrowBC ↑\uparrowDiv →\rightarrow
GT------91.794
SemGes5.451$^{\pm0.000}$7.939$^{\pm0.000}$0.597$^{\pm0.000}$51.641$^{\pm0.000}$
GestureLSM3.067$^{\pm0.038}$3.950$^{\pm0.028}$0.607$^{\pm0.000}$97.457$^{\pm0.138}$
GlobalDiff3.162$^{\pm0.028}$4.623$^{\pm0.041}$0.605$^{\pm0.000}$75.412$^{\pm0.151}$
ConvoFusion3.693$^{\pm0.055}$6.278$^{\pm0.089}$0.598$^{\pm0.000}$85.682$^{\pm0.129}$
Co$^{3}$Gesture4.187^{\pm0.023}\5.644$^{\pm0.033}$0.603$^{\pm0.001}$105.633$^{\pm0.114}$
DyaDiT3.628$^{\pm0.046}$4.570$^{\pm0.047}$0.592$^{\pm0.000}$98.779$^{\pm0.287}$
Beyond Talking3.026$^{\pm0.041}$3.511$^{\pm0.045}$0.606$^{\pm0.001}$98.980$^{\pm0.182}$
DyaChain (Ours)1.505$^{\pm0.017}$2.273±0.015^{\pm0.015}0.609±0.000^{\pm0.000}88.266^{\pm0.209}\

[A] Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset, in arXiv, 2025.

AuthorsOpenReview ↗

Rebuttal by Authors of W1 (Part 3) and Q3

W1 (Part 3) & Q3: Repeated evaluations and confidence intervals across Tables 1, 2, and 3

R3: Thanks for your suggestion. Following your recommendation, we repeated all evaluations across 10 independent sampling runs and updated Tables 1, 4, and 5 in the revised manuscript (corresponding to Tables 1, 2, and 3 in the initial submission), as well as the newly added Tables 2, 3, and 7, to report means and 95% confidence intervals (±0.000\pm0.000 for deterministic SemGes). As shown in the updated tables below, sampling variability remains small across runs, and DyaChain's 95% CIs do not overlap with those of any competing baseline across all metrics in Table 1, nor with ablated variants on pairwise gesture and holistic metrics in Tables 4 and 5, confirming that our improvements are statistically reliable.

Updated Table 1 (SOTA comparison on HOWDY; G-Div and F-Div denote Gesture and Face Diversity):

MethodsP-FGD ↓\downarrowS-FGD ↓\downarrowBC ↑\uparrowG-Div →\rightarrowP-FED ↓\downarrowS-FED ↓\downarrowMSE ↓\downarrowF-Div →\rightarrowP-FHD ↓\downarrowS-FHD ↓\downarrow
GT------143.942------786.462----
SemGes3.936$^{\pm0.000}$3.178$^{\pm0.000}$0.589$^{\pm0.000}$107.845$^{\pm0.000}$8.239$^{\pm0.000}$3.253$^{\pm0.000}$10.663$^{\pm0.000}$583.897$^{\pm0.000}$14.596$^{\pm0.000}$7.536$^{\pm0.000}$
GestureLSM1.492$^{\pm0.020}$2.952$^{\pm0.085}$0.604$^{\pm0.001}$134.080$^{\pm0.704}$5.197$^{\pm0.075}$0.923$^{\pm0.008}$10.905$^{\pm0.015}$718.290$^{\pm3.838}$10.067$^{\pm0.083}$5.146$^{\pm0.088}$
GlobalDiff1.896$^{\pm0.039}$1.264$^{\pm0.049}$0.597$^{\pm0.001}$138.545$^{\pm0.768}$5.219$^{\pm0.037}$1.580$^{\pm0.011}$11.787$^{\pm0.045}$1021.156$^{\pm6.332}$17.623$^{\pm0.150}$8.385$^{\pm0.097}$
ConvoFusion1.194$^{\pm0.010}$1.238$^{\pm0.018}$0.596$^{\pm0.001}$133.859$^{\pm0.590}$6.592$^{\pm0.089}$1.181$^{\pm0.018}$9.181$^{\pm0.032}$773.699$^{\pm5.519}$11.746$^{\pm0.113}$4.032$^{\pm0.039}$
Co$^{3}$Gesture1.186^{\pm0.024}\1.590$^{\pm0.086}$0.603$^{\pm0.001}$138.085$^{\pm0.623}$7.801$^{\pm0.053}$3.631$^{\pm0.025}$9.456$^{\pm0.033}$705.492$^{\pm2.583}$10.162$^{\pm0.297}$5.135$^{\pm0.116}$
DyaDiT1.211$^{\pm0.026}$1.790$^{\pm0.043}$0.592$^{\pm0.001}$150.448$^{\pm0.923}$7.956$^{\pm0.097}$2.755$^{\pm0.027}$9.102$^{\pm0.049}$828.382$^{\pm4.733}$11.192$^{\pm0.128}$5.375$^{\pm0.074}$
Beyond Talking1.169$^{\pm0.029}$1.285$^{\pm0.032}$0.601$^{\pm0.001}$136.004$^{\pm0.949}$6.423$^{\pm0.061}$2.150$^{\pm0.013}$9.127$^{\pm0.023}$764.988$^{\pm3.975}$11.072$^{\pm0.073}$4.255$^{\pm0.030}$
DyaChain (Ours)0.824$^{\pm0.021}$1.043±0.045^{\pm0.045}0.609±0.001^{\pm0.001}140.964±0.251^{\pm0.251}4.009±0.057^{\pm0.057}0.865±0.014^{\pm0.014}8.962±0.031^{\pm0.031}782.051±1.414^{\pm1.414}7.482±0.066^{\pm0.066}2.938^{\pm0.054}\

Updated Table 4 (originally Table 2; Ablation of PSR and TRI on HOWDY):

MethodsP-FGD ↓\downarrowDiv →\rightarrowP-FED ↓\downarrowP-FHD ↓\downarrow
w/o PSR1.135$^{\pm0.033}$139.082$^{\pm0.548}$6.027$^{\pm0.136}$10.858$^{\pm0.166}$
w/o Text1.063$^{\pm0.027}$137.856$^{\pm0.514}$5.074$^{\pm0.092}$9.758$^{\pm0.180}$
w/o Routing0.993$^{\pm0.015}$132.298$^{\pm0.525}$4.580$^{\pm0.123}$9.493$^{\pm0.133}$
w/o TRI1.076$^{\pm0.017}$137.303$^{\pm0.332}$4.295$^{\pm0.067}$9.339$^{\pm0.099}$
w/o Gate0.965$^{\pm0.018}$136.237$^{\pm0.239}$4.092$^{\pm0.048}$8.619$^{\pm0.097}$
DyaChain (Full)0.824$^{\pm0.021}$140.964±0.251^{\pm0.251}4.009±0.057^{\pm0.057}7.482^{\pm0.066}\

Updated Table 5 (originally Table 3; Ablation of the Progressive Interaction Chain on HOWDY):

MethodsP-FGD ↓\downarrowDiv →\rightarrowP-FED ↓\downarrowP-FHD ↓\downarrow
w/ GT text0.684$^{\pm0.022}$142.589$^{\pm0.221}$3.130$^{\pm0.065}$5.674$^{\pm0.093}$
w/ Random text1.454$^{\pm0.021}$132.000$^{\pm0.201}$5.379$^{\pm0.097}$10.179$^{\pm0.064}$
w/ Full retrieval1.042$^{\pm0.017}$136.407$^{\pm0.255}$4.299$^{\pm0.064}$8.391$^{\pm0.034}$
w/ Direct event inference0.964$^{\pm0.021}$134.162$^{\pm0.211}$4.137$^{\pm0.061}$7.932$^{\pm0.066}$
DyaChain (Full)0.824$^{\pm0.021}$140.964±0.251^{\pm0.251}4.009±0.057^{\pm0.057}7.482^{\pm0.066}\
Reviewer_P5W9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DyaChain, a system that takes two speakers’ audio and timestamped transcripts as input and simultaneously generates six-second sequences of upper-body, hand, and facial motion for both speakers.

The system has two main components:

  • A fine-tuned LLM predicts interaction relations and timed motion events, using posture descriptions retrieved from the training set.
  • A diffusion model generates both participants’ motions, using cross-person attention and body-part-specific text conditioning.

The paper also introduces HOWDY, a dataset reconstructed from conversational videos with paired motion, body-part descriptions, and interaction annotations.

Critical strengths
  1. HOWDY provides paired speech and motion with detailed interaction annotations. This is useful for studying coordinated speaking and listening behavior beyond single-person gesture generation.
  2. The evaluation includes quantitative comparisons, component ablations, and a user study against six baselines.
  3. The generated motions look ok to me in the supplementary videos.
Critical weaknesses
  1. The current method generates complete clips offline. Exploring online generation would be an interesting direction for future work and could make the system more useful for interactive applications.
  2. Releasing the code would provide a useful reference and make the method easier to reproduce.
Questions for authors

None.

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors of W1 to W2

We thank Reviewer P5W9 for your positive assessment and constructive suggestions. We are glad that you recognized the value of HOWDY's paired speech-motion data and detailed interaction annotations for "studying coordinated speaking and listening behavior beyond single-person gesture generation", as well as our evaluation across quantitative comparisons, component ablations, and the user study. Below, we address your comments point by point.

W1: Exploring online generation for interactive applications

R1: Thanks for highlighting this valuable direction. Currently, DyaChain generates complete clips offline because the Progressive Interaction Chain infers clip-level dyadic relations and timed motion events prior to full-sequence diffusion denoising. Nevertheless, TRI restricts counterpart attention to the current and preceding frames, providing a useful starting point for future streaming extensions. Following your suggestion, we have added a dedicated Limitations and Future Work discussion in Appendix A.6 of the revised manuscript on extending DyaChain to low-latency online generation (e.g., combining rolling-window causal diffusion with incremental LLM planning, complementing concurrent streaming explorations discussed in Section 2, Lines 155–157).

W2: Code and dataset release for reproducibility

R2: Yes, we will publicly release the HOWDY dataset and complete codebase for academic research. While we have shared the core code of DyaChain via an anonymous repository (https://anonymous.4open.science/r/DyaChain-FA5A/), hosting the full dataset is currently not feasible considering the anonymity principles during the review period and its storage scale. Currently, the dataset—including all processed 3D SMPL-X motion parameters, hierarchical annotations, and aligned metadata—and our data processing scripts are being organized, and we will open-source them as soon as possible.

We hope our responses and the revised manuscript address your comments, and we are happy to clarify any further questions.

Thank you very much.

Best regards,

Authors

Reviewer_xRhVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper generates upper-body, hand, and face motion for both people in a conversation at the same time, from their speech and transcripts. It introduces HOWDY, about 42k six-second clips reconstructed from talk-show and interview videos with SAM 3D Body and EMOCA. The clips are annotated, mostly with rules and LLMs plus human review, with body-part motion descriptions, conversation states, and dyadic relations. DyaChain first uses a fine-tuned Qwen3-4B to predict relations and timed motion events and retrieves static posture descriptions from the training set. A diffusion model then uses these conditions through two new modules: TRI (causal cross-person attention with a learned response delay and a gate) and PSR (routing text to each body part). Trained and tested only on HOWDY, it beats six retrained baselines on FD metrics, ablations support each component, and a user study (20 participants, 20 clips) prefers it.

Critical strengths
  1. A useful dataset direction. Paired dyadic motion with body-part descriptions, conversation states, and relation labels is rich supervision that other dyadic datasets mostly lack.

  2. Consistent results. DyaChain beats all six retrained baselines on nearly every metric, the ablations show each component helps, and a user study (20 people) agrees.

Critical weaknesses
  1. Evaluated only on its own dataset. Everything is trained and tested on HOWDY. There are no results on existing dyadic datasets (GES-Inter, HoCo, Seamless Interaction, Talking With Hands), so it’s unclear whether the gains carry over.

  2. Missing the closest baseline. Beyond Talking (IJCV 2025) also generates both people’s holistic motion with chain-like coupling, and the paper cites it, but it isn’t compared.

  3. Baseline implementations are unclear. The paper says all baselines use their official code, but Co3 Gesture’s code doesn’t appear to be public: its project page and repository contain only the webpage. It’s unclear how this baseline was implemented, which makes its comparison hard to verify.

  4. Interaction is never measured directly. All metrics are distribution-based (FD, Div, BC). Nothing checks response timing, turning or gaze toward the partner, or backchannels, even though asynchronous response is the main claim. The learned response delays are never analyzed.

  5. Data quality is unverified, and “holistic” is overstated. The ground truth is monocular pseudo-labels (SAM 3D Body + EMOCA on talk-show footage) with no accuracy check. Most annotations come from rules and LLMs (MotionScript, GPT-5.6, Qwen), and the amount of human review and the annotator agreement aren’t reported. The output covers only upper body, hands, and face (no legs or global translation).

  6. Missing related work. Not discussed: DualTalk (CVPR 2025), which models both speakers switching between speaking and listening; Can Language Models Learn to Listen? (ICCV 2023); DIM (ECCV 2024); LoM (CVPR 2025), which models speech, text, and holistic motion together; SocialGen (3DV 2026), multi-human social interaction with a motion–language model; ViBES (CVPR 2026), joint speech and behavior planning for a conversational 3D agent; and the concurrent DyaPlex (2026).

Questions for authors
  1. The paper states that all baselines use their official code, but I could not find a public code release for Co3 Gesture (its project page and repository contain only the webpage). How was Co3 Gesture implemented: obtained from its authors, or reimplemented? If reimplemented, how was correctness checked (e.g., by reproducing its reported numbers on GES-Inter)? The same question applies to any other baseline without public code.

  2. Can you evaluate on an existing dyadic dataset (e.g. seamless interatcion), and compare with Beyond Talking?

  3. Can you add metrics that measure interaction directly, such as response latency, head or gaze orientation toward the partner, or timing of backchannels? What response delays does TRI actually learn?

  4. Will HOWDY be released?

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors of W3, W1, Q1 and Q2

We thank Reviewer xRhV for your valuable time and constructive comments, and for recognizing that HOWDY provides "rich supervision that other dyadic datasets mostly lack" and that DyaChain achieves "consistent results" across quantitative metrics, ablations, and the user study. We have addressed your concerns in the revised manuscript (highlighted in blue), and our point-by-point responses are listed below.

W3 & Q1: Clarification on baseline implementations (especially Co$^{3}$Gesture)

R1: Thanks for your careful check and comments. The implementation details of all baselines are listed point by point as below.

a) Indeed, the public GitHub repository of Co$^{3}$Gesture currently contains only its project webpage. During our experiments, we contacted the original authors via email and obtained their official source code directly for academic benchmarking.

b) The remaining five baselines in our initial submission (SemGes, GestureLSM, GlobalDiff, ConvoFusion, and DyaDiT) all have publicly released official repositories, which we used directly. For Beyond Talking, which is newly added during the rebuttal period and does not have publicly released code, we re-implemented the method following the original paper (please refer to R3 for details).

c) To ensure a fair comparison, all baselines retain their original pre-trained audio encoders following their official settings, and aligned speech transcripts are also provided to baselines that originally take transcripts as input. Apart from this, all remaining motion generation components are trained from scratch on HOWDY, with their final output layers adapted to predict our 326-dimensional motion representation for both interlocutors. We have clarified these details in the revised manuscript (Section 5.2, Lines 409–414).

W1 & Q2 (Part 1): Cross-dataset evaluation on Seamless Interaction

R2: Thanks for your valuable suggestion. As suggested, we newly conducted a zero-shot cross-dataset evaluation on the Seamless Interaction dataset [A] to examine whether DyaChain's improvements carry over to an existing dyadic benchmark. The details are listed point by point as below.

a) Following our data protocol, we synchronized the dyadic audio, word-level transcripts, and motion sequences from batch 0000 of the naturalistic test split (naturalistic/test/0000) of Seamless Interaction into 6-second (90-frame) clips at 15 FPS, yielding 7,013 valid dyadic clips. All compared baselines and DyaChain (trained on HOWDY) were directly evaluated across all 7,013 clips without any target-domain fine-tuning. During inference, DyaChain inferred its Progressive Interaction Chain conditions from Seamless Interaction's audio and transcripts while retrieving static posture priors from the HOWDY training set, without requiring any target-domain annotations.

b) Since Seamless Interaction provides SMPL-H body and hand poses without 50D FLAME expression parameters, facial and FLAME-dependent holistic metrics (FED, MSE, and FHD) are not applicable; we therefore report all four upper-body and hand gesture metrics (P-FGD, S-FGD, BC, and Div) across 10 sampling runs. As shown in the table below, DyaChain consistently outperforms all compared baselines across every metric under this zero-shot setting, confirming strong generalization to unseen dyadic conversations. We have added this experiment and discussion in the revised manuscript (Appendix A.4.1, Lines 1099–1116, Table 7).

MethodsP-FGD ↓\downarrowS-FGD ↓\downarrowBC ↑\uparrowDiv →\rightarrow
GT------91.794
SemGes5.451$^{\pm0.000}$7.939$^{\pm0.000}$0.597$^{\pm0.000}$51.641$^{\pm0.000}$
GestureLSM3.067$^{\pm0.038}$3.950$^{\pm0.028}$0.607$^{\pm0.000}$97.457$^{\pm0.138}$
GlobalDiff3.162$^{\pm0.028}$4.623$^{\pm0.041}$0.605$^{\pm0.000}$75.412$^{\pm0.151}$
ConvoFusion3.693$^{\pm0.055}$6.278$^{\pm0.089}$0.598$^{\pm0.000}$85.682$^{\pm0.129}$
Co$^{3}$Gesture4.187^{\pm0.023}\5.644$^{\pm0.033}$0.603$^{\pm0.001}$105.633$^{\pm0.114}$
DyaDiT3.628$^{\pm0.046}$4.570$^{\pm0.047}$0.592$^{\pm0.000}$98.779$^{\pm0.287}$
Beyond Talking3.026$^{\pm0.041}$3.511$^{\pm0.045}$0.606$^{\pm0.001}$98.980$^{\pm0.182}$
DyaChain (Ours)1.505$^{\pm0.017}$2.273±0.015^{\pm0.015}0.609±0.000^{\pm0.000}88.266^{\pm0.209}\

[A] Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset, in arXiv, 2025.

AuthorsOpenReview ↗

Rebuttal by Authors of W2, Q1 and Q2

W2, Q1 (Part 2) & Q2 (Part 2): Discussion and quantitative comparison with Beyond Talking (IJCV 2025)

R3: Thanks for highlighting this closely related work. As suggested, we have added discussions and quantitative comparisons with Beyond Talking [B] in the revised manuscript. The details are listed point by point as below.

a) We have included discussions of both Beyond Talking [B] and its HoCo dataset in the main text (Section 2, Lines 153–155 and 179–181). Beyond Talking explores holistic dyadic motion generation from audio via chain-like autoregressive coupling at the discrete motion-token level. Complementing this paradigm, DyaChain constructs an explicit Progressive Interaction Chain (from conversational states and dyadic relations to timed part-level motion events), paired with causal counterpart attention with learnable relative temporal biases and relation/state-conditioned gating (TRI) and part-aligned semantic routing (PSR).

b) As the official codebase and HoCo dataset of Beyond Talking are not yet publicly released, we carefully re-implemented its two-stage framework—training part-wise VQ-VAEs for the speaker’s body and hands and the listener’s body, hands, and face, followed by its chain-coupled autoregressive Transformer—together with its separate CNN-based speaker-face regression branch on HOWDY following the architectural and training details in [B]. As shown in the table below (reporting 10-run means for brevity; full 95% CIs are in Table 1) as well as the zero-shot results in R2, Beyond Talking achieves competitive results among conversational baselines, confirming the effectiveness of its chain-like token coupling. Incorporating explicit hierarchical interaction guidance and part-aligned semantic routing via TRI and PSR allows DyaChain to achieve further improvements across gesture, facial, and holistic metrics. We have incorporated Beyond Talking into Section 5.2 (Tables 1 and 2) and Appendix A.4.1 (Table 7). We have also included Beyond Talking in the qualitative comparisons in Section 5.3 (Figure 4).

MethodsP-FGD$\downarrow$S-FGD$\downarrow$BC$\uparrow$G-Div$\rightarrow$P-FED$\downarrow$S-FED$\downarrow$MSE$\downarrow$F-Div$\rightarrow$P-FHD$\downarrow$S-FHD$\downarrow$
GT------143.942------786.462----
ConvoFusion1.1941.2380.596133.8596.5921.1819.181773.69911.7464.032
Co$^{3}$Gesture1.1861.5900.603138.0857.8013.6319.456705.49210.1625.135
DyaDiT1.2111.7900.592150.4487.9562.7559.102828.38211.1925.375
Beyond Talking1.1691.2850.601136.0046.4232.1509.127764.98811.0724.255
DyaChain (Ours)0.8241.0430.609140.9644.0090.8658.962782.0517.4822.938

[B] Beyond Talking – Generating Holistic 3D Human Dyadic Motion for Communication, in IJCV, 2025.

AuthorsOpenReview ↗

Rebuttal by Authors of W4 and Q3: Interaction Metrics

W4 & Q3: Interaction metrics and learned response delays in TRI

R4: Thanks for your insightful comments and suggestions. Drawing on established evaluation paradigms across dyadic behavioral synchrony [C, D], visual mutual-attention and 3D rotation geometry [E, F], and listener-reaction benchmarks [G, H], we adopt three literature-grounded interpersonal coordination metrics alongside a quantitative analysis of TRI's learned response delays. The details are listed point by point as below.

a) Grounded in prior literature across these domains, we evaluate response timing, head-orientation alignment, and backchannel timing on the HOWDY test set: (1) Building on Windowed Time-Lagged Cross-Correlation (TLCC) for dyadic leader–follower synchrony in behavioral time-series analysis and listener modeling [C, D], we adopt Δ\DeltaTLCC (↓\downarrow) to measure the mean absolute discrepancy between generated and ground-truth cross-person joint angular-velocity TLCC curves across symmetric frame lags, assessing fidelity to asynchronous lead–lag coupling; (2) Motivated by reciprocal head-orientation cues for dyadic mutual attention [E] and standard SO(3)SO(3) rotation distance theory [F], we adopt Relative Orientation Error (ROE ↓\downarrow) to compute the mean per-frame SO(3)SO(3) geodesic error of interpersonal relative head rotation Rrel,t=R0,t⊤R1,tR_{\mathrm{rel},t} = R_{0,t}^\top R_{1,t} against paired ground truth in the shared camera frame; and (3) Following established correlation-based listener-reaction and backchannel evaluation protocols [G, H], we adopt Pearson Correlation Coefficient (PCC ↑\uparrow) over listening and brief-response intervals to measure the temporal alignment of nonverbal backchannels. As shown in the table below, DyaChain consistently achieves the best performance across all three metrics.

MethodsΔ\DeltaTLCC ↓\downarrowROE ↓\downarrowPCC ↑\uparrow
ConvoFusion0.234$^{\pm0.000}$0.818$^{\pm0.003}$0.117$^{\pm0.003}$
Co$^{3}$Gesture0.221^{\pm0.001}\0.749$^{\pm0.005}$0.090$^{\pm0.002}$
DyaDiT0.234$^{\pm0.001}$0.757$^{\pm0.004}$0.079$^{\pm0.005}$
Beyond Talking0.217$^{\pm0.000}$0.738$^{\pm0.002}$0.092$^{\pm0.002}$
DyaChain (Ours)0.207$^{\pm0.000}$0.713±0.004^{\pm0.004}0.124^{\pm0.002}\

[C] Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series, in Psychological Methods, 2002. [D] Learning to listen: Modeling non-deterministic dyadic facial motion, in CVPR, 2022. [E] LAEO-Net: Revisiting people looking at each other in videos, in CVPR, 2019. [F] Metrics for 3D rotations: Comparison and analysis, in JMIV, 2009. [G] REACT2023: The first multiple appropriate facial reaction generation challenge, in ACM MM, 2023. [H] VividListener: Expressive and controllable listener dynamics modeling for multi-modal responsive interaction, in AAAI, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Sincere Appreciation to Reviewers and ACs

Dear Reviewers and ACs,

We would like to extend our sincere appreciation for your time and valuable comments throughout the review and discussion process.

Guided by your constructive suggestions, we have substantially strengthened our manuscript with additional cross-dataset and baseline experiments, interaction metrics and analyses, and dataset quality audits.

Once more, thank you very much for the time and effort you have dedicated to helping us improve our paper.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.