- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces HOWDY, a new two-person conversation dataset that contains synchronized speech, two-person motion, and annotations for body-part motion and interaction. It additionally introduces DyaChain, a full-body (body + hands + face) motion generation method from synchronized speech and a paired transcript. Dyachain reasons about a progressive interaction chain: dyadic relations, motion events, and conversational states. The paper further proposes Temporal-Relational Interactor and Part-Aligned Semantic Routing, which gate interaction features and inject motion events + conversational state information into the generation process. DyaChain sets the new SOTA in the HOWDY dataset across gesture, face, and holistic interaction generation metrics.
Critical strengths
- The HOWDY dataset pairs full-body movements of the two interactants with body part-level motion annotations, frame-level conversational states, and relation attributes; Its rich and hierarchical annotations position it uniquely in the speech-interaction dataset realm, which is a sound contribution to the community if open-sourced.
- DyaChain is the first method that generates full-body interaction (body + hand + face) from speech and text, and its evaluation is extensive and shows strong empirical results.
Critical weaknesses
Unfair comparison and unclear gains: DyaChain makes use of motion transcripts, LLM-labeled motion events, and dataset-retrieved motion descriptions, which the baseline methods cannot leverage. When removing text input (w/o PSR in Table 2, P-FHD 10.920 vs Co3 10.055) or using random text (P-FHD 10.255 vs Co3 10.055), the P-FHD metric is worse than the Co3 baseline, and the P-FGD is similar or worse. Combined with the absence of confidence intervals, the fact that the proposed Dyachain is only tested on the in-house HOWDY dataset and a shot-level data split (the same video might appear in both the training and test sets), the SOTA metrics are not backed up by significant evidence.
Questions for authors
- What input configurations are the baselines evaluated under? Can the transcript, relation attributes, and part-level motion annotation be provided to ConvoFusion and DyaDiT (which are able to receive text input) and evaluated again? "w/o PSR" in Table 2 is inferior to GestureLSM and Co3 Gesture.
- Since train/test/validation is split at the shot level, does the same pair of persons appear in both train and test sets?
- Can you repeat the experiments in Tables 1, 2, and 3 and report variance and confidence intervals?
- Learnable response delays are claimed but not analyzed. Can the authors provide analysis to support this claim? For example, show the distribution of the listener's response latency in the generated interactions, and visualize this along with the learned ?
6 comments
Rebuttal by Authors of W1 (Part 2) and Q2
R2: Thanks for raising this concern. We clarify our shot-level split protocol on HOWDY and report a newly added zero-shot cross-dataset evaluation on the Seamless Interaction dataset [A] point by point as below.
a) Yes, the same pair of interlocutors may appear in both the training and test sets under our shot-level split. Nevertheless, all 6-second clips segmented from the same continuous shot are strictly assigned to a single split to prevent temporal-window overlap (Appendix A.1.3, Lines 925–930), and when partitioning the test set, we also avoid including test clips that are temporally adjacent to training clips. Moreover, all baselines are trained and evaluated on the identical split, and DyaChain takes only audio and transcripts without speaker-identity conditioning, so the comparison remains fair.
b) Regarding your concerns, we newly conducted a zero-shot cross-dataset evaluation on the Seamless Interaction dataset [A] to examine whether DyaChain's improvements carry over to an existing dyadic benchmark with unseen pairs of interlocutors. Following our data protocol, we synchronized the dyadic audio, word-level transcripts, and motion sequences from batch 0000 of the naturalistic test split (naturalistic/test/0000) of Seamless Interaction into 6-second (90-frame) clips at 15 FPS, yielding 7,013 valid dyadic clips. All compared baselines and DyaChain (trained on HOWDY) were directly evaluated across all 7,013 clips without any target-domain fine-tuning. During inference, DyaChain inferred its Progressive Interaction Chain conditions from Seamless Interaction's audio and transcripts while retrieving static posture priors from the HOWDY training set, without requiring any target-domain annotations.
c) Since Seamless Interaction provides SMPL-H body and hand poses without 50D FLAME expression parameters, facial and FLAME-dependent holistic metrics (FED, MSE, and FHD) are not applicable; we therefore report all four upper-body and hand gesture metrics (P-FGD, S-FGD, BC, and Div) across 10 sampling runs. As shown in the table below, DyaChain consistently outperforms all compared baselines across every metric under this zero-shot setting, confirming strong generalization to unseen dyadic conversations. We have added this experiment and discussion in the revised manuscript (Appendix A.4.1, Lines 1099–1116, Table 7).
| Methods | P-FGD | S-FGD | BC | Div |
|---|---|---|---|---|
| GT | -- | -- | -- | 91.794 |
| SemGes | 5.451$^{\pm0.000}$ | 7.939$^{\pm0.000}$ | 0.597$^{\pm0.000}$ | 51.641$^{\pm0.000}$ |
| GestureLSM | 3.067$^{\pm0.038}$ | 3.950$^{\pm0.028}$ | 0.607$^{\pm0.000}$ | 97.457$^{\pm0.138}$ |
| GlobalDiff | 3.162$^{\pm0.028}$ | 4.623$^{\pm0.041}$ | 0.605$^{\pm0.000}$ | 75.412$^{\pm0.151}$ |
| ConvoFusion | 3.693$^{\pm0.055}$ | 6.278$^{\pm0.089}$ | 0.598$^{\pm0.000}$ | 85.682$^{\pm0.129}$ |
| Co$^{3}$Gesture | 4.187^{\pm0.023}\ | 5.644$^{\pm0.033}$ | 0.603$^{\pm0.001}$ | 105.633$^{\pm0.114}$ |
| DyaDiT | 3.628$^{\pm0.046}$ | 4.570$^{\pm0.047}$ | 0.592$^{\pm0.000}$ | 98.779$^{\pm0.287}$ |
| Beyond Talking | 3.026$^{\pm0.041}$ | 3.511$^{\pm0.045}$ | 0.606$^{\pm0.001}$ | 98.980$^{\pm0.182}$ |
| DyaChain (Ours) | 1.505$^{\pm0.017}$ | 2.273 | 0.609 | 88.266^{\pm0.209}\ |
[A] Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset, in arXiv, 2025.
Rebuttal by Authors of W1 (Part 3) and Q3
W1 (Part 3) & Q3: Repeated evaluations and confidence intervals across Tables 1, 2, and 3
R3: Thanks for your suggestion. Following your recommendation, we repeated all evaluations across 10 independent sampling runs and updated Tables 1, 4, and 5 in the revised manuscript (corresponding to Tables 1, 2, and 3 in the initial submission), as well as the newly added Tables 2, 3, and 7, to report means and 95% confidence intervals ( for deterministic SemGes). As shown in the updated tables below, sampling variability remains small across runs, and DyaChain's 95% CIs do not overlap with those of any competing baseline across all metrics in Table 1, nor with ablated variants on pairwise gesture and holistic metrics in Tables 4 and 5, confirming that our improvements are statistically reliable.
Updated Table 1 (SOTA comparison on HOWDY; G-Div and F-Div denote Gesture and Face Diversity):
| Methods | P-FGD | S-FGD | BC | G-Div | P-FED | S-FED | MSE | F-Div | P-FHD | S-FHD |
|---|---|---|---|---|---|---|---|---|---|---|
| GT | -- | -- | -- | 143.942 | -- | -- | -- | 786.462 | -- | -- |
| SemGes | 3.936$^{\pm0.000}$ | 3.178$^{\pm0.000}$ | 0.589$^{\pm0.000}$ | 107.845$^{\pm0.000}$ | 8.239$^{\pm0.000}$ | 3.253$^{\pm0.000}$ | 10.663$^{\pm0.000}$ | 583.897$^{\pm0.000}$ | 14.596$^{\pm0.000}$ | 7.536$^{\pm0.000}$ |
| GestureLSM | 1.492$^{\pm0.020}$ | 2.952$^{\pm0.085}$ | 0.604$^{\pm0.001}$ | 134.080$^{\pm0.704}$ | 5.197$^{\pm0.075}$ | 0.923$^{\pm0.008}$ | 10.905$^{\pm0.015}$ | 718.290$^{\pm3.838}$ | 10.067$^{\pm0.083}$ | 5.146$^{\pm0.088}$ |
| GlobalDiff | 1.896$^{\pm0.039}$ | 1.264$^{\pm0.049}$ | 0.597$^{\pm0.001}$ | 138.545$^{\pm0.768}$ | 5.219$^{\pm0.037}$ | 1.580$^{\pm0.011}$ | 11.787$^{\pm0.045}$ | 1021.156$^{\pm6.332}$ | 17.623$^{\pm0.150}$ | 8.385$^{\pm0.097}$ |
| ConvoFusion | 1.194$^{\pm0.010}$ | 1.238$^{\pm0.018}$ | 0.596$^{\pm0.001}$ | 133.859$^{\pm0.590}$ | 6.592$^{\pm0.089}$ | 1.181$^{\pm0.018}$ | 9.181$^{\pm0.032}$ | 773.699$^{\pm5.519}$ | 11.746$^{\pm0.113}$ | 4.032$^{\pm0.039}$ |
| Co$^{3}$Gesture | 1.186^{\pm0.024}\ | 1.590$^{\pm0.086}$ | 0.603$^{\pm0.001}$ | 138.085$^{\pm0.623}$ | 7.801$^{\pm0.053}$ | 3.631$^{\pm0.025}$ | 9.456$^{\pm0.033}$ | 705.492$^{\pm2.583}$ | 10.162$^{\pm0.297}$ | 5.135$^{\pm0.116}$ |
| DyaDiT | 1.211$^{\pm0.026}$ | 1.790$^{\pm0.043}$ | 0.592$^{\pm0.001}$ | 150.448$^{\pm0.923}$ | 7.956$^{\pm0.097}$ | 2.755$^{\pm0.027}$ | 9.102$^{\pm0.049}$ | 828.382$^{\pm4.733}$ | 11.192$^{\pm0.128}$ | 5.375$^{\pm0.074}$ |
| Beyond Talking | 1.169$^{\pm0.029}$ | 1.285$^{\pm0.032}$ | 0.601$^{\pm0.001}$ | 136.004$^{\pm0.949}$ | 6.423$^{\pm0.061}$ | 2.150$^{\pm0.013}$ | 9.127$^{\pm0.023}$ | 764.988$^{\pm3.975}$ | 11.072$^{\pm0.073}$ | 4.255$^{\pm0.030}$ |
| DyaChain (Ours) | 0.824$^{\pm0.021}$ | 1.043 | 0.609 | 140.964 | 4.009 | 0.865 | 8.962 | 782.051 | 7.482 | 2.938^{\pm0.054}\ |
Updated Table 4 (originally Table 2; Ablation of PSR and TRI on HOWDY):
| Methods | P-FGD | Div | P-FED | P-FHD |
|---|---|---|---|---|
| w/o PSR | 1.135$^{\pm0.033}$ | 139.082$^{\pm0.548}$ | 6.027$^{\pm0.136}$ | 10.858$^{\pm0.166}$ |
| w/o Text | 1.063$^{\pm0.027}$ | 137.856$^{\pm0.514}$ | 5.074$^{\pm0.092}$ | 9.758$^{\pm0.180}$ |
| w/o Routing | 0.993$^{\pm0.015}$ | 132.298$^{\pm0.525}$ | 4.580$^{\pm0.123}$ | 9.493$^{\pm0.133}$ |
| w/o TRI | 1.076$^{\pm0.017}$ | 137.303$^{\pm0.332}$ | 4.295$^{\pm0.067}$ | 9.339$^{\pm0.099}$ |
| w/o Gate | 0.965$^{\pm0.018}$ | 136.237$^{\pm0.239}$ | 4.092$^{\pm0.048}$ | 8.619$^{\pm0.097}$ |
| DyaChain (Full) | 0.824$^{\pm0.021}$ | 140.964 | 4.009 | 7.482^{\pm0.066}\ |
Updated Table 5 (originally Table 3; Ablation of the Progressive Interaction Chain on HOWDY):
| Methods | P-FGD | Div | P-FED | P-FHD |
|---|---|---|---|---|
| w/ GT text | 0.684$^{\pm0.022}$ | 142.589$^{\pm0.221}$ | 3.130$^{\pm0.065}$ | 5.674$^{\pm0.093}$ |
| w/ Random text | 1.454$^{\pm0.021}$ | 132.000$^{\pm0.201}$ | 5.379$^{\pm0.097}$ | 10.179$^{\pm0.064}$ |
| w/ Full retrieval | 1.042$^{\pm0.017}$ | 136.407$^{\pm0.255}$ | 4.299$^{\pm0.064}$ | 8.391$^{\pm0.034}$ |
| w/ Direct event inference | 0.964$^{\pm0.021}$ | 134.162$^{\pm0.211}$ | 4.137$^{\pm0.061}$ | 7.932$^{\pm0.066}$ |
| DyaChain (Full) | 0.824$^{\pm0.021}$ | 140.964 | 4.009 | 7.482^{\pm0.066}\ |
Rebuttal by Authors of W1 (Part 1) and Q1
We thank Reviewer 76nd for your thoughtful review and constructive feedback. We are glad that you recognized that HOWDY's rich, hierarchical annotations "position it uniquely in the speech-interaction dataset realm", and found DyaChain's evaluation extensive with strong empirical results. To fulfill HOWDY's value to the community, we confirm that we will publicly release the full dataset and codebase (with the core code of DyaChain shared at https://anonymous.4open.science/r/DyaChain-FA5A/). Below, we address your concerns and questions point by point.
W1 (Part 1) & Q1: Baseline input configurations, text-augmented baselines, and "w/o PSR" vs. "w/o Text"
R1: Thanks for raising these important questions. We clarify the baseline input configurations, evaluate text-augmented baselines, and distinguish
w/o PSRfromw/o Textpoint by point as below.a) For fair comparisons, as described for Table 1 (Section 5.2, Lines 409–414), all methods retain their original audio encoders, and speech transcripts are also provided to baselines that originally take them as input (SemGes, GestureLSM, ConvoFusion, and Beyond Talking).
b) Following your suggestion, to examine whether our gains mainly stem from extra semantic inputs, we newly added Table 3 (Section 5.2, Lines 435–447), augmenting ConvoFusion and DyaDiT under the same text protocol as DyaChain—trained with dataset annotations and evaluated with the identical inferred Progressive Interaction Chain descriptions encoded via frozen FLAN-T5-base and injected through each baseline's original conditioning mechanism (cross-attention for both methods), denoted as ConvoFusion$^{+}$ and DyaDiT$^{+}.Asshownbelow(with95^{+}$ and DyaDiT$^{+}$, underscoring the effectiveness of PSR and TRI.
c) In our original ablation table (now Table 4, Lines 453–472),
w/o PSRremoves PSR altogether—not only dropping text injection (GRCA), but also replacing part-specific motion encoders and body-part identity embeddings (Table 6) with a single shared motion encoder. Collapsing heterogeneous upper-body, finger, and facial parameters into a shared encoder may impair both motion representation and subsequent cross-person interaction modeling in TRI, causingw/o PSRto underperform GestureLSM and Co$^{3}$Gesture on P-FHD. To isolate the effect of removing motion descriptions alone, we newly addedDyaChain (w/o Text)in Tables 3 and 4, which removes GRCA while retaining part-specific encoding and identity embeddings. As shown below and compared with Table 1, without motion descriptions,DyaChain (w/o Text)already achieves lower P-FGD, P-FED, and P-FHD than GestureLSM, Co3Gesture, and all other unaugmented baselines. Meanwhile, inw/ Random text(Table 5, formerly Table 3), randomly mismatched static posture priors may conflict with the current conversation, causing the model to generate poses that deviate further from the ground-truth distribution thanDyaChain (w/o Text).