acceptodds
Withdrawn from ICLR 2027

ReDyFlow: Temporal-Pyramid Residual Flow Matching for Dyadic Conversational Avatar Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

20 views

Abstract

Dyadic conversational avatar generation requires speech-aligned articulation and coordinated behaviors that are not uniquely determined by audio. We introduce ReDyFlow, a two-stage framework that jointly generates the speaking and listening head-motion sequences of both participants, without requiring observed partner motion as input. Given paired audio, speaking activity, and reference motion styles, a role-aware deterministic predictor constructs paired motion anchors through multi-scale temporal modeling and interaction between preliminary predictions. An anchor-conditioned joint flow then stochastically refines both trajectories with a temporal-pyramid velocity network that captures multi-resolution context and bidirectional participant interactions. We further introduce rollout-based dynamics supervision to regularize the temporal statistics of integrated samples. Experiments on the UniLS benchmark show that ReDyFlow achieves better results on the majority of quantitative metrics, while qualitative comparisons and user preferences indicate more natural speaking motion and listening reactions.

Primary area:
generative models
Keywords:
Flow Matching, Conversational Avatar

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zx8XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper generates 3D head motion for both people in a conversation at the same time, without needing the partner's motion as input. It has two stages: first a deterministic model predicts rough anchor motions for both people using audio, who is speaking, and a style clip; then a flow model refines both motions together, with attention between the two people.

Critical strengths

I like the setting which is practical since it doesn't need the partner's motion. The idea of letting the deterministic part handle lip movement and the flow handle the less predictable parts (expressions, head reactions) makes sense. Besides, results on UniLS are good on most metrics, especially F-FID/P-FID, and there is a user study and extra results on DualTalk.

Critical weaknesses
  • I think there may be ground-truth leakage in the evaluation. Test sequences are 400 frames and the style reference starts at min(L/2, L-100) = 200, so the reference is frames 200-300, which is exactly one of the four test windows. The model also gets the mean/std of this reference. This could be why even Stage 1 alone has about 5x better F-FID than UniLS. I'd like to see results with a reference from a different clip, or with no reference.
  • The contribution is somewhat incremental. Deterministic prior + residual flow is from SubtleTalk, cross-person attention is from InterGen, and pyramid flows already exist.
  • The system is very complicated: six training phases, more than 20 losses and lots of hand-set weights, with no sensitivity study. It's hard to know what actually brings the improvement.
Questions for authors

Please refer to weaknesses.

Reviewer_SSLZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReDyFlow jointly generates both participants’ conversational head motions without observed partner motion. It combines role-aware, multi-scale motion anchors with an anchor-conditioned joint flow that uses temporal-pyramid modeling and bidirectional interaction. Rollout-based supervision further improves motion dynamics, with experiments showing gains on most UniLS metrics.

Critical strengths
  1. Deterministic anchors provide stable motion predictions, while the joint flow models variability. Multi-scale interaction and rollout supervision further support coordinated motion generation.
  2. The method improves seven of ten metrics over complete baselines in Table 1, with supporting ablations and favorable user preferences.
  3. The framework jointly generates both participants’ speaking and listening motions without requiring synchronized partner-motion observations.
Critical weaknesses
  1. The architecture mainly combines established ideas, including deterministic prediction, residual flow matching, temporal pyramids, and cross-attention. Its distinction from prior combinations could be demonstrated more clearly.
  2. Style references come from the same source sequence and may overlap with the target interval (Appendix A.1). Although this follows UniLS, evaluation with disjoint or cross-sequence references would better establish generalization.
  3. Related to weakness 1, the interaction and temporal-pyramid modules are removed from both stages simultaneously. These experiments do not distinguish their stage-specific contributions, and role-aware routing lacks a dedicated ablation.
  4. Generation uses separate 100-frame windows without previous-motion conditioning or cross-window blending. Continuity across boundaries and consistency over longer conversations remain insufficiently evaluated.
Questions for authors
  1. Regarding weakness 2, since style references may overlap with target intervals, do the reported gains persist when overlapping frames are excluded from evaluation? This would help determine whether the improvements depend on access to target motion information.
  2. Were the imported baseline scores computed using exactly the same metric implementation as ReDyFlow? To me, the video results do not seem too different between UniLS and ReDyFlow. How was comparability verified for the large F-FID/P-FID improvements?
  3. Regarding weakness 1, do you have ablations that remove interaction or temporal-pyramid modeling from each stage separately, ideally with comparable model capacity? The current joint removals leave the contributions of the anchor predictor and flow network unresolved.

My confidence is limited by unresolved questions about reference-motion overlap and metric comparability. I would consider raising my score if the rebuttal provides convincing clarification or evidence addressing these concerns.

Reviewer_7t13OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Paper proposes ReDyFlow, two-stage method for dyadic conversational avatar generation, jointly generating speaking and listening head motion of both participants without observed partner motion.

Method consists of 2 main stages: Stage 1: role-aware deterministic predictor with audio routing, temporal pyramid, and predicted-motion interaction; Stage 2, anchor-conditioned joint flow with temporal-pyramid velocity network. Main contributions are joint flow refinement around deterministic anchors, and rollout-based dynamics supervision. Evaluation covers DiffPoseTalk, ARTalk, DualTalk, UniLS methods, ablations, and user study

Critical strengths
  1. Method is well-motivated: deterministic anchors predict speech-driven signal, while flow refinement models multi-modal expressions and reactions, giving better results compared to use of only regression or only flow. Rollout supervision regularizes std and velocity std of generated motion, improving dynamics metrics.
  2. Extensive ablation (Table 1): isolates both stages, interaction, temporal pyramid, rollout supervision, and motion-only data. Broad comparison with baselines: 2 benchmarks, 5 methods, user study. Achieving best results on 7 of 10 metrics while inference with only 10 steps
Critical weaknesses
  1. Possible ground-truth leakage through style reference. Test style encoding is computed from statistics of the segment that may overlap with the target (Appendix A.1).
  2. Ablations do not cover all components. Conversational audio routing (Sec. 3.2) is listed in contributions, but not isolated in Table 1.
Questions for authors
  1. Could results be reported with and without style reference (e.g. null-style embedding, Appendix A.2) to clarify contribution of the method without possible ground-truth leakage?
  2. Additional ablation. Could results be reported without conversational audio routing (Sec. 3.2), to isolate component contribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.