acceptodds
Under review as a conference paper at ICLR 2027

ConfabMuse: Structured Interaction Control for Multi-Person Talking Avatars

22 views

Abstract

Generating realistic multi-avatar conversational videos requires more than synchronizing multiple audio streams with visible avatars. A multi-avatar conversational generation framework must determine when each stream should influence motion, who should be animated by each stream, and how avatar behavior should evolve as the interaction unfolds. Existing methods largely treat multi-avatar generation as audio routing. They associate audio streams with avatars, but provide limited modeling of cross-stream turn dynamics and little structured control over how actions and relations evolve over time. To address this, we introduce ConfabMuse, a diffusion Transformer framework for controllable two-person conversational video generation from a reference image, two driving audio streams, and optional text guidance. ConfabMuse introduces a Turn-Context Audio Adapter for within and cross-stream speech modeling, Audio-Vision Binding for learned stream-to-avatar correspondence, and a Scene-Grounded Interaction Planner for structured control of evolving avatar actions and relations. Experiments across single and multi-person talking-avatar benchmarks show that ConfabMuse achieves state-of-the-art conversational interactivity, improves structured interaction control, and maintains strong audio-visual synchronization and generation quality. These results demonstrate the benefit of jointly modeling conversational timing, avatar correspondence, and interaction dynamics for controllable multi-person talking-avatar generation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Learning, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SEN7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ConfabMuse, a diffusion Transformer framework for controllable two-person conversational video generation. Given a reference image, two audio streams, and optional text guidance, the framework aims to generate realistic talking avatars with appropriate speech synchronization and coordinated interactions. The proposed method consists of three components: a Turn-Context Audio Adapter that models speech activity and conversational turns, an Audio-Vision Binding module that associates audio streams with their corresponding avatars, and a Scene-Grounded Interaction Planner that uses structured interaction graphs to control evolving actions and relations. The authors conduct extensive experiments across single-person and multi-person talking-avatar benchmarks. The results demonstrate competitive visual quality and audio-visual synchronization, along with improvements in conversational interactivity and structured interaction control.

Critical strengths
  1. Clear motivation and coherent framework design. The paper addresses an important challenge in multi-person conversational video generation: generating natural interactions requires more than simply associating audio streams with visible avatars. The decomposition into conversational timing, audio-avatar binding, and evolving interaction control is intuitive and well motivated. I particularly appreciate the Scene-Grounded Interaction Planner, which explicitly models avatar actions and relations across conversational turns while preserving ongoing behaviors.
  2. Comprehensive experimental evaluation and promising performance. The paper provides extensive evaluations across multiple single-person and multi-person benchmarks, covering generation quality, audio-visual synchronization, motion coordination, and interaction controllability. The proposed framework achieves strong results across most reported metrics. The ablation studies further demonstrate the effectiveness of the proposed components, particularly in reducing unintended listener activation and improving interaction transitions. The experimental section is a significant strength of the paper.
Critical weaknesses
  1. The contribution of the structured interaction planner could be more clearly isolated. Although the Scene-Grounded Interaction Planner is an interesting component, its advantages over simpler forms of temporally structured conditioning are not fully established. In Table 6, the authors compare the proposed planner with simple and expanded text prompts. However, it remains unclear whether these baselines receive equivalent information about avatar-specific actions and transition timing. A comparison with turn-aligned text guidance containing the same interaction information would better clarify the benefit of the proposed graph representation and planning mechanism.
  2. The proposed interaction-control metrics lack human validation. The paper introduces several MLLM-based metrics to evaluate interaction states, transitions, gaze, and temporal consistency. While these metrics are well motivated and provide more targeted measurements than conventional video-generation metrics, their reliability remains somewhat uncertain. The authors appropriately include a cross-evaluator comparison with InternVL3.5, which produces consistent method rankings. Nevertheless, agreement between automated evaluators does not fully establish their correspondence with human judgments, particularly for subtle behaviors such as gaze and conversational responses. A small human evaluation would provide additional confidence in the reported improvements.
Questions for authors

Q1. Could the authors provide a more controlled comparison to clarify the contribution of the Scene-Grounded Interaction Planner? For example, how would the framework perform with turn-aligned text instructions containing the same avatar actions and transition timing, but without the structured graph representation? Q2. The paper introduces several MLLM-based metrics for interaction controllability. Although the cross-evaluator analysis is encouraging, have the authors considered validating these metrics against human judgments? In particular, how reliably do the proposed metrics capture subtle conversational behaviors such as gaze direction, listener responses, and interaction transitions?

Reviewer_T3LpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the interesting problem of multi-avatar conversational video generations, which is of great importance to the industry applications. In this paper, it presents a new method called, ConfabMuse, which is based on a diffusion transformer framework for controllable conversational video generation from a reference image. Experimental results show reasonable performance of the proposed ConfabMuse. Also, a set of ablations are reported to further validate the effectiveness of the method.

Critical strengths
  • The paper targets the interesting problem of controllable two-person conversational video generation from multiple conditions like a reference image, two driving audio streams, and optional text guidance, which is of great importance to industry community.

  • The decomposition of multi-person conversational avatar generaiton into trhee coupled questions, when (which stream should drive motion), who (which stream drives which avatar), and how (how actions and relations evolve across turns) , is interesting and new to the cummunity.

  • The paper presents an extensive evaluation of the proposed algorithm with a set of evaluation metrics are involved.

Critical weaknesses
  • For the experiments, I would suggest to have an user-study. For example, the evaluation can involve human assessment. Currently, the metrics like ISCS / ITS / GTA / TIF may not well reflect the actual human judgement.

  • The scenario coverage is not enough. For example, currently, there are two people with one reference image and no overlapping speech. It may have large distribution gap with the the video distribution.

  • There are too many evaluation metrics which are reported in the paper. For example, in Table 5-7, the evaluation are reported based on Motion Bind, Spatial Rel., ISCS, ITS, GTA, TIF, TSA, TTE, FSC, Sync-C, Active-Speaker AP. I would wonder is it possible to have a unified metric to well reflect different overall performance.

  • How about the performance on different diffusion transformer? Currently, it utilizes the Wan model and further trained based on three stages. How about the robustness of the method on different models?

Questions for authors

Please mainly address the questions in the weakness section.

Reviewer_iATbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ConfabMuse, a framework for generating two-person conversational videos from a reference image, two audio streams, and optional text. It has three components: 1) a Turn-Context Audio Adapter, which models when each stream should drive motion using within- and cross-stream speech activity; 2) Audio-Vision Binding, which learns audio–avatar compatibility as a cross-attention bias to decide who each stream animates; and 3) a Scene-Grounded Interaction Planner, a tuned VLM with a frozen reflector that predicts sparse interaction-graph updates at turn boundaries to model how behavior evolves. Experiments cover single-person, two-person, motion-coordination, and interaction-control benchmarks show clear gains.

Critical strengths
  1. Well-motivated problem: Natural conversation needs more than assigning speech to the right face: listeners should react, ongoing actions should persist, and gaze should shift across turns. Splitting the problem into when / who / how gives a clear structure.
  2. Consistent multi-person gains and informative ablation. Method beats baselines in metrics evaluating visual quality, motion-coordination and interactivity. Ablation experiments support the claim that the other speaker's activity helps control listener behavior.
Critical weaknesses
  1. Audio–avatar binding need more details to validate. It is unclear how the intended correspondence between arbitrary voices and avatars is established, or how avatar regions are maintained during motion. In the bias-removal ablation (Table 9), Sync-C and Active-Speaker AP are lower than even the Wav2Vec2-only baseline (Table 7). This suggests the bias may have been removed only at inference, which would introduce a training–inference mismatch. Matched comparisons with explicit routing and tests involving voice swaps would better establish the module's contribution.
  2. Ablation on benefit of structured representation. Graph guidance appears to provide more explicit actions and transitions than the text baselines (e.g., Fig. 6), so the gains may partly reflect richer instructions. Two additions would strengthen the evidence: a comparison that gives plain text the same content and timing, and separate ablations of GRPO, reflection, and sparse updates.
Questions for authors

See weakness above,

Minor:

  1. The scope is limited to two-person conversations, as the authors acknowledge. It remains unclear how the binding and planner would scale to three or more people.
  2. Graph updates occur only at speech-turn boundaries. The paper should also evaluate gestures and actions that change independently of turn transitions.

Overall, This is an interesting paper with a well-motivated decomposition of conversational avatar generation. However, several key aspects are unclear or insufficiently validated, including the binding ablation, the fairness of the planner comparison, and the human validation of the interaction metrics. I would be willing to raise my score if these concerns are adequately addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.