acceptodds
Under review as a conference paper at ICLR 2027

InterRole: Role-Aware Speech Injection for Multi-Character Interaction

Abstract

Audio-driven character animation has achieved impressive progress in single-character scenarios, but remains challenging in multi-character settings. Existing methods often rely on region-exclusive speech injection to prevent identity leakage, which improves speaker-audio binding but limits listener access to speech cues and weakens speaker-listener interactions. In this paper, we identify listener-side behavior as a central component of multi-character audio-driven animation and formulate the key thesis that speech should be globally accessible but decoded in a role-specific manner. Guided by this thesis, we analyze several speech injection schemes and propose InterRole, a complete framework for role-aware multi-character audio-driven animation. Its core module, RoleMixAttn, separates speech conditioning into expressive and receptive attention branches and uses dynamic soft routing to adaptively mix them for each video token. This design jointly models speaker articulation and listener responses while improving robustness to character movement and occlusion. To support training and evaluation, we construct the RoleTalk-AV dataset, adopt a staged training strategy, and introduce interaction-oriented metrics for gaze alignment and prosody-correlated listener motion. Experiments on Wan2.1 and LTX-2.3 show that InterRole generates more coherent speaker-listener interactions than existing multi-character baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Model, Character Animation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bkM3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a role-aware speech injection mechanism for audio-driven multi-character animation. The paper provides a reasonable motivation on why it is necessary to disentangle the injected speech into expressive (speaker) and receptive branches (listener). The most important contribution is the proposed attention mechanism termed RoleMixAttn, which uses dynamic soft routing to adaptively mix the expressive and receptive branches. To accomplish the proposed task, the paper also self-collects the RoleTalk-AV dataset. Extensive experiments show the proposed method achieve comparable results against the established works.

Critical strengths

The paper offers a good motivation on why the injected speech needs to be separated into two functional branches: an expressive branch for self-driven speaking behaviors and a receptive branch for listener-side responses. This motivation is sound and clear to understand.

In light of this motivation, the paper presents the RoleMixAttn, which uses dynamic soft routing to adaptively mix the expressive and receptive branches. The RoleMixAttn starts with the basic dual-branch with fixed masks, then is refined to replace the binary character masks with dynamic soft routing. Logically, it makes sense for the development of such a method. Compared to the existing works which treat the audio signal as a global context, the proposed RoleMixAttn offers a more reasonable approach to processing the audio signal in the context of multi-character audio-driven generation, which inherently requires the audio signals to be attached to different characters during the interaction.

The paper provides a three-stage training on the self-collected ROLETALK-AV dataset. This three-stage training was also tightly connected to the role-aware audio injection.

Critical weaknesses

Though the motivation of this work, particularly the proposed core module RoleMixAttn makes sense, there are several missing details. The paper does not clearly explain how to initialize the character anchor C to suppress speech-induced artifacts in background regions. This was only stated as a learnable character anchor. The ablation study in Table 3 does not clearly demonstrate the performance gap when using the soft routing only, compared to the soft routing and the character gate G_c.

The RoleMixAttn relies on a user-supplied bounding box. This box was used to crop the corresponding identity image from the input image. Can this bounding box be obtained from a person detector? After that, was this used to generate the human mask M_i? This key implementation detail is also missing here. The paper could provide some visualization results to indicate what this bounding box and human mask look like. Otherwise, the following G_i and G_c all look confusing.

The paper does not provide details regarding the self-collected ROLETALK-AV dataset from the supplementary materials. The supplementary includes the video demo results, along with some partial code. Maybe I was missing something here. But it does not seem to provide details about the self-collected ROLETALK-AV dataset. I am not against using a private dataset. However, the proposed method should also evaluate on the public datasets, such that the results can be reproducible. Without evaluating on the public datasets, the effectiveness of the proposed method cannot be well validated.

The paper also provides incomplete evaluation results. It uses commonly used metrics such as FID, FVD, and Sync-C, along with its own proposed two metrics GSA and APC that were specifically designed to evaluate the interactions. However, it should provide the complete evaluation metrics by following the practice of the existing works. For instance, in the InfiniteTalk, it uses 5 different metrics such as FID, FVD, Sync-C, Sync-D, and CSIM.

The paper offers an extensive list of video demo results from the supplementary repository. I have checked some of the video results. And it seems to have certain issues. For instance, in the case of Scalable Capacity, the input audio for the InfiniteTalk and InterRole(ours) are not the same.

Questions for authors

How to initialize the character anchor C?

How to generate the bounding box and its follow-up human mask? Are there any visualization results available?

How the proposed method perform on public datasets?

How the proposed method perform under the complete evaluation metrics?

What is the importance of the character gate G_c?

Reviewer_yWhdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper introduces InterRole, an audio-driven framework for multi-character animation. It focuses on modeling expressive and receptive states together. A soft routing method called RoleMixAttn is also proposed to address the limitation of a fixed face/body mask. The experimental results show that the framework can achieve identity scalability and preserve good interactivity.

Critical strengths
  1. RoleMixAttn is a well-conceived design. By simultaneously modeling both "speak" and "listen" states, it distinguishes itself from the mask predictor approach used in InteractHuman, while also partially resolving the occlusion issues associated with hard masks.

  2. The experiments are comprehensive, and the paper extends its framework to the LTX model, thereby demonstrating the generality of the framework.

Critical weaknesses

Main Concern:

My main concern is that the unusually fast speech in the demos may obscure weaknesses in lip synchronization and interaction quality.

After watching the demos, I am not convinced that InterRole’s lip synchronization is as good as the reported Sync-C scores suggest. Many examples show imperfect alignment. Most use rapid Mandarin speech, making synchronization difficult to assess. Moreover, Sync-C relies on a model trained on English, so its reliability on Mandarin is uncertain. A large proportion of Mandarin clips in the test set could therefore affect the reliability of the reported scores.

The interaction demos are mostly only 3–5 seconds long. Three seconds may not even allow a person to finish an expressive sentence at a natural speaking rate. These examples leave me unconvinced that InterRole can sustain coherent interactions over longer durations. In contrast, the demos of InfiniteTalk, MultiTalk, and AnyTalker show interactions lasting 10–15 seconds or longer at natural speaking rates. The authors should provide longer examples with natural-paced speech, as the current rapid delivery makes interaction quality difficult to judge.

Other Concerns:

Jointly modeling speaking and listening is not new; LPM-1.0 has already explored this successfully. Moreover, MultiTalk and AnyTalker retain global self-attention, which allows each character to indirectly access other characters’ audio-conditioned features. These methods can therefore, in principle, model cross-character interactions during listening.

Then, for the benchmark, I noticed the authors report a test set of 300 clips. How many contain one, two, three, or more people? What is the language distribution?

Questions for authors

See Critical Weaknesses. At present, I am inclined to reject this paper unless the authors can provide compelling evidence.

2 comments

Dear Reviewer yWhd,

Thank you again for carefully reviewing our additional results and acknowledging that they address your earlier concerns about lip-sync evaluation and video duration. Your feedback has helped us clarify and better substantiate our central contribution.

In our latest response, we have added controlled qualitative comparisons and listener-side speech-conditioning interventions to provide further evidence of speech-responsive listener behavior, which is the main focus of our work.

As the discussion period draws to a close, we would greatly appreciate your thoughts on this additional evidence and whether it helps clarify our listener-side contribution. We would welcome the opportunity to discuss any remaining questions or concerns.

Best regards,

The Authors

Thank you for the additional experiments and detailed responses. However, the ablation studies leave my concerns regarding the significance of the contribution unresolved. I therefore cannot recommend acceptance and will maintain my "Weak Reject" recommendation. That said, I have raised my assessment of the paper's soundness.

Reviewer_Ber8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

InterRole is a framework for multi-character audio-driven animation that makes speech globally accessible while modeling speakers and listeners differently. Its RoleMixAttn module adaptively combines expressive and receptive attention to coordinate speech articulation and listener responses. Supported by the RoleTalk-AV dataset and interaction-focused metrics, experiments show more coherent speaker–listener interactions than existing baselines.

Critical strengths
  1. This paper addresses a practical problem. Realistic multi-party conversations require both accurate speaker lip synchronization and natural listener responses, giving listener behavior modeling clear practical value.
  2. This method can directly condition listener responses on speech. Compare to previous methods, the receptive branch uses the speaker’s audio to guide listener nods, gaze, and subtle expressions, making interactions more natural. Listener responsiveness also shows the largest improvement in the user study.
  3. This paper can handle character movement and occlusion more flexibly. Dynamic soft routing updates where audio conditioning applies based on character identity and current video features, making it better suited than fixed-region injection for movement, crossing, and mutual occlusion.
Critical weaknesses
  • Three-stage training introduces a complex training process that is difficult to tune.
  • Can GSA and APC distinguish natural listener responses from mechanical behaviors? Constantly looking at the speaker or nodding in sync with speech energy could yield high scores without appearing natural.
Questions for authors
  • What drives the improvement from soft routing? Better character localization or continuous mixing of expressive and receptive behaviors? Could the authors provide a controlled ablation that keeps character localization fixed while comparing hard and soft branch mixing?
  • Nodding or facial expressions that appear natural may also learn from motion priors prevalent in the training data. With the reference image, text prompt, and random seed fixed, will the listener behaviors change when the speaker’s audio is replaced, temporally shuffled, or removed? (To make sure the listener truly responds based on the current speech.)
  • Can we add GSA/APC reference values for GT videos? These numbers would help us better understand these metrics.
  • How robust is the method to input errors—for example, if the initial bounding box for a person is blurry or incomplete?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.