- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a role-aware speech injection mechanism for audio-driven multi-character animation. The paper provides a reasonable motivation on why it is necessary to disentangle the injected speech into expressive (speaker) and receptive branches (listener). The most important contribution is the proposed attention mechanism termed RoleMixAttn, which uses dynamic soft routing to adaptively mix the expressive and receptive branches. To accomplish the proposed task, the paper also self-collects the RoleTalk-AV dataset. Extensive experiments show the proposed method achieve comparable results against the established works.
Critical strengths
The paper offers a good motivation on why the injected speech needs to be separated into two functional branches: an expressive branch for self-driven speaking behaviors and a receptive branch for listener-side responses. This motivation is sound and clear to understand.
In light of this motivation, the paper presents the RoleMixAttn, which uses dynamic soft routing to adaptively mix the expressive and receptive branches. The RoleMixAttn starts with the basic dual-branch with fixed masks, then is refined to replace the binary character masks with dynamic soft routing. Logically, it makes sense for the development of such a method. Compared to the existing works which treat the audio signal as a global context, the proposed RoleMixAttn offers a more reasonable approach to processing the audio signal in the context of multi-character audio-driven generation, which inherently requires the audio signals to be attached to different characters during the interaction.
The paper provides a three-stage training on the self-collected ROLETALK-AV dataset. This three-stage training was also tightly connected to the role-aware audio injection.
Critical weaknesses
Though the motivation of this work, particularly the proposed core module RoleMixAttn makes sense, there are several missing details. The paper does not clearly explain how to initialize the character anchor C to suppress speech-induced artifacts in background regions. This was only stated as a learnable character anchor. The ablation study in Table 3 does not clearly demonstrate the performance gap when using the soft routing only, compared to the soft routing and the character gate G_c.
The RoleMixAttn relies on a user-supplied bounding box. This box was used to crop the corresponding identity image from the input image. Can this bounding box be obtained from a person detector? After that, was this used to generate the human mask M_i? This key implementation detail is also missing here. The paper could provide some visualization results to indicate what this bounding box and human mask look like. Otherwise, the following G_i and G_c all look confusing.
The paper does not provide details regarding the self-collected ROLETALK-AV dataset from the supplementary materials. The supplementary includes the video demo results, along with some partial code. Maybe I was missing something here. But it does not seem to provide details about the self-collected ROLETALK-AV dataset. I am not against using a private dataset. However, the proposed method should also evaluate on the public datasets, such that the results can be reproducible. Without evaluating on the public datasets, the effectiveness of the proposed method cannot be well validated.
The paper also provides incomplete evaluation results. It uses commonly used metrics such as FID, FVD, and Sync-C, along with its own proposed two metrics GSA and APC that were specifically designed to evaluate the interactions. However, it should provide the complete evaluation metrics by following the practice of the existing works. For instance, in the InfiniteTalk, it uses 5 different metrics such as FID, FVD, Sync-C, Sync-D, and CSIM.
The paper offers an extensive list of video demo results from the supplementary repository. I have checked some of the video results. And it seems to have certain issues. For instance, in the case of Scalable Capacity, the input audio for the InfiniteTalk and InterRole(ours) are not the same.
Questions for authors
How to initialize the character anchor C?
How to generate the bounding box and its follow-up human mask? Are there any visualization results available?
How the proposed method perform on public datasets?
How the proposed method perform under the complete evaluation metrics?
What is the importance of the character gate G_c?
Dear Reviewer yWhd,
Thank you again for carefully reviewing our additional results and acknowledging that they address your earlier concerns about lip-sync evaluation and video duration. Your feedback has helped us clarify and better substantiate our central contribution.
In our latest response, we have added controlled qualitative comparisons and listener-side speech-conditioning interventions to provide further evidence of speech-responsive listener behavior, which is the main focus of our work.
As the discussion period draws to a close, we would greatly appreciate your thoughts on this additional evidence and whether it helps clarify our listener-side contribution. We would welcome the opportunity to discuss any remaining questions or concerns.
Best regards,
The Authors