- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses an important problem of spatial-speaker binding in the multi-person video generation domains. Other than using person's location information, the paper presents a speaker-aware audio representation that allows the proposed method to track who is speaking and when directly in the acoustic representation. Thus, the motivation of the work is clear, and the addressed problem is also important. The most important contribution of this work is a new proposed speaker-aware audio modulation (SAM) to achieve spatial-mask-free speaker conditioning, which is an improvement to the existing spatial-masking paradigms that explicitly uses the person's locations.
Critical strengths
The most important strength of this work is a new proposed speaker-aware audio modulation (SAM) to achieve spatial-mask-free speaker conditioning, which is an improvement to the existing spatial-masking paradigms that explicitly use the person's locations.
The other important strength is the MULTI-REFERENCE VISUAL CONDITIONING (MVC) that adds an ID board of person crops across shots, explicitly maintaining identity cues throughout the entire generation.
The entire framework of the M2Talk could also be considered as one of the strengths, as this pipeline shows effectiveness compared to the existing works.
Critical weaknesses
Though the paper shows clear motivation for the proposed method, it also has several key concerns to be addressed.
The paper lists different flowchart comparisons in Figure 2 regarding the audio conditioning for single-person and conversational generation. However, this proposed flowchart, especially the bottom illustration, does not align correctly with the pipeline of the proposed approaches, which involves SAM, TCP, and MVC. After reviewing the Figure 2, it looks like the Speaker-Aware Audio Modulation (f) is the illustration that corresponds to the SAM description. What are these c, d, and e? These are not part of the proposed M2Talk work?
The core idea of this work is the proposed Speaker-aware audio modulation (SAM). Inside this module, Speaker activity timeline (SAT) also plays a vital role. The paper explains that the initial SAT uses a binary representation, M, which records the speaker's activity across the timelines and shots. One intriguing question is how to initialize the mask M. Does this still rely on applying person tracking techniques across the videos and associating them with the speaker information in the audio? If that is the case, then, this is still a traditional way of spatial-masking paradigms that explicitly use the person's locations. The difference seems to lie in the representations.
Although the paper claims the TEMPORAL CONTEXT PROPAGATION (TCP) to be one of the contributions. But this TCP just uses the trail frames of the previously generated trunk as the context for the new chunk generation. This has been regularly explored in the existing chunk-wise generation frameworks. The paper needs to show what the major difference is here.
The paper also needs to better explain the MULTI-REFERENCE VISUAL CONDITIONING (MVC) , where it concatenates the ID board information with the shot information. This also seems to be a regular concatenation operation. The difference is the choice of inputs here. Could you also explain what m_orig is and what role it plays?
The paper shows many video demos in the provided link. However, it does not seem to have a single video demo where it shows the evolving Speaker activity timeline (SAT) with the generated video. This should also apply to the comparisons against the baselines such that we can clearly see the drawbacks of existing spatial-masking schemes and how the proposed SAT can effectively address them to keep the speaker-aware audio injection.
Questions for authors
There are many questions for the authors. I have highlighted a few of them here.
Could you explain what are the purposes of listing the c, d, and e in Figure 2 and how are they aligned with the proposed method in this work?
Could you explain on how to generate the binary representation M in SAT? Whether there is a person tracking technique used?
Could you explain what are the major difference of MVC compared to the existing chunk-wise generations?
Please explain what m_orig is and what role it plays.
Please provide clear evolving Speaker activity timeline (SAT) with the generated video in the demo links.
5 comments
Response to Reviewer bwpM(2/2)
W4 (Part 2) & Q4: Definition and Role of
As defined in Section 3.4 (Eq. 7, lines 331–339) and Appendix A.4 (lines 913–919), is the binary image-condition availability mask of the Wan2.1-I2V [7] reference pathway (1 at the supplied initial frame in , 0 at zero-padded frames), extended by MVC to . It marks only which latent positions hold reference images, distinct from SAT and the loss ignore mask (Appendix A.5, line 961).
W5 & Q5: Synchronized Evolving SAT Visualizations in Video Demos
We have updated our anonymous project page with synchronized evolving SAT timelines and a moving playback cursor directly beneath generated videos, both for the side-by-side baseline comparisons (30s Baseline Comparisons, noting in captions that each baseline uses its native input interface) and multi-party conversations (3–4 Person Demos, plus interactive exploration at SAT Studio), enabling frame-by-frame verification of active person slots, silence, and overlaps against mouth movements.
References:
[1] Kong et al. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation. arXiv:2505.22647, 2025.
[2] Huang et al. Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-Mask-Based Embedding Router. arXiv:2506.19833, 2025.
[3] Zhong et al. AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement. arXiv:2511.23475, 2025.
[4] Perez et al. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI, 2018.
[5] Han et al. Leveraging Self-Supervised Learning for Speaker Diarization. ICASSP, 2025.
[6] Chung & Zisserman. Out of Time: Automated Lip Sync in the Wild. ACCVW, 2017.
[7] Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.
[8] Yang et al. InfiniteTalk: Audio-Driven Video Generation for Sparse-Frame Video Dubbing. arXiv:2508.14033, 2025.
Dear Reviewer bwpM,
Thank you again for your thoughtful and constructive comments. We understand the considerable time and effort involved in the reviewing process and sincerely appreciate your helpful suggestions.
Following your suggestions, we have:
- clarified how Fig. 2 contrasts prior spatial-binding baselines (Panels c–e) against SAM (Panel f) and aligns with the full M$^2$Talk pipeline in Fig. 3
- illustrated with a concrete inference example how binary SAT is built from a 1D timestamp schedule and shared audio without person tracking or 2D masks
- clarified TCP's cross-modal alignment role, ablated MVC's shot references vs. ID board (Appendix G.2, Table 11), and explained (Eq. 7; Appendices A.4–A.5)
- updated our Project Page with synchronized evolving SAT timelines and a live playback cursor across baseline and multi-party video demos
Given the limited discussion period, we sincerely hope to have further discussions if you have any remaining questions or concerns. We would greatly appreciate any further feedback you may have, as it would help us further improve our work.
Thank you again for your time and valuable feedback.
Response to Reviewer bwpM(1/2)
We thank Reviewer bwpM for recognizing our motivation and the effectiveness of SAM, MVC, and the M$^2$Talk pipeline. Below we address Weaknesses W1–W5 and Questions Q1–Q5 (line numbers refer to the revised manuscript).
Figure 2 (lines 181–185) contrasts prior audio-conditioning paradigms against SAM, whereas Figure 3 (lines 230–232) shows the full M$^2$Talk pipeline (SAM + TCP + MVC; visual overview at Method Overview):
Panels (a)–(b): Prior single-person audio conditioning (Section 2.1, lines 130–132).
Panels (c)–(e): Prior spatial-binding baselines rather than components of M$^2$Talk: (c) MultiTalk [1], (d) Bind-Your-Avatar [2], and (e) AnyTalker [3] (Section 2.1, lines 137–139; Section 3.2, lines 238–244).
Panel (f): Our SAM module, which modulates a single shared multi-speaker audio track via FiLM [4] before one unified audio cross-attention pass without per-speaker audio splitting or spatial masks (Fig. 2f, lines 184–185; Section 3.2, lines 253–257).
At inference, M$^2$Talk uses no person tracking, face detection, or 2D spatial masks, taking only one shared mono audio track plus a 1D speaking-interval schedule for the fixed slots defined in Section 3.2 (lines 258–264; interactive demo at SAT Studio):
Here,
speaker_id0 and 1 assign one-time discrete IDs from left-to-right in the static reference/ID board (2–3 unused; 4 shared off-screen; lines 259–264), remaining invariant as characters move, turn, or change camera angles across shots. Rasterizing onto length L yields M∈0,1L×S (S=5, line 277), e.g.,[1, 1, 0, 0, 0](1.2–1.5 s overlap),[0, 0, 0, 0, 1](3.5–4.0 s off-screen), and[0, 0, 0, 0, 0](silence), with zero pixel coordinates. Offline tracking and audio–visual alignment [5, 6] are used only when curating training labels, never at inference.As noted in Section 3.3 (lines 294–296), TCP builds on established trailing-frame reuse [1, 7, 8] rather than claiming frame reuse as a new primitive; its role is synchronizing trailing motion anchors (Zanc(k), r=9; Eqs. 5–6, lines 299–312; line 848) with the global audio/SAT/SAE window Wk (lines 313–316). Removing TCP drops MTHM Sync-C from 8.54 to 7.91, raises FVD from 695.91 to 862.36 (Table 3, line 465), and loses background motion continuity (Section 4.3, lines 478–482; Appendix C.2, Fig. 10, lines 1404–1425).
Trailing frames in TCP cannot supply appearance cues for new camera angles or characters absent from Shot 1; MVC pairs shot composition references (I1…N) with a cross-shot ID board (IID) aligned to SAT slots (Section 3.4, lines 320–342). Separating these components on Movie Dialogues (Section 4.3, lines 493–495; variant setups in Appendix G.2, Table 11, lines 1687–1720) confirms each input's role: