acceptodds
Under review as a conference paper at ICLR 2027

MTalk: Audio-Driven Multi-Person Multi-Shot Conversational Video Generation via Speaker-Identity Conditioning

Abstract

Audio-driven human video generation has progressed from animating single person to synthesizing multi-person conversations. Prevailing multi-person talking video frameworks rely heavily on spatially grounded audio binding, where speech features are hard-routed to target visual regions via spatial masks. However, this rigid spatial coupling fundamentally limits natural human interactions, frequently causing binding failures under dynamic movements, suppressing non-verbal listener dynamics, and failing during off-screen speech or shot transitions. To overcome these boundaries, we propose MTalk, a framework for multi-person multi-shot conversational video generation built on speaker-identity conditioning. Our key insight is to decouple who is speaking and when from visual localization. Specifically, Speaker-Aware Audio Modulation (SAM) encodes a supplied Speaker Activity Timeline (SAT) into time-aligned Speaker-Aware Embeddings (SAEs) that modulate acoustic features without per-person spatial masks. Temporal Context Propagation (TCP) preserves visual coherence across consecutive generation chunks, while Multi-Reference Visual Conditioning (MVC) supplies shot reference images and ID board via dedicated condition slots to support multi-shot transitions. Trained on diverse natural conversational videos, MTalk effectively handles complex turn-taking, off-screen speech, and multi-shot, achieving state-of-the-art performance on audio-driven video generation benchmarks. Project page: https://anonymous.4open.science/w/M2Talk-F011.

Primary area:
generative models
Keywords:
Audio-Driven Human Video Generation, Multi-Person Dialogue Generation, Multi-Person Multi-Shot Conversational Video Generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_bwpMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses an important problem of spatial-speaker binding in the multi-person video generation domains. Other than using person's location information, the paper presents a speaker-aware audio representation that allows the proposed method to track who is speaking and when directly in the acoustic representation. Thus, the motivation of the work is clear, and the addressed problem is also important. The most important contribution of this work is a new proposed speaker-aware audio modulation (SAM) to achieve spatial-mask-free speaker conditioning, which is an improvement to the existing spatial-masking paradigms that explicitly uses the person's locations.

Critical strengths

The most important strength of this work is a new proposed speaker-aware audio modulation (SAM) to achieve spatial-mask-free speaker conditioning, which is an improvement to the existing spatial-masking paradigms that explicitly use the person's locations.

The other important strength is the MULTI-REFERENCE VISUAL CONDITIONING (MVC) that adds an ID board of person crops across shots, explicitly maintaining identity cues throughout the entire generation.

The entire framework of the M2Talk could also be considered as one of the strengths, as this pipeline shows effectiveness compared to the existing works.

Critical weaknesses

Though the paper shows clear motivation for the proposed method, it also has several key concerns to be addressed.

The paper lists different flowchart comparisons in Figure 2 regarding the audio conditioning for single-person and conversational generation. However, this proposed flowchart, especially the bottom illustration, does not align correctly with the pipeline of the proposed approaches, which involves SAM, TCP, and MVC. After reviewing the Figure 2, it looks like the Speaker-Aware Audio Modulation (f) is the illustration that corresponds to the SAM description. What are these c, d, and e? These are not part of the proposed M2Talk work?

The core idea of this work is the proposed Speaker-aware audio modulation (SAM). Inside this module, Speaker activity timeline (SAT) also plays a vital role. The paper explains that the initial SAT uses a binary representation, M, which records the speaker's activity across the timelines and shots. One intriguing question is how to initialize the mask M. Does this still rely on applying person tracking techniques across the videos and associating them with the speaker information in the audio? If that is the case, then, this is still a traditional way of spatial-masking paradigms that explicitly use the person's locations. The difference seems to lie in the representations.

Although the paper claims the TEMPORAL CONTEXT PROPAGATION (TCP) to be one of the contributions. But this TCP just uses the trail frames of the previously generated trunk as the context for the new chunk generation. This has been regularly explored in the existing chunk-wise generation frameworks. The paper needs to show what the major difference is here.

The paper also needs to better explain the MULTI-REFERENCE VISUAL CONDITIONING (MVC) , where it concatenates the ID board information with the shot information. This also seems to be a regular concatenation operation. The difference is the choice of inputs here. Could you also explain what m_orig is and what role it plays?

The paper shows many video demos in the provided link. However, it does not seem to have a single video demo where it shows the evolving Speaker activity timeline (SAT) with the generated video. This should also apply to the comparisons against the baselines such that we can clearly see the drawbacks of existing spatial-masking schemes and how the proposed SAT can effectively address them to keep the speaker-aware audio injection.

Questions for authors

There are many questions for the authors. I have highlighted a few of them here.

Could you explain what are the purposes of listing the c, d, and e in Figure 2 and how are they aligned with the proposed method in this work?

Could you explain on how to generate the binary representation M in SAT? Whether there is a person tracking technique used?

Could you explain what are the major difference of MVC compared to the existing chunk-wise generations?

Please explain what m_orig is and what role it plays.

Please provide clear evolving Speaker activity timeline (SAT) with the generated video in the demo links.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bwpM(1/2)

We thank Reviewer bwpM for recognizing our motivation and the effectiveness of SAM, MVC, and the M$^2$Talk pipeline. Below we address Weaknesses W1–W5 and Questions Q1–Q5 (line numbers refer to the revised manuscript).

W1 & Q1: Role of Panels (c), (d), and (e) in Figure 2 vs. Figure 3

Figure 2 (lines 181–185) contrasts prior audio-conditioning paradigms against SAM, whereas Figure 3 (lines 230–232) shows the full M$^2$Talk pipeline (SAM + TCP + MVC; visual overview at Method Overview):

Panels (a)–(b): Prior single-person audio conditioning (Section 2.1, lines 130–132).

Panels (c)–(e): Prior spatial-binding baselines rather than components of M$^2$Talk: (c) MultiTalk [1], (d) Bind-Your-Avatar [2], and (e) AnyTalker [3] (Section 2.1, lines 137–139; Section 3.2, lines 238–244).

Panel (f): Our SAM module, which modulates a single shared multi-speaker audio track via FiLM [4] before one unified audio cross-attention pass without per-speaker audio splitting or spatial masks (Fig. 2f, lines 184–185; Section 3.2, lines 253–257).

W2 & Q2: How Binary SAT M\mathbf{M} Is Constructed and Why Inference Is Spatial-Mask-Free

At inference, M$^2$Talk uses no person tracking, face detection, or 2D spatial masks, taking only one shared mono audio track plus a 1D speaking-interval schedule for the fixed slots defined in Section 3.2 (lines 258–264; interactive demo at SAT Studio):

{
  "audio": "conversation.wav",
  "duration": 4.5,
  "speaker_intervals": [
    {"speaker_id": 0, "start_time": 0.0, "end_time": 1.5},
    {"speaker_id": 1, "start_time": 1.2, "end_time": 3.0},
    {"speaker_id": 4, "start_time": 3.5, "end_time": 4.0}
  ]
}

Here, speaker_id 0 and 1 assign one-time discrete IDs from left-to-right in the static reference/ID board (2–3 unused; 4 shared off-screen; lines 259–264), remaining invariant as characters move, turn, or change camera angles across shots. Rasterizing onto length LL yields M∈0,1L×S\mathbf{M}\in\\{0,1\\}^{L\times S} (S=5S=5, line 277), e.g., [1, 1, 0, 0, 0] (1.2–1.5 s overlap), [0, 0, 0, 0, 1] (3.5–4.0 s off-screen), and [0, 0, 0, 0, 0] (silence), with zero pixel coordinates. Offline tracking and audio–visual alignment [5, 6] are used only when curating training labels, never at inference.

W3: Role of TCP Relative to Existing Chunk-Wise Continuation

As noted in Section 3.3 (lines 294–296), TCP builds on established trailing-frame reuse [1, 7, 8] rather than claiming frame reuse as a new primitive; its role is synchronizing trailing motion anchors (Zanc(k)\mathbf{Z}_{\text{anc}}^{(k)}, r=9r=9; Eqs. 5–6, lines 299–312; line 848) with the global audio/SAT/SAE window Wk\mathcal{W}_k (lines 313–316). Removing TCP drops MTHM Sync-C from 8.54 to 7.91, raises FVD from 695.91 to 862.36 (Table 3, line 465), and loses background motion continuity (Section 4.3, lines 478–482; Appendix C.2, Fig. 10, lines 1404–1425).

W4 (Part 1) & Q3: How MVC Differs from Naive Concatenation and Chunk-Wise Continuation

Trailing frames in TCP cannot supply appearance cues for new camera angles or characters absent from Shot 1; MVC pairs shot composition references (I1…N\mathbf{I}_{1\dots N}) with a cross-shot ID board (IID\mathbf{I}_{\text{ID}}) aligned to SAT slots (Section 3.4, lines 320–342). Separating these components on Movie Dialogues (Section 4.3, lines 493–495; variant setups in Appendix G.2, Table 11, lines 1687–1720) confirms each input's role:

MethodBA ↑ (%)BA near shot transitions ↑ (%)OLR ↓ (%)
M$^2$Talk (Ours)74.2679.963.78
w/o ID board56.6755.9511.66
w/o MVC31.1823.1511.43
AuthorsOpenReview ↗

Response to Reviewer bwpM(2/2)

W4 (Part 2) & Q4: Definition and Role of morig\mathbf{m}_{\mathrm{orig}}

As defined in Section 3.4 (Eq. 7, lines 331–339) and Appendix A.4 (lines 913–919), morig\mathbf{m}_{\mathrm{orig}} is the binary image-condition availability mask of the Wan2.1-I2V [7] reference pathway (1 at the supplied initial frame in Iorig\mathbf{I}_{\mathrm{orig}}, 0 at zero-padded frames), extended by MVC to m~ref=[1K∣morig]\widetilde{\mathbf{m}}_{\mathrm{ref}} = [\mathbf{1}_K \mid \mathbf{m}_{\mathrm{orig}}]. It marks only which latent positions hold reference images, distinct from SAT M\mathbf{M} and the loss ignore mask Ωk\Omega_k (Appendix A.5, line 961).

W5 & Q5: Synchronized Evolving SAT Visualizations in Video Demos

We have updated our anonymous project page with synchronized evolving SAT timelines and a moving playback cursor directly beneath generated videos, both for the side-by-side baseline comparisons (30s Baseline Comparisons, noting in captions that each baseline uses its native input interface) and multi-party conversations (3–4 Person Demos, plus interactive exploration at SAT Studio), enabling frame-by-frame verification of active person slots, silence, and overlaps against mouth movements.

References:

[1] Kong et al. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation. arXiv:2505.22647, 2025.

[2] Huang et al. Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-Mask-Based Embedding Router. arXiv:2506.19833, 2025.

[3] Zhong et al. AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement. arXiv:2511.23475, 2025.

[4] Perez et al. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI, 2018.

[5] Han et al. Leveraging Self-Supervised Learning for Speaker Diarization. ICASSP, 2025.

[6] Chung & Zisserman. Out of Time: Automated Lip Sync in the Wild. ACCVW, 2017.

[7] Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[8] Yang et al. InfiniteTalk: Audio-Driven Video Generation for Sparse-Frame Video Dubbing. arXiv:2508.14033, 2025.

Dear Reviewer bwpM,

Thank you again for your thoughtful and constructive comments. We understand the considerable time and effort involved in the reviewing process and sincerely appreciate your helpful suggestions.

Following your suggestions, we have:

  • clarified how Fig. 2 contrasts prior spatial-binding baselines (Panels c–e) against SAM (Panel f) and aligns with the full M$^2$Talk pipeline in Fig. 3
  • illustrated with a concrete inference example how binary SAT M\mathbf{M} is built from a 1D timestamp schedule and shared audio without person tracking or 2D masks
  • clarified TCP's cross-modal alignment role, ablated MVC's shot references vs. ID board (Appendix G.2, Table 11), and explained morig\mathbf{m}_{\mathrm{orig}} (Eq. 7; Appendices A.4–A.5)
  • updated our Project Page with synchronized evolving SAT timelines and a live playback cursor across baseline and multi-party video demos

Given the limited discussion period, we sincerely hope to have further discussions if you have any remaining questions or concerns. We would greatly appreciate any further feedback you may have, as it would help us further improve our work.

Thank you again for your time and valuable feedback.

Reviewer_AbXwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents M²Talk, an audio-driven framework for multi-person, multi-shot conversational video generation built on Wan2.1-I2V-14B. Speaker-Aware Audio Modulation encodes an externally supplied speaker activity timeline and modulates acoustic features through FiLM, avoiding explicit per-person spatial masking of audio-attention outputs. Temporal Context Propagation reuses recent generated frames, while Multi-Reference Visual Conditioning incorporates shot references and an identity board. Experiments evaluate generation quality, speaker binding, shot transitions, and human preferences.

Critical strengths

SAM provides a simple and effective mechanism for explicit speaker control. It uses speaker-activity embeddings to modulate acoustic features without explicit per-person spatial masking of audio attention. The controlled ablation in Table 3 provides strong evidence of its usefulness: removing SAM reduces BA from 91.11 to 34.00 and PBA from 88.22 to 25.77, while increasing OLR from 0 to 100. These results support the effectiveness of activity-conditioned modulation for controlling visible and off-screen speech.

The evaluation includes useful controlled tests and transparent metric interpretation. MPMS tests speaker assignment by swapping activity annotations while keeping the reference image and audio unchanged, providing a targeted assessment beyond visual quality. The multi-shot experiments cover three datasets and demonstrate substantial improvements in cut timing and face coverage with MVC. Appendix B also explicitly acknowledges that high cross-shot consistency can result from failing to produce requested cuts, and reports coverage alongside identity metrics, helping readers interpret the results appropriately.

Critical weaknesses

1.Can you evaluate visibility-aware speaker binding, off-screen leakage, and per-speaker lip synchronization on multi-shot clips, particularly around cuts and speaker re-entry? Please report detection coverage and complement unreliable automatic measurements with human evaluation. PBA would also be informative if paired assignment-swapped samples are constructed.

2.How was speaker-to-person correspondence supplied to each MPMS baseline? Please clarify the conversion of SAT into each method’s native conditions, including overlapping and off-screen speech, and report differences in reference inputs, prompts, resolutions, and sampling settings.

3.Were Tables 1–4 computed using one generation per example or multiple seeds? Could you provide per-video results and video/pair-level confidence intervals for the principal comparisons, particularly BA/PBA against AnyTalker, while distinguishing variation across examples from sampling variability?

4.What specific advantage does SAM provide over CovOG-style speaking-score modulation? Could you compare alternative conditioning mechanisms under the same SAT supervision, backbone, training data, and budget to separate the benefits of modulation design from those of removing spatial routing?

Questions for authors

Can you report per-speaker lip synchronization and binding accuracy on the multi-shot sets, particularly around cuts, off-screen transitions, and speaker re-entry?

Under the same training data, supervision, and budget, how does SAM compare with spatial routing and simpler SAT-conditioning alternatives, such as additive or concatenated speaker embeddings? Does the listener-motion advantage persist?

How are shot references injected in “w/o MVC”? Can you separately ablate the ID board and reference-prefix mechanism, and include a shot-wise audio-driven baseline using the same references and audio segments?

During overlapping speech, how does the model determine which acoustic component belongs to each active person? Can you evaluate speaker-specific synchronization separately for overlapping speech and for three- or four-person scenes?

How were the training SAT annotations and cross-shot identity correspondences obtained? Please clarify the annotation pipeline and train–test separation, and report uncertainty across test scenes for the main binding and human-preference results.

9 comments
AuthorsOpenReview ↗

Response to Reviewer AbXw (1/3)

We thank Reviewer AbXw for recognizing SAM's speaker-control design and paired MPMS protocol, and for the constructive suggestions on cross-shot evaluation, baseline interfaces, and ablations. Below we address W1–W4 and Q1–Q5 (line numbers refer to the revised manuscript).

W1 & Q1: Multi-Shot Speaker Binding, Off-Screen Leakage, and Lip Synchronization Around Cuts

While Table 4 evaluates cut timing and identity/frame quality (where M$^2$Talk reaches 89.6% face coverage and 100.0% cross-shot pair coverage; Table 4, line 472; Appendix B.2, lines 1018–1046), we additionally evaluate Movie Dialogues on overall BA, transition-adjacent BA (around cuts, off-screen transitions, and speaker re-entry), OLR, and active-speaker Sync-C / Sync-D [1] (Section 4.3, lines 493–495; Appendix G.2, Table 11, lines 1687–1724):

MethodBA ↑ (%)BA near shot transitions ↑ (%)OLR ↓ (%)Sync-C ↑Sync-D ↓
M$^2$Talk (Ours)74.2679.963.787.757.92
w/o ID board56.6755.9511.66––
w/o MVC31.1823.1511.43––
H3 [2]64.8260.0269.504.269.56

Missing TalkNet detections remain in the BA denominator as incorrect (Appendix A.3, Eq. 9, lines 872–878), and representative transition frames were manually checked against detector errors (lines 1715–1716). Sync-C/D are omitted for w/o MVC and w/o ID board due to cross-shot misattribution; H3 [2] receives speaker turns via the shared timestamped script (Table 5, lines 972–983) yet suffers 69.50% OLR vs. 3.78% for M$^2$Talk; and paired swapped PBA is evaluated on co-visible MPMS pairs (Table 2) because single-character Movie Dialogues close-ups conflict with swapped assignments (lines 1719–1724).

W2: Supplying Speaker Correspondence and Inference Settings for MPMS Baselines

SAT-to-baseline conversion: Across the 20 MPMS original/swapped pairs (Appendix A.1, lines 832–838), M$^2$Talk takes the single mixed audio track and SAT directly, whereas spatial baselines require per-person audio streams paired with localized person/face regions. We mask each speaker's audio stream by their active SAT intervals: inactive and off-screen-only intervals are zeroed out to silence (preventing off-screen routing to visible listeners), while overlapping intervals retain speech in both streams.

Reference, prompt, resolution, and sampling settings: MultiTalk [3], Bind-Your-Avatar [4], and AnyTalker-14B [5] all run via their official pipelines using native localization on the shared reference image, shared scene prompts with native negative prompts, default sampling settings, and native resolutions (including 896×448 for MultiTalk [3] and M$^2$Talk and 864×416 for AnyTalker-14B [5]; Table 9, lines 1620–1626); M$^2$Talk's exact sampling settings and negative prompts are in Appendix A.2 (lines 841–858).

AuthorsOpenReview ↗

Response to Reviewer AbXw (2/3)

W3 & Q5 (Part 2): Seed Protocol and Scene/Pair-Level Uncertainty

Seed protocol & MPMS bootstrap CIs (Appendix G.3, Table 12, lines 1728–1744): Tables 1–4 share one deterministic seed per example across methods over distinct test scenes. Pair-preserving bootstrap over the 20 MPMS scene pairs (capturing cross-scene variation rather than multi-seed sampling variance) gives M$^2$Talk − AnyTalker-14B [5] differences of −1.25-1.25\\% (95% CI [−2.51[-2.51\\%, +0.06\\%], crossing zero) on BA (N=40N=40) and −2.56-2.56\\% (95% CI [−4.43[-4.43\\%, -0.70\\%]) on PBA (N=20N=20), showing statistically indistinguishable overall binding alongside AnyTalker's small mouth-crop PBA edge from explicit face masking (Section 4.2, lines 407–414).

Human-preference scene-level uncertainty: Applying scene-cluster bootstrap across the 10 30-second test scenes (resampling entire scenes while preserving all 10 participants and 6 dimensions per scene; Table 2, lines 401–404; Table 8, lines 1350–1361), M2Talk\text{M}^2\text{Talk} achieves a six-dimension mean first-place share of 91.3391.33\\% (95% CI [83.83[83.83\\%, 96.67\\%]; two-way participant$\times$scene bootstrap [80.00[80.00\\%, 98.33\\%]), compared with 6.836.83\\% ([2.33[2.33\\%, 13.33\\%]) for MultiTalk [3] and 1.831.83\\% ([0.67[0.67\\%, 3.33\\%]) for AnyTalker-14B [5]. Paired scene-level differences remain strictly positive at +84.50+84.50 pp (95% CI [+70.33,+94.17][+70.33, +94.17]) over MultiTalk and +89.50+89.50 pp ([+80.83,+96.00][+80.83, +96.00]) over AnyTalker-14B (with standalone Overall Preference at 94.0094.00\\%, 95% CI [86.00[86.00\\%, 100.00\\%]; and 77.5077.50\\%–86.2586.25\\% across dimensions in the controlled study of Table 10, lines 1674–1685).

W4 & Q2: Comparison with CovOG, Simpler SAT Conditioning, and Spatial Routing

CovOG and linear/additive alternatives: CovOG [6] (Sec. 4.1, Eq. 2) scales per-person audio features by speaking scores, eaudioi=eaudio⋅σ(MLP(ai))e_{\text{audio}}^i = e_{\text{audio}} \cdot \sigma(\text{MLP}(a^i)), but still relies on head-keypoint face masks in cross-attention and skeletal pose guidance (added in Section 2.1, lines 140–142), whereas SAM modulates shared audio before unmasked global cross-attention (lines 253–257). Unlike addition or concatenation before a linear layer (W[at;st]+b=Waat+Wsst+b\mathbf{W}[\mathbf{a}_t; \mathbf{s}_t] + \mathbf{b} = \mathbf{W}_a \mathbf{a}_t + \mathbf{W}_s \mathbf{s}_t + \mathbf{b}, where ∂a^t/∂at=Wa\partial \widehat{\mathbf{a}}_t / \partial \mathbf{a}_t = \mathbf{W}_a is independent of st\mathbf{s}_t, so a state-dependent offset cannot directly gate time-varying phonetic fluctuations WaΔat\mathbf{W}_a \Delta \mathbf{a}_t at this layer during active off-screen speech), FiLM [7] (Eq. 4, lines 279–286) introduces state-dependent multiplicative gating ∂a^t/∂at=diag(1+γt(st))\partial \widehat{\mathbf{a}}_t / \partial \mathbf{a}_t = \mathrm{diag}(1 + \boldsymbol{\gamma}_t(\mathbf{s}_t)), allowing active off-screen speech (eS\mathbf{e}_S) and background noise (e0\mathbf{e}_0) to be attenuated at the feature level before cross-attention rather than relying solely on downstream DiT blocks to cancel leaked speech variance (our empirical ablation below isolates SAM against spatial routing rather than separately training 14B additive/concatenated variants).

Controlled spatial-routing study: Under identical Wan2.1-I2V-14B backbone [8], training data, TCP, and Stage 1 budget (200k steps, batch size 32), M$^2$Talk (TCP + SAM) outperforms Reproduced (TCP + MultiTalk Spatial Routing) [3] across 80 blind judgments (Section 4.3, lines 430–431, 476–477; Appendix G.1, Table 10, lines 1674–1710):

Evaluation DimensionM$^2$Talk (TCP + SAM) First ↑Reproduced (TCP + Spatial Routing) First ↑Undecided
Identity Consistency81.25%16.25%2.50%
Temporal Continuity77.50%21.25%1.25%
Sustained Visual Quality77.50%22.50%0.00%
Speaker Motion Naturalness83.75%16.25%0.00%
Listener Motion Naturalness86.25%11.25%2.50%
Overall Preference80.00%20.00%0.00%
AuthorsOpenReview ↗

Response to Reviewer AbXw (3/3)

Q3: w/o MVC vs. w/o ID board Setup and Shot-Wise Stitched Baseline

w/o MVC vs. w/o ID board (Appendix G.2, lines 1717–1720): All M$^2$Talk variants in W1 prepend I1…N\mathbf{I}_{1\dots N} via Eq. (7) (lines 329–342) at inference: w/o MVC evaluates the Stage 1 checkpoint (trained without MVC), w/o ID board trains Stage 2 LoRA (10k steps) with I1…N\mathbf{I}_{1\dots N} only (without IID\mathbf{I}_{\text{ID}}), and M$^2$Talk trains Stage 2 LoRA with both IID\mathbf{I}_{\text{ID}} and I1…N\mathbf{I}_{1\dots N}.

Shot-wise stitched baseline: Splitting a clip at ground-truth cut timestamps reduces cut timing to manual post-processing and turns each segment into single-shot generation, already evaluated on HDTF, EMTD, MTHM, and MPMS (Tables 1–2, lines 378–404). Table 4 (lines 483–487) evaluates unified end-to-end multi-shot synthesis across cuts.

Q4: Overlapping Speech Attribution and 3–4 Person Scenes

During overlap, SAM sums active slot embeddings (st=∑smt,ses\mathbf{s}_t = \sum_s m_{t,s}\mathbf{e}_s, Eq. 3, lines 270–274) on the shared audio stream without explicit source separation. Voice-to-person attribution is anchored via temporal continuity from surrounding solo turns: natural overlaps (interruptions, backchannels, or turn handoffs) typically adjoin single-speaker frames τ\tau where sτ=es\mathbf{s}_\tau = \mathbf{e}_s uniquely binds slot ss to one speaker's multi-layer Wav2Vec acoustic cues (pitch and vocal timbre; Eq. 1, lines 195–203) and lip trajectory; when entering an overlap (st=e1+e2\mathbf{s}_t = \mathbf{e}_1 + \mathbf{e}_2), DiT 3D spatiotemporal self-attention and TCP (Section 3.3, lines 294–316) propagate each character's established voice–identity association across frames (whereas cold-start overlap without solo context remains an inherent limitation of unseparated audio). Because SyncNet [1] on unseparated mixtures suffers cross-speaker interference and public benchmarks (MTHM, MPMS) have two visible speakers, we provide qualitative overlapping-speech and 3–4 person videos (with source audio and synchronized SATs) at 3–4 Person Demos and SAT Studio.

Q5 (Part 1): Training Annotations, Cross-Shot Correspondence, and Train–Test Disjointness

Stage 1 (~2K hours, lines 347–349) follows MultiTalk's curation pipeline [3]; Stage 2 adds cross-shot clips. YOLO person/face detection and ByteTrack extract up to 4 active character tracks (S−1=4S{-}1=4, line 277; filtering out short, low-visibility, or static bystander tracks) to select ID-board portraits and link identities across shots for fixed left-to-right slots and per-shot references. DiariZen [9] and SyncNet [1] bind speech intervals to tracks and label verified off-screen speech, while QC rejects clips with unresolved identities, low sync, small per-shot face area, or uncovered characters. All seven evaluation sets (lines 349–356) are strictly disjoint from Stage 1 and Stage 2 training sources.

References:

[1] Chung & Zisserman. Out of Time: Automated Lip Sync in the Wild. ACCVW, 2017.

[2] MiniMax. Open General Intelligence: MiniMax H3 Is Now Open Source. 2026.

[3] Kong et al. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation. arXiv:2505.22647, 2025.

[4] Huang et al. Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-Mask-Based Embedding Router. arXiv:2506.19833, 2025.

[5] Zhong et al. AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement. arXiv:2511.23475, 2025.

[6] Zhu et al. Multi-Human Interactive Talking Dataset. arXiv:2508.03050, 2025.

[7] Perez et al. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI, 2018.

[8] Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[9] Han et al. Leveraging Self-Supervised Learning for Speaker Diarization. ICASSP, 2025.

Reviewer_Uu8LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

M2Talk targets at generating videos in audio-driven multi-person, multi-shot context. Its core technology involves a system architecture that combines multiple components: MVC, SAM, TCP and MVC, among which SAM and TCP are considered as novel elements. The experiments show M2Talk achieve SOTA in multiple aspects and the ablation further show the effect of SAM and TCP.

Critical strengths

The proposed solution tackles a complex real-world challenge of multi-person, multi-shot conversational video generation. The dynamics of such video scenes make it challenging for computational models. The proposed architecture consists of dedicated modules to tackle various aspects of the challenge, making it a sound solution. The authors conducted extensive experiments with comparisons with competitive benchmarks/baselines. The proposed solution excels in multiple metrics.

Critical weaknesses

While the overall solution consists of complicated technical components, the methodological merit is unclear due to the intricate mechanisms and unclear flow of logic. The mathematical development is obscure with weak logical connections between the equations and reasoning, despite a modular process in Figure 3. In essence, it is difficult to follow and validate if the derivation is correct or not. Compared to the relatively clear, strong practical relevance, the theoretical contribution is not substantial, which is a critical factor for ICLR community. The authors provide a bunch of examples in the anonymous website. However, they only show cherry-picked appealing examples and do not explain adequately how the proposed solution translates to visible gain in the actual tasks. How to interpret the output with respect to the solution and against other methods?

Questions for authors

What are the methodological/theoretical contributions other than a complicated solution framework? How to interpret the examples with respect to the method and in comparison with other methods? (provide more explanations) Show negative cases and explain the limitations of the proposed solution accordingly.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Uu8L (1/2)

We thank Reviewer Uu8L for the positive assessment and constructive suggestions on clarifying our methodological formulation, mathematical flow, and failure analysis (line numbers refer to the revised manuscript).

W1 (Part 1), W2 & Q1: Representation Disentanglement Beyond System Integration

For the ICLR community, our core contribution is not an engineering stack of modules, but a condition-side representation disentanglement paradigm [1, 2] for multi-speaker generative modeling: disentangling temporal speaker-identity attribution (who speaks when) from 2D spatial layout (where subjects appear in pixel space (x,y)(x, y)) (Section 1, lines 096–109; Section 3.2, lines 237–257):

  1. Spatial-entanglement bottleneck in 2D masking: Prior multi-person methods (Fig. 2c–e, lines 171–179, 238–244) entangle speaker identity with pixel coordinates (x,y)(x, y) via post-attention spatial masks:

    Hvout=∑pMspatial(p)(x,y,t)⊙Attn(Qv,Ka,p,Va,p)\mathbf{H}_v^{\text{out}} = \sum_{p} \mathbf{M}_{\text{spatial}}^{(p)}(x, y, t) \odot \text{Attn}\big(\mathbf{Q}_v, \mathbf{K}_{a,p}, \mathbf{V}_{a,p}\big)

    Entangling identity with (x,y)(x, y) causes three structural failures: (i) across shot cuts or off-screen speech, Mspatial(p)(x,y,t)\mathbf{M}_{\text{spatial}}^{(p)}(x, y, t) jumps discontinuously or becomes empty (∅\emptyset), causing routing breakdown or off-screen lip leakage onto visible listeners; (ii) regional masking factorizes pixel space and disrupts joint modeling of pθ(Xlistener,Xspeaker∣a)p_\theta(\mathbf{X}_{\text{listener}}, \mathbf{X}_{\text{speaker}} \mid \mathbf{a}), suppressing listener reactions and co-speech body dynamics (lines 086–091, 245–249); and (iii) online spatial localization degrades under head turns or occlusions.

  2. Condition-side representation disentanglement via SAM: Consistent with disentanglement theory showing that separating generative factors requires explicit inductive biases and factor supervision [1], SAM disentangles speaker attribution from spatial coordinates directly in the 1D conditioning space rather than constraining 2D visual tokens. Discrete slot states mt\mathbf{m}_t (active speakers, silence e0\mathbf{e}_0, off-screen eoff\mathbf{e}_{\text{off}}; lines 258–278) modulate 1D acoustic tokens via FiLM [3] before unmasked global DiT cross-attention [4] (Eq. 4, lines 279–290):

    a^t=(1+γt(mt))⊙at+βt(mt),Hvout=Attn(Qv,Ka(a^),Va(a^))\widehat{\mathbf{a}}_t = \big(1 + \boldsymbol{\gamma}_t(\mathbf{m}_t)\big) \odot \mathbf{a}_t + \boldsymbol{\beta}_t(\mathbf{m}_t), \qquad \mathbf{H}_v^{\text{out}} = \text{Attn}\big(\mathbf{Q}_v, \mathbf{K}_a(\widehat{\mathbf{a}}), \mathbf{V}_a(\widehat{\mathbf{a}})\big)

Visual tokens interact globally in a single cross-attention pass over one shared mixed audio stream (eliminating per-speaker stem separation and separate per-person attention branches; Fig. 2f, lines 184–185), so spatial layout and listener dynamics emerge jointly from pθ(X∣a^)p_\theta(\mathbf{X} \mid \widehat{\mathbf{a}}). Holding the Wan2.1-I2V-14B backbone [5], training data, TCP, and 200k steps fixed, replacing spatial routing with SAM yields an 86.25% vs. 11.25% blind preference on Listener Motion Naturalness and 80.00% vs. 20.00% overall (Section 4.3, lines 430–431, 476–477; Appendix G.1, Table 10, lines 1674–1710).

  1. Generalizability beyond video generation: Encoding entity attribution into 1D conditioning representations prior to global cross-attention extends naturally to broader Diffusion Transformers [4] (such as multi-subject image/3D synthesis and multi-source audio–visual generation) whenever entity-specific control must be bound without rigid 2D layout masks.
AuthorsOpenReview ↗

Response to Reviewer Uu8L (2/2)

W1 (Part 2): Clarifying the Mathematical Flow Across Equations (1)–(7)

Equations (1)–(7) form a single three-stage forward pass (M,at→acond(k)→X^(k)\mathbf{M}, \mathbf{a}_t \to \mathbf{a}_{\text{cond}}^{(k)} \to \widehat{\mathbf{X}}^{(k)}; interactive illustration at SAT Studio):

  1. SAM audio modulation: Windowed audio features at\mathbf{a}_t (Eq. 1, lines 195–203) are modulated by summed slot embeddings st\mathbf{s}_t (Eq. 3, lines 270–278) via FiLM into a^t\widehat{\mathbf{a}}_t (Eq. 4, lines 279–286) and temporally grouped onto the latent grid as acond(k)\mathbf{a}_{\text{cond}}^{(k)} (Eq. 13, Appendix A.4, lines 921–931).

  2. TCP & MVC latent prefixing: Trailing frames yield clean motion anchors Zanc(k)\mathbf{Z}_{\text{anc}}^{(k)} (Eqs. 5–6, lines 299–312), while IID\mathbf{I}_{\text{ID}} and I1…N\mathbf{I}_{1\dots N} form the length-KK reference prefix prepended to X~τ(k)\widetilde{\mathbf{X}}_\tau^{(k)}, I~ref\widetilde{\mathbf{I}}_{\text{ref}}, and m~ref\widetilde{\mathbf{m}}_{\text{ref}} (Eq. 7, lines 324–342) under shared window Wk\mathcal{W}_k (lines 313–316).

  3. Denoising, decoding to X^(k)\widehat{\mathbf{X}}^{(k)} & masked loss: Conditioned on (X~τ(k),m~ref,I~ref,acond(k))(\widetilde{\mathbf{X}}_\tau^{(k)}, \widetilde{\mathbf{m}}_{\text{ref}}, \widetilde{\mathbf{I}}_{\text{ref}}, \mathbf{a}_{\text{cond}}^{(k)}), the DiT predicts rectified-flow velocity v^θ,rest(k)\widehat{\mathbf{v}}_{\theta,\text{rest}}^{(k)} (Eqs. 14–16, Appendix A.5, lines 935–961); after denoising and removing the KK reference-prefix positions (lines 340–341), VAE decoding produces the output chunk X^(k)\widehat{\mathbf{X}}^{(k)} (Eq. 2, lines 208–215), keeping the reference-availability mask m~ref\widetilde{\mathbf{m}}_{\text{ref}} (lines 337–339, 915–919), SAT M\mathbf{M} (lines 258–264), and loss ignore mask Ωk\Omega_k (lines 952–961) distinct.

W3 & Q2: Interpreting Qualitative Demos Relative to Baselines

Our updated project page directly illustrates the failure modes analyzed in Section 1 (lines 081–095) and addresses cherry-picking concerns:

  1. Side-by-side 30-second comparisons vs. MultiTalk [6] and AnyTalker [7] (30s Comparisons): Across the scenes (Section 4.2, Fig. 4, lines 415–423; Appendix C.1 & C.3, Table 8 and Figs. 11–13, lines 1350–1361, 1458–1608), spatial-mask baselines show frozen listener poses (AnyTalker) or late-stage drift (MultiTalk at 29 s), whereas M$^2$Talk sustains natural listener reactions and co-speech gestures, directly validating our design motivation for SAM.

  2. 3–4 person & multi-shot transitions (3–4 Person Demos, Multi-Shot Demos): Across multi-party turn-taking, camera cuts, and character entry/re-entry (Fig. 1; Appendix B.4, Figs. 7–9, lines 1188–1332), SAM + MVC preserves turn assignment, introduces characters absent from Shot 1 via the ID board, and prevents lip leakage on silent listeners during off-screen speech.

  3. Randomly sampled additional cases (More Videos): To further rule out cherry-picking, we have added 25 randomly selected conversational cases on the project page.

Q3: Negative Cases, Limitations, and Open-Source Release

While SAM and MVC reliably maintain speaker attribution and lip synchronization, M$^2$Talk inherits visual generation priors from the underlying Wan2.1-I2V-14B [5] video backbone, which may occasionally exhibit minor hand or small-object artifacts during rapid gestures or fine-grained hand–object interactions (see Failure Cases), and will naturally benefit from stronger future video backbones. Alongside the 25 randomly sampled cases (More Videos), we will publicly release our inference code and model weights to share our technical and theoretical advances with the community.

References:

[1] Locatello et al. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. ICML, 2019.

[2] Denton & Birodkar. Unsupervised Learning of Disentangled Representations from Video. NeurIPS, 2017.

[3] Perez et al. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI, 2018.

[4] Peebles & Xie. Scalable Diffusion Models with Transformers. ICCV, 2023.

[5] Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[6] Kong et al. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation. arXiv:2505.22647, 2025.

[7] Zhong et al. AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement. arXiv:2511.23475, 2025.

Dear Reviewer Uu8L,

Thank you again for your thoughtful and insightful comments. We understand the considerable time and effort involved in the reviewing process and sincerely appreciate your constructive suggestions.

Following your suggestions, we have:

  • formulated our core contribution through condition-side representation disentanglement and discussed its generalizability across Diffusion Transformers
  • clarified the three-stage mathematical flow across Eqs. (1)–(7) and separated the roles of all masks (Sec. 3.4; Appendices A.4–A.5)
  • interpreted our side-by-side 30s baseline and multi-shot comparisons with synchronized SAT timelines on our Project Page
  • added 25 randomly sampled cases and backbone boundary cases on the project page, alongside our commitment to release inference code and weights

Given the limited discussion period, we sincerely hope to have further discussions if you have any remaining questions or concerns. We would greatly appreciate any further feedback you may have, as it would help us further improve our work.

Thank you again for your time and valuable feedback.

Reviewer_6zgWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies audio-driven conversational video generation with multiple participants, off-screen speech, and changing camera shots. M2Talk uses Speaker-Aware Audio Modulation (SAM) to convert a supplied Speaker Activity Timeline (SAT) into learned embeddings that modulate acoustic features through FiLM. Temporal Context Propagation (TCP) carries recent generated frames across chunks, while Multi-Reference Visual Conditioning (MVC) introduces shot references and an identity board. Experiments report substantial speaker-binding improvements over a no-SAM ablation, strong requested-cut timing on three multi-shot evaluation sets, and pronounced human preference for the model's 30-second outputs. The multi-shot evaluation lacks quantitative speaker-assignment results around cuts, and the experiments do not isolate the proposed causal explanation for improved listener motion.

Critical strengths

A useful conversational control interface with substantial ablation gains. Section 3.2 and Eqs. (3)–(4) explicitly represent visible-person activity, off-screen speech, overlapping activity, and non-speech. With training data and steps fixed, Table 3 reports that SAM improves binding accuracy (BA) from 34.00% to 91.11% and paired binding accuracy (PBA) from 25.77% to 88.22%, while reducing off-screen leakage rate (OLR) from 100% to 0%. These results support the importance of activity-conditioned audio within the proposed model and address a practical problem beyond single-person lip synchronization. Strong shot-control results and encouraging perceptual evidence, supported by detailed evaluation protocols. Table 6 reports cut F1 scores of 97.62%, 100.00%, and 96.55% on the AVA-AVD and UnityShots evaluation subsets and the curated Movie Dialogues set, respectively. The numbers of clips with every requested cut matched and no extra detected cut increase from 9/40, 0/12, and 0/40 without MVC to 39/40, 12/12, and 37/40. Appendix C.1 reports 91% first-place preference for listener naturalness and 94% overall preference from ten participants ranking ten 30-second samples. The randomized anonymous presentation, explicit acknowledgment of repeated judgments, and detailed metric and coverage definitions in Appendices A–C improve the interpretability of these results. Together, these findings support the practical value of the complete system.

Critical weaknesses

The multi-shot evaluation lacks quantitative evidence of speaker assignment across cuts. Figure 1 illustrates multi-shot generation with supplied activity labels, including a participant appearing in a later shot and off-screen speech. The quantitative speaker-binding evaluation on MPMS tests two visible-person channels and their swapped assignments (Appendices A.1 and A.3), without reporting results specifically around shot transitions. The multi-shot results in Tables 4, 6, and 7 measure cuts, reference similarity, identity coverage, consistency, and visual quality. Appendix B.2 explicitly states that no audio-quality or lip-sync scores are computed in that evaluation. These reported metrics leave speaker assignment and visible-listener leakage around cuts unmeasured. This gap directly affects the central motivation in Sections 1 and 2.2: maintaining speech assignment when participants disappear, reappear, or first enter in later shots. Please evaluate speaker assignment and off-screen leakage around transitions on a predefined subset of the existing multi-shot outputs, and check detector reliability on representative frames. The experiments do not isolate the effect of spatial masking on listener motion. Sections 3.2 and 4.2 attribute rigid listener behavior to spatial constraints. Figure 4 and Appendix C provide qualitative comparisons and human preferences for listener naturalness. Table 3 evaluates SAM and TCP removal using generation-quality and speaker-binding metrics, without a listener-motion score. Appendix C.1 compares M2Talk, MultiTalk, and AnyTalker as complete systems. A matched comparison that varies spatial routing while holding the remaining training and conditioning setup fixed is absent, leaving the proposed cause of the listener-motion advantage unresolved. The superiority claims in Section 1's contribution list and Section 4.2 also need qualification against Table 2. M2Talk has higher MTHM Sync-C than AnyTalker-14B (8.54 versus 8.41), while AnyTalker-14B has higher MPMS BA/PBA (92.36/90.78 versus 91.11/88.22) and lower MTHM FID/FVD (46.24/468.97 versus 89.81/695.91). All four methods in Table 2 have 0% MPMS OLR. The paper should distinguish the observed perceptual preference from these speaker-binding and visual-quality results, and support the proposed causal explanation with controlled evidence.

Questions for authors

Speech assignment across shot boundaries. How reliably does speaker assignment survive shot transitions and the first appearance of a participant absent from the initial image? On a predefined subset of existing multi-shot outputs, can you report assignment accuracy and visible-speaker leakage during off-screen speech around cuts, together with lip-sync measurements where valid? Please distinguish visibility changes from ordinary within-shot turns. Such evidence would directly address the largest gap in the central contribution. Evidence for the proposed listener-motion mechanism. What existing evidence distinguishes the effect of removing spatial routing from the effects of training data, temporal continuation, and other conditioning differences? How do these results support attributing the listener-motion advantage to spatial-mask removal? Training supervision and evaluation independence. Section 4.1 describes approximately 2K hours of authorized training videos covering single-person faces or bodies and multi-person conversations. Section 3.2 defines the activity channels and left-to-right person-slot ordering, and Section 3.4 explains the use of shot references and an ID board. Which sources and mixture proportions make up the training data, and how were SAT labels, visibility changes, shot references, and identity-board crops annotated or extracted and quality-checked for training? What checks were performed for overlap with evaluation sources? Please also provide each stage's training-step count, effective batch size, and the data-bucket weights referenced in Appendix A.5.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6zgW (1/2)

We thank Reviewer 6zgW for the detailed feedback and for recognizing SAM's ablation gains, shot-control performance, and evaluation protocols. Below we address Weaknesses W1–W3 and Questions Q1–Q3 (line numbers refer to the revised manuscript).

W1 & Q1: Speaker Assignment, Off-Screen Leakage, and Lip Synchronization Across Shot Cuts

On Movie Dialogues (Table 4), we evaluate overall binding accuracy across all turns (BA), transition-adjacent binding accuracy around cuts, off-screen transitions, and character entry/re-entry (BA near shot transitions, with representative transition frames checked against detector errors), off-screen leakage rate (OLR; definitions in Appendix A.3, lines 872–904), and active-speaker Sync-C / Sync-D [1] (Section 4.3, lines 493–495; Appendix G.2, Table 11, lines 1687–1724):

MethodBA ↑ (%)BA near shot transitions ↑ (%)OLR ↓ (%)Sync-C ↑Sync-D ↓
M$^2$Talk (Ours)74.2679.963.787.757.92
w/o ID board56.6755.9511.66––
w/o MVC31.1823.1511.43––
H3 [2]64.8260.0269.504.269.56

As detailed in Appendix G.2 (lines 1717–1723), all M$^2$Talk variants prepend I1…N\mathbf{I}_{1\dots N} via Eq. (7) (lines 329–342) at inference (w/o MVC uses the Stage 1 checkpoint; w/o ID board trains Stage 2 LoRA without IID\mathbf{I}_{\text{ID}}): adding IID\mathbf{I}_{\text{ID}} boosts transition BA from 55.95% to 79.96% when participants absent from Shot 1 appear in later shots, while Sync-C/D are omitted for both ablations due to frequent cross-shot misattribution. Although H3 [2] receives the shared timestamped dialogue script (Table 5, lines 972–983), text-only turn conditioning yields 69.50% OLR during off-screen speech vs. 3.78% for M$^2$Talk (Sync-C 7.75 vs. 4.26).

W2 & Q2: Isolating Spatial-Mask Removal on Listener Motion (Controlled Experiment)

Holding the single-shot dataset, Wan2.1-I2V-14B backbone [3], TCP continuation, Stage 1 budget (200,000 steps, batch size 32), and inference settings (896×448) fixed, we trained Reproduced (TCP + MultiTalk Spatial Routing) [4] by replacing only SAM with MultiTalk's spatial routing (Section 4.3, lines 430–431, 476–477; Appendix G.1, Table 10, lines 1674–1710):

Evaluation DimensionM$^2$Talk (TCP + SAM) First ↑Reproduced (TCP + Spatial Routing) First ↑Undecided
Identity Consistency81.25%16.25%2.50%
Temporal Continuity77.50%21.25%1.25%
Sustained Visual Quality77.50%22.50%0.00%
Speaker Motion Naturalness83.75%16.25%0.00%
Listener Motion Naturalness86.25%11.25%2.50%
Overall Preference80.00%20.00%0.00%

Across 80 blind judgments (8 participants × 10 30-second videos), M$^2$Talk is preferred 86.25% vs. 11.25% on Listener Motion Naturalness and 80.00% vs. 20.00% overall, confirming that removing 2D spatial routing drives natural listener behavior.

AuthorsOpenReview ↗

Response to Reviewer 6zgW (2/2)

W3: Qualifying Claims Against Table 2

  • Metric vs. perceptual trade-off (Section 4.2, lines 376–377, 407–414; Appendix G.3, lines 1728–1744): AnyTalker-14B [5] routes audio strictly within face masks (Fig. 2e, lines 171–178, 242–244), keeping non-speaking regions static. Because TalkNet BA/PBA [6] and SyncNet [1] evaluate cropped faces, facial masking slightly favors mouth-crop PBA while BA is statistically indistinguishable (9595\\% CI [−2.51[-2.51\\%, +0.06\\%] on M$^2$Talk − AnyTalker-14B, Table 12; all four methods reach 0.00% MPMS OLR because spatial baselines receive per-speaker audio with off-screen intervals zeroed out, whereas M$^2$Talk suppresses leakage directly from the shared track, vs. 100.00% w/o SAM in Table 3). Static non-speaking bodies/backgrounds also keep frames closer to the reference distribution (lower FID/FVD) despite frozen listeners over 30-second turns (0.0% vs. 91.0% in Table 8, lines 1350–1361), similar to localized portrait models Sonic [7] and FantasyTalking [8] achieving lower FVD than full-body models in Table 1 (lines 387, 389).

  • Scope of claims: Section 1 (lines 121–122) summarizes overall performance across the majority of benchmarks and baselines rather than every individual metric, and Section 4.2 (lines 407–414) explicitly states this Table 2 trade-off.

Q3: Training Data, Annotation & QC Pipeline, Train–Test Separation, and Training Settings

(1) Data sources & bucket weights: Stage 1 pools ~2K hours of authorized single-shot videos across single-person talking-head/body and multi-person conversation branches (lines 347–349) processed via MultiTalk's curation workflow [4]; Stage 2 adds cross-shot clips for MVC, with uniform loss averaging (weight 1.0) across buckets (Appendix A.5, line 960).

(2) Annotation, MVC construction, & QC: Candidate clips run YOLO person/face detection and ByteTrack tracking, filtering out short, low-visibility, or static bystander tracks and retaining up to 4 active character trajectories (S−1=4S{-}1=4, line 277) to select ID-board portraits and per-shot reference frames across visibility changes. DiariZen [9] extracts speech intervals and SyncNet [1] binds them to character tracks to form M∈0,1L×S\mathbf{M} \in \\{0,1\\}^{L \times S} (lines 258–264); only verified off-screen speech enters the off-screen slot, and QC discards clips with unresolved identities, low sync, insufficient per-shot face area, or uncovered target characters.

(3) Train–test separation: Verified against source video/film manifests and audio/portrait records, all evaluation audio tracks, synthetic/real reference portraits, and source videos/films across all seven benchmarks (lines 349–356; Appendix A.1, lines 825–838; Appendix B.1, lines 990–995) are strictly disjoint from training data.

(4) Training settings: Stage 1 trains for 200,000 steps and Stage 2 (rank-32 LoRA) for 10,000 steps on Wan2.1-I2V-14B [3], both with AdamW, lr 5×10−55 \times 10^{-5}, 100 warm-up steps, global batch size 32, and uniform bucket averaging (Section 4.1, lines 364–370; Appendix A.5, line 960).

References:

[1] Chung & Zisserman. Out of Time: Automated Lip Sync in the Wild. ACCVW, 2017.

[2] MiniMax. Open General Intelligence: MiniMax H3 Is Now Open Source. 2026.

[3] Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[4] Kong et al. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation. arXiv:2505.22647, 2025.

[5] Zhong et al. AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement. arXiv:2511.23475, 2025.

[6] Tao et al. Is Someone Speaking? Exploring Long-Term Temporal Features for Audio-Visual Active Speaker Detection. ACM MM, 2021.

[7] Ji et al. Sonic: Shifting Focus to Global Audio Perception in Portrait Animation. CVPR, 2025.

[8] Wang et al. FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis. arXiv:2504.04842, 2025.

[9] Han et al. Leveraging Self-Supervised Learning for Speaker Diarization. ICASSP, 2025.

Dear Reviewer 6zgW,

Thank you again for your thoughtful and detailed comments. We understand the considerable time and effort involved in the reviewing process and sincerely appreciate your constructive suggestions.

Following your suggestions, we have:

  • evaluated speaker binding, off-screen leakage, and SyncNet lip-sync around shot cuts on Movie Dialogues (Sec. 4.3; Appendix G.2, Table 11)
  • conducted a controlled 80-judgment study isolating SAM against 2D spatial routing under identical backbone, data, TCP, and budget (Appendix G.1, Table 10)
  • qualified our claims against Table 2 with MPMS bootstrap CIs and analyzed the metric vs. perceptual trade-off (Sec. 4.2; Appendix G.3, Table 12)
  • documented our data curation, SAT/MVC annotation and QC pipeline, strict train–test separation, and two-stage training settings

Given the limited discussion period, we sincerely hope to have further discussions if you have any remaining questions or concerns. We would greatly appreciate any further feedback you may have, as it would help us further improve our work.

Thank you again for your time and valuable feedback.

Dear Reviewer 6zgW,

As the discussion period comes to the end, we wanted to briefly share that we have uploaded an updated manuscript incorporating all revisions and new controlled experiments addressing your feedback (highlighted in blue):

  1. Multi-Shot Speaker Binding and Off-Screen Leakage Around Cuts (W1 & Q1): Section 4.3 (Page 10, lines 505–508) and Appendix G.2 (Table 12, Page 33) report overall BA (74.26%), transition-adjacent BA (79.96% vs. 55.95% w/o ID board and 23.15% w/o MVC), OLR (3.78% vs. 69.50% for H3), and active-speaker Sync-C/D on Movie Dialogues.
  2. Controlled Spatial-Routing Study — Both Human Preference and Quantitative Metrics (W2 & Q2): Holding the backbone, single-shot dataset, TCP continuation, and 200,000-step Stage 1 budget fixed, Section 4.3 (Page 9, lines 477–481) and Appendix G.1 (Tables 10–11, Page 32) now report both blind 30-second motion preferences (86.25% vs. 11.25% on listener motion) and newly added quantitative comparisons across MTHM, MPMS, and EMTD (e.g., 91.11% vs. 90.05% BA, 88.22% vs. 87.44% PBA, and 8.54 vs. 8.24 MTHM Sync-C).
  3. Qualified Claims and Training Supervision Details (W3 & Q3): Section 4.2 (Page 8, lines 416–424) and Appendix G.3 (Table 13, Page 33) qualify Table 2 with pair-preserving bootstrap CIs, while Section 4.1 (Pages 7–8, lines 376, 407–408) and Appendix A.5 (Page 18, line 960) specify all batch-size, step-count, and bucket-averaging details.

Thank you again for your time and detailed review!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Area Chair, Reviewers, and Readers

Dear Area Chair, Reviewers, and Readers,

We sincerely thank all four reviewers for their constructive feedback and encouraging recognition of M$^2$Talk across multiple fronts. Specifically, the reviewers highlighted (1) our clear motivation and practical relevance (6zgW, Uu8L, bwpM); (2) SAM's spatial-mask-free conversational control interface (6zgW, AbXw, bwpM); (3) the effectiveness of MVC and the ID board for multi-shot synthesis (6zgW, Uu8L, bwpM); and (4) our rigorous, interpretable evaluation design (6zgW, Uu8L, AbXw). Their thoughtful suggestions have helped us further strengthen our causal and cross-shot validations, sharpen our methodological presentation, and expand our visual demonstrations.

In particular, we have addressed the shared concerns regarding cross-shot speaker assignment around camera cuts, causal isolation of spatial-mask removal on listener motion, and metric trade-offs and statistical uncertainty. We also thank Reviewer Uu8L for inspiring a deeper discussion on theoretical innovation and broader generalizability, and we provide dedicated clarifications on condition-side representation disentanglement.

Our major revisions include:

  1. Cross-shot speaker binding and lip-sync validation: We added transition-adjacent speaker-binding, off-screen leakage, and SyncNet evaluations on Movie Dialogues (Sec. 4.3; Appendix G.2, Table 11) across camera cuts and newly entering speakers.

  2. Controlled causal evidence for spatial-mask removal: We clarified that our training data was curated in alignment with MultiTalk's workflow after consulting its authors, and reproduced MultiTalk's spatial-mask mechanism on top of TCP under identical backbone, data, and training budget for blind preference comparison (Sec. 4.3; Appendix G.1, Table 10).

  3. Statistical uncertainty and baseline trade-off analyses: We included pair-preserving bootstrap confidence intervals on MPMS (Sec. 4.2; Appendix G.3, Table 12), expanded the discussion of CovOG (Sec. 2.1), and detailed data curation, train–test separation, and baseline configurations (Sec. 4.1; Appendices A–B).

  4. Methodological clarifications and Project Page updates: We clarified Fig. 2 vs. Fig. 3, Eqs. (1)–(7), and mask roles (Sec. 3.4; Appendices A.4–A.5), and updated our project page with live-cursor SAT timelines, 25 randomly sampled cases, and backbone boundary cases.

Overall, by formulating speaker binding as condition-side representation disentanglement rather than 2D spatial masking, M$^2$Talk enables novel single-track multi-speaker control, natural speaker–listener dynamics, and seamless cross-shot attribution. All revisions are highlighted in blue in the revised manuscript. We hope our clarifications and additions address the reviewers' concerns and provide a clearer understanding of our work.

AuthorsOpenReview ↗

Project Page Summary

Project Page SectionContent SummaryCorresponding Manuscript Sections (Revised version)
SAT StudioInteractive 1D SAT interval & JSON explorer (SAT -> SAE -> FiLM)Sec. 3.2, Eqs. (3)–(4) (lines 258–290); App. A.4, Eq. (13) (lines 921–931)
Two-Person 30sSide-by-side 30s user-study videos (M$^2$Talk, MultiTalk, AnyTalker) + live SAT cursorSec. 4.2, Table 2 & Fig. 4 (lines 400–423); App. C.1 & C.3, Table 8 & Figs. 11–13
3–4 Person Demos8 natural 3–4 person videos with turn-taking, overlaps, and live SAT barsSec. 3.2, 4 visible + 1 off-screen slots (S=5S=5, lines 258–278)
More Videos (25 Random)25 uncurated, randomly sampled conversational videos across diverse scenesSec. 4.2, Qualitative Evaluation (lines 415–419)
Multi-Shot DemosMulti-shot videos & comparison panels on Movie Dialogues, AVA-AVD, UnityShotsFig. 1; Sec. 3.4, Eq. (7); Sec. 4.3, Table 4; App. B.1–B.4 (Tables 5–7, Figs. 7–9); App. G.2, Table 11
Component AblationsSide-by-side Full model vs. Without SAM, Without TCP, and Without MVC videosSec. 4.3, Tables 3–4 (lines 425–495); App. C.2, Fig. 10; App. G.1–G.2, Tables 10–11
Method OverviewArchitecture diagram & conditioning walkthrough of SAM, TCP, and MVCFigs. 2–3; Sec. 3.1–3.4, Eqs. (1)–(7) (lines 181–342); App. A.4–A.5, Eqs. (13)–(16)
Quantitative ResultsSingle-person, multi-person, multi-shot, user-study, CI, and latency benchmarksSec. 4.2–4.3, Tables 1–4 & Fig. 5; App. B.3, C.1, E, & G.1–G.3, Tables 6–12
Failure Cases3 complete 30s videos showing fine-grained hand/object backbone artifactsApp. F.1, Limitations (lines 1642–1645)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.