acceptodds
Withdrawn from ICLR 2027

AnyTalker: Data-Efficient Scaling of Multi-Person Talking Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Generating multi-person talking videos requires modeling not only per-speaker lip synchronization but also cross-speaker interactions, including gaze shifts, nods, and listening reactions. Existing methods often rely on hundreds or thousands of hours of curated multi-person video, which is costly to annotate and concentrated in domains such as studio podcasts. In this paper, we introduce AnyTalker, a data-efficient framework that shows how to build multi-person generation largely from abundant, open-domain single-person data. To this end, we concatenate independent single-person clips to teach the model to bind each audio stream to its corresponding face. To extend this learned binding beyond the two-person training setting, each DiT block uses an Audio-Face Cross Attention module with parameters shared across identity-audio pairs, allowing the number of identities to vary at inference time. We then refine listener behavior using 12 hours of real two-person conversations. Our evaluation combines human and MLLM assessments of listener reactions with emotion metrics and a calibrated responsiveness metric that compares listener motion with real videos. Despite training on at most two people, AnyTalker generates three- and four-person conversations with listener interactions. Across the evaluated benchmarks, it matches or surpasses methods trained on more real multi-person data in lip synchronization, visual quality, and human judgments of naturalness.

Primary area:
generative models
Keywords:
Video Generation, Virtual Human

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q7VaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method for multi-person talking-head video generation and argues that their approach is more data-efficient in terms of scaling. Their major contribution is the two-stage training approach, which sounds practical; their validation indicates its superiority over the previous baseline. Nevertheless, I overall feel it offers an incremental contribution. It also needs more evidence to support whether those generated animations are socially appropriate beyond just naturalism. As in a real multiperson conversation, the listener's behavior is merely conditioned on the speaker's audio and facial behaviors; in many cases, I would argue that if the listener's social interactions are not even predictable. This paper clearly simplified a lot of those assumptions.

Critical strengths
  1. A practical training strategy for multi-person talking-head video generation, which is scalable for 2-4 # of persons and also applies to other animals. The first-stage training seems to be straightforward to scale up (i.e., 2000 hours), while the second stage, as the author claims, only requires a very small set of two-person interaction video data (i.e., 12 hours). So apparently the paper is arguing that the first-stage training gains more generalized audio-face sync-up, although it has no or little idea how the cross-identity interaction works. But, for some unknown magic, only 12 hours of training could enable the richness interaction learning sounds incredible? It is, however, again unclear what the pretrained representations (from the first stage) are that can accelerate the second-stage training.

  2. The results are stronger than any previous baseline based on the metrics they reported. The relevant ablations are helpful.

Critical weaknesses

The overall approach feels like an engineering-level trick, as opposed to a principled new technique that has a novel breakthrough. The two-stage training makes sense overall. However, I'd like to challenge that the two-person or multiple-person videos are also widely available, based on previous method, despite it requires annoation. Can the authors provide more specific unit economic numbers to justify that training on 2000 hours of video data is better than asking people to do the annotation? Also, the multiple-person interactions from real videos are more nuanced than any synthetic dataset, would a smaller set of real data but with rich annoation would beat the proposed methodology from both economic and performance perspective? The authors should provide this discussion.

The proposed method quantifies listener motion, but its validity as a measure of natural interactivity is not sufficiently established. I personally feel the resulting video is very unnatural (i.e., gaze directions & facial expressions). This metric should be validated against blinded human ratings.

Besides, I don't see any deep, insightful analysis of why the first-stage training would "ease" the second-stage training.

Questions for authors

Do you see any scaling law applies in either the first stage or the second stage training? Why 2000 hours, why not 200 hours or 20000 hours, how does that curve look like for the final performance.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer q7Va

W1/W3/Q1. Data economics and transfer from stage 1. First, single-person training adapts an image-to-video (I2V) backbone to audio-driven generation. MultiTalk uses about 2,000 h of single-person videos with Wan2.1-I2V-14B; InfiniteTalk also reports about 2,000 h. Both of them are famous multi-person driving methods. Our 2,000 h covers all usable single-person clips after filtering, matching these recipes.

In fact, large single-person datasets are publicly available: SpeakerVid-5M provides 8.7K h of single-speaker videos with captions, speech transcripts and pose annotations; TalkVid offers 1,244 h with dense captions. Concatenated multi-person (CM) training reuses single-person recordings without new interaction annotations.

Single-person training can also support multi-person driving. Playmate2 trains only on single-person videos and enables multi-character generation at inference. HunyuanVideo-Avatar does not use multi-person data for fine-tuning, while its project demos show multi-character animation, including joint-singing examples.

However, natural interaction remains a challenge. The four-person demo below the Abstract on the Playmate2 project page shows limited visible interaction, consistent with our pre-refinement result in Fig. 14. This motivates real-conversation refinement to learn joint speaker–listener behavior.

Given that thousands of hours of single person data pretrain cannot be avoided, we therefore test whether CM pretraining reduces the real-conversation data needed for refinement. Table E7 and Figure E8 examine stage 2 with a 1.3B model, comparing initialization from real single-person (RS) data with the RS+CM mixture. Table E7 fixes 5,000 optimizer updates. At each size, both runs use the same real two-person subset, 8 H200 accelerators, gradient accumulation 2 and global batch size 16, evaluated on the same 60 two-person inputs.

Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).

E7: real data (hours)RSRS+CM
33.67/0.514.85/0.45
63.78/0.424.99/0.38
123.83/0.375.07/0.33
244.29/0.345.05/0.32
484.65/0.325.12/0.30
964.96/0.325.10/0.32

RS+CM improves Sync-C* at every size and ΔResp at five (tie at 96 h). With 24 h, it exceeds RS's 96 h synchronization at the same ΔResp: one-quarter as much real data under the same refinement budget. Further gains beyond 12 h are small within this budget.

Beyond E7's fixed-update endpoints, Figure E8 tracks refinement through a 24 h real-video pool; Table E8 reports all 11 checkpoints. Both runs share the sampling seed and receive identical samples at every iteration. The horizontal axis counts cumulative unique video hours; repeated clips count once.

Each cell reports (Sync-C*, ↑) / (ΔResp, ↓).

E8: unique data seen (hours)RSRS+CM
43.71/0.504.57/0.44
63.64/0.484.52/0.45
83.82/0.474.68/0.42
103.89/0.444.97/0.37
123.81/0.425.06/0.32
143.95/0.425.01/0.35
164.07/0.395.10/0.32
184.03/0.385.13/0.31
204.08/0.365.09/0.33
224.15/0.375.05/0.32
244.24/0.365.09/0.31

RS+CM leads on both metrics at all 11 checkpoints. At 12 h it is close to its 24 h result and already better than RS at 24 h. These results suggest that AnyTalker provides better initialization and achieves comparable performance on these metrics with less real-conversation data, highlighting its data efficiency.

W2. Metric validation. Table E10 compares ΔResp with 100 individual preference votes per person count from the paper's blinded study (25 non-author raters, randomized order, hidden method names). A vote matches when the chosen method attains the lowest ΔResp among that input's candidates; ties count. There are four candidates for two people, two for three/four. Different raters may evaluate the same input.

E10: peopleVotesAgreement (%) ↑
210089.00
310084.00
410077.00

Video E12 shows a silent listener looking toward the speaker; E13 shows orientation toward a subsequent speaker; E1 shows a stronger listener smile after a happy edit of the other audio under fixed visual/text conditions.

Agreement of 77–89% supports ΔResp's relevance to human preference in these comparisons; the videos illustrate listener gaze and expression responses.

AuthorsOpenReview ↗

Follow up Response to Reviewer q7Va

Dear Reviewer q7Va,

Following up on our rebuttal, we would appreciate your feedback on the additional explanations and experiments. We have clarified the roles of single-person pretraining and real-conversation refinement, provided matched-budget data-efficiency comparisons, and examined agreement between our listener-response metric and human preferences.

Please let us know if any concerns remain or need further clarification. Thank you again for your constructive suggestions and your time in considering our responses.

Best regards, The Authors

I appreciate the authors' additional results in the rebuttal. I see the point of the single-person pretraining; however, this further reinforces the incremental nature of this work compared with prior work. The authors basically try to argue that all those prior works are doing similar things by using single-person data pretraining. Even though I appreciate the extensive work, I don't see sufficient exciting new insights or very compelling visual quality being demonstrated. Just showing those minor numeric improvements does not make the whole narrative or results very compelling. I still can't raise my score based on those considerations.

AuthorsOpenReview ↗

(Round 2) Reply to Reviewer q7Va

Thank you for your follow-up. We would like to clarify two points.

(1) Our core contribution is constructing multi-person training supervision by composing independent single-person videos. The composed samples teach multi-stream audio–identity binding before real-conversation refinement. A small real-conversation set then supplies the cross-person reactions absent from the synthetic pairs. Both Reviewers 9eqr and BdXy explicitly recognize this complementary data strategy as a strength. We cited prior single-person pretraining to contextualize the data requirements.

(2) The improvement is also directly visible. We provide a new side-by-side comparison with MultiTalk and Playmate2 (E16). At 2–4 s, the two participants in our result turn toward one another, while the participants in both baselines remain largely camera-facing. This provides a concrete example of improved cross-person interaction beyond numerical gains.

We hope these clarifications and the additional evidence will encourage you to reconsider your rating.

Reviewer_BdXyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AnyTalker generates multi-person talking videos from a reference image, separate audio streams, and a text prompt. It first learns audio-to-face binding using single-person videos and artificially concatenated pairs, then refines listener behavior using real two-person conversations. Its Audio-Face Cross Attention module shares parameters across participants and uses face masks to assign each audio stream to its corresponding person. The paper evaluates generation with up to four people, including lip synchronization, visual quality, listener reactions, human preferences, and inference cost.

Critical strengths

Learning audio binding from concatenated single-person clips reduces reliance on real conversational recordings. Table 3 supports the usefulness of combining synthetic pairs with a small real-conversation refinement set.

Although training includes at most two people, the model generates three and four person videos with competitive synchronization and human preferences. The evaluation also includes MultiTalk and other relevant baselines, alongside both human and automatic assessments.

Critical weaknesses

Figure 4 checks whether reactions occur, but not whether they happen at the right time or match what the speaker says. The responsiveness metric measures average movement during listening, and human preferences measure overall naturalness. Neither establishes that reactions depend on the speaker’s audio. The model also accepts text, but only generic prompts are tested. It therefore remains unclear whether the listener movements are meaningful responses or general listening behaviors.

Table 4 tests the architectural components only on single-person videos using the smaller model. This does not show how these components prevent one person from following another person’s audio or support generation with more people. Table 3 tests training data choices with two people, but not larger groups. The data-efficiency claim also needs comparisons using different amounts of real conversation data under the same training budget.

Questions for authors

If one speaker’s audio is changed while the listener’s audio, reference image, prompt, and random seed remain fixed, do the listener’s reactions change appropriately? Please evaluate reaction timing and relevance, rather than occurrence alone.

Can text prompts control a specific listener’s gaze, nods, or other reactions? Please show matched examples that vary only the instruction, or clarify that explicit reaction control is outside the method’s scope.

Can the architectural and data ablations be evaluated on three, and four person inputs, including incorrect mouth activation during silence and responses to another person’s audio?

The three- and four-person results show visible listener movements, but do not establish whether each listener responds to the appropriate speaker. In larger groups, who determines which participant a speaker addresses and which listeners should react? Please assess directed gaze and listener responses around speaker changes, rather than only whether reactions occur somewhere in the video.

How does performance change with different amounts of real-conversation refinement data under matched training budgets? Does concatenated pretraining consistently reduce the amount needed?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer BdXy: (1/2)

W1 / Q1. Audio interventions. Thank you for the suggestion. We added audio-intervention experiments E1–E4 using AnyTalker-14B. We change one participant's audio while keeping the reference image, the other participant's audio, the prompt ("These people are talking to each other.") and random seed fixed. E1/E2 use Step-Audio-EditX for happy and sad emotion edits. In E1, editing the right speaker's delivery produces a more pronounced left-listener smile during 0–4.19 s. E3 mutes the right track: the left woman no longer makes the head turn seen in the baseline. E3 compares the common 0–7.875 s at original speed. E4 delays the left track by 1 s; the left man, whose audio is edited, shows a wider mouth opening before speaking. E1 and E3 therefore provide qualitative evidence that listener behavior depends on the other participant's audio under fixed visual and text conditions.

W1 / Q2. Text control. We added text-control comparisons E14/E15, each pairing a generic-prompt baseline with one gaze instruction. The reference image, both audio tracks, identity order, seed and sampling settings stay fixed; videos display the exact prompts. In E14, the instruction targets the right participant, who turns toward the partner. In E15, the instruction targets the left participant, who faces the camera. These matched examples show that text can steer the orientation of a specified participant.

W2 / Q3. Components and training data. We added component and data ablations E5/E6 using a 1.3B model on 60/20/20 two-/three-/four-person inputs. Table E5 reports the component ablations. Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓). Its first four rows are stage-1 models; the last adds 12 h of real two-person conversation refinement for 50,000 optimizer updates, following the paper's Table 3. Training uses at most two people. The Audio-Face Cross Attention (AFCA) ablation removes face-token concatenation, retaining per-identity audio attention, masks and accumulation. Concatenated multi-person (CM) data pair two single-person clips with separate audio.

E5: 1.3B model2 people3 people4 people
without AFCA4.42/0.523.94/0.593.60/0.66
without CM4.39/0.533.88/0.613.53/0.69
without Face Mask4.65/0.494.01/0.563.71/0.65
Full stage 14.97/0.474.24/0.523.96/0.62
Full +12 h refinement5.02/0.324.36/0.384.15/0.49

Table E6 compares real single-person (RS) and CM pretraining, followed where indicated by real multi-person (RM) refinement. Rows without RM are stage-1 models. Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).

E6: training data2 people3 people4 people
CM3.34/0.493.10/0.602.99/0.67
RS4.39/0.533.88/0.613.53/0.69
RS+CM4.97/0.474.24/0.523.96/0.62
RS+RM3.88/0.353.69/0.473.44/0.58
RS+CM+RM5.02/0.324.36/0.384.15/0.49

Full stage 1 beats each component ablation on both metrics. Mixed pretraining improves both metrics before and after refinement, including unseen three/four people.

AuthorsOpenReview ↗

Reply to Reviewer BdXy: (2/2)

Q4. Directed gaze. We added gaze examples E11–E13, using participant audio tracks to identify who speaks and when. The generic prompt gives no addressee or listener-specific gaze instruction. E13 shows the central participant speaking at source time 4.56–6.48 s, followed by the black-coat participant from 6.72 s; the generated central participant later turns toward that speaker. E12 shows a silent listener raising her gaze toward the speaker, without a speaker change. These examples include ground truth and the generated clip for the same interval. They show listener orientation toward a speaking participant under a generic prompt, without explicit gaze instructions.

W2 / Q5. Matched-budget data scaling. We added data-scaling experiments E7/E8 comparing real single-person (RS) pretraining with RS plus concatenated multi-person (CM) data using a 1.3B model. Table E7 fixes 5,000 refinement updates. At each size, the two initializations use the same real two-person subset, 8 NVIDIA H200 accelerators, gradient accumulation 2 and global batch size 16, evaluated on the same 60 two-person inputs.

Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).

E7: real data (hours)RSRS+CM
33.67/0.514.85/0.45
63.78/0.424.99/0.38
123.83/0.375.07/0.33
244.29/0.345.05/0.32
484.65/0.325.12/0.30
964.96/0.325.10/0.32

RS+CM improves Sync-C* at all six sizes and ΔResp at five (tie at 96 h). At 24 h, it exceeds RS's 96 h Sync-C* with the same ΔResp, using a quarter of the real data.

Figure E8 plots learning trajectories through a 24-hour real-video pool; Table E8 below lists all 11 plotted checkpoints. Both runs use the same sampling seed; at every iteration, their samplers return identical training samples. The horizontal axis counts cumulative unique video hours, counting repeated clips only once.

Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).

E8: unique data seen (hours)RSRS+CM
43.71/0.504.57/0.44
63.64/0.484.52/0.45
83.82/0.474.68/0.42
103.89/0.444.97/0.37
123.81/0.425.06/0.32
143.95/0.425.01/0.35
164.07/0.395.10/0.32
184.03/0.385.13/0.31
204.08/0.365.09/0.33
224.15/0.375.05/0.32
244.24/0.365.09/0.31

RS+CM leads at all 11 checkpoints; its 12 h result is already close to its 24 h result and better than RS at 24 h on both metrics.

E7 uses 5,000-update endpoints; E8 selects checkpoints by unique-data exposure. Together, E7/E8 show that concatenated pretraining reduces the real-conversation data needed to reach comparable performance on these metrics under the tested refinement settings.

AuthorsOpenReview ↗

Follow up Response to Reviewer BdXy

Dear Reviewer BdXy,

Following up on our rebuttal, we would appreciate your feedback on the additional experiments. Your questions motivated audio and text interventions, component and training-data ablations, listener-gaze examples and matched-budget data-scaling comparisons to further examine interaction and data efficiency.

Please let us know if any concerns remain or need further clarification. Thank you again for your helpful suggestions and your time in reviewing these additions.

Best regards, The Authors

Reviewer_9eqrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Targeting the task of audio-driven multi-person talking video generation, this paper proposes AnyTalker, a framework that combines a two-stage data training strategy with a local audio-face binding module. The main contribution of this work lies in decomposing multi-person independent control and interpersonal interaction into different data stages for learning, and providing empirical support on single-person benchmarks as well as InteractivePerson, a self-constructed multi-person test set containing 100 samples. Overall, this is a systematic work of practical value.

Critical strengths
  1. The paper decomposes independent audio binding and cross-person interactive reactions in multi-person video generation, learning them separately using mixed single-person/concatenated data and a small amount of real dialogue data, and the data construction method matches well with the regional constraint and parameter-sharing design of the AFCA module. The data ablation in Table 3 indicates that the two types of data provide complementary supervision: adding concatenated data mainly improves lip synchronization in two-person tests, while further adding real two-person dialogue fine-tuning reduces listener motion deviation and improves the emotion classification label consistency rate with GT videos.
  2. Under a training setup containing at most two people, the model can extrapolate to three-person and four-person testing scenarios, achieving consistent performance advantages across both single-person test sets and the evaluated multi-person settings.
Critical weaknesses
  1. I note that the authors have clearly stated the limitations of ∆Resp in the appendix. Existing human preference, MLLM behavior detection, and motion evaluations support that the generated results exhibit more visible listener behaviors and higher naturalness, but they are not yet sufficient to distinguish whether these behaviors undergo meaningful temporal changes in response to another speaker's audio track, or mainly stem from a conversational scene motion prior that does not depend on the other party's audio track.

  2. The architectural ablations in Table 4 are conducted solely under the single-person HDTF setting using the Stage-1 1.3B model. While these ablations support the contribution of relevant components to single-person generation quality, they have not directly verified the respective effects of reference face features and spatial region constraints on multi-person audio binding, listener behaviors, and person-count extrapolation, which limits the explanation of the mechanism behind the performance gains.

  3. Table 3 has supported the effectiveness of concatenated pre-training under the same 1.3B model scale, but performance variation curves across different real two-person data scales under a fixed training budget have not yet been reported; therefore, existing results can support that the strategy is effective under the 12-hour real data setting, but are still insufficient to fully characterize how its real-data efficiency varies with data scale, and it is also difficult to quantify the extent of real data saved solely through cross-system comparisons with MultiTalk. Regarding person-count extrapolation, the three-person and four-person tests each contain only 20 videos, and confidence intervals are not reported for the main results.

Questions for authors
  1. Under the premise of fixing the reference image, the listener's own audio track, and the random seed, if interventions are applied only to the other speaker's audio track (such as temporal shifting, altering pause/turn-taking rhythms, or muting? ), will the generated listener actions (nodding, head-turning, or gaze events) exhibit interpretable temporal changes accordingly?
  2. If the architectural ablation variants already defined in Table 4 are directly extended to the two person as well as three and four person test sets of InteractivePerson, what are the performance differences among the components in multi person lip binding, listener reaction metrics, and person-count extrapolation?
  3. Please additionally provide the confidence intervals on the main objective metrics and human preferences for the InteractivePerson test set.
  4. As a suggestion to more fully characterize the boundaries of the method, providing performance trends across different real two person data scales under a fixed 1.3B model and training budget, or offering a stratified quantitative analysis regarding the degree of substantial subject deviation from the first-frame fixed mask and regional overlap, would help more accurately define the data efficiency and scope of applicability.
  5. The paper currently only promises to release inference code, but I believe that open-sourcing the training and concatenated data construction pipeline code in the future would help the community further reproduce and verify this strategy. Please note that this will not affect my rating, and I am simply hoping to learn about the relevant code out of interest.
3 comments
AuthorsOpenReview ↗

Reply to Reviewer 9eqr: (1/2)

W1 / Q1. Audio-dependent listener behavior. Thank you for the suggestion. We added audio-intervention experiments E1–E4 using AnyTalker-14B in two-person settings. We change one participant's audio while keeping the reference image, the other participant's audio, the prompt ("These people are talking to each other.") and random seed fixed. The emotion edits in E1/E2 use Step-Audio-EditX. In E1, a happy edit of the right speaker produces a more pronounced smile in the left listener; E2 uses a sad edit of the left speaker. In E3, muting the right track removes the left woman's head turn seen in the baseline. In E4, delaying the left track by 1 s is accompanied by a wider mouth-opening motion in the left man before he speaks. The E4 observation concerns the participant whose audio is edited. E1 and E3 therefore provide qualitative evidence that listener behavior depends on the other participant's audio under fixed visual and text conditions.

W2 / Q2. Multi-person components. We added component ablations E5 using a 1.3B model on 60/20/20 two-/three-/four-person InteractivePerson inputs. The Audio-Face Cross Attention (AFCA) ablation removes face-token concatenation while retaining per-identity audio attention, masks and accumulation. Concatenated multi-person (CM) data pair two single-person clips with separate audio tracks. The first four rows are stage-1 models. The final row additionally receives 12 hours of real two-person conversation refinement for 50,000 optimizer updates, as in the paper's Table 3. Training uses at most two people.

Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).

E5: 1.3B model2 people3 people4 people
without AFCA4.42/0.523.94/0.593.60/0.66
without CM4.39/0.533.88/0.613.53/0.69
without Face Mask4.65/0.494.01/0.563.71/0.65
Full stage 14.97/0.474.24/0.523.96/0.62
Full +12 h refinement5.02/0.324.36/0.384.15/0.49

Full stage 1 improves both metrics over every component ablation at each person count, including unseen three- and four-person settings.

W3 / Q3. Confidence intervals. We added confidence intervals E9 for AnyTalker-14B minus MultiTalk for two people and minus Playmate2 for three/four. We use 10,000 bootstrap resamples: paired input resampling within each 60/20/20 subset, retaining phase-duration weights for objective metrics, and two-way rater × input resampling of the original preference votes. Entries are differences [95% confidence interval (CI)], using the 2.5th/97.5th percentiles; preference differences use percentage points (pp). All methods shown in the original ballots remain in the preference denominator.

PeopleSync-C* ↑ΔResp ↓Preference (pp) ↑
2+0.14 [+0.05, +0.22]−0.06 [−0.11, −0.04]+11.20 [+6.20, +16.40]
3+1.69 [+1.15, +2.42]−0.15 [−0.24, −0.11]+53.12 [+38.25, +65.10]
4+1.74 [+1.12, +2.36]−0.18 [−0.27, −0.08]+56.48 [+41.08, +70.40]

All nine intervals exclude zero in the direction favoring AnyTalker, supporting its stronger performance on the reported metrics against these comparators.

AuthorsOpenReview ↗

Reply to Reviewer 9eqr: (2/2)

W3 / Q4. Real-data efficiency. We added data-scaling experiments E7/E8 comparing real single-person (RS) pretraining with a mixture of RS and concatenated multi-person (CM) data. Table E7 fixes the 1.3B architecture and refinement at 5,000 optimizer updates. At each data size, both initializations use the same real two-person subset, 8 NVIDIA H200 accelerators, gradient accumulation 2 and global batch size 16, evaluated on the same 60 two-person inputs.

Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).

E7: real data (hours)RSRS+CM
33.67/0.514.85/0.45
63.78/0.424.99/0.38
123.83/0.375.07/0.33
244.29/0.345.05/0.32
484.65/0.325.12/0.30
964.96/0.325.10/0.32

RS+CM improves Sync-C* at all six sizes and ΔResp at five, tying at 96 h. Its 24 h result exceeds RS's 96 h synchronization with the same reported ΔResp, using one quarter of the real data at the same refinement budget.

Figure E8 plots learning trajectories through a 24-hour real-video pool; Table E8 below gives all 11 plotted checkpoints. Both runs use the same sampling seed; at every iteration, their samplers return identical training samples. The horizontal axis counts cumulative unique video hours, counting repeated clips only once.

Each cell reports duration-weighted speaking-phase lip-synchronization confidence (Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).

E8: unique data seen (hours)RSRS+CM
43.71/0.504.57/0.44
63.64/0.484.52/0.45
83.82/0.474.68/0.42
103.89/0.444.97/0.37
123.81/0.425.06/0.32
143.95/0.425.01/0.35
164.07/0.395.10/0.32
184.03/0.385.13/0.31
204.08/0.365.09/0.33
224.15/0.375.05/0.32
244.24/0.365.09/0.31

RS+CM leads on both metrics at all 11 checkpoints. Its 12 h result is close to its 24 h result and already better than RS at 24 h. E7 uses 5,000-update endpoints; E8 selects checkpoints by unique-data exposure. These are separate from the 50,000-update refinement endpoints in the paper's Table 3 and Table E5 in Part 1. Together, E7/E8 support a reduction in the real-conversation data needed to reach comparable performance on these metrics when using concatenated pretraining.

Q5. Code release. Thank you for the suggestion. Alongside inference, we will release training code and the data preparation pipeline, including construction of concatenated multi-person clips.

AuthorsOpenReview ↗

Follow up Response to Reviewer 9eqr

Dear Reviewer 9eqr,

Following up on our rebuttal, we would appreciate your feedback on the additional experiments. We especially thank you for motivating the audio interventions, component ablations for two to four people, confidence intervals and matched-budget data-scaling comparisons, which strengthened our evaluation.

Please let us know if any concerns remain or need further clarification. Thank you again for your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your careful review. We have provided detailed responses and additional experiments on listener behavior, multi-person generalization, data efficiency and evaluation reliability. Supporting tables, figures and videos are available on our rebuttal page.

We would appreciate your feedback on whether these responses address your concerns or whether any points need further clarification. Thank you for your time and consideration.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.