- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new method for multi-person talking-head video generation and argues that their approach is more data-efficient in terms of scaling. Their major contribution is the two-stage training approach, which sounds practical; their validation indicates its superiority over the previous baseline. Nevertheless, I overall feel it offers an incremental contribution. It also needs more evidence to support whether those generated animations are socially appropriate beyond just naturalism. As in a real multiperson conversation, the listener's behavior is merely conditioned on the speaker's audio and facial behaviors; in many cases, I would argue that if the listener's social interactions are not even predictable. This paper clearly simplified a lot of those assumptions.
Critical strengths
-
A practical training strategy for multi-person talking-head video generation, which is scalable for 2-4 # of persons and also applies to other animals. The first-stage training seems to be straightforward to scale up (i.e., 2000 hours), while the second stage, as the author claims, only requires a very small set of two-person interaction video data (i.e., 12 hours). So apparently the paper is arguing that the first-stage training gains more generalized audio-face sync-up, although it has no or little idea how the cross-identity interaction works. But, for some unknown magic, only 12 hours of training could enable the richness interaction learning sounds incredible? It is, however, again unclear what the pretrained representations (from the first stage) are that can accelerate the second-stage training.
-
The results are stronger than any previous baseline based on the metrics they reported. The relevant ablations are helpful.
Critical weaknesses
The overall approach feels like an engineering-level trick, as opposed to a principled new technique that has a novel breakthrough. The two-stage training makes sense overall. However, I'd like to challenge that the two-person or multiple-person videos are also widely available, based on previous method, despite it requires annoation. Can the authors provide more specific unit economic numbers to justify that training on 2000 hours of video data is better than asking people to do the annotation? Also, the multiple-person interactions from real videos are more nuanced than any synthetic dataset, would a smaller set of real data but with rich annoation would beat the proposed methodology from both economic and performance perspective? The authors should provide this discussion.
The proposed method quantifies listener motion, but its validity as a measure of natural interactivity is not sufficiently established. I personally feel the resulting video is very unnatural (i.e., gaze directions & facial expressions). This metric should be validated against blinded human ratings.
Besides, I don't see any deep, insightful analysis of why the first-stage training would "ease" the second-stage training.
Questions for authors
Do you see any scaling law applies in either the first stage or the second stage training? Why 2000 hours, why not 200 hours or 20000 hours, how does that curve look like for the final performance.
4 comments
Follow up Response to Reviewer q7Va
Dear Reviewer q7Va,
Following up on our rebuttal, we would appreciate your feedback on the additional explanations and experiments. We have clarified the roles of single-person pretraining and real-conversation refinement, provided matched-budget data-efficiency comparisons, and examined agreement between our listener-response metric and human preferences.
Please let us know if any concerns remain or need further clarification. Thank you again for your constructive suggestions and your time in considering our responses.
Best regards, The Authors
I appreciate the authors' additional results in the rebuttal. I see the point of the single-person pretraining; however, this further reinforces the incremental nature of this work compared with prior work. The authors basically try to argue that all those prior works are doing similar things by using single-person data pretraining. Even though I appreciate the extensive work, I don't see sufficient exciting new insights or very compelling visual quality being demonstrated. Just showing those minor numeric improvements does not make the whole narrative or results very compelling. I still can't raise my score based on those considerations.
(Round 2) Reply to Reviewer q7Va
Thank you for your follow-up. We would like to clarify two points.
(1) Our core contribution is constructing multi-person training supervision by composing independent single-person videos. The composed samples teach multi-stream audio–identity binding before real-conversation refinement. A small real-conversation set then supplies the cross-person reactions absent from the synthetic pairs. Both Reviewers 9eqr and BdXy explicitly recognize this complementary data strategy as a strength. We cited prior single-person pretraining to contextualize the data requirements.
(2) The improvement is also directly visible. We provide a new side-by-side comparison with MultiTalk and Playmate2 (E16). At 2–4 s, the two participants in our result turn toward one another, while the participants in both baselines remain largely camera-facing. This provides a concrete example of improved cross-person interaction beyond numerical gains.
We hope these clarifications and the additional evidence will encourage you to reconsider your rating.
Reply to Reviewer q7Va
W1/W3/Q1. Data economics and transfer from stage 1. First, single-person training adapts an image-to-video (I2V) backbone to audio-driven generation. MultiTalk uses about 2,000 h of single-person videos with Wan2.1-I2V-14B; InfiniteTalk also reports about 2,000 h. Both of them are famous multi-person driving methods. Our 2,000 h covers all usable single-person clips after filtering, matching these recipes.
In fact, large single-person datasets are publicly available: SpeakerVid-5M provides 8.7K h of single-speaker videos with captions, speech transcripts and pose annotations; TalkVid offers 1,244 h with dense captions. Concatenated multi-person (CM) training reuses single-person recordings without new interaction annotations.
Single-person training can also support multi-person driving. Playmate2 trains only on single-person videos and enables multi-character generation at inference. HunyuanVideo-Avatar does not use multi-person data for fine-tuning, while its project demos show multi-character animation, including joint-singing examples.
However, natural interaction remains a challenge. The four-person demo below the Abstract on the Playmate2 project page shows limited visible interaction, consistent with our pre-refinement result in Fig. 14. This motivates real-conversation refinement to learn joint speaker–listener behavior.
Given that thousands of hours of single person data pretrain cannot be avoided, we therefore test whether CM pretraining reduces the real-conversation data needed for refinement. Table E7 and Figure E8 examine stage 2 with a 1.3B model, comparing initialization from real single-person (RS) data with the RS+CM mixture. Table E7 fixes 5,000 optimizer updates. At each size, both runs use the same real two-person subset, 8 H200 accelerators, gradient accumulation 2 and global batch size 16, evaluated on the same 60 two-person inputs.
Each cell reports duration-weighted speaking-phase lip-synchronization confidence (
Sync-C*, ↑) / listening-phase eye-motion gap to ground truth (ΔResp, ↓).RS+CM improves Sync-C* at every size and ΔResp at five (tie at 96 h). With 24 h, it exceeds RS's 96 h synchronization at the same ΔResp: one-quarter as much real data under the same refinement budget. Further gains beyond 12 h are small within this budget.
Beyond E7's fixed-update endpoints, Figure E8 tracks refinement through a 24 h real-video pool; Table E8 reports all 11 checkpoints. Both runs share the sampling seed and receive identical samples at every iteration. The horizontal axis counts cumulative unique video hours; repeated clips count once.
Each cell reports (
Sync-C*, ↑) / (ΔResp, ↓).RS+CM leads on both metrics at all 11 checkpoints. At 12 h it is close to its 24 h result and already better than RS at 24 h. These results suggest that AnyTalker provides better initialization and achieves comparable performance on these metrics with less real-conversation data, highlighting its data efficiency.
W2. Metric validation. Table E10 compares ΔResp with 100 individual preference votes per person count from the paper's blinded study (25 non-author raters, randomized order, hidden method names). A vote matches when the chosen method attains the lowest ΔResp among that input's candidates; ties count. There are four candidates for two people, two for three/four. Different raters may evaluate the same input.
Video E12 shows a silent listener looking toward the speaker; E13 shows orientation toward a subsequent speaker; E1 shows a stronger listener smile after a happy edit of the other audio under fixed visual/text conditions.
Agreement of 77–89% supports ΔResp's relevance to human preference in these comparisons; the videos illustrate listener gaze and expression responses.