- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VidVoice-Ref, a dataset of approximately one million reference–target pairs covering more than 1,000 hours of video for audio-visual reference-conditioned human video generation. The dataset retains reference videos and synchronized audio, providing both appearance variation and voice information. Its curation pipeline combines cross-clip identity matching, visual diversity selection, speaker filtering, and voice matching for single- and multi-subject scenarios. The evaluation includes dataset statistics, qualitative examples, and a 120-case benchmark comparing four existing generators on visual quality, identity and voice consistency, audio-visual synchronization, and prompt adherence.
Critical strengths
- A useful extension beyond image-based references. Preserving reference videos together with synchronized audio is a meaningful feature of this dataset. Compared with isolated reference images, this format offers a richer description of a subject’s appearance and voice, and could support research on preserving visual and vocal identity jointly. The inclusion of both single- and multi-subject settings also broadens its potential applications.
- A concrete curation pipeline operating at substantial scale. The paper describes how identity retrieval, visual diversity selection, and speaker and voice-consistency filtering are combined to construct approximately one million reference–target pairs. The explicit filtering criteria make the construction process easier to inspect and provide a practical starting point for future work on audio-visual reference data.
Critical weaknesses
- The evaluation provides limited evidence for the added value of the richer reference format. The 120-case benchmark helps characterize the performance of existing generators, but it does not establish the benefits of training on VidVoice-Ref or isolate the contributions of static-image, video-only, and audio-visual references. In particular, it remains unclear whether temporal information provides an advantage beyond simply supplying more visual observations of the subject. I do not consider downstream training a prerequisite for a dataset paper. Nevertheless, a focused comparison would strengthen the case for the proposed reference format, and any claims of improved downstream generation should be qualified where they are not directly supported by the current experiments.
- Subject–voice correspondence is not sufficiently verified. The pipeline selects the target speech segment most similar to the reference voice, while the appendix notes that per-subject audio-visual association is not enforced through lip-motion tracking. Voice similarity alone does not establish that the selected speech belongs to the visually matched person. This is particularly concerning in multi-subject scenes, where a pair could pass the visual and voice filters without the two modalities referring to the same individual. The absence of an independent audit of this correspondence leaves uncertainty about a central property of the dataset. The concern is not the lack of lip-motion tracking itself, but the lack of evidence that the resulting associations are reliable.
Questions for authors
- What evidence supports the added value of video and audio-visual references? Have you conducted any training experiments using VidVoice-Ref, or controlled comparisons among static-image, video-only, and audio-visual references? Existing results would be helpful, particularly comparisons that control for the amount of visual input. If these experiments have not been conducted, please clarify which proposed benefits are demonstrated by the current evaluation and which remain to be tested.
- How reliable is the subject–voice correspondence, especially in multi-subject scenes? What prevents the selected target speech segment from belonging to someone other than the visually referenced person? Have you independently audited the final reference–target pairs for this type of mismatch? Please report the audit protocol, sample size, and accuracy, ideally separating single- and multi-subject cases.
My main concern is whether the dataset’s defining properties are reliably established, rather than whether the authors have trained a new generator. Direct evidence of pairing quality would therefore be more important to my assessment than a large-scale downstream training study.
2 comments
The rebuttal partially addresses my concerns through the correspondence audit and reference comparison, but the benefits of training on VidVoice-Ref and the contribution of temporal information remain unverified, leaving my recommendation at Borderline.
Audit of Subject-Voice Correspondence and Evidence for Audio-Visual References
We thank the reviewer for the clear review and for stating which evidence matters most. Since the reviewer weighs pairing quality above a downstream training study, we answer Question 2 first, and we will add both tables below to the revised paper.
1. Reliability of the subject-voice correspondence (Question 2)
The link rests on two conditions. First, the reference voice belongs to the visible person. The reference is a single shot, no sampled frame shows more than one face, and diarization finds one speaker. A SyncNet lip-sync test on both reference and target clips (Section 3.2) then discards clips without a visibly synchronized speaker and requires exactly one synchronized face track in a reference (audio-visual offset at most 3 frames at 25 frames per second, confidence at least 2.0), which removes voice-over and shot/reverse-shot dialogue. Second, the reference voice must recur in the target with similarity of at least 0.6 on a segment longer than one second, and in multi-person targets this voice match selects the referenced subject's speech.
Three annotators independently and blindly reviewed the decisions of the lip-sync test, with disagreements resolved by majority vote:
We also audited the final pairs independently. Three annotators labeled 100 randomly drawn reference-target pairs, with majority vote and no indeterminate cases, and a pair counts as correct only if the target contains the referenced person and that person speaks with the reference voice. Pairing accuracy was 98% (49/50, 95% confidence interval 89.5-99.6%) for single-person targets and 92% (46/50, 81.2-96.8%) for multi-person targets. Examples of reference-target pairs are available at VidVoice-Ref audio-video examples.
2. Evidence for video and audio-visual references (Question 1)
We ran a controlled comparison: Wan 3.0 is held fixed on 100 single-subject benchmark cases with the same prompts, and only the reference changes, from a single frame (the middle frame of the reference video, without audio) to the full reference video with its audio, the setting of Table 1.
The audio-visual reference roughly doubles voice similarity (ECAPA 0.22 to 0.45, SIM-o 0.25 to 0.51) and raises SyncNet confidence (3.74 to 4.04), which an image reference cannot provide, while face similarity stays on par and appearance similarity is slightly higher. This directly supports keeping each reference video with its synchronized audio, the defining property of VidVoice-Ref.
Prior work that trains on video references adds visual gains. With the same frozen backbone, Slot-ID (Table 3 of that paper) reports a face similarity of 0.642 with a portrait, 0.658 with a tiled pseudo-video of equal length and 0.729 with the real video. TAVR (Table 3 of that paper) reports gains from adding the reference audio (identity similarity 0.64 to 0.66, Sync-C 7.55 to 7.60). These gains come with training, which VidVoice-Ref enables at scale for multiple subjects with synchronized audio.
3. What is shown and what comes next
The evidence covers the composition of the corpus (about one million samples from about 134,000 source videos), the accuracy of the pairs (the audit above), their audio quality (a blind three-annotator rating of 50 samples, with speech intelligibility rated good for 80%), the differences among existing systems (Table 1), and the value of audio-visual references at inference time (the comparison above). The next steps, which the revised discussion will state, are training on VidVoice-Ref and testing temporal order beyond frame count with video-only and frame-shuffled conditions.
We again thank the reviewer for focusing on the properties that define the dataset. The audit and the controlled comparison above will be added to the revised paper.