acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static References: Audio-Visual Reference Conditioning for Human Video Generation

Abstract

Reference-conditioned human video generation has advanced rapidly, yet existing datasets still predominantly represent human subjects using one or a few static reference images. Such references provide only sparse observations of subject appearance while discarding temporal visual cues and voice characteristics naturally available in videos. Large-scale training data with richer audio-visual references for human video generation therefore remain limited. We introduce VidVoice-Ref, a large-scale dataset for human video generation with audio-visual subject references. Instead of reducing a subject to isolated frames, VidVoice-Ref preserves complete reference videos together with synchronized audio, capturing multi-view appearance, pose and expression variations, motion patterns, and voice timbre. We construct identity-consistent cross-video samples by identifying target subjects, retrieving temporally separated reference clips, and selecting visually diverse references across viewpoints, poses, expressions, motions, and scenes. Audio references are further refined through speaker diarization, speech validity filtering, intra-clip speaker consistency, voice quality filtering, and cross-video voice matching. VidVoice-Ref contains approximately 1M reference–target samples and over 1K hours of video, covering both single- and multi-subject scenarios. VidVoice-Ref provides a scalable data foundation for studying richer human subject representations beyond static image references, particularly for identity preservation, subject binding, and audio-visual reference-conditioned human video generation.

Primary area:
datasets and benchmarks
Keywords:
Video Generation, Audio-Visual References

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RC9YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VidVoice-Ref, a dataset of approximately one million reference–target pairs covering more than 1,000 hours of video for audio-visual reference-conditioned human video generation. The dataset retains reference videos and synchronized audio, providing both appearance variation and voice information. Its curation pipeline combines cross-clip identity matching, visual diversity selection, speaker filtering, and voice matching for single- and multi-subject scenarios. The evaluation includes dataset statistics, qualitative examples, and a 120-case benchmark comparing four existing generators on visual quality, identity and voice consistency, audio-visual synchronization, and prompt adherence.

Critical strengths
  1. A useful extension beyond image-based references. Preserving reference videos together with synchronized audio is a meaningful feature of this dataset. Compared with isolated reference images, this format offers a richer description of a subject’s appearance and voice, and could support research on preserving visual and vocal identity jointly. The inclusion of both single- and multi-subject settings also broadens its potential applications.
  2. A concrete curation pipeline operating at substantial scale. The paper describes how identity retrieval, visual diversity selection, and speaker and voice-consistency filtering are combined to construct approximately one million reference–target pairs. The explicit filtering criteria make the construction process easier to inspect and provide a practical starting point for future work on audio-visual reference data.
Critical weaknesses
  1. The evaluation provides limited evidence for the added value of the richer reference format. The 120-case benchmark helps characterize the performance of existing generators, but it does not establish the benefits of training on VidVoice-Ref or isolate the contributions of static-image, video-only, and audio-visual references. In particular, it remains unclear whether temporal information provides an advantage beyond simply supplying more visual observations of the subject. I do not consider downstream training a prerequisite for a dataset paper. Nevertheless, a focused comparison would strengthen the case for the proposed reference format, and any claims of improved downstream generation should be qualified where they are not directly supported by the current experiments.
  2. Subject–voice correspondence is not sufficiently verified. The pipeline selects the target speech segment most similar to the reference voice, while the appendix notes that per-subject audio-visual association is not enforced through lip-motion tracking. Voice similarity alone does not establish that the selected speech belongs to the visually matched person. This is particularly concerning in multi-subject scenes, where a pair could pass the visual and voice filters without the two modalities referring to the same individual. The absence of an independent audit of this correspondence leaves uncertainty about a central property of the dataset. The concern is not the lack of lip-motion tracking itself, but the lack of evidence that the resulting associations are reliable.
Questions for authors
  1. What evidence supports the added value of video and audio-visual references? Have you conducted any training experiments using VidVoice-Ref, or controlled comparisons among static-image, video-only, and audio-visual references? Existing results would be helpful, particularly comparisons that control for the amount of visual input. If these experiments have not been conducted, please clarify which proposed benefits are demonstrated by the current evaluation and which remain to be tested.
  2. How reliable is the subject–voice correspondence, especially in multi-subject scenes? What prevents the selected target speech segment from belonging to someone other than the visually referenced person? Have you independently audited the final reference–target pairs for this type of mismatch? Please report the audit protocol, sample size, and accuracy, ideally separating single- and multi-subject cases.

My main concern is whether the dataset’s defining properties are reliably established, rather than whether the authors have trained a new generator. Direct evidence of pairing quality would therefore be more important to my assessment than a large-scale downstream training study.

2 comments
AuthorsOpenReview ↗

Audit of Subject-Voice Correspondence and Evidence for Audio-Visual References

We thank the reviewer for the clear review and for stating which evidence matters most. Since the reviewer weighs pairing quality above a downstream training study, we answer Question 2 first, and we will add both tables below to the revised paper.

1. Reliability of the subject-voice correspondence (Question 2)

The link rests on two conditions. First, the reference voice belongs to the visible person. The reference is a single shot, no sampled frame shows more than one face, and diarization finds one speaker. A SyncNet lip-sync test on both reference and target clips (Section 3.2) then discards clips without a visibly synchronized speaker and requires exactly one synchronized face track in a reference (audio-visual offset at most 3 frames at 25 frames per second, confidence at least 2.0), which removes voice-over and shot/reverse-shot dialogue. Second, the reference voice must recur in the target with similarity of at least 0.6 on a segment longer than one second, and in multi-person targets this voice match selects the referenced subject's speech.

Three annotators independently and blindly reviewed the decisions of the lip-sync test, with disagreements resolved by majority vote:

Audit settingSamplesAgreementPrecisionRecall
Single-person reference5090.0%97.0%88.9%
Multi-person target5086.0%88.9%57.1%

We also audited the final pairs independently. Three annotators labeled 100 randomly drawn reference-target pairs, with majority vote and no indeterminate cases, and a pair counts as correct only if the target contains the referenced person and that person speaks with the reference voice. Pairing accuracy was 98% (49/50, 95% confidence interval 89.5-99.6%) for single-person targets and 92% (46/50, 81.2-96.8%) for multi-person targets. Examples of reference-target pairs are available at VidVoice-Ref audio-video examples.

2. Evidence for video and audio-visual references (Question 1)

We ran a controlled comparison: Wan 3.0 is held fixed on 100 single-subject benchmark cases with the same prompts, and only the reference changes, from a single frame (the middle frame of the reference video, without audio) to the full reference video with its audio, the setting of Table 1.

ReferenceCurricularFaceArcFaceR-CLIPDINOECAPASIM-oSyncNet confidence
Single frame, no audio0.6370.5920.7540.3730.2170.2503.74
Video with audio0.6420.5900.7620.3790.4530.5054.04

The audio-visual reference roughly doubles voice similarity (ECAPA 0.22 to 0.45, SIM-o 0.25 to 0.51) and raises SyncNet confidence (3.74 to 4.04), which an image reference cannot provide, while face similarity stays on par and appearance similarity is slightly higher. This directly supports keeping each reference video with its synchronized audio, the defining property of VidVoice-Ref.

Prior work that trains on video references adds visual gains. With the same frozen backbone, Slot-ID (Table 3 of that paper) reports a face similarity of 0.642 with a portrait, 0.658 with a tiled pseudo-video of equal length and 0.729 with the real video. TAVR (Table 3 of that paper) reports gains from adding the reference audio (identity similarity 0.64 to 0.66, Sync-C 7.55 to 7.60). These gains come with training, which VidVoice-Ref enables at scale for multiple subjects with synchronized audio.

3. What is shown and what comes next

The evidence covers the composition of the corpus (about one million samples from about 134,000 source videos), the accuracy of the pairs (the audit above), their audio quality (a blind three-annotator rating of 50 samples, with speech intelligibility rated good for 80%), the differences among existing systems (Table 1), and the value of audio-visual references at inference time (the comparison above). The next steps, which the revised discussion will state, are training on VidVoice-Ref and testing temporal order beyond frame count with video-only and frame-shuffled conditions.

We again thank the reviewer for focusing on the properties that define the dataset. The audit and the controlled comparison above will be added to the revised paper.

The rebuttal partially addresses my concerns through the correspondence audit and reference comparison, but the benefits of training on VidVoice-Ref and the contribution of temporal information remain unverified, leaving my recommendation at Borderline.

Reviewer_Y5SUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VidVoice-Ref is a dataset of approximately one million reference–target samples, spanning over 1,000 hours, for human video generation with video and voice references. Its construction combines face-based retrieval, visual diversity filtering, speaker filtering, and reference–target voice matching. The paper also introduces a 120-case benchmark and evaluates four existing generation systems.

Critical strengths

Keeping the reference video and its audio is a sensible extension of image-based subject conditioning. Appendix A clearly distinguishes using speech to specify vocal identity from using it to drive animation. The single- and two-subject examples in Figures 7–8 illustrate why this formulation could be useful across changes of scene and appearance.

The curation description contains practical detail. Table 2 collects the thresholds, and Appendices B.2–B.3 explain the visual and audio filters, including the distinction between separated vocals used for assessment and original audio delivered as the reference. I also appreciate the separate evaluation of facial identity, broader appearance, voice, and synchronization; one score would hide meaningful differences here.

Critical weaknesses

My main concern is the face–voice association. Equation (3), p. 6, and Equation (7), p. 16, take the maximum speaker similarity over all target speech segments, accepting a match at similarity 0.6 with a segment longer than one second. This establishes that a sufficiently similar voice occurs somewhere in the target, but does not establish which visible person produced it. Appendix B.3 acknowledges that per-subject audiovisual association is not explicitly enforced through lip-motion tracking. Section 3.2 says off-screen speech is removed, without explaining the procedure or measuring its accuracy. Single-face filtering and one-speaker diarization do not resolve this association by themselves. I would expect a manual audit of identity matching and speaker binding, particularly for multi-person targets. An accurate alternative to lip tracking would be entirely acceptable; what is missing is evidence that the labels are correct.

The corpus is also difficult to assess beyond its headline size. Figure 2 gives useful duration and resolution statistics, but there are no counts of unique identities, source videos, targets, or references. Appendix B.5 allows up to three references per subject, so it matters whether reported hours count unique footage or repeated use across pairs. “Temporally separated” in Section 3.3 has no specified minimum gap, and the paper does not quantify how often the reference and target actually differ in scene or viewpoint. Nor does Section 5.1 explain the relationship between the 120 benchmark cases and the full corpus, or provide an identity/source-disjoint split. These are central dataset facts. The absence of a release location or concrete access plan further limits what another group can presently build on.

I do not find a direct test of the dataset's intended benefit. Table 1 compares existing systems, rather than holding a system fixed and varying static images, sampled frames, full videos, and reference audio. It therefore cannot tell us what the additional temporal information or the proposed curation contributes. A compatible existing model would suffice for a controlled comparison. Together with curation precision and coverage measurements, this would make a much stronger case for the dataset. The present paper supplies neither this comparison nor a sufficiently strong data audit.

Questions for authors
  1. How accurate are face–voice association and target speaker binding on a manually checked sample, separately for single- and multi-subject cases? How is off-screen speech filtered?
  2. What are the unique identity/source/clip counts, separation rules, benchmark split, and release plan?
  3. Can a controlled existing-model comparison or a stronger data audit show what full audiovisual references and the proposed filtering add over simpler references?
  4. What exact inputs and settings produced Table 1, and do its rankings survive paired uncertainty analysis? Did the evaluated prompts contain the person names shown in Figures 5–6?
2 comments
AuthorsOpenReview ↗

Speaker Binding Audit, Corpus Statistics, Reference-Format Comparison, and Table 1 Uncertainty

We thank the reviewer for the detailed review and answer the four questions in order. All new results will be added to the revised paper.

1. Face-voice association and off-screen speech.

A reference is kept only if it is a single shot, never shows more than one face in a sampled frame, has exactly one diarized speaker with more than one second of speech, and has a consistent voiceprint. Its voice must then recur in the target with similarity of at least 0.6 on a segment longer than one second. Off-screen speech is removed by a SyncNet lip-sync test on both reference and target clips (Section 3.2): every face track is tested against every speaker at 25 frames per second, and a pair passes if the audio-visual offset is at most 3 frames (about 120 milliseconds) and the confidence at least 2.0. Clips without a visibly synchronized speaker are discarded, and a reference must have exactly one passing face track. In multi-person targets, the voice match selects the referenced subject's speech. Three annotators independently and blindly reviewed the lip-sync decisions, with majority vote: agreement 90.0%, precision 97.0% and recall 88.9% on 50 single-person references, and 86.0%, 88.9% and 57.1% on 50 multi-person targets. Three annotators also audited 100 randomly drawn final pairs, with no indeterminate cases. A pair is correct only if the target contains the referenced person and that person speaks with the reference voice.

TargetPairsCorrectAccuracy (95% confidence interval)
Single-person504998% (89.5-99.6%)
Multi-person504692% (81.2-96.8%)

2. Counts, separation, split and release.

For the full corpus of about one million samples (rounded): about 134,000 source videos, 548,000 unique target clips and 344,000 unique reference clips, 912,000 single-subject and 88,000 multi-subject samples, 978,000 distinct reference-target pairs, and at most 189,000 distinct identities (identity clusters, an upper bound). A reference and its target are different, temporally separated clips of the same source video (Section 3.3). Near-duplicates are excluded by a 0.90 cap on face similarity to the target subject and by the DINO diversity filter at cosine distance 0.4 (Appendix B.2), so references vary in viewpoint, pose, expression or scene. The 120 benchmark cases (Section 5.1) form an evaluation set, and before release we will remove every corpus sample that shares a source video or a clip with them. VidVoice-Ref has not yet been released. Before release, we will verify the authorization of each source, withhold raw media without confirmed redistribution rights, and define usage terms and a removal procedure.

3. Controlled comparison.

We held Wan 3.0 fixed on 100 single-subject benchmark cases with the same prompts and changed only the reference: a single frame (the middle frame of the reference video, without audio), or the full reference video with its audio, the setting of Table 1.

ReferenceCurricularFaceArcFaceR-CLIPDINOECAPASIM-oSyncNet confidence
Single frame, no audio0.6370.5920.7540.3730.2170.2503.74
Video with audio0.6420.5900.7620.3790.4530.5054.04

The audio-visual reference roughly doubles voice similarity and improves lip sync, which an image reference cannot provide, while preserving face identity and slightly raising appearance similarity. Methods trained on video references report further identity gains (Slot-ID, TAVR), and VidVoice-Ref enables such training at scale.

4. Table 1 settings, uncertainty and names.

Names: no. As stated in Appendix B.4, explicit person names are excluded from the final generation prompts. Figures 5 and 6 mistakenly show an annotation from before this step and will be replaced. Every system receives the same prompt and the reference video with its audio, and generates one 4-second sample per case with audio (720p for Kling 3.0, Seedance 2.5 and Wan 3.0, 768p for MiniMax H3, and a 16:9 aspect ratio except for the adaptive Seedance 2.5). Face similarity compares 16 generated frames with the mean of 4 frames per reference video, with Hungarian one-to-one matching for multiple subjects. R-CLIP (CLIP ViT-B/32) and DINO (DINOv2-Base) use full frames, and Naturalness scores 10 frames on a 1 to 5 scale. On the 55 single-subject cases scored for all four systems, a paired bootstrap (10,000 resamples) with Holm-corrected Wilcoxon signed-rank tests confirms three findings: MiniMax H3 has the highest UTMOS (p < 0.001), Wan 3.0 is more natural than Kling 3.0 and MiniMax H3 (p < 0.05), and Kling 3.0 trails the other systems on R-CLIP, ECAPA, SIM-o and SyncNet confidence (p < 0.001). The remaining differences among Seedance 2.5, Wan 3.0 and MiniMax H3 are within the confidence intervals and will be shown as ties, with the intervals added to Table 1.

Dear Reviewer Y5SU,

We would like to kindly follow up and ask whether our latest response has addressed your concerns. As the discussion deadline is approaching, we would be very grateful if you could review our clarification when convenient. We sincerely appreciate your continued engagement and genuinely look forward to further communication with you. Thank you again for your valuable feedback.

Reviewer_z7AzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript introduces VidVoice-Ref, a dataset containing approximately one million reference–target samples and over 1,000 hours of human video. The claimed contribution is the preservation of complete reference video clips together with synchronized audio, supported by a dedicated curation pipeline incorporating identity retrieval, visual-diversity filtering, and speaker-consistency filtering. The work also introduces a 120-case benchmark and compares four existing audio-visual generation models across five dimensions.

Critical strengths

The main strength is a clear pipeline for constructing identity-consistent reference–target pairs with synchronized audio, combining identity retrieval, visual-diversity filtering, and speaker-consistency filtering.

Critical weaknesses

My primary concern is the dataset’s usability and practical value. Static examples cannot establish audio quality or person–voice correspondence, and the PDF provides neither inspectable audio examples nor an independent quality assessment. Furthermore, no empirical dataset comparison or downstream training evaluation demonstrates the benefit of the proposed data. Evaluating existing video gen models does not address these gaps.

Questions for authors

How is speech verified to come from the intended visible subject? Section 3.2 states that off-screen speech is removed, but the described single-face/single-speaker filters and voice-similarity matching do not establish this correspondence. The person–voice mismatch rate should be reported, particularly for multi-person targets.

3 comments
AuthorsOpenReview ↗

Clarifications on Speaker Attribution, Audio Quality, and Dataset Scope

We thank the reviewer for carefully reading our paper and recognizing the value of its identity-consistent pairing pipeline. The concerns about usability, person–voice correspondence, audio quality, and release ethics identify aspects that need clearer evidence or explanation. We respond to each point below and will revise the paper accordingly.

1. Visible-speaker filtering and person–voice correspondence

Both reference and target videos undergo SyncNet audio–visual synchronization filtering. Detected faces are linked across frames into visual speaker tracks, and synchronization is evaluated for each track–diarized speaker pair. Videos are standardized to 25 fps; a pair passes if the absolute offset is at most 3 frames (about 120 ms) and confidence is at least 2.0. Clips without a visibly synchronized speaker are discarded; references must have exactly one passing visual track.

For cross-video pairing, we first match the person visually and then compare the voice characteristics of the corresponding audio segments. Together, these steps support person–voice correspondence across the pair. This filtering removes clips with no visibly synchronized speaker; it does not imply that every speech segment in a target video comes from an on-screen person.

In a manual audit of reference–target pairs, pairing accuracy was 98% (49/50) for single-person targets and 92% (46/50) for multi-person targets, with no indeterminate cases. Three annotators independently and blindly reviewed the SyncNet filtering results; disagreements were resolved by majority vote:

Audit settingSamplesAgreementPrecision among passed samplesRecall
Single-person reference5090.0%97.0%88.9%
Multi-person target5086.0%88.9%57.1%

The threshold prioritizes reducing false acceptance, and therefore rejects some valid samples. We will clarify this trade-off and correct inaccurate descriptions in the paper and supplement.

2. Audio–video examples and audio-quality assessment

We provide an interactive page with target–reference audio–video examples for inspecting the content, audio quality, and their correspondence: VidVoice-Ref audio–video examples.

We manually assessed audio quality on 50 samples. The dimensions follow the ITU-T P.835 distinction between speech quality and background noise, using a simplified three-level rubric rather than a full P.835 test. ITU-T P.835 “Good” means clear speech with minor interference; “Fair” means some utterances require effort, but the main content remains intelligible; “Poor” means speech is difficult to understand or interference substantially affects listening. Three annotators rated samples blindly; disagreements were resolved by majority vote.

Human-evaluation dimensionSamplesGoodFairPoor
Speech intelligibility5040 (80%)7 (14%)3 (6%)
Background interference and recording artifacts5042 (84%)6 (12%)2 (4%)

3. Dataset contribution and downstream utility

VidVoice-Ref contains approximately one million reference–target pairs and over 1,000 hours of video. It preserves complete reference videos with synchronized audio and constructs pairs using identity retrieval, visual-diversity filtering, and speaker-consistency filtering. The paper also introduces a 120-case benchmark and evaluates four existing audio–visual generation models across five dimensions.

Evaluating existing models does not establish that training on VidVoice-Ref improves downstream performance; this paper includes no controlled training study. We will clarify this limitation and not present benchmark results as evidence of training gains. Downstream training evaluation is future work.

4. Data release and ethics

VidVoice-Ref has not yet been publicly released. We will clarify release conditions, identifiable faces and voices, and potential misuse. Before release, we will verify each source’s applicable authorization and distribution conditions; raw media without confirmed redistribution rights will not be released. We will also specify usage boundaries and procedures for removal requests and potential misuse.

We again thank the reviewer for these specific and constructive comments. The questions about visible-speaker verification and mismatch rates prompted us to add the corresponding audit and clarify the pipeline. We will revise the paper and supplement to make the evidence, scope, and limitations easier to assess.

Dear Reviewer z7Az,

We would like to kindly follow up and ask whether our latest response has addressed your concerns. As the discussion deadline is approaching, we would be very grateful if you could review our clarification when convenient. We sincerely appreciate your continued engagement and genuinely look forward to further communication with you. Thank you again for your valuable feedback.

Thanks for adding the video examples and explaining how to make the match in multi-person clips. The vis url give me a clearer picture of the data quality (while most of the examples are close-up shots, the overall quality is good). One of my main concerns remains, I’d still like to know how much a model benefits from training on VidVoice-Ref, which will make the contribution of this work much stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.