acceptodds
Under review as a conference paper at ICLR 2027

VESTA: Timbre-Faithful and Lip-Synced Joint Talking Audio-Video Generation

Abstract

Recent joint talking audio–video generation produces realistic human videos, yet falls short on both the voice and the lips: the timbre is uncontrolled rather than specified, and the lips stay imperfectly synchronized. Existing timbre-injection methods feed the reference into the token stream, which in a coupled generator leaks time-varying content and unsettles the lips; they supervise both modalities with reconstruction losses alone, and serve every speaker with one fixed pass. We present VESTA (Video-audio Expert-guided Synchronized Timbre-faithful Avatar), which injects timbre off the time axis and supervises perception directly on the decoded face and voice, through three components. (1) StatShift modulates the audio branch's channel statistics (AdaLN) uniformly across time with a reference speaker embedding; carrying no time-varying signal, it controls timbre while empirically preserving the content the lips track. (2) RewardSteer adds the perceptual supervision joint generators lack: a timbre reward, and for lip-sync a noise-aware critic scored on a face-localized decode of one-step predictions, since both streams are generated and only the face carries the signal. (3) TimbreTune is an optional per-sample test-time step for the hard-speaker tail. On the open-source EMTD benchmark, VESTA raises reference-timbre similarity by nearly 50% (from 0.40 to 0.60), a gain ablations attribute to the mechanism rather than the data; it also leads on lip-sync (Sync-C 7.43 vs. 6.55) and stays competitive on PQ, CU and FVD. TimbreTune adds +6.2%, most on the hardest speakers.

Primary area:
generative models
Keywords:
Human Video Generation, Joint Audio–video Generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_7MhhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VESTA, a joint talking audio–video generation model with reference-timbre control, built on a MOVA-style dual-tower DiT. It has three components: StatShift injects an ECAPA speaker embedding into the audio DiT’s AdaLN through a shared MLP (time-invariant modulation); RewardSteer adds a timbre reward and a lip-sync reward on one-step denoised predictions, the latter using a self-trained critic and a face-localized partial decode; TimbreTune performs test-time LoRA adaptation using self-paired segments of the reference audio.

Critical strengths
  1. The paper is well structured and easy to follow: the problem is decomposed into three clearly stated gaps, each matched by one component, and Fig. 2 lays out the structure and role of every component clearly enough to understand the overall design without the text.
  2. The 2×2 ablation in Table 4 is clean and makes the contribution of each reward directly visible: the timbre reward mainly improves SIM, the lip-sync reward mainly improves Sync-C/Sync-D, and the two do not interfere with each other.
Critical weaknesses
  1. Table 3 does not test its own central claim. The paper attributes StatShift’s advantage to token injection “leaking the words and prosody of the reference clip and thereby disrupting lip-sync,” yet the table reports only SIM and Sync. Under the evaluation protocol with cyclically shifted target text, WER is the most direct measure of content leakage, but it is omitted; WER should be a required metric for this ablation.
  2. Training details of the lip-sync critic f_v/f_a are missing, including the training data and how positive and negative pairs are constructed. These details determine whether the critic measures fine-grained synchronization or only coarse audio–visual correspondence, and are therefore essential to the paper’s lip-sync claims; they should at least be given in the supplementary material.
  3. Key comparison methods are missing from the main table: (1) a cascade baseline (zero-shot TTS + audio-driven video generation); (2) TTS methods such as the CosyVoice series should be included as references for audio quality and SIM, so that the actual level of the proposed method can be established; (3) MiniMax H3 supports reference-timbre input but is neither compared against nor mentioned in the related work, which is inappropriate.
  4. No details of the training data are provided. For a generative method whose results depend heavily on subjective judgment, the paper also provides no video demo.
  5. The origin of the audio branch’s DiT and VAE is unclear: their architecture, whether they are pretrained, and whether they are trained by the authors or adopted from existing models are all unspecified. The paper places audio generation and timbre control at the center of its contribution, yet omits the most basic description of these models.
Questions for authors
  1. During training, is a_ref the audio of the target clip itself, or a different utterance from the same speaker?
  2. Which ECAPA-TDNN checkpoint is used for the timbre reward, and on what data was it trained? Is it the same network as the conditioning encoder used by StatShift?
  3. For the remaining questions, please refer to the weaknesses above.

Although the novelty of the proposed method is moderate, it offers a low-cost solution in a very narrow domain (the compute and data costs should be reported) and has considerable value for engineering applications. The main method is clearly presented. If the authors can supplement the missing experiments and implementation details and demonstrate that the main method is solid and effective, I will consider raising my score.

Reviewer_psSSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VESTA, a framework for jointly generating talking audio and video from a reference portrait, a text script, and a reference voice. It addresses the difficulty of preserving speaker timbre while maintaining lip synchronization when introducing voice conditioning into a joint audio-video diffusion model. The central component, StatShift, extracts a global speaker embedding and injects it through the scale, shift, and gating parameters of the audio transformer's adaptive normalization layers, using the same conditioning across temporal positions. RewardSteer further trains the model with timbre and synchronization rewards computed from decoded one-step estimates, with a noise-aware synchronization critic and face-aware partial decoding to reduce training cost. An optional TimbreTune procedure adapts the speaker projection using three optimization steps on the reference audio. Experiments on 110 English EMTD samples report higher speaker similarity and synchronization scores than the compared joint generation systems, supported by component ablations and human preference judgments. The results establish a promising approach to speaker-conditioned joint generation, although the proposed explanation of content disentanglement and the robustness of several design choices require further evidence.

Critical strengths
  1. The paper addresses a practical conflict in talking-head generation: a model must reproduce a reference voice without allowing its spoken content or timing to interfere with the requested script and lip movements. Conditioning the normalization pathway with a global speaker embedding is a clear architectural choice. It provides a compact mechanism for controlling speaker identity within a model that jointly generates both modalities.

  2. The controlled ablations provide useful evidence for the proposed conditioning and reward mechanisms. Under the same backbone, data, and reward setting, StatShift achieves a speaker similarity of 0.522, compared with 0.354 for cross-attention and 0.384 for additive conditioning, while retaining a higher synchronization score (Table 3). The separate timbre-only, synchronization-only, and combined reward experiments in Table 4 also help distinguish the contributions of the two training signals.

  3. The evaluation uses different networks for training rewards and final metrics: ECAPA-based timbre supervision is assessed with WavLM similarity, and the learned synchronization critic is assessed with an independent SyncNet. This reduces the concern that the reported improvements merely reflect optimization of the evaluation network itself. The main comparison reports strong timbre and synchronization results, and the double-blind human study provides complementary perceptual evidence (Tables 1–2).

Critical weaknesses
  1. The evidence for speaker-content disentanglement is weaker than the architectural interpretation suggests. The VCTK probe demonstrates that channel statistics can identify speakers under its stated protocol, but it does not establish that speech content is absent from those statistics. Moreover, the probe studies audio VAE latents, whereas StatShift modulates DiT hidden states. Temporally uniform conditioning alone does not guarantee content independence after subsequent nonlinear processing. The authors should clarify the probe's speaker and utterance counts and test whether changing reference words or prosody, while preserving speaker identity, changes generated content or timing (Section 4.3.1).

  2. The reward ablation establish the usefulness of timbre and synchronization supervision, but do not separately validate two central RewardSteer design choices. The paper would benefit from comparisons between clean and noise-aware synchronization critics, and between full decoding followed by cropping and the proposed latent partial decoding, under comparable training budgets. The existing spectral-flux sampling ablation addresses a different question. Reporting actual memory and runtime savings, together with the frequency of skipped rewards when face tracking fails, would clarify whether the efficiency mechanism changes the distribution of examples receiving synchronization supervision (Section 3.3).

  3. The evaluation is narrow relative to the breadth of the claimed generation improvements. The main test set contains 110 English samples, while the training corpus is internal and its composition is insufficiently characterized for independent reproduction. The stated train-test separation is useful, but speaker counts, data sources, and deduplication procedures would make that claim more assessable. Confidence intervals over clips are particularly important for FVD/FID and model comparisons at this scale. Likewise, the 1,040 human judgments should be accompanied by the number of unique clips and uncertainty accounting for repeated ratings.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.