- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VESTA, a joint talking audio–video generation model with reference-timbre control, built on a MOVA-style dual-tower DiT. It has three components: StatShift injects an ECAPA speaker embedding into the audio DiT’s AdaLN through a shared MLP (time-invariant modulation); RewardSteer adds a timbre reward and a lip-sync reward on one-step denoised predictions, the latter using a self-trained critic and a face-localized partial decode; TimbreTune performs test-time LoRA adaptation using self-paired segments of the reference audio.
Critical strengths
- The paper is well structured and easy to follow: the problem is decomposed into three clearly stated gaps, each matched by one component, and Fig. 2 lays out the structure and role of every component clearly enough to understand the overall design without the text.
- The 2×2 ablation in Table 4 is clean and makes the contribution of each reward directly visible: the timbre reward mainly improves SIM, the lip-sync reward mainly improves Sync-C/Sync-D, and the two do not interfere with each other.
Critical weaknesses
- Table 3 does not test its own central claim. The paper attributes StatShift’s advantage to token injection “leaking the words and prosody of the reference clip and thereby disrupting lip-sync,” yet the table reports only SIM and Sync. Under the evaluation protocol with cyclically shifted target text, WER is the most direct measure of content leakage, but it is omitted; WER should be a required metric for this ablation.
- Training details of the lip-sync critic f_v/f_a are missing, including the training data and how positive and negative pairs are constructed. These details determine whether the critic measures fine-grained synchronization or only coarse audio–visual correspondence, and are therefore essential to the paper’s lip-sync claims; they should at least be given in the supplementary material.
- Key comparison methods are missing from the main table: (1) a cascade baseline (zero-shot TTS + audio-driven video generation); (2) TTS methods such as the CosyVoice series should be included as references for audio quality and SIM, so that the actual level of the proposed method can be established; (3) MiniMax H3 supports reference-timbre input but is neither compared against nor mentioned in the related work, which is inappropriate.
- No details of the training data are provided. For a generative method whose results depend heavily on subjective judgment, the paper also provides no video demo.
- The origin of the audio branch’s DiT and VAE is unclear: their architecture, whether they are pretrained, and whether they are trained by the authors or adopted from existing models are all unspecified. The paper places audio generation and timbre control at the center of its contribution, yet omits the most basic description of these models.
Questions for authors
- During training, is a_ref the audio of the target clip itself, or a different utterance from the same speaker?
- Which ECAPA-TDNN checkpoint is used for the timbre reward, and on what data was it trained? Is it the same network as the conditioning encoder used by StatShift?
- For the remaining questions, please refer to the weaknesses above.
Although the novelty of the proposed method is moderate, it offers a low-cost solution in a very narrow domain (the compute and data costs should be reported) and has considerable value for engineering applications. The main method is clearly presented. If the authors can supplement the missing experiments and implementation details and demonstrate that the main method is solid and effective, I will consider raising my score.