- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
UltraDub is a visual voice cloning (V2C) framework designed to generate authentic dubbing speech from silent video, text scripts, and reference audio, demonstrating state-of-the-art performance across four benchmark datasets. The paper addresses two core challenges in existing V2C systems: cross-modal interference caused by sequential conditioning and lip-sync distortion induced by imbalanced Classifier-Free Guidance (CFG) during inference. To resolve these issues, the authors propose Motion-guided Dual-context Retrieving (MDR) during training to dynamically anchor linguistic and speaker features to shared visual motion, along with Rhythm-anchored Trajectory Guidance (RTG) during inference to prevent generative drift using a visual midpoint anchor. Furthermore, the authors introduce DiverseDub, a curated benchmark of 2,595 clips across eight real-world categories constructed with DNSMOS quality filtering and SyncNet offset verification.
Critical strengths
This work makes a solid technical contribution by diagnosing the trade-off between intelligibility and visual synchronization in multi-condition Flow Matching while introducing a retraining-free trajectory guidance mechanism and a practical benchmark. The RTG mechanism provides an elegant approach to stabilizing inference trajectories by computing a visual rhythm anchor and applying hierarchical guidance corrections without requiring model retraining. Additionally, the authors establish rigorous evaluation standards through strict cross-utterance evaluation protocols and contribute the DiverseDub benchmark, which covers diverse real-world domains filtered via DNSMOS and provides a valuable resource for the audiovisual research community.
Critical weaknesses
Major Weakness
-
The paper completely lacks component-wise ablation experiments for its two core contributions, MDR and RTG. Without isolating the impact of individual architectural modules or comparing RTG against standard CFG on the same backbone, it is impossible to verify whether performance gains stem from the training architecture or the inference guidance.
-
The paper claims that MDR mitigates cross-modal interference, yet provides no empirical evidence to support this mechanism. Incorporating visualizations such as attention maps or layer-wise feature retention analyses would validate that MDR successfully prevents feature corruption across layers.
Minor Weakness
-
The paper relies solely on automated metrics, lacking subjective human evaluations (e.g., MOS or AB preference tests) to confirm user-perceived naturalness and lip-sync quality.
-
Figure 2(c) does not explicitly label the three delta vectors defined in Equation (6), making it harder to map the schematic diagram directly to the mathematical equations.
-
References [52] and [53] contain duplicate citations for the same paper under different publication years and invalid page ranges, requiring cleanup in the final version.
Questions for authors
-
Could the authors include a Limitations and Future Work section addressing the increased computational latency (4-5x NFE) of RTG and discussing potential inference acceleration strategies?
-
In Equation (7), the guidance schedule alpha(t_n) = lambda_max * t_n^gamma defines a time-varying exponent gamma, but the implementation uses a constant schedule (gamma = 0). Did the authors explore time-varying schedules (gamma > 0) to adjust guidance strength across flow steps, and what were the empirical trade-offs?
-
In Table 1, UltraDub achieves lower LSE-D scores (tighter lip-sync) than Ground Truth real videos. Do the authors attribute this to hyper-precise alignment from RTG or to a metric artifact of SyncNet where synthetic audio over-fits to visual features?
2 comments
Response to Reviewer 9CSP (Part 2 of 2)
Q1 RTG's Latency and Potential Acceleration
We compare RTG with two representative CFG schemes for flow-based dubbing: standard two-branch CFG and AlignDiT-style three-branch CFG (i.e., the Fig. 5 baseline). All methods use the same UltraDub checkpoint and are evaluated on CelebV-Dub in the cross-utterance setting, with one RTX 4090 per run.
We provide two potential acceleration strategies for RTG:
-
(1) replaying captured GPU operations using CUDA Graphs while reusing fixed conditions
-
(2) additionally caching static cross-attention K/V and precomputing timestep-dependent quantities for the fixed sampling schedule.
Both acceleration strategies preserve the same results with original RTG. The second strategy further reduces the latency to a level close to that of standard two-branch CFG and even surpasses AlignDiT-style three-branch CFG in inference efficiency. Importantly, these are optional for acceleration discuss. RTG’s core methodological contribution remains its rhythm anchored guidance mechanism.
| Method | Latency (s/utterance)↓ | RTF↓ |
|---|---|---|
| Standard two-branch CFG | 2.0870 | 0.6184 |
| AlignDiT-style three-branch CFG | 2.2513 | 0.6671 |
| RTG (original) | 4.5995 | 1.3628 |
| RTG (CUDA Graph + fixed-condition reuse) | 2.2444 | 0.6650 |
| RTG (CUDA Graph + condition, K/V, and timestep caching) | 2.1588 | 0.6396 |
We thank the reviewer again for this valuable suggestion. We have incorporated the above analysis into the paper and added a Limitations and Future Work (see Apendix A.11) section discussing latency in the supplementary material. We will release all training, inference, and acceleration code to support further research by the community.
Q2. Time-Varying Guidance Schedule
We evaluated in the CelebVDub cross-utterance setting. All variants use RTG with the same checkpoint, generation seed, 32-step sampler, and .
| γ | WER (%)↓ | SPKSIM (%)↑ | EmoSIM (%)↑ | AVSync↑ | LSE-C↑ | LSE-D↓ |
|---|---|---|---|---|---|---|
| 0 | 9.21 | 56.91 | 80.91 | 0.428 | 7.38 | 7.09 |
| 0.5 | 10.32 | 54.44 | 81.27 | 0.420 | 7.22 | 7.22 |
| 1 | 12.61 | 51.21 | 79.72 | 0.411 | 7.02 | 7.37 |
| 2 | 15.22 | 47.41 | 78.62 | 0.404 | 6.84 | 7.51 |
Compared with γ = 0, γ = 0.5 improves EmoSIM by 0.36 percentage points, but reduces SPKSIM by 2.47 percentage points and worsens all three synchronization metrics. Under the same ASR evaluation, WER also increases slightly from 9.21% to 10.32%. For γ = 1 and 2, all six point estimates are worse than γ = 0. Overall, none of the tested positive exponents provides a compelling trade-off, at least in terms of objective metrics.
Q3. Better than Ground Truth's LSE-D(↓)
This phenomenon is not unique to UltraDub and has also been reported in prior audiovisual generation work. For example, VoiceCraft-Dub (ICCV 2025) [1] reports an LSE-D of 6.59, compared with 6.88 for GT. DiFlowDubber (CVPR 2026) [2] reports 6.73, compared with 6.78 for GT. Similar observations extend to talking-face generation: EcoFace (ICLR 2025) [3] reports 10.253 vs. 11.535 for GT, and TransFace (ACL 2024) [4] reports 7.193 vs. 7.407 for GT.
One possible explanation is that LSE-C/D are computed in SyncNet's learned audiovisual feature space, where ground truth (GT) does not necessarily achieve the best score. Nevertheless, LSE-C/D remain well-established metrics. For a more comprehensive evaluation, we have also added AVSync (ACM MM 2025)[5], an AV-HuBERT-based metric, which includes a co-developer of SyncNet (ACCV 2016)[6]. As result, UltraDub achieved the highest AVSync scores among the evaluated dubbing methods, further supporting its audiovisual synchronization performance (please see Appendix A.8).
[1] Sung-Bin Kim et al. VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models. ICCV, 2025. Table 3.
[2] Ngoc-Son Nguyen et al. DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization. CVPR Findings, 2026. Table 2.
[3] Jiajian Xie et al. EcoFace: Audio-Visual Emotional Co-Disentanglement Speech-Driven 3D Talking Face Generation. ICLR, 2025. Table 1.
[4] Xize Cheng et al. TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation. Findings of ACL, 2024. Table 1.
[5] Jeongsoo Choi et al. AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation. ACM Multimedia, 2025. Section 4.2.
[6] Joon Son Chung and Andrew Zisserman. Out of Time: Automated Lip Sync in the Wild. ACCV Workshops, 2016.
Response to Reviewer 9CSP (Part 1 of 2)
We sincerely thank the Reviewer 9CSP for recognizing our RTG and multi-condition flow matching dubbing framework (methodological innovation), strict cross-utterance evaluation (experimental validity), and DiverseDub with diverse real-world scenarios (benchmark contribution). We address all weaknesses (W) and questions (Q) point by point. We hope the clarifications and supporting evidence below will help resolve potential misunderstandings and support a reassessment of our work.
W1. MDR and RTG Component Ablation
We respectfully clarify that the component ablations were already reported in Table X1 on our supplementary project page, whose link was provided with the original submission. Reviewer Dszy also acknowledged this, stating: “The supplementary component ablations support the contributions of MDR and RTG, ...” and “Beyond the memory-configuration ablations, ...” We thank you again for your valuable suggestion. To highlight these results, we have now moved the ablation study (Table X1) into the main text (Sec.5).
On LRS3-Cross, under conventional CFG, dual-memory MDR improves LSE-C from 3.17 to 8.18 and SPKSIM from 47.12% to 51.49% over SA+FFN. Notably, MDR already outperforms the recent CoSyncDiT baseline across all four metrics without RTG, demonstrating the advantage of the MDR architecture independently of RTG. The single-memory variants further examine the contributions of each retrieval branch. On the same dual-memory backbone, RTG further improves performance. More importantly, our text-guidance analysis shows that RTG preserves audiovisual synchronization more effectively than conventional CFG as text guidance increases (Figs.
5 and9 of the revised manuscript).W2. Mitigates Interference by MDR
As suggested, we have added attention maps in Fig. 6 (Sec.5 Attention Visualization), with additional examples in Appendix A.10, to examine MDR’s ability to mitigate interference. This visualization shows whether target-text tokens align with the target-frame or reference-frame region, providing a qualitative view of potential cross-boundary interference (see boundary details).
In the boundary details of Fig. 6, CoSyncDiT shows reference-frame attention to target-text tokens, while AlignDiT deviates further. In contrast, UltraDub’s attention path passes precisely through the intersection of the reference–target boundaries. These observations are consistent with MDR’s intended role in reducing interference and may help explain UltraDub’s strong WER and speaker-similarity results, including on multi-scenario benchmark (DiverseDub).
W3. Subjective Evaluation
In fact, we previously reported the subjective MOS results on our supplementary project page (Table X4). Now, we have included them in the revision (Appendix A.9). Following AlignDiT, a five-point scale for naturalness (MOS-N) and speaker similarity (MOS-S), using the same protocol for all methods. Results were reported as mean ± 95% confidence interval (CI). UltraDub achieved the highest mean MOS-N and MOS-S among the evaluated dubbing methods, indicating natural speech and strong speaker similarity.
W4. Figure and Type Corrections
We have explicitly labeled the three delta vectors from Eq. (6) in Fig. 2(c) and explain the caption to clarify their scaled contributions to the state update. Furthermore, we have also added detailed pseudocode as Algorithm 1 in Appendix A.2. RTG addresses a key challenge in V2C dubbing by anchoring hierarchical corrections to visual rhythm, helping balance intelligibility with audiovisual synchronization without retraining. We have corrected the reference in the revision.