acceptodds
Withdrawn from ICLR 2027

UltraDub: Towards Authentic Dubbing by Unifying Visually-Steered Flow Learning and Trajectory Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual voice cloning requires intelligible, speaker-consistent speech synchronized with visible articulation. However, sequential multimodal conditioning can disrupt previously established temporal and speaker cues, while imbalanced inference guidance can improve linguistic accuracy at the expense of lip synchronization. In this paper, we propose UltraDub, a unifying visually steered flow learning and trajectory Guidance dubbing framework that leverages vision in two ways: as continuous motion for multimodal context aggregation, and as structural rhythm for trajectory rectification. Specifically, we introduce the Motion-guided Dual-context Retrieving (MDR) module, which continually recalibrates linguistic and speaker-style retrieval through shared lip-motion query residuals, utilizing independent time-conditioned gates to regulate their contributions. Furthermore, we propose Rhythm-anchored Trajectory Guidance (RTG), a training-free mechanism that evaluates hierarchical multimodal corrections at a visual-only predictive midpoint, safely strengthening semantic conditioning without sacrificing temporal alignment. Finally, we construct DiverseDub, a multi-scenario benchmark to evaluate video dubbing in the wild. Extensive experiments demonstrate that UltraDub achieves state-of-the-art performance across four datasets. Our anonymous project page is available at: https://ultradub2026.github.io.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Movie Dubbing, Visual Voice Cloning, Audio-Visual Aligning, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9CSPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

UltraDub is a visual voice cloning (V2C) framework designed to generate authentic dubbing speech from silent video, text scripts, and reference audio, demonstrating state-of-the-art performance across four benchmark datasets. The paper addresses two core challenges in existing V2C systems: cross-modal interference caused by sequential conditioning and lip-sync distortion induced by imbalanced Classifier-Free Guidance (CFG) during inference. To resolve these issues, the authors propose Motion-guided Dual-context Retrieving (MDR) during training to dynamically anchor linguistic and speaker features to shared visual motion, along with Rhythm-anchored Trajectory Guidance (RTG) during inference to prevent generative drift using a visual midpoint anchor. Furthermore, the authors introduce DiverseDub, a curated benchmark of 2,595 clips across eight real-world categories constructed with DNSMOS quality filtering and SyncNet offset verification.

Critical strengths

This work makes a solid technical contribution by diagnosing the trade-off between intelligibility and visual synchronization in multi-condition Flow Matching while introducing a retraining-free trajectory guidance mechanism and a practical benchmark. The RTG mechanism provides an elegant approach to stabilizing inference trajectories by computing a visual rhythm anchor and applying hierarchical guidance corrections without requiring model retraining. Additionally, the authors establish rigorous evaluation standards through strict cross-utterance evaluation protocols and contribute the DiverseDub benchmark, which covers diverse real-world domains filtered via DNSMOS and provides a valuable resource for the audiovisual research community.

Critical weaknesses

Major Weakness

  • The paper completely lacks component-wise ablation experiments for its two core contributions, MDR and RTG. Without isolating the impact of individual architectural modules or comparing RTG against standard CFG on the same backbone, it is impossible to verify whether performance gains stem from the training architecture or the inference guidance.

  • The paper claims that MDR mitigates cross-modal interference, yet provides no empirical evidence to support this mechanism. Incorporating visualizations such as attention maps or layer-wise feature retention analyses would validate that MDR successfully prevents feature corruption across layers.

Minor Weakness

  • The paper relies solely on automated metrics, lacking subjective human evaluations (e.g., MOS or AB preference tests) to confirm user-perceived naturalness and lip-sync quality.

  • Figure 2(c) does not explicitly label the three delta vectors defined in Equation (6), making it harder to map the schematic diagram directly to the mathematical equations.

  • References [52] and [53] contain duplicate citations for the same paper under different publication years and invalid page ranges, requiring cleanup in the final version.

Questions for authors
  • Could the authors include a Limitations and Future Work section addressing the increased computational latency (4-5x NFE) of RTG and discussing potential inference acceleration strategies?

  • In Equation (7), the guidance schedule alpha(t_n) = lambda_max * t_n^gamma defines a time-varying exponent gamma, but the implementation uses a constant schedule (gamma = 0). Did the authors explore time-varying schedules (gamma > 0) to adjust guidance strength across flow steps, and what were the empirical trade-offs?

  • In Table 1, UltraDub achieves lower LSE-D scores (tighter lip-sync) than Ground Truth real videos. Do the authors attribute this to hyper-precise alignment from RTG or to a metric artifact of SyncNet where synthetic audio over-fits to visual features?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9CSP (Part 1 of 2)

We sincerely thank the Reviewer 9CSP for recognizing our RTG and multi-condition flow matching dubbing framework (methodological innovation), strict cross-utterance evaluation (experimental validity), and DiverseDub with diverse real-world scenarios (benchmark contribution). We address all weaknesses (W) and questions (Q) point by point. We hope the clarifications and supporting evidence below will help resolve potential misunderstandings and support a reassessment of our work.

W1. MDR and RTG Component Ablation

We respectfully clarify that the component ablations were already reported in Table X1 on our supplementary project page, whose link was provided with the original submission. Reviewer Dszy also acknowledged this, stating: “The supplementary component ablations support the contributions of MDR and RTG, ...” and “Beyond the memory-configuration ablations, ...” We thank you again for your valuable suggestion. To highlight these results, we have now moved the ablation study (Table X1) into the main text (Sec.5).

MethodWER(%)↓SPKSIM(%)↑LSE-C↑LSE-D↓
GT2.3356.187.806.85
CoSyncDiT2.0947.537.836.88
SA+FFN(base)1.5247.123.1711.25
+MDR(linguistic-only)1.4346.538.176.59
+MDR(speaker-style-only)1.5851.928.136.64
+MDR(shared-dual)1.3951.498.186.56
+MDR(shared-dual)+RTG1.3752.058.236.50

On LRS3-Cross, under conventional CFG, dual-memory MDR improves LSE-C from 3.17 to 8.18 and SPKSIM from 47.12% to 51.49% over SA+FFN. Notably, MDR already outperforms the recent CoSyncDiT baseline across all four metrics without RTG, demonstrating the advantage of the MDR architecture independently of RTG. The single-memory variants further examine the contributions of each retrieval branch. On the same dual-memory backbone, RTG further improves performance. More importantly, our text-guidance analysis shows that RTG preserves audiovisual synchronization more effectively than conventional CFG as text guidance increases (Figs.5 and9 of the revised manuscript).

W2. Mitigates Interference by MDR

As suggested, we have added attention maps in Fig. 6 (Sec.5 Attention Visualization), with additional examples in Appendix A.10, to examine MDR’s ability to mitigate interference. This visualization shows whether target-text tokens align with the target-frame or reference-frame region, providing a qualitative view of potential cross-boundary interference (see boundary details).

In the boundary details of Fig. 6, CoSyncDiT shows reference-frame attention to target-text tokens, while AlignDiT deviates further. In contrast, UltraDub’s attention path passes precisely through the intersection of the reference–target boundaries. These observations are consistent with MDR’s intended role in reducing interference and may help explain UltraDub’s strong WER and speaker-similarity results, including on multi-scenario benchmark (DiverseDub).

W3. Subjective Evaluation

In fact, we previously reported the subjective MOS results on our supplementary project page (Table X4). Now, we have included them in the revision (Appendix A.9). Following AlignDiT, a five-point scale for naturalness (MOS-N) and speaker similarity (MOS-S), using the same protocol for all methods. Results were reported as mean ± 95% confidence interval (CI). UltraDub achieved the highest mean MOS-N and MOS-S among the evaluated dubbing methods, indicating natural speech and strong speaker similarity.

MethodMOS-N↑MOS-S↑
Ground Truth4.15±0.124.01±0.13
ProDubber3.10±0.172.55±0.13
VoiceCraftDub3.31±0.093.15±0.15
InstructDubber3.36±0.122.60±0.14
AlignDiT3.82±0.143.99±0.13
CoSyncDiT3.86±0.123.98±0.15
UltraDub3.96±0.134.01±0.12

W4. Figure and Type Corrections

We have explicitly labeled the three delta vectors from Eq. (6) in Fig. 2(c) and explain the caption to clarify their scaled contributions to the state update. Furthermore, we have also added detailed pseudocode as Algorithm 1 in Appendix A.2. RTG addresses a key challenge in V2C dubbing by anchoring hierarchical corrections to visual rhythm, helping balance intelligibility with audiovisual synchronization without retraining. We have corrected the reference in the revision.

AuthorsOpenReview ↗

Response to Reviewer 9CSP (Part 2 of 2)

Q1 RTG's Latency and Potential Acceleration

We compare RTG with two representative CFG schemes for flow-based dubbing: standard two-branch CFG and AlignDiT-style three-branch CFG (i.e., the Fig. 5 baseline). All methods use the same UltraDub checkpoint and are evaluated on CelebV-Dub in the cross-utterance setting, with one RTX 4090 per run.

We provide two potential acceleration strategies for RTG:

  • (1) replaying captured GPU operations using CUDA Graphs while reusing fixed conditions

  • (2) additionally caching static cross-attention K/V and precomputing timestep-dependent quantities for the fixed sampling schedule.

Both acceleration strategies preserve the same results with original RTG. The second strategy further reduces the latency to a level close to that of standard two-branch CFG and even surpasses AlignDiT-style three-branch CFG in inference efficiency. Importantly, these are optional for acceleration discuss. RTG’s core methodological contribution remains its rhythm anchored guidance mechanism.

MethodLatency (s/utterance)↓RTF↓
Standard two-branch CFG2.08700.6184
AlignDiT-style three-branch CFG2.25130.6671
RTG (original)4.59951.3628
RTG (CUDA Graph + fixed-condition reuse)2.24440.6650
RTG (CUDA Graph + condition, K/V, and timestep caching)2.15880.6396

We thank the reviewer again for this valuable suggestion. We have incorporated the above analysis into the paper and added a Limitations and Future Work (see Apendix A.11) section discussing latency in the supplementary material. We will release all training, inference, and acceleration code to support further research by the community.

Q2. Time-Varying Guidance Schedule

We evaluated γ∈{0.5,1,2}\gamma \in \{0.5, 1, 2\} in the CelebVDub cross-utterance setting. All variants use RTG with the same checkpoint, generation seed, 32-step sampler, and λmax⁡=1\lambda_{\max}=1.

γWER (%)↓SPKSIM (%)↑EmoSIM (%)↑AVSync↑LSE-C↑LSE-D↓
09.2156.9180.910.4287.387.09
0.510.3254.4481.270.4207.227.22
112.6151.2179.720.4117.027.37
215.2247.4178.620.4046.847.51

Compared with γ = 0, γ = 0.5 improves EmoSIM by 0.36 percentage points, but reduces SPKSIM by 2.47 percentage points and worsens all three synchronization metrics. Under the same ASR evaluation, WER also increases slightly from 9.21% to 10.32%. For γ = 1 and 2, all six point estimates are worse than γ = 0. Overall, none of the tested positive exponents provides a compelling trade-off, at least in terms of objective metrics.

Q3. Better than Ground Truth's LSE-D(↓)

This phenomenon is not unique to UltraDub and has also been reported in prior audiovisual generation work. For example, VoiceCraft-Dub (ICCV 2025) [1] reports an LSE-D of 6.59, compared with 6.88 for GT. DiFlowDubber (CVPR 2026) [2] reports 6.73, compared with 6.78 for GT. Similar observations extend to talking-face generation: EcoFace (ICLR 2025) [3] reports 10.253 vs. 11.535 for GT, and TransFace (ACL 2024) [4] reports 7.193 vs. 7.407 for GT.

One possible explanation is that LSE-C/D are computed in SyncNet's learned audiovisual feature space, where ground truth (GT) does not necessarily achieve the best score. Nevertheless, LSE-C/D remain well-established metrics. For a more comprehensive evaluation, we have also added AVSync (ACM MM 2025)[5], an AV-HuBERT-based metric, which includes a co-developer of SyncNet (ACCV 2016)[6]. As result, UltraDub achieved the highest AVSync scores among the evaluated dubbing methods, further supporting its audiovisual synchronization performance (please see Appendix A.8).

[1] Sung-Bin Kim et al. VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models. ICCV, 2025. Table 3.

[2] Ngoc-Son Nguyen et al. DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization. CVPR Findings, 2026. Table 2.

[3] Jiajian Xie et al. EcoFace: Audio-Visual Emotional Co-Disentanglement Speech-Driven 3D Talking Face Generation. ICLR, 2025. Table 1.

[4] Xize Cheng et al. TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation. Findings of ACL, 2024. Table 1.

[5] Jeongsoo Choi et al. AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation. ACM Multimedia, 2025. Section 4.2.

[6] Joon Son Chung and Andrew Zisserman. Out of Time: Automated Lip Sync in the Wild. ACCV Workshops, 2016.

Reviewer_DszyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes UltraDub for visual voice cloning. It introduces MDR to coordinate text and speaker-style retrieval using shared lip-motion features, and RTG, a training-free sampling strategy that evaluates multimodal corrections at a visual anchor. The authors also introduce DiverseDub, a benchmark spanning eight categories. Experiments on four datasets report improved speaker similarity and lip synchronization while maintaining competitive speech intelligibility.

Critical strengths
  1. MDR and RTG use lip-motion information to coordinate both context retrieval and sampling, providing a coherent approach to balancing linguistic accuracy, speaker identity, and audiovisual synchronization.
  2. Evaluation on four datasets shows leading speaker similarity and lip-synchronization results among the compared methods while maintaining competitive intelligibility. On GRID, UltraDub reduces WER from AlignDiT's 19.47% to 14.53%.
  3. DiverseDub contains 2,595 clips across eight categories, broadening evaluation to varied real-world content and complementing existing dubbing benchmarks.
Critical weaknesses
  1. The supplementary component ablations support the contributions of MDR and RTG, but do not directly isolate the benefits of shared lip-motion query residuals, independent time-conditioned gates, or the visual-only midpoint anchor against simpler alternatives. This limits the evidence for the individual contributions of these specific design choices.
  2. Although the architecture and sampling settings are described, the training configuration remains insufficiently specified. In particular, the training datasets and splits, explicit loss objective, optimization settings, and modality-dropout probabilities are not clearly reported. These omissions limit reproducibility and make it difficult to assess the experimental setup.
  3. Although the supplementary page identifies the Figure 5 baseline as conventional CFG, the evaluation dataset and precise baseline configuration remain unclear. Furthermore, RTG requires additional conditional network evaluations at the midpoint, but no comparison under matched computational budgets or measurement of inference overhead is reported. Consequently, it remains unclear how much of the improvement is attributable to the guidance strategy rather than additional computation.
  4. The authors acknowledge that stronger text guidance decreases LSE-C and increases LSE-D, although RTG mitigates this degradation relative to the baseline. These findings support improved robustness, but the abstract’s claim of strengthening conditioning “without sacrificing temporal alignment” should be qualified. The differing trends in AVSync and the SyncNet-based metrics also warrant further discussion.
Questions for authors
  1. Could the authors provide the complete training setup, including training datasets and splits, backbone initialization, loss objective, optimizer and learning-rate schedule, training duration, and modality-dropout probabilities?
  2. Beyond the memory-configuration ablations, could the authors isolate the contributions of shared lip-motion query residuals and independent time-conditioned gates through comparisons with alternative visual-conditioning schemes and shared versus independent gating?
  3. Could the authors compare visual-only and fully conditioned midpoint anchors using the same backbone and sampling settings? Additionally, could RTG and conventional CFG be compared under matched computational budgets, with the numbers of network evaluations and inference latency reported?
  4. The supplementary page identifies the baseline as conventional CFG. For Figure 5, which dataset and split are evaluated, and are both sampling strategies applied to the same trained backbone with identical settings apart from the guidance strategy?
  5. How should the improving AVSync but decreasing LSE-C and increasing LSE-D under stronger text guidance be interpreted? Does preserving temporal alignment refer to reduced degradation relative to conventional CFG, and could the abstract’s claim be clarified accordingly?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Dszy (Part 1 of 2)

We sincerely thank the reviewer for recognizing the coherent and novel designs of MDR and RTG, UltraDub’s strong performance across four datasets, and particularly the value of DiverseDub as a multi-scenario benchmark covering in-the-wild dubbing content across eight real-world categories. We address all questions (Q) and weaknesses (W) point by point. We hope the clarifications and supporting evidence below will help resolve potential misunderstandings and support a reassessment of our work.

Q1&W2. Training Reproducibility

Thank you for your valuable feedback. We have added the loss function details in Appendix A.3. Under the same loss-function as AlignDiT, our method achieves strong results across four dubbing datasets, highlighting the effectiveness of our novel architecture and guidance design.

We summarize the training implementation and available records below. If accepted, we will open-source all code to support reproducibility. Further details of architecture were already provided in main paper Sec.5 "Implementation Details"

ComponentConfiguration
Training data139,786 LRS3 clips: pretrain=108,058; trainval=31,728.
BackboneDiT: layers=22; hidden dim=1,024; heads=16.
InitializationCompatible weights from EMA of a 250k-step, 16 kHz F5-TTS checkpoint.
Audio features16 kHz; Mel bins=100; window/FFT=640; hop=160.
OptimizationAdamW; peak LR=10−510^{-5}; linear warmup/decay; warmup argument=20k updates.
Batch3,200 Mel frames/GPU; ≤64 utterances/batch.
Training duration104.6 h wall-clock on 1× RTX 4090, including pauses and I/O.
Condition dropoutFollowing AlignDiT: all/text/video=0.2/0.2/0.2; reference=0.3 independently.

Q2 & W1. More Ablations

As suggested, we extended the original ablations. All experiments start from UltraDub’s dual-memory baseline (the penultimate row of the original ablation table), with RTG disabled to isolate architectural effects. First, we replace the shared lip-motion query residuals with CoSyncDiT-style non-shared visual injection.

Methods (LRS3-Cross)Params (M)WER (%)↓SPKSIM (%)↑LSE-C↑LSE-D↓
UltraDub (CoSyncDiT-style visual injection)609.7901.8950.137.956.82
UltraDub (shared query residuals)586.6991.3951.498.186.56

Shared query residuals achieve better results across all four metrics with fewer parameters, supporting both the effectiveness and parameter efficiency of this design. Then, we replace the independent time-conditioned gates with a gate shared across the two retrieval branches.

Methods (LRS3-Cross)Params (M)WER (%)↓SPKSIM (%)↑LSE-C↑LSE-D↓
UltraDub (shared gating)563.6081.4851.078.166.59
UltraDub (independent gating)586.6991.3951.498.186.56

The results show that shared gating reduces the parameter count but slightly degrades performance, suggesting that it is less effective at aggregating linguistic and speaker-style contexts. So, we retain it to balance performance.

Q3&W3. Ablation on Midpoint Anchor

As suggested, we isolate the midpoint predictor by comparing visual-only and fully conditioned anchors within the same RTG framework on CelebVDub-S2, using the same sampling steps and model weights.

Midpoint anchor (CelebVDub dataset)WER (%)↓SPKSIM (%)↑LSE-C ↑LSE-D ↓
Visual-only (sT=4s_T=4)9.21056.9077.3767.091
Fully conditioned (sT=4s_T=4)9.62156.7137.3637.103

The results show that visual anchoring gives better WER, speaker similarity, emotion similarity, and synchronization point estimates. We further increase text guidance to sT=5s_T=5, keeping all other settings unchanged.

Midpoint anchor (CelebVDub dataset)WER (%)↓SPKSIM (%)↑LSE-C ↑LSE-D ↓
Visual-only (sT=5s_T=5)7.81255.9497.3227.138
Fully conditioned (sT=5s_T=5)8.20555.7467.3087.153

At sT=5s_T=5, visual anchoring lowers WER and improves speaker similarity, with particularly favorable audiovisual alignment scores (LSE-C/D). These results support RTG’s visual-only velocity to locate a shared midpoint for hierarchical multimodal corrections.

AuthorsOpenReview ↗

Response to Reviewer Dszy (Part 2 of 2)

Q3&W3. Efficiency Analysis

To assess inference cost, we compare RTG with two conventional CFG schemes using the same checkpoint and 32 sampling steps. All evaluations use a single RTX 4090 per run.

MethodLatency (s/utterance)↓RTF↓
Standard two-branch CFG2.08700.6184
AlignDiT-style three-branch CFG (Fig.5 Baseline)2.25130.6671
RTG (original)4.59951.3628

Given the higher latency of the original RTG, we further investigate whether this overhead can be reduced without changing the guidance mechanism. We evaluate two optional implementation-level optimizations:

RTG acceleration strategyLatency (s/utterance)↓RTF↓
CUDA Graph + fixed-condition reuse2.24440.6650
CUDA Graph + condition, K/V, and timestep caching2.15880.6396

Both strategies preserve the outputs of the original RTG. The second reduces latency from 4.5995 to 2.1588 seconds per utterance, bringing it close to two-branch CFG and below three-branch CFG in this implementation. Further details are provided in Appendix A.11.

Q4&W3. Clarifying the Baseline in Fig. 5

The baseline in Fig. 5 is AlignDiT-style three-branch CFG, a conventional extrapolation-based CFG method.

The evaluation is conducted on the LRS3 dataset using the same trained UltraDub backbone, checkpoint, and numerical guidance-scale range, with only the guidance mechanism varied. Therefore, the improvement is attributable solely to RTG’s guidance strategy.

SettingBaseline in Fig.5RTG (Ours)
Same dataset: LRS3-Cross, Setting 2✓✓
Same trained backbone and checkpoint of UltraDub✓✓
Same numerical guidance scale range: 1.0–5.0✓✓
Guidance mechanismThree branch AlignDiT-Style CFGThe proposed Visual-anchored RTG

We have further added comparisons on the challenging CelebVDub dataset in Appendix A.7, using the same evaluation setup as Fig. 5. The results similarly show that RTG better preserves audiovisual synchronization under stronger text guidance, further supporting its effectiveness.

Q5&W4. Clarifying RTG’s Synchronization Advantage under Strong Text Guidance

Although LSE-C/D show some degradation as guidance strengthens, RTG consistently outperforms the current guidance baseline (AlignDiT-style CFG) at every tested scale. The table below reports the LSE-D results from Fig. 5 and their changes relative to each method's scale-1 score.

Guidance scaleAlignDiT-style CFG ↓RTG ↓CFG performance change Δ∗\Delta^*RTG performance change Δ∗\Delta^*Reduction in degradation GG ↑
16.461116.427180.000000.00000—
26.468286.45368-0.00717-0.02650-269.6%
36.543486.47404-0.08237-0.04686+43.1%
46.664686.49697-0.20357-0.06979+65.7%
56.784786.51286-0.32367-0.08568+73.5%

Here, Δ(s)=D(1)−D(s) and G(s)=100×[ΔRTG(s)−ΔCFG(s)]/|ΔCFG(s)|. From scale 1 to 5, RTG's LSE-D increases by only 0.08568 (Ours), versus 0.32367 (Baseline) for CFG, corresponding to a 73.5% smaller increase.

Although AVSync (ACM MM 2025, whose authors include a co-developer of SyncNet) has been suggested to provide a more accurate and robust assessment[1], LSE-C/D (ACCV 2016) remain widely used metrics for audiovisual alignment. To make our claim more precise, we have adopted the reviewer’s suggestion and revised the abstract: enabling stronger semantic conditioning while better preserving temporal alignment than conventional CFG.

[1] J. Choi et al. “AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation.” ACM Multimedia, 2025.

Reviewer_jJdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes UltraDub, which unifies multimodal context retrieval and trajectory guidance for voice cloning. It also introduces a new dubbing benchmark which consists of videos from 8 categories. Experiments are conducted on 3 public datasets and the introduced dataset and the proposed approach is compared against other SOTA methods.

Critical strengths

New challenging dataset is introduced.

SOTA performance is achieved.

Critical weaknesses

Major Weaknesses

  • There is no user study in the paper. A user study is necessary in generative works as quality cannot be judged only by automated metrics, which are not always highly correlated with human perception.
  • There is no ablation study so the contribution of each of the proposed components cannot be quantified.
  • It is not clear what data were used for training. Details about the training data should be provided, e.g., which parts of the datasets are used for training the model.
  • Assuming that the model has been trained on the training sets of the public sets used for evaluation, it is unfair to compare against models that have not been trained on the same data. The proposed model has been trained with in-domain data whereas the other models have not.
  • The overall loss function is not mentioned, so it’s not clear how the model was trained.
  • All the examples in the supplementary material use the same sentence for the ground truth and generated video. This is an easy scenario. The model should also be tested on unseen sentences.

Other weaknesses

  • There are no details about the vocoder used and the other encoders used in the model. A table or a figure explaining the architectures of encoders/decoders would be helpful.
  • A pretrained vocoder is used, please mention how it has been pretrained.
  • Will the dataset become publicly available?
  • Which model is used for estimating the WER? The reference used in Section 5 (metrics) on p. 7 is irrelevant.
Questions for authors

Please see the Major weaknesses section above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer jJdt (Part 1 of 2)

We sincerely thank the reviewer for recognizing our challenging dubbing benchmark and UltraDub’s state-of-the-art performance. We address all weaknesses (W) point by point below. We hope the clarifications and supporting evidence below will help resolve potential misunderstandings and support a reassessment of our work.

W1. Human Evaluation

We appreciate the reviewer’s emphasis on human evaluation. In fact, the subjective MOS study was included on our supplementary project page at the time of submission (Table X4). Now, we have now added its protocol and results to Appendix A.9 of the revised manuscript.

Following AlignDiT, listeners rated speech naturalness (MOS-N) and speaker similarity (MOS-S) on a five-point scale. All methods were evaluated using the same protocol. Results are reported as mean ± 95% confidence interval. Among the evaluated dubbing methods, UltraDub achieves the highest mean MOS-N and MOS-S, providing complementary human evidence for its speech naturalness and speaker similarity.

MethodMOS-N↑MOS-S↑
Ground Truth4.15±0.124.01±0.13
ProDubber3.10±0.172.55±0.13
VoiceCraftDub3.31±0.093.15±0.15
InstructDubber3.36±0.122.60±0.14
AlignDiT3.82±0.143.99±0.13
CoSyncDiT3.86±0.123.98±0.15
UltraDub3.96±0.134.01±0.12

W1. MDR and RTG Component Ablation

To be honest, the ablations were included at submission under supplementary project page “More Experimental Results” (Table X1). Reviewer Dszy also noted that “The supplementary component ablations support the contributions of MDR and RTG.” Now, we have now incorporated it into the main text (Sec.5 Ablation study).

MethodWER(%)↓SPKSIM(%)↑LSE-C↑LSE-D↓
GT2.3356.187.806.85
SA+FFN(base)1.5247.123.1711.25
+MDR(linguistic-only)1.4346.538.176.59
+MDR(speaker-style-only)1.5851.928.136.64
+MDR(shared-dual)1.3951.498.186.56
+MDR(shared-dual)+RTG1.3752.058.236.50

On LRS3-Cross, with conventional CFG, dual-memory MDR improved SPKSIM from 47.12% to 51.49% and LSE-C from 3.17 to 8.18 over SA+FFN. The single-memory variants further examined the contributions of linguistic and speaker-style retrieval. Adding RTG to the same dual-memory backbone reduced WER from 1.39% to 1.37%, improved SPKSIM from 51.49% to 52.05%, and reduced LSE-D from 6.56 to 6.50. Besides, we have added detailed ablations of visual query residuals and gating configurations to further validate the effectiveness of our design choices.

W3&4. Clarifying Training Details and Comparison Principles

All methods are retrained on the LRS3 pretrain and trainval splits. For AlignDiT, we directly use the authors’ official LRS3 checkpoint. This ensures a rigorous and fair comparison. For CelebV-Dub, GRID, and DiverseDub, all methods are evaluated without dataset-specific fine-tuning to directly assess out-of-domain (OOD) generalization. We thank the reviewer again for the suggestion. We have added a Training and Evaluation Protocol subsection in Appendix A.1 to clarify these training and evaluation details.

W5 Loss Function.

We have added the full loss formulation and implementation details in Appendix A.3, “Details of Training Loss” (Eqs. 10–11). UltraDub follows AlignDiT’s training objective, combining flow matching with auxiliary CTC supervision. Under the same loss-function setup as AlignDiT, UltraDub achieves strong results across four dubbing datasets, highlighting the effectiveness of our architecture and guidance design.

AuthorsOpenReview ↗

Response to Reviewer jJdt (Part 2 of 2)

W6. Clarifying the Dubbing Evaluation Settings

Unlike TTS, V2C requires evaluating audiovisual alignment without modifying the original lip, so the input text must match the original utterance, as same as in all V2C dubbing paper. But the original utterance has three aspects:

  • Held-out target utterances: test clips excluded from training set.
  • Zero-shot (unseen cross-dataset) evaluation: direct evaluation on another dataset without dataset-specific training or fine-tuning.
  • Cross-utterance setting: using a different utterance of the same speaker as the reference, rather than the target ground-truth recording.
Evaluation datasetHeld-out target utterancesZero-shot (unseen cross-dataset) evaluationCross-utterance setting
LRS3-Cross✓✘(because all methods were trained on LRS3)✓
CelebV-Dub✓✓✓
GRID✓✓✓
DiverseDub✓✓✓

As clarified in W3 and Appendix A.1, all compared methods were trained on LRS3. LRS3-Cross uses the held-out LRS3 test split, while CelebV-Dub, GRID, and DiverseDub were not used for training or fine-tuning, enabling zero-shot evaluation. Besides, in our paper, all dataset evaluations follow the cross-utterance setting.

To our knowledge, combining zero-shot and cross-utterance evaluation (the last three rows) constitutes a particularly challenging dubbing setting. We encourage the reviewer to listen to the examples on our demo page, which follow the corresponding evaluation settings.

W6&7. Details of vocoder and the other encoders

Following the recent dubbing method CoSyncDiT, we use the same pretrained 16 kHz Vocos checkpoint and encoder configurations. The tables below summarize the component architectures and vocoder pretraining configuration. Upon acceptance, we will release the pretrained checkpoints and configuration files to reproducte our results.

  • Configuration of the 16 kHz Vocos Vocoder
ItemSetting
Vocoder16 kHz Vocos; 8 ConvNeXt blocks; hidden 512, FFN 1,536
Mel input100 bins; FFT 640; hop 160
OptimizationAdamW; LR 5e-4; cosine decay; no warmup
Batch / crop16 / 2 seconds
LossAdversarial + feature matching + Mel L1; Mel weight 45, MR weight 0.1
Training1.12M updates per optimizer
Pretraining dataMixed corpus; historical membership unverified
  • Configuration of the Encoder
ComponentArchitecture
Visual feature extractorAV-HuBERT-Large; 24 layers; 1,024-D features at 25 FPS
Visual temporal encoderTwo transposed convolutions (4× upsampling) + two Conformer blocks; hidden 1,024
Text encoderToken embedding + four ConvNeXtV2 blocks; hidden 512
Reference-speech encoderTwo Conv1d layers; 100→256→256; kernel 3
Acoustic backbone22 DiT blocks; hidden 1,024; 16 heads
Vocoder (Waveform Decoder)16 kHz Vocos; eight ConvNeXt blocks + inverse STFT

W8

Yes. We will publicly release DiverseDub (across eight real-world challenging dubbing scenarios) upon acceptance, together with its annotations and evaluation pipline. We hope this release will benefit the multimedia community and advance research on AI dubbing and audiovisual synthesis.

W9

Following AlignDiT, we use Whisper-large-v3 to transcribe generated speech and compute WER against the target transcript. We would like to clarify that the citation to Radford et al. (2023) refers to Whisper model rather than the definition of WER, as same as in AlignDiT’s Evaluation Metrics section[1]. The citation is appropriate; we have clarified these in the revision.

[1] J. Choi et al. “AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation.” ACM Multimedia, 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Gentle Follow-up on Recent Rebuttal Submission and Discussion

Dear Reviewers,

Thank you again for the time and care you have devoted to reviewing our manuscript. I wanted to extend a gentle follow-up on the rebuttal we submitted a few days ago concerning our manuscript.

We are eager to hear your thoughts and understand if the additional clarifications and explanations provided have addressed the concerns you raised. We deeply value your insights and are fully open to further discussion should there be any outstanding issues or new queries.

We appreciate the time and effort you invested in this review process and look forward to your valuable feedback.

Best regards,

Authors of Paper 5006

AuthorsOpenReview ↗

General response (updated 16 Nov)

Dear PC, SAC, AC, Reviewers:

We sincerely thank all reviewers again for their time and careful consideration of our work. Their thoughtful comments and suggestions have helped us improve the paper and present our contributions more clearly. Reviewer 9CSP recognizes the technical contribution of training-free RTG, the strict cross-utterance evaluation, and DiverseDub’s practical value. Reviewer Dszy highlights the coherent use of lip motion in MDR and RTG, strong results across four datasets, and the supporting supplementary component ablations. Reviewer jJdt acknowledges the challenging new audiovisual dubbing benchmark and state-of-the-art performance. However, Reviewers 9CSP and jJdt may not have noticed that the component ablations and subjective MOS evaluation were already available on our supplementary project at submission; both have now been included in the revised manuscript for clarity.

The revisions below substantiate and further clarify our original contributions:

(☑) 1. Visualizing attention maps within the DiT. Alignment visualizations examine potential cross-segment interference, while additional AVSync evaluations complement the SyncNet-based synchronization results.

(☑) 2. Adding detailed ablations of shared queries and independent gating. Shared query residuals outperform CoSyncDiT-style visual injection across all four metrics while reducing parameters from 609.790M to 586.699M. Independent gating outperforms shared gating across all four metrics, supporting separate regulation of linguistic and speaker-style retrieval.

(☑) 3. Re-highlighting the original MDR and RTG component ablations. These ablations were already provided at submission and are now included in the main text. Without RTG, MDR improves SPKSIM from 47.12% to 51.49% and LSE-C from 3.17 to 8.18. RTG further improves all four metrics on the same backbone.

(☑) 4. Analyzing visual-only vs. fully conditioned midpoint anchors. With identical weights and sampling steps, visual-only anchors improve WER, speaker similarity, and synchronization over fully conditioned anchors.

(☑) 5. Clarifying RTG’s guidance-schedule choice. Comparisons of γ=0,0.5,1,2\gamma=0,0.5,1,2 support constant guidance as a balanced default under the tested conditions.

(☑) 6. Analyzing RTG’s inference cost. With the same checkpoint and sampling steps on one RTX 4090, CUDA Graphs with condition reuse reduce RTG latency from 4.60 to 2.24 seconds per utterance, comparable to AlignDiT-style three-branch CFG (2.25 seconds). Additional K/V and timestep caching further reduce latency to 2.16 seconds.

(☑) 7. Re-highlighting the original subjective MOS evaluation. These results were already provided at submission and are now included in the manuscript. UltraDub achieves the highest mean MOS-N (3.96±0.13) and MOS-S (4.01±0.12) among the evaluated dubbing methods.

(☑) 8. Clarifying unseen-data and cross-utterance evaluation. All four datasets use held-out target utterances and cross-utterance references. CelebV-Dub, GRID, and DiverseDub additionally test zero-shot generalization without dataset-specific fine-tuning.

(☑) 9. Adding training and implementation details. We provide more explicit training, loss, encoder, and vocoder specifications, add RTG pseudocode, clarify evaluation models, and correct figures and references.

(☑) 10. Highlighting DiverseDub’s benchmark value. DiverseDub broadens evaluation across eight real-world categories with varied speaking styles and production scenarios. The benchmark, annotations, and evaluation pipeline will be publicly released upon acceptance.

Changes in the main paper are highlighted in blue. We hope these clarifications and supporting evidence will be helpful to the Reviewers and Area Chair in their further consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.