- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies synchronous audio and video customization. Built on OVI, OmniCustom uses modality specific LoRA modules to incorporate reference images and audio, together with facial features, timbre features, and contrastive flow matching losses. The framework jointly generates video and speech with specified identity, voice timbre, and spoken content.
Critical strengths
The method has practical value. Users can jointly control generation through images, audio, and text without additional fine tuning for each new identity.
The ablations demonstrate clear improvements. Adding the contrastive losses improves both facial and speaker similarity, while inference latency remains close to that of the original OVI model, suggesting limited additional inference overhead.ons.
Critical weaknesses
Critical Weaknesses
The independent contribution of the contrastive losses is insufficiently established. These losses separate predictions with and without reference conditions, but the resulting differences are not necessarily specific to identity and timbre. The paper lacks comparisons against stronger reference guidance and separate ablations of the identity and timbre losses. It therefore remains unclear whether the gains arise from the particular contrastive formulation or from generally strengthening reference conditioning.
The sources of the performance and efficiency advantages are not sufficiently disentangled. The method is trained on one million task specific examples, but the contributions of data processing and model design are not clearly separated. Inference takes approximately 4.02 minutes, close to the 4.16 minutes reported for DreamID Omni, while the substantial speed advantage over MiniMax H3 coincides with a difference in backbone size. The results support low additional inference overhead, but do not isolate an efficiency advantage attributable to the proposed design.
Important evaluation details remain unclear. For the additional 70 test videos, the paper establishes that the videos are absent from the training set, but does not explicitly establish that their identities are also unseen. This leaves the generalization claim insufficiently verified. The user study also does not explicitly state whether participants received the reference audio, making the interpretation of timbre consistency judgments uncertain.
Minor Weaknesses
The user study treats repeated judgments from the same participant as independent observations. Its significance analysis does not account for correlations associated with participants and videos.
The efficiency analysis omits the number of trainable parameters, peak GPU memory usage, and the additional computational cost of contrastive training.
Questions for authors
Questions For Authors
Could the authors compare the contrastive losses against adjustments to reference guidance strength and separately report the effects of the identity and timbre losses.
Could controlled experiments distinguish the contributions of data filtering, audio captioning, and model design, together with a comparable analysis of computational costs.
Are all test identities excluded from the training set, and how was identity deduplication performed. Did participants receive reference audio during the user study, and can the statistical analysis account for dependencies between repeated judgments.
3 comments
Rebuttal for Reviewer sCCi
Q3: The contributions of data processing and model design are not separated.
A3: We study the effect of data filtering and audio captions in Tab.A. Specifically, we use a small data subset (50K) from our OmniCustom-1M to perform controlled experiments, where we train each ablation setting with 200K steps.
Table B: Ablation study on data filtering and audio captions using a small data subset (50K) from our OmniCustom-1M. All results are generated using the same inference strategies.
| Settings | Sim-Arc↑ | Sim-Cur↑ | FID↓ | FVD↓ | CLIP↑ | Speaker-Sim↑ | FAD↓ | WER↓ | LSE-C↑ | LSE-D↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| Unfiltered + No audio captions | 0.49 | 0.51 | 126 | 520 | 27.07 | 0.41 | 4.05 | 3.00 | 5.35 | 9.15 |
| Unfiltered + Audio captions | 0.50 | 0.52 | 124 | 514 | 27.10 | 0.44 | 3.86 | 2.85 | 5.40 | 9.10 |
| Filtered + No audio captions | 0.52 | 0.54 | 111 | 485 | 27.23 | 0.44 | 3.82 | 2.82 | 5.65 | 8.78 |
| Filtered + Audio captions | 0.54 | 0.56 | 108 | 477 | 27.25 | 0.47 | 3.64 | 2.65 | 5.70 | 8.72 |
It can be seen that without data filtering and audio captions, we obtain the worst results. Introducing audio captions yields improvements in Speaker-Sim, FAD, and WER. Applying data filtering brings gains on all metrics, especially for FID, FVD, FAD, CLIP, LSE-C and LSE-D. Combining both achieves the best performance.
We have added this ablation to our Sec 6.4.
Q4: trainable parameters, peak GPU memory, and computational cost of contrastive training.
A4:
| Backbone Parameters | 11.66B |
| Our Parameters | 12.63B |
| peak GPU memory | 72GB |
| Cost of contrastive training | +234.7 TFLOP (+23.6%) |
Q5: 70 Youtube Evaluation and User study.
A5: We make sure the identities of additional 70 Youtube videos are unseen by in our training dataset, using clips with ArcFace similarity to the training set below 0.1. As for user study, each user are given two reference conditions (identity and audio) and two outputs (ours and the competing method). We have added A5 in our Sec 6.1 and Appendix B.
Q6: Correlations associated with participants and videos in user study.
A6: Our original binomial analysis treated repeated judgments as independent and therefore did not account for participant- and video-specific correlations. We have corrected this analysis by reanalyzing all 18 method--criterion comparisons separately using intercept-only logistic regression with two-way cluster-robust standard errors, clustered simultaneously by participant and evaluated video. Each comparison contains 200 judgments from 20 participants, with each participant evaluating 10 randomly sampled videos. We used one-sided -based inference to test whether the preference probability for OmniCustom exceeded the chance level of .
Using an unadjusted per-comparison significance level of , 16 of the 18 comparisons were statistically significant. The two non-significant comparisons were both against MiniMax-H3: Timbre Consistency (48%, ) and Audio-Video Synchronization (54%, ). For conciseness, we report the two significant comparisons with the largest -values: Audio-Video Synchronization against DreamID-Omni (65%, ) and ID Consistency against MiniMax-H3 (67%, ). All other significant comparisons had smaller cluster-robust -values. Thus, accounting for the repeated-measures structure weakens the statistical evidence relative to the original independent binomial analysis but preserves its qualitative conclusions.
We have replaced the independence-based binomial analysis with this cluster-corrected analysis in our Appendix B.
Q7: Ethics
A7: (1) Data permissions. Our data are derived solely from SpeakerVid-5M, which is released for academic research; we strictly adhere to its license and terms of use and use it exclusively for non-commercial research. (2) Subject consent and opt-out. The identities in our data are inherited from SpeakerVid-5M, and individual-level consent is governed by that upstream source; we do not claim to have separately obtained per-subject consent. To provide subjects with agency, we will offer a public opt-out mechanism on our release page: any individual who identifies their likeness or voice may request removal, upon which we will delete the corresponding data from OmniCustom-1M and its derivatives and re-release a cleaned version. The dataset is used strictly for research and not for commercial purposes. (3) Safeguards against impersonation. We will revise our original unrestricted release plan. Access to the full model checkpoint will be gated, requiring agreement to a use policy that prohibits impersonation, fraud, harassment, non-consensual sexual content, political manipulation, and biometric surveillance.
We have revised the Ethics Statement section in our paper.
Second-round Rebuttal for Reviewer sCCi
Q1+: Contrastive losses against adjustments to reference guidance strength
A1+: In A1 of our first rebuttal, we aim to demonstrate that ours training-time contrastive regularizer is different from text-guided CFG. Note that we do not adopt reference guidance during inference as our default setting. In this second response, we explore the effect of the reference guidance scale. To this end, we extend the standard CFG formulation with an additional direction that explicitly controls the influence of the reference conditions . Specifically, we introduce a third prediction obtained by dropping the reference conditions while keeping the positive text: \begin{equation} v_m^{\phi} = v_{\theta}^{m}(Z_t, \phi, \phi, C^{+}, t), \end{equation} and compute the guided prediction as follows: \begin{equation} \hat{v}_m = v_m^{-} + s_m\left(v_m^{+} - v_m^{-}\right) + s_m^{r}\left(v_m^{+} - v_m^{\phi}\right), \quad m \in \{V, A\}. \end{equation} The two scales and independently control adherence to the text prompt and to the reference conditions, respectively. Note that our contrastive regularizer is orthogonal to such guidance on the reference condition. That is, the reference guidance scale can be applied on top of our OmniCustom. Nonetheless, we use as our default, which only requires two forward passes with only text-guided CFG, whereas using an extra guidance scale for reference would lead to an average 40% latency increase, from 4.02 minutes to 5.62 minutes. In contrast, our contrastive regularizer does not add latency but only adds training cost by 23.6%.
Table C: Ablation study using different guidance scales on reference-image () and reference-audio (). An excessively large scale leads to performance degradation.
| Model | Metric | 0 | 1 | 2 | 4 | 6 |
|---|---|---|---|---|---|---|
| Ours (+ Reference embeddings) | Sim-Arc↑ | 0.48 | 0.50 | 0.55 | 0.54 | 0.51 |
| Speaker-Sim ↑ | 0.38 | 0.45 | 0.42 | 0.41 | 0.39 | |
| Ours (+ Contrastive losses) | Sim-Arc↑ | 0.60 | 0.63 | 0.63 | 0.62 | no-face |
| Speaker-Sim ↑ | 0.51 | 0.52 | 0.54 | 0.53 | 0.51 |
Tab. C illustrates how "Ours (+ Reference embeddings)'' and "Ours (+ Contrastive losses)'' respond to different inference-time guidance scales applied to the reference image () and the reference audio (). As grows from 0 to 2, "Ours (+ Reference embeddings)'' yields better identity similarity in Sim-Arc. However, a stronger guidance scale above 2 leads to performance degradation. A similar pattern can be observed for "Ours (+ Contrastive losses)''. The corresponding visualization results are shown in A-Fig2 of Appendix, which also support our findings. In particular, causes unacceptable distortion and the customized identity vanishes. Furthermore, for , the audio performance peaks at under "Ours (+ Reference embeddings)'' and at under "Ours (+ Contrastive losses)''.
In summary, although an appropriate guidance scale can further improve the fidelity of identity and timbre, based on ``Ours (+ Reference embeddings)'', its best performance (0.55 of Sim-Arc and 0.45 of Speaker-Sim) still underperforms our contrastive regularizer with (0.60 of Sim-Arc and 0.51 of Speaker-Sim) while introducing an average 40% latency increase. We have added this ablation to our Appendix B.
We thank the reviewer again for valuable comments. The additional ablation studies indeed help us further improve the quality of this paper.
Rebuttal for Reviewer sCCi
We appreciate the insightful comments from the reviewer. We will address the reviewer’s concerns one by one. Should the reviewers still have any confusion or new questions regarding our responses, we welcome them and would be pleased to provide further clarifications.
Q1: contrastive losses against adjustments to reference guidance strength and separate ablations of the identity and timbre losses.
A1: We respectfully point out that we did not leverage CFG for reference guidance in our main paper. Our contrastive regularizer (Eq.8 and Eq.9) is a training-time mechanism along the reference-conditioning axis. It is different from CFG used only during sampling. Specifically, CFG scales in Sec. 6.1 are inherited from OVI and are applied only to text conditioning. During both CFG forward passes, the reference-image/audio latents and their identity/timbre embeddings are retained. The two predictions differ only in their text conditions: \begin{equation} v_m^{+}=v_{\theta}^{m}(Z_t, I^r, A^r, C^{+}, t), \qquad v_m^{-}=v_{\theta}^{m}(Z_t, I^r, A^r, C^{-}, t), \end{equation} where C− denotes negative text condition and m∈V,A. The modality-specific guided prediction is as follows: \begin{equation} \hat{v}_m=v_m^{-}+s_m\left(v_m^{+}-v_m^{-}\right), \end{equation} where sV=4.0 and sA=3.0 are the video and audio guidance scales.
In contrast, our contrastive regularizer is used only during training and compares vθ(Zt,Ir,C,t) and vθ(Zt,ϕ,C,t) while keeping the text condition fixed. It encourages the learned velocity field to respond distinctly to the reference condition. All ablations in Tab.1 use the same text-CFG scales and do not adopt reference guidance scale during inference. Adding the contrastive losses improves Sim-Arc from 0.48 to 0.60, Sim-Cur from 0.49 to 0.62, and Speaker-Sim from 0.38 to 0.51. Therefore, these improvements result from the training regularizer rather than a change in inference-time CFG. The ablation of reference guidance strength can be found in Second-round Rebuttal.
We have added this comparison to Sec. 7.
Besides, We argue that our contrastive losses separate predictions with and without reference conditions, which makes the resulting differences are pecific to identity and timbre. As described in Lines 311--314 and Fig. 3, the positive and negative predictions share the same timestep, and text condition, and differ only in reference injection. For the positive prediction, the generation tokens attend to the VAE-encoded reference latents, and the projected global face and timbre embeddings are added to the corresponding self-attention outputs. For the negative prediction, attention to the reference latents is masked, and the global reference embeddings are set to zero.
We also provide separate ablations of the identity and timbre losses in the Tab. A.
Table A: Ablation study on separate contrastive loss.
We perform an ablation study to demonstrate the effects of the contrastive identity loss and the contrastive timbre loss separately. The results are reported in Tab. A. We find that "Only contrastive identity loss" achieves identity similarity comparable to our standard OmniCustom and timbre similarity comparable to ours (+ Reference embeddings). ``Only contrastive timbre loss" achieves timbre performance identical to our standard OmniCustom and identity similarity comparable to ours (+ Reference embeddings). This ablation study indicates that the contrastive identity loss can independently enhance identity preservation, whereas the contrastive timbre loss independently improves timbre fidelity.
We have added this ablation to our Appendix B.
Q2: Isolate efficiency advantage to the proposed design.
A2: Generally speaking, the main efficiency advantage stems from backbone size. Our reference embeddings and contrastive losses aim to improve identity and timbre fidelity rather than reduce time cost. Nevertheless, our overall framework remains lightweight thanks to LoRA: it yields only 0.97B trainable parameters on top of the 11B OVI backbone. Under identical hardware and generation settings, OVI requires 3.97 minutes, whereas OmniCustom requires 4.02 minutes, corresponding to a small extra latency of approximately 1.3%. Besides, our contrastive losses add 234.7 TFLOPs during training, which corresponds to a 23.6% increase. Note that our contrastive losses do not add inference latency but only training cost. Our OmniCustom achieves the most favorable latency for the sync audio–video customization task. We have added this analysis to Sec. 6.3.