acceptodds
Desk rejected at ICLR 2027

OmniCustom: Sync Audio-Video Customization Using Contrastive Flow Matching

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Given a reference image and a reference audio , sync audio-video customization task aims to generate videos that maintain the identity of the reference image while imitating the timbre of the reference audio, with spoken content freely specifiable through user-provided textual prompts. While existing methods have achieved promising progress, a system that provides high-quality preservation of given references while yielding user-friendly inference latency remains a challenge. To deal with this, we propose OmniCustom, a powerful DiT-based sync audio-video customization framework that can synthesize a video following reference image identity, audio timbre, and text prompts all at once in a zero-shot manner. Our framework is built on three designs. First, identity and audio timbre control are achieved by modality-specific LoRA modules that operate through self-attention layers within the joint audio-video generation backbone. Second, we introduce a contrastive learning objective alongside the standard flow matching objective. It uses predicted flows conditioned on reference inputs as positive samples and those without reference conditions as negative samples, enhancing the model’s ability to preserve identity and timbre. Third, we train OmniCustom on our constructed large-scale audio-visual human dataset OmniCustom-1M. Extensive experiments demonstrate that OmniCustom achieves state-of-the-art customization performance while providing user-friendly inference runtime.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sync Audio-Video Customization, Contrastive Learning, Flow matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sCCiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies synchronous audio and video customization. Built on OVI, OmniCustom uses modality specific LoRA modules to incorporate reference images and audio, together with facial features, timbre features, and contrastive flow matching losses. The framework jointly generates video and speech with specified identity, voice timbre, and spoken content.

Critical strengths

The method has practical value. Users can jointly control generation through images, audio, and text without additional fine tuning for each new identity.

The ablations demonstrate clear improvements. Adding the contrastive losses improves both facial and speaker similarity, while inference latency remains close to that of the original OVI model, suggesting limited additional inference overhead.ons.

Critical weaknesses

Critical Weaknesses

The independent contribution of the contrastive losses is insufficiently established. These losses separate predictions with and without reference conditions, but the resulting differences are not necessarily specific to identity and timbre. The paper lacks comparisons against stronger reference guidance and separate ablations of the identity and timbre losses. It therefore remains unclear whether the gains arise from the particular contrastive formulation or from generally strengthening reference conditioning.

The sources of the performance and efficiency advantages are not sufficiently disentangled. The method is trained on one million task specific examples, but the contributions of data processing and model design are not clearly separated. Inference takes approximately 4.02 minutes, close to the 4.16 minutes reported for DreamID Omni, while the substantial speed advantage over MiniMax H3 coincides with a difference in backbone size. The results support low additional inference overhead, but do not isolate an efficiency advantage attributable to the proposed design.

Important evaluation details remain unclear. For the additional 70 test videos, the paper establishes that the videos are absent from the training set, but does not explicitly establish that their identities are also unseen. This leaves the generalization claim insufficiently verified. The user study also does not explicitly state whether participants received the reference audio, making the interpretation of timbre consistency judgments uncertain.

Minor Weaknesses

The user study treats repeated judgments from the same participant as independent observations. Its significance analysis does not account for correlations associated with participants and videos.

The efficiency analysis omits the number of trainable parameters, peak GPU memory usage, and the additional computational cost of contrastive training.

Questions for authors

Questions For Authors

Could the authors compare the contrastive losses against adjustments to reference guidance strength and separately report the effects of the identity and timbre losses.

Could controlled experiments distinguish the contributions of data filtering, audio captioning, and model design, together with a comparable analysis of computational costs.

Are all test identities excluded from the training set, and how was identity deduplication performed. Did participants receive reference audio during the user study, and can the statistical analysis account for dependencies between repeated judgments.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer sCCi

We appreciate the insightful comments from the reviewer. We will address the reviewer’s concerns one by one. Should the reviewers still have any confusion or new questions regarding our responses, we welcome them and would be pleased to provide further clarifications.

Q1: contrastive losses against adjustments to reference guidance strength and separate ablations of the identity and timbre losses.

A1: We respectfully point out that we did not leverage CFG for reference guidance in our main paper. Our contrastive regularizer (Eq.8 and Eq.9) is a training-time mechanism along the reference-conditioning axis. It is different from CFG used only during sampling. Specifically, CFG scales in Sec. 6.1 are inherited from OVI and are applied only to text conditioning. During both CFG forward passes, the reference-image/audio latents and their identity/timbre embeddings are retained. The two predictions differ only in their text conditions: \begin{equation} v_m^{+}=v_{\theta}^{m}(Z_t, I^r, A^r, C^{+}, t), \qquad v_m^{-}=v_{\theta}^{m}(Z_t, I^r, A^r, C^{-}, t), \end{equation} where C−C^{-} denotes negative text condition and m∈V,Am\in \\{V,A \\}. The modality-specific guided prediction is as follows: \begin{equation} \hat{v}_m=v_m^{-}+s_m\left(v_m^{+}-v_m^{-}\right), \end{equation} where sV=4.0s_V=4.0 and sA=3.0s_A=3.0 are the video and audio guidance scales.

In contrast, our contrastive regularizer is used only during training and compares vθ(Zt,Ir,C,t)v_{\theta}(Z_t,I^r,C,t) and vθ(Zt,ϕ,C,t)v_{\theta}(Z_t,\phi,C,t) while keeping the text condition fixed. It encourages the learned velocity field to respond distinctly to the reference condition. All ablations in Tab.1 use the same text-CFG scales and do not adopt reference guidance scale during inference. Adding the contrastive losses improves Sim-Arc from 0.480.48 to 0.600.60, Sim-Cur from 0.490.49 to 0.620.62, and Speaker-Sim from 0.380.38 to 0.510.51. Therefore, these improvements result from the training regularizer rather than a change in inference-time CFG. The ablation of reference guidance strength can be found in Second-round Rebuttal.

We have added this comparison to Sec. 7.

Besides, We argue that our contrastive losses separate predictions with and without reference conditions, which makes the resulting differences are pecific to identity and timbre. As described in Lines 311--314 and Fig. 3, the positive and negative predictions share the same timestep, and text condition, and differ only in reference injection. For the positive prediction, the generation tokens attend to the VAE-encoded reference latents, and the projected global face and timbre embeddings are added to the corresponding self-attention outputs. For the negative prediction, attention to the reference latents is masked, and the global reference embeddings are set to zero.

We also provide separate ablations of the identity and timbre losses in the Tab. A.

Table A: Ablation study on separate contrastive loss.

ModelSim-Arc↑Speaker-Sim↑
No contrastive losses (Ours (+ Reference embeddings))0.480.38
Only contrastive identity loss0.610.37
Only contrastive timbre loss0.460.51
Contrastive identity and timbre losses (Ours (+ Contrastive losses))0.600.51

We perform an ablation study to demonstrate the effects of the contrastive identity loss and the contrastive timbre loss separately. The results are reported in Tab. A. We find that "Only contrastive identity loss" achieves identity similarity comparable to our standard OmniCustom and timbre similarity comparable to ours (+ Reference embeddings). ``Only contrastive timbre loss" achieves timbre performance identical to our standard OmniCustom and identity similarity comparable to ours (+ Reference embeddings). This ablation study indicates that the contrastive identity loss can independently enhance identity preservation, whereas the contrastive timbre loss independently improves timbre fidelity.

We have added this ablation to our Appendix B.

Q2: Isolate efficiency advantage to the proposed design.

A2: Generally speaking, the main efficiency advantage stems from backbone size. Our reference embeddings and contrastive losses aim to improve identity and timbre fidelity rather than reduce time cost. Nevertheless, our overall framework remains lightweight thanks to LoRA: it yields only 0.97B trainable parameters on top of the 11B OVI backbone. Under identical hardware and generation settings, OVI requires 3.97 minutes, whereas OmniCustom requires 4.02 minutes, corresponding to a small extra latency of approximately 1.3%. Besides, our contrastive losses add 234.7 TFLOPs during training, which corresponds to a 23.6% increase. Note that our contrastive losses do not add inference latency but only training cost. Our OmniCustom achieves the most favorable latency for the sync audio–video customization task. We have added this analysis to Sec. 6.3.

AuthorsOpenReview ↗

Rebuttal for Reviewer sCCi

Q3: The contributions of data processing and model design are not separated.

A3: We study the effect of data filtering and audio captions in Tab.A. Specifically, we use a small data subset (50K) from our OmniCustom-1M to perform controlled experiments, where we train each ablation setting with 200K steps.

Table B: Ablation study on data filtering and audio captions using a small data subset (50K) from our OmniCustom-1M. All results are generated using the same inference strategies.

SettingsSim-Arc↑Sim-Cur↑FID↓FVD↓CLIP↑Speaker-Sim↑FAD↓WER↓LSE-C↑LSE-D↓
Unfiltered + No audio captions0.490.5112652027.070.414.053.005.359.15
Unfiltered + Audio captions0.500.5212451427.100.443.862.855.409.10
Filtered + No audio captions0.520.5411148527.230.443.822.825.658.78
Filtered + Audio captions0.540.5610847727.250.473.642.655.708.72

It can be seen that without data filtering and audio captions, we obtain the worst results. Introducing audio captions yields improvements in Speaker-Sim, FAD, and WER. Applying data filtering brings gains on all metrics, especially for FID, FVD, FAD, CLIP, LSE-C and LSE-D. Combining both achieves the best performance.

We have added this ablation to our Sec 6.4.

Q4: trainable parameters, peak GPU memory, and computational cost of contrastive training.

A4:

Backbone Parameters11.66B
Our Parameters12.63B
peak GPU memory72GB
Cost of contrastive training+234.7 TFLOP (+23.6%)

Q5: 70 Youtube Evaluation and User study.

A5: We make sure the identities of additional 70 Youtube videos are unseen by in our training dataset, using clips with ArcFace similarity to the training set below 0.1. As for user study, each user are given two reference conditions (identity and audio) and two outputs (ours and the competing method). We have added A5 in our Sec 6.1 and Appendix B.

Q6: Correlations associated with participants and videos in user study.

A6: Our original binomial analysis treated repeated judgments as independent and therefore did not account for participant- and video-specific correlations. We have corrected this analysis by reanalyzing all 18 method--criterion comparisons separately using intercept-only logistic regression with two-way cluster-robust standard errors, clustered simultaneously by participant and evaluated video. Each comparison contains 200 judgments from 20 participants, with each participant evaluating 10 randomly sampled videos. We used one-sided t19t_{19}-based inference to test whether the preference probability for OmniCustom exceeded the chance level of 0.50.5.

Using an unadjusted per-comparison significance level of α=0.05\alpha=0.05, 16 of the 18 comparisons were statistically significant. The two non-significant comparisons were both against MiniMax-H3: Timbre Consistency (48%, p=0.621p=0.621) and Audio-Video Synchronization (54%, p=0.271p=0.271). For conciseness, we report the two significant comparisons with the largest pp-values: Audio-Video Synchronization against DreamID-Omni (65%, p=0.0277p=0.0277) and ID Consistency against MiniMax-H3 (67%, p=0.0148p=0.0148). All other significant comparisons had smaller cluster-robust pp-values. Thus, accounting for the repeated-measures structure weakens the statistical evidence relative to the original independent binomial analysis but preserves its qualitative conclusions.

We have replaced the independence-based binomial analysis with this cluster-corrected analysis in our Appendix B.

Q7: Ethics

A7: (1) Data permissions. Our data are derived solely from SpeakerVid-5M, which is released for academic research; we strictly adhere to its license and terms of use and use it exclusively for non-commercial research. (2) Subject consent and opt-out. The identities in our data are inherited from SpeakerVid-5M, and individual-level consent is governed by that upstream source; we do not claim to have separately obtained per-subject consent. To provide subjects with agency, we will offer a public opt-out mechanism on our release page: any individual who identifies their likeness or voice may request removal, upon which we will delete the corresponding data from OmniCustom-1M and its derivatives and re-release a cleaned version. The dataset is used strictly for research and not for commercial purposes. (3) Safeguards against impersonation. We will revise our original unrestricted release plan. Access to the full model checkpoint will be gated, requiring agreement to a use policy that prohibits impersonation, fraud, harassment, non-consensual sexual content, political manipulation, and biometric surveillance.

We have revised the Ethics Statement section in our paper.

AuthorsOpenReview ↗

Second-round Rebuttal for Reviewer sCCi

Q1+: Contrastive losses against adjustments to reference guidance strength

A1+: In A1 of our first rebuttal, we aim to demonstrate that ours training-time contrastive regularizer is different from text-guided CFG. Note that we do not adopt reference guidance during inference as our default setting. In this second response, we explore the effect of the reference guidance scale. To this end, we extend the standard CFG formulation with an additional direction that explicitly controls the influence of the reference conditions (Ir,Ar)(I^r, A^r). Specifically, we introduce a third prediction obtained by dropping the reference conditions while keeping the positive text: \begin{equation} v_m^{\phi} = v_{\theta}^{m}(Z_t, \phi, \phi, C^{+}, t), \end{equation} and compute the guided prediction as follows: \begin{equation} \hat{v}_m = v_m^{-} + s_m\left(v_m^{+} - v_m^{-}\right) + s_m^{r}\left(v_m^{+} - v_m^{\phi}\right), \quad m \in \{V, A\}. \end{equation} The two scales sms_m and smrs_m^{r} independently control adherence to the text prompt and to the reference conditions, respectively. Note that our contrastive regularizer is orthogonal to such guidance on the reference condition. That is, the reference guidance scale can be applied on top of our OmniCustom. Nonetheless, we use smr=0s_m^r=0 as our default, which only requires two forward passes with only text-guided CFG, whereas using an extra guidance scale for reference would lead to an average 40% latency increase, from 4.02 minutes to 5.62 minutes. In contrast, our contrastive regularizer does not add latency but only adds training cost by 23.6%.

Table C: Ablation study using different guidance scales on reference-image (svrs_v^{r}) and reference-audio (sars_a^{r}). An excessively large scale leads to performance degradation.

ModelMetric01246
Ours (+ Reference embeddings)Sim-Arc↑0.480.500.550.540.51
Speaker-Sim ↑0.380.450.420.410.39
Ours (+ Contrastive losses)Sim-Arc↑0.600.630.630.62no-face
Speaker-Sim ↑0.510.520.540.530.51

Tab. C illustrates how "Ours (+ Reference embeddings)'' and "Ours (+ Contrastive losses)'' respond to different inference-time guidance scales applied to the reference image (svrs_v^{r}) and the reference audio (sars_a^{r}). As svrs_v^{r} grows from 0 to 2, "Ours (+ Reference embeddings)'' yields better identity similarity in Sim-Arc. However, a stronger guidance scale above 2 leads to performance degradation. A similar pattern can be observed for "Ours (+ Contrastive losses)''. The corresponding visualization results are shown in A-Fig2 of Appendix, which also support our findings. In particular, svr=6s_v^{r}=6 causes unacceptable distortion and the customized identity vanishes. Furthermore, for sars_a^{r}, the audio performance peaks at sar=1s_a^r = 1 under "Ours (+ Reference embeddings)'' and at sar=2s_a^r = 2 under "Ours (+ Contrastive losses)''.

In summary, although an appropriate guidance scale can further improve the fidelity of identity and timbre, based on ``Ours (+ Reference embeddings)'', its best performance (0.55 of Sim-Arc and 0.45 of Speaker-Sim) still underperforms our contrastive regularizer with smr=0s_m^r=0 (0.60 of Sim-Arc and 0.51 of Speaker-Sim) while introducing an average 40% latency increase. We have added this ablation to our Appendix B.

We thank the reviewer again for valuable comments. The additional ablation studies indeed help us further improve the quality of this paper.

Reviewer_QEDROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present OmniCustom, a parameter-efficient DiT framework for zero-shot synchronous customization of audio and video: from a reference portrait, a reference audio sample, and a text prompt, it produces in one pass a video that preserves the reference identity while rendering an audio track that mimics the reference timbre. Constructed on the OVI joint audio-video backbone, it attaches reference-image and reference-audio branches through two modality-specific LoRA adapters applied to the QKV projections of self-attention, enriches the flow-matching loss with a contrastive term, and trains on OmniCustom-1M, a filtered collection of roughly one million human audio-visual clips. Evaluated on a 100-identity benchmark, it achieves the strongest identity preservation, FID/FVD, and FAD while running with substantially lower latency than MiniMax-H3.

Critical strengths
  1. The problem is convincingly motivated and practically relevant, with a clear account of the shortcomings being addressed — the unwieldy TTS-plus-audio-driven cascade, DreamID-Omni's limited fidelity to the reference, and the prohibitive inference cost of MiniMax-H3.
  2. The architecture is indeed economical: introducing reference branches through modality-specific LoRA on the existing QKV projections leaves the OVI structure intact with few additional parameters, and the measured overhead is negligible, which supports the claim of a user-friendly runtime.
  3. The contrastive flow-matching objective is an elegant reformulation that builds positive and negative pairs directly in flow space instead of measuring feature-space distances, and the ablation demonstrates its effectiveness.
Critical weaknesses
  1. The technical contribution is incremental. Each ingredient — the OVI backbone, the LoRA-based parameter efficiency, and the contrastive regularizer adapted from Contrastive Flow Matching — already exists, and the reference-branch concatenation parallels prior customization methods such as Phantom and VACE; what is offered is chiefly a well-engineered combination together with a filtered dataset.
  2. A number of comparisons are structurally imbalanced, which undermines the state-of-the-art framing. Audio-driven baselines are routed through a CosyVoice TTS front-end that is inherently unable to clone the reference timbre or produce background sound, handicapping them on Speaker-Sim and synchronization by design; moreover, OmniCustom's own Speaker-Sim (0.51) remains below MiniMax-H3 (0.55), so the SOTA claim applies only to a subset of metrics.
  3. The evaluation is limited in scale and rigor: the benchmark comprises only 100 self-constructed identities, results averaged over three seeds are reported without variance or significance analysis, the user study and most qualitative evidence are relegated to the appendix, and the resolution mismatch between training and inference (480p vs. 720p) is never examined.
  4. Robustness and generalization remain unexplored. All training data consist of single-speaker talking-head videos, with no treatment of failure cases, non-speech or multi-speaker audio, extreme pose or identity difficulty, or mismatches between timbre and text; the dataset's claimed identity and timbre diversity is never quantified.
  5. Improvements attributable to the method are not separated from those attributable to the data. Because OmniCustom is fine-tuned on the large, task-matched OmniCustom-1M corpus while baselines rely on off-the-shelf weights, Table 1 cannot disentangle gains from the design itself (reference LoRA plus contrastive flow matching) from gains due to in-domain training, risking an overstated methodological contribution; a matched-data, method-removed control (e.g., vanilla OVI fine-tuned on OmniCustom-1M) is required.
  6. The environmental-audio capability is asserted rather than measured. Generating contextual background sound (e.g., ocean waves) is described as a property inherited from OVI and is evidenced only through textual statements and qualitative or appendix demos, whereas the quantitative metrics in Table 1 (FAD, WER, LSE) assess speech and lip-sync rather than ambient audio. Consequently, the ability to produce ocean-wave sound remains a claim accompanied by a demo rather than a rigorously evaluated finding.
Questions for authors

None

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer QEDR

We appreciate the insightful comments from the reviewer. We will address the reviewer’s concerns one by one. Should the reviewers still have any confusion or new questions regarding our responses, we welcome them and would be pleased to provide further clarifications.

Q1: Incremental contribution.

A1: We respectfully disagree that the contribution is merely engineering.

(1) The reference design is not Phantom/VACE reuse: These two methods are single-modality silent-video models with one stream. On OVI's twin-backbone joint model, we must route modality-specific references. Besides, OmniCustom does not rely solely on reference-feature concatenation. It additionally processes identity and timbre embeddings through several trainable linear layers and adds the projected features to the self-attention outputs of the corresponding video and audio branches.

(2) Contrastive Flow Matching (CFM) is a general training objective for conditional generative modeling rather than a method designed for customization. CFM contrasts flows across different conditioning samples to enforce uniqueness. It addresses the overlap between velocity fields associated with different samples or conditions, which can lead to ambiguous conditional generation. In contrast, for the same sample, we use the reference-conditioned flow against the reference-absent flow. That is, we fix the noisy sample, timestep, and text condition and comparing velocity predictions with and without reference injection. Besides, unlike prior customization contrast (e.g., PuLID), we do not need feature-distance computation.

(3) OmniCustom-1M is reorganized into task-specific reference-target pairs: the first four seconds provide the reference audio, while the final five seconds provide the training audio/video clips. The dataset additionally includes synchronization and quality filtering, ASR transcripts, structured audio captions.

We also study the effect of data filtering and audio captions in Tab. A. Specifically, we use a small data subset (50K) from our OmniCustom-1M to perform controlled experiments, where we train each ablation setting with 200K steps.

Table A: Ablation study on data filtering and audio captions using a small data subset (50K) from our OmniCustom-1M. All results are generated using the same inference strategies.

SettingsSim-Arc↑Sim-Cur↑FID↓FVD↓CLIP↑Speaker-Sim↑FAD↓WER↓LSE-C↑LSE-D↓
Unfiltered + No audio captions0.490.5112652027.070.414.053.005.359.15
Unfiltered + Audio captions0.500.5212451427.100.443.862.855.409.10
Filtered + No audio captions0.520.5411148527.230.443.822.825.658.78
Filtered + Audio captions0.540.5610847727.250.473.642.655.708.72

It can be seen that without data filtering and audio captions, we obtain the worst results. Introducing audio captions yields improvements in Speaker-Sim, FAD, and WER. Applying data filtering brings gains on all metrics, especially for FID, FVD, FAD, CLIP, LSE-C and LSE-D. Combining both achieves the best performance.

We have added this ablation to our Sec 6.4.

Q2: Imbalanced comparison.

A2: We respectfully point out that the comparison is about two paradigms, which is structurally balanced. Audio-driven baseline can conduct sync audio-video customization by a serial pipeline, where TTS front-end can mimic reference timbre but cannot generate background sound. This is the weakness of this paradigm.

Although we underperform MiniMax-H3 on Speaker-Sim (0.51 vs. 0.55), MiniMax-H3 is a 33B omni-generation model trained on large-scale undisclosed data. Our sync customization framework, including LoRA, reference embeddings, and contrastive losses is backbone-agnostic and could, in principle, be applied to MiniMax-H3. However, its released checkpoint is a CFG-distilled Omni Transformer. Naïvely fine-tuning it may pull the model back toward a standard conditional predictor, causing its distilled CFG behavior to drift or disappear.

AuthorsOpenReview ↗

Rebuttal for Reviewer QEDR

Q3: Variance or significance analysis.

A3: Table B: Revised Table 1 using mean ±\pm standard deviation.

ModelSim-Arc↑Sim-Cur↑FID↓FVD↓CLIP↑Speaker-Sim↑FAD↓WER(%)↓LSE-C↑LSE-D↓Latency↓
Phantom0.56±0.18‾\underline{0.56 \pm 0.18}0.57±0.18‾\underline{0.57 \pm 0.18}169.69±41.04169.69\pm41.04478.36±86.64478.36\pm 86.6426.72±3.7126.72 \pm 3.71–––––4.24±0.164.24\pm0.16
VACE0.22±0.110.22 \pm 0.110.23±0.120.23 \pm 0.12183.85±42.91183.85 \pm 42.91575.09±77.21575.09 \pm 77.2128.03±3.56‾\underline{28.03 \pm 3.56}–––––3.81±0.15\boldsymbol{3.81 \pm 0.15}
HunyuanCustom0.53±0.160.53 \pm 0.160.55±0.150.55 \pm 0.15130.29±48.67130.29 \pm 48.67517.12±66.03517.12 \pm 66.0323.98±4.3723.98 \pm 4.37–––4.87±1.454.87 \pm 1.4510.47±2.1910.47 \pm 2.198.72±0.178.72 \pm 0.17
Humo0.50±0.170.50 \pm 0.170.52±0.180.52 \pm 0.18181.01±34.56181.01 \pm 34.56475.94±79.03‾\underline{475.94 \pm 79.03}27.23±4.0127.23 \pm 4.01–––5.42±1.735.42 \pm 1.739.31±2.169.31 \pm 2.167.19±0.117.19 \pm 0.11
DreamID-Omni0.48±0.180.48 \pm 0.180.50±0.180.50 \pm 0.18183.42±51.72183.42 \pm 51.72509.16±92.91509.16 \pm 92.9127.56±3.9227.56 \pm 3.920.39±0.130.39 \pm 0.135.64±1.065.64 \pm 1.063.16±1.413.16 \pm 1.414.51±1.934.51 \pm 1.939.75±2.549.75 \pm 2.544.16±0.144.16 \pm 0.14
MiniMax-H30.56±0.190.56 \pm 0.190.57±0.180.57 \pm 0.18181.08±42.15181.08 \pm 42.15504.28±95.36504.28 \pm 95.3627.09±4.1227.09 \pm 4.120.55±0.28\boldsymbol{0.55 \pm 0.28}3.98±0.923.98 \pm 0.922.23±1.90\boldsymbol{2.23 \pm 1.90}5.82±1.75\boldsymbol{5.82 \pm 1.75}8.31±1.37‾\underline{8.31 \pm 1.37}15.34±0.1415.34 \pm 0.14
Ours (Baseline)0.39±0.170.39 \pm 0.170.41±0.170.41 \pm 0.17137.18±27.66137.18 \pm 27.66559.96±86.20559.96 \pm 86.2028.31±3.98\boldsymbol{28.31 \pm 3.98}0.30±0.190.30 \pm 0.194.32±0.874.32 \pm 0.872.40±1.82‾\underline{2.40 \pm 1.82}5.72±1.845.72 \pm 1.848.45±2.098.45 \pm 2.094.05±0.164.05 \pm 0.16
Ours (+ Reference embeddings)0.48±0.180.48 \pm 0.180.49±0.180.49 \pm 0.18105.62±31.25‾\underline{105.62 \pm 31.25}506.33±78.60506.33 \pm 78.6027.68±3.0427.68 \pm 3.040.38±0.210.38 \pm 0.213.67±0.82‾\underline{3.67 \pm 0.82}2.78±1.772.78 \pm 1.775.68±1.675.68 \pm 1.678.27±1.82\boldsymbol{8.27 \pm 1.82}4.09±0.154.09 \pm 0.15
Ours0.60±0.15\boldsymbol{0.60 \pm 0.15}0.62±0.15\boldsymbol{0.62 \pm 0.15}95.57±24.63\boldsymbol{95.57 \pm 24.63}440.49±63.52\boldsymbol{440.49 \pm 63.52}27.45±3.6627.45 \pm 3.660.51±0.25‾\underline{0.51 \pm 0.25}3.44±0.81\boldsymbol{3.44 \pm 0.81}2.51±1.852.51 \pm 1.855.76±1.35‾\underline{5.76 \pm 1.35}8.53±1.798.53 \pm 1.794.02±0.13‾\underline{4.02 \pm 0.13}

Table C: Pairwise statistical significance analysis between our method and existing methods.

MetricMethodOurs / MethodΔ95% CIp
Sim-Arc↑Phantom0.60 / 0.56+0.04[+0.01, +0.07]0.003**
VACE0.60 / 0.22+0.38[+0.36, +0.40]<0.001***
HunyuanCustom0.60 / 0.53+0.07[+0.05, +0.09]<0.001***
Humo0.60 / 0.50+0.10[+0.08, +0.12]<0.001***
DreamID-Omni0.60 / 0.48+0.12[+0.10, +0.14]<0.001***
MiniMax-H30.60 / 0.56+0.04[+0.01, +0.07]0.004**
CLIP↑Phantom27.45 / 26.72+0.73[+0.14, +1.32]0.015*
VACE27.45 / 28.03−0.58[−1.17, +0.01]0.054 (n.s.)
HunyuanCustom27.45 / 23.98+3.47[+2.86, +4.08]<0.001***
Humo27.45 / 27.23+0.22[−0.45, +0.89]0.519 (n.s.)
DreamID-Omni27.45 / 27.56−0.11[−0.75, +0.53]0.736 (n.s.)
MiniMax-H327.45 / 27.09+0.36[−0.26, +0.98]0.258 (n.s.)
Speaker-Sim↑DreamID-Omni0.51 / 0.39+0.12[+0.09, +0.15]<0.001***
MiniMax-H30.51 / 0.55−0.04[−0.08, +0.00]0.066 (n.s.)
WER(%)↓DreamID-Omni2.51 / 3.16−0.65[−0.79, −0.51]<0.001***
MiniMax-H32.51 / 2.23+0.28[−0.02, +0.58]0.068 (n.s.)
LSE-C↑DreamID-Omni5.76 / 4.51+1.25[+1.03, +1.47]<0.001***
MiniMax-H35.76 / 5.82−0.06[−0.28, +0.16]0.589 (n.s.)
LSE-D↓DreamID-Omni8.53 / 9.75−1.22[−1.48, −0.96]<0.001***
MiniMax-H38.53 / 8.31+0.22[−0.04, +0.48]0.091 (n.s.)

Here, *p<0.05, **p<0.01, and ***p<0.001 show the level of statistical significance. n.s. = not significant.

We show significant superiority in visual fidelity (Sim-Arc, p<0.01) over both Phantom and MiniMax-H3. While MiniMax-H3 exhibits marginal numerical advantages in certain audio and synchronization metrics (Speaker-Sim, WER, LSE-C, LSE-D), these differences do not reach statistical significance. Due to the long-tail distribution and high variance of audio error metrics (e.g., WER) across diverse complex scenarios, the marginal advantages of MiniMax-H3 are statistically negligible. This confirms that our model achieves on-par performance in audio fidelity and audio-visual synchronization, while maintaining a distinct advantage in visual subject consistency and inference latency. Note that FID, FVD, and FAD are distribution-level metrics, where significance tests are not applicable.

We have added Tab. C to our Appendix B.

Besides, the training data of OVI 1.0-11B is 720720. Our OmniCustom-1M use the training data of 480854. Our experimental results demonstrate that our method can generalize to 720P.

AuthorsOpenReview ↗

Rebuttal for Reviewer QEDR

Q4: Robustness and generalization.

A4: We respectfully point out we have provided the limitation of our method in A-Fig6, where our OmniCustom struggles to preserve the identity of profile faces (extreme pose).

Recall that in Appendix C, we give additional applications. (1) Music imitation using non-speech audio. (2) Cross-gender and Cross-age Customization. (3) Non-human Objects.

Besides, as shown in Tab. D, mismatch between timbre and text leads to a lower \textit{Speaker-Sim} score. However, such mismatch does not hurt identity preservation. We attribute this degradation to the model implicitly encoding certain priors regarding text descriptions and timbre ranges. We have added Tab. D in our Appendix C.

Table D: Text-timbre matching analysis.

Text-timbre match0.51
Text-timbre mismatche0.40

Then, identity and timbre diversity are as follows:

Table E: Diversity of our dataset.

IDs/timbre19653
Hours2500
clips1M

We have added descriptions of diversity to our Sec 5.

Q5: Method and data.

A5: We respectfully point out that our baseline does not rely on off-the-shelf weights. As seen in Lines 199–201, Lines 251–253, and Fig. 2(a), the reference image and audio are converted into reference latents and then concatenated with the original video and audio latents. Here, we enforce reference images and trained video clips to share identities to learn ID customization. Similarly, we enforce reference audio and training audio to share timbres to learn timbre customization.

That is, our baseline is trained with “modality-aware LoRAs with reference features”; the same training data and configuration are used for “Ours (+ Reference embeddings)” and “Ours (+ Contrastive losses)”. Thus, the gains from each design can be properly established. As for method-removed control, the OVI backbone itself only performs joint audio–video generation; vanilla OVI fine-tuning cannot endow it with customization capability.

We welcome further elaboration if the reviewers still have questions regarding this issue.

Q6: Environmental Audio.

A6: Finetuned upon the OVI backbone, our method inherits its audio generation capability. We utilize CLAP [a] to quantify the semantic alignment between generated audio and input text prompts over 300 test videos.

Table F: Comparison between our and OVI on Audio-text alignment.

SettingsCLAP
OVI Backbone0.41
Ours0.44

Our method achieves audio-prompt alignment comparable to OVI, verifying its capability to generate background sounds.

[a] Yusong Wu, Ke Chen, Tianyu Zhang, et al. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023.

We have added A6 to our Sec 6.3.

AuthorsOpenReview ↗

Concern Check

Dear Reviewer QEDR,

As the ICLR rebuttal period is drawing to a close, we wonder whether our responses have addressed your concerns. If any concerns remain unresolved, please kindly let us know, and we are happy to provide further clarifications. We are glad to see that Reviewer Jo1Q has raised score during the rebuttal. We look forward to your reply.

In A2 of our initial rebuttal for you, we pointed out that naïvely fine-tuning MiniMax-H3 with our data and method may pull it back toward a standard conditional predictor, causing its distilled CFG behavior to drift or disappear. Here, we want to provide more explanations.

As observed in [a], MiniMax-H3 itself is a CFG-distilled model, and training it quickly breaks the distillation and degrades quality. As a workaround, they first un-distill the model and then perform fine-tuning on it. We leave exploring this feasibility for future work. We have updated this discussion in Sec. 7.

[a] https://huggingface.co/circlestone-labs/MiniMax-H3-Image-Training-Adapter

Reviewer_Jo1QOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper addresses sync audio-video customization. Given one reference face image, a short reference voice clip and a text prompt containing the spoken line, the model generates a talking-person video and its audio in a single pass. The video should keep the reference identity. The speech should imitate the reference timbre and say the prompted words, with plausible background sound.

Reference branches. The reference image goes through the video VAE and the reference audio goes through an audio VAE. The resulting latents are concatenated to the video and audio latents at timestep 0 and attended to in each branch’s self-attention layers. Modality-specific LoRAs are applied to the QKV projections of the reference tokens. The cross-modal attention stays frozen to preserve the audio-video alignment. Global embeddings. A face embedding (InsightFace) and a timbre embedding (NaturalSpeech3) pass through trainable linear layers and are added to the self-attention output. Contrastive flow-matching regularizer. The standard flow-matching losses for video and audio are kept. A term is added that maximizes the L2 distance between the velocity predicted with the reference and a stop-gradient velocity predicted with the reference masked and its embedding zeroed. The authors present this as a feature-free variant of Contrastive Flow Matching, with the reference-conditioned flow as the positive and the reference-free flow as the negative. Data. The authors build OmniCustom-1M, about 1M single-speaker clips filtered from SpeakerVid-5M by SyncNet and aesthetic scores, with Qwen3-Omni audio captions. The reference image and audio come from the same video as the training clip.

Critical strengths
  1. A simple, effective way to preserve identity and timbre. The paper adds a global InsightFace face embedding (and a NaturalSpeech3 timbre embedding) to the self-attention outputs, on top of reference-token concatenation. The embedding goes through a few linear layers and needs no extra module or heavy finetuning. This gives a compact identity signal that complements the spatial reference tokens.

  2. A lightweight contrastive flow-matching regularizer that gives large further gains. The paper treats the reference-conditioned velocity as the positive and the no-reference velocity (reference tokens masked, embeddings zeroed) as the negative. It then pushes the two apart directly in flow space. It needs no feature extractor, no extra negative samples, and no extra network.

Critical weaknesses
  1. Metric circularity. The identity conditioning is an InsightFace embedding, and the headline identity metric (Sim-Arc) uses ArcFace. Sim-Cur uses CurricularFace, a closely related face-recognition model that has been trained on the same data. they are almost the same systems, so it gives little independent evidence.

  2. Mixed results. Against MiniMax-H3, the model is worse on Speaker-Sim (0.51 vs 0.55), WER (2.51 vs 2.23), LSE-D (8.53 vs 8.31) and LSE-C (5.76 vs 5.82). CLIP is lower than VACE and than the paper’s own baseline (27.45 vs 28.31). The ablation also shows the final model is worse than the intermediate variant on LSE-D and than the baseline on WER, which “little variation” glosses over. No error bars. Table 1 reports single numbers with no standard deviations or confidence intervals over seeds or identities, and the benchmark is only 100 identities. I therefore cannot tell which of these differences (including the 0.04–0.05 identity margins and the small LSE/WER gaps) are significant. Qualitative results do not clearly favor the method. In the supplementary comparisons. MiniMax-H3 looked better to me than OmniCustom.

  3. Novelty. Reference-token concatenation with a zero timestep and LoRA, plus face-embedding injection, are established in image and video customization. The remaining novelty is the contrastive term.

Questions for authors
  1. Relation between the contrastive regularizer and classifier-free guidance. The contrastive term (Eq. 9) pushes the reference-conditioned velocity v_θ(Z, I^r, C, t) away from the no-reference velocity v_θ(Z, φ, C, t). This closely resembles what CFG does at sampling time, which extrapolates from an unconditional prediction toward a conditional one. Sec. 6.1 reports CFG scales of 3.0 (audio) and 4.0 (video), but does not say what the guidance is applied to. What is happening here?

  2. How were the 70 YouTube identities checked for overlap with SpeakerVid-5M speakers or videos?

  3. LoRA is emphasized throughout the paper, but the design (adapters on the reference-token QKV projections and the output projection) appears to be a standard adaptation of the reference tokens. What is specific to your design that other reference-conditioning LoRA methods lack, and what evidence shows it matters?

9 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Jo1Q

We appreciate the insightful comments from the reviewer. We will address the reviewer’s concerns one by one. Should the reviewers still have any confusion or new questions regarding our responses, we welcome them and would be pleased to provide further clarifications.

Q1: Metric circularity.

A1: Our video evaluation follows the protocol of ConsisID (CVPR 2025), which also uses an ArcFace-based face extractor while reporting FaceSim-Arc and FaceSim-Cur. Specifically, ConsisID extracts multi-scale ArcFace features, fuses them with CLIP features, and injects them through cross-attention. In contrast, OmniCustom uses a facial-recognition embedding as a global condition, and adds it to the output of self-attention. To ease concern, our user study shows that OmniCustom is preferred for identity consistency over all competing methods.

Q3: Novelty.

A3: We propose modality-specific LoRAs for sync customization, where the lightweight training yields a user-friendly latency compared with MiniMax-H3 (4.02 min vs. 15.34 min). Generally speaking, the main efficiency advantage stems from backbone size. Our reference embedding and contrastive losses aim to improve identity and timbre fidelity rather than reduce time cost. Nevertheless, our overall framework remains lightweight thanks to LoRA: it yields only 0.97B trainable parameters on top of the 11B OVI backbone. Under identical settings, OVI requires 3.97 minutes, whereas OmniCustom requires 4.02 minutes, corresponding to a small extra latency of approximately 1.3%.

The primary algorithmic novelty is auxiliary contrastive learning objectives, which forces the model to learn the difference from non-reference injection, thus enhancing the fidelity of given identity and timbre. Notably,existing contrastive schemes operated in customization tasks always operate in the form of feature-level contrast, while our method does not require the computation of feature distance between positive and negative samples.

OmniCustom-1M is a task-specific resource constructed for synchronous identity and timbre customization. It contains approximately one million clips spanning 2,500 hours. Its reference–target pairing protocol uses temporally disjoint segments from the same speaker, thereby preserving timbre while changing linguistic content and discouraging speech-content copying. The dataset further provides synchronization and quality filtering, independently sampled face references, ASR transcripts, and structured audio annotations covering accent, pitch, prosody, emotion, and speaking rate.

We also study the effect of data filtering and audio captions in Tab. A. Specifically, we use a small data subset (50K) from our OmniCustom-1M to perform controlled experiments, where we train each ablation setting with 200K steps.

Table A: Ablation study on data filtering and audio captions using a small data subset (50K) from our OmniCustom-1M. All results are generated using the same inference strategies.

SettingsSim-Arc↑Sim-Cur↑FID↓FVD↓CLIP↑Speaker-Sim↑FAD↓WER↓LSE-C↑LSE-D↓
Unfiltered + No audio captions0.490.5112652027.070.414.053.005.359.15
Unfiltered + Audio captions0.500.5212451427.100.443.862.855.409.10
Filtered + No audio captions0.520.5411148527.230.443.822.825.658.78
Filtered + Audio captions0.540.5610847727.250.473.642.655.708.72

It can be seen that without data filtering and audio captions, we obtain the worst results. Introducing audio captions yields improvements in Speaker-Sim, FAD, and WER. Applying data filtering brings gains on all metrics, especially for FID, FVD, FAD, CLIP, LSE-C and LSE-D. Combining both achieves the best performance.

We have added this ablation to our Sec 6.4.

AuthorsOpenReview ↗

Rebuttal for Reviewer Jo1Q

Q2: Mixed results.

A2: Table B: Revised Table 1 using mean ±\pm standard deviation.

ModelSim-Arc↑Sim-Cur↑FID↓FVD↓CLIP↑Speaker-Sim↑FAD↓WER(%)↓LSE-C↑LSE-D↓Latency↓
Phantom0.56±0.18‾\underline{0.56 \pm 0.18}0.57±0.18‾\underline{0.57 \pm 0.18}169.69±41.04169.69\pm41.04478.36±86.64478.36\pm 86.6426.72±3.7126.72 \pm 3.71–––––4.24±0.164.24\pm0.16
VACE0.22±0.110.22 \pm 0.110.23±0.120.23 \pm 0.12183.85±42.91183.85 \pm 42.91575.09±77.21575.09 \pm 77.2128.03±3.56‾\underline{28.03 \pm 3.56}–––––3.81±0.15\boldsymbol{3.81 \pm 0.15}
HunyuanCustom0.53±0.160.53 \pm 0.160.55±0.150.55 \pm 0.15130.29±48.67130.29 \pm 48.67517.12±66.03517.12 \pm 66.0323.98±4.3723.98 \pm 4.37–––4.87±1.454.87 \pm 1.4510.47±2.1910.47 \pm 2.198.72±0.178.72 \pm 0.17
Humo0.50±0.170.50 \pm 0.170.52±0.180.52 \pm 0.18181.01±34.56181.01 \pm 34.56475.94±79.03‾\underline{475.94 \pm 79.03}27.23±4.0127.23 \pm 4.01–––5.42±1.735.42 \pm 1.739.31±2.169.31 \pm 2.167.19±0.117.19 \pm 0.11
DreamID-Omni0.48±0.180.48 \pm 0.180.50±0.180.50 \pm 0.18183.42±51.72183.42 \pm 51.72509.16±92.91509.16 \pm 92.9127.56±3.9227.56 \pm 3.920.39±0.130.39 \pm 0.135.64±1.065.64 \pm 1.063.16±1.413.16 \pm 1.414.51±1.934.51 \pm 1.939.75±2.549.75 \pm 2.544.16±0.144.16 \pm 0.14
MiniMax-H30.56±0.190.56 \pm 0.190.57±0.180.57 \pm 0.18181.08±42.15181.08 \pm 42.15504.28±95.36504.28 \pm 95.3627.09±4.1227.09 \pm 4.120.55±0.28\boldsymbol{0.55 \pm 0.28}3.98±0.923.98 \pm 0.922.23±1.90\boldsymbol{2.23 \pm 1.90}5.82±1.75\boldsymbol{5.82 \pm 1.75}8.31±1.37‾\underline{8.31 \pm 1.37}15.34±0.1415.34 \pm 0.14
Ours (Baseline)0.39±0.170.39 \pm 0.170.41±0.170.41 \pm 0.17137.18±27.66137.18 \pm 27.66559.96±86.20559.96 \pm 86.2028.31±3.98\boldsymbol{28.31 \pm 3.98}0.30±0.190.30 \pm 0.194.32±0.874.32 \pm 0.872.40±1.82‾\underline{2.40 \pm 1.82}5.72±1.845.72 \pm 1.848.45±2.098.45 \pm 2.094.05±0.164.05 \pm 0.16
Ours (+ Reference embeddings)0.48±0.180.48 \pm 0.180.49±0.180.49 \pm 0.18105.62±31.25‾\underline{105.62 \pm 31.25}506.33±78.60506.33 \pm 78.6027.68±3.0427.68 \pm 3.040.38±0.210.38 \pm 0.213.67±0.82‾\underline{3.67 \pm 0.82}2.78±1.772.78 \pm 1.775.68±1.675.68 \pm 1.678.27±1.82\boldsymbol{8.27 \pm 1.82}4.09±0.154.09 \pm 0.15
Ours0.60±0.15\boldsymbol{0.60 \pm 0.15}0.62±0.15\boldsymbol{0.62 \pm 0.15}95.57±24.63\boldsymbol{95.57 \pm 24.63}440.49±63.52\boldsymbol{440.49 \pm 63.52}27.45±3.6627.45 \pm 3.660.51±0.25‾\underline{0.51 \pm 0.25}3.44±0.81\boldsymbol{3.44 \pm 0.81}2.51±1.852.51 \pm 1.855.76±1.35‾\underline{5.76 \pm 1.35}8.53±1.798.53 \pm 1.794.02±0.13‾\underline{4.02 \pm 0.13}

In Tab.C, we also perform statistical significance analysis.

Table C: Pairwise statistical significance analysis between our method and existing methods.

MetricMethodOurs / MethodΔ95% CIp
Sim-Arc↑Phantom0.60 / 0.56+0.04[+0.01, +0.07]0.003**
VACE0.60 / 0.22+0.38[+0.36, +0.40]<0.001***
HunyuanCustom0.60 / 0.53+0.07[+0.05, +0.09]<0.001***
Humo0.60 / 0.50+0.10[+0.08, +0.12]<0.001***
DreamID-Omni0.60 / 0.48+0.12[+0.10, +0.14]<0.001***
MiniMax-H30.60 / 0.56+0.04[+0.01, +0.07]0.004**
CLIP↑Phantom27.45 / 26.72+0.73[+0.14, +1.32]0.015*
VACE27.45 / 28.03−0.58[−1.17, +0.01]0.054 (n.s.)
HunyuanCustom27.45 / 23.98+3.47[+2.86, +4.08]<0.001***
Humo27.45 / 27.23+0.22[−0.45, +0.89]0.519 (n.s.)
DreamID-Omni27.45 / 27.56−0.11[−0.75, +0.53]0.736 (n.s.)
MiniMax-H327.45 / 27.09+0.36[−0.26, +0.98]0.258 (n.s.)
Speaker-Sim↑DreamID-Omni0.51 / 0.39+0.12[+0.09, +0.15]<0.001***
MiniMax-H30.51 / 0.55−0.04[−0.08, +0.00]0.066 (n.s.)
WER(%)↓DreamID-Omni2.51 / 3.16−0.65[−0.79, −0.51]<0.001***
MiniMax-H32.51 / 2.23+0.28[−0.02, +0.58]0.068 (n.s.)
LSE-C↑DreamID-Omni5.76 / 4.51+1.25[+1.03, +1.47]<0.001***
MiniMax-H35.76 / 5.82−0.06[−0.28, +0.16]0.589 (n.s.)
LSE-D↓DreamID-Omni8.53 / 9.75−1.22[−1.48, −0.96]<0.001***
MiniMax-H38.53 / 8.31+0.22[−0.04, +0.48]0.091 (n.s.)

Here, *p<0.05, **p<0.01, and ***p<0.001 show the level of statistical significance. n.s. = not significant.

Our method demonstrates significant superiority in visual fidelity (Sim-Arc, p<0.01) over both Phantom and MiniMax-H3. While MiniMax-H3 exhibits marginal numerical advantages in certain audio and synchronization metrics (Speaker-Sim, WER, LSE-C, LSE-D), these differences do not reach statistical significance (n.s., p > 0.05). Due to the long-tail distribution and high variance of audio error metrics (e.g., WER) across diverse complex scenarios, the marginal advantages of MiniMax-H3 are statistically negligible. This confirms that our model achieves on-par performance in audio fidelity and audio-visual synchronization, while maintaining a distinct advantage in visual subject consistency and inference latency. Note that FID, FVD, and FAD are distribution-level metrics, where significance tests are not applicable.

Besides, as seen in 4-th example in A-Fig. 8, MiniMax-H3 cannot generate a "lab coat" for the woman.

We have added Tab. C to our Appendix B.

AuthorsOpenReview ↗

Rebuttal for Reviewer Jo1Q

Q2+: Mixed results on ablation.

A2+: To assess whether the small changes in CLIP, WER, LSE-C, and LSE-D during our ablation study are significant, we conducted a per-prompt paired t-test with 95% confidence intervals. As shown in Tab. D, the performance fluctuations after adding each contribution (i.e., Reference embeddings and Contrastive losses) are marginal. None of these changes are statistically significant (all p>0.05p > 0.05). This indicates that while these modules significantly improve the visual generation metrics (e.g., Sim-Arc, FID, FVD), they do not introduce statistical degradation to text alignment and audio-visual synchronization capabilities.

Table D: Significance analysis of the ablations on audio and text metrics.

MetricSettingValueΔ95% CIp
CLIP↑Baseline28.31–––
+ Reference embeddings27.68−0.63[−1.42, +0.16]0.117 (n.s.)
+ Contrastive Losses27.45−0.23[−1.09, +0.63]0.596 (n.s.)
WER (%)↓Baseline2.40–––
+ Reference embeddings2.78+0.38[−0.12, +0.88]0.135 (n.s.)
+ Contrastive Losses2.51−0.27[−0.74, +0.20]0.261 (n.s.)
LSE-C↑Baseline5.72–––
+ Reference embeddings5.68−0.04[−0.39, +0.31]0.824 (n.s.)
+ Contrastive Losses5.76+0.08[−0.35, +0.51]0.719 (n.s.)
LSE-D↓Baseline8.45–––
+ Reference embeddings8.27−0.18[−0.61, +0.25]0.412 (n.s.)
+ Contrastive Losses8.53+0.26[−0.22, +0.74]0.288 (n.s.)

Here, difference (Δ = after − before). Δ and p are relative to the previous row. n.s. = not significant.

We have added Tab. D to our Appendix B.

Q4: Relation between the contrastive regularizer and classifier-free guidance.

A4: Our contrastive regularizer (Eq.8 and Eq.9) is a training-time mechanism along the reference-conditioning axis. It is different from CFG used only during sampling. Specifically, CFG scales in Sec. 6.1 are inherited from OVI and are applied only to text conditioning. During both CFG forward passes, the reference-image/audio latents and their identity/timbre embeddings are retained. The two predictions differ only in their text conditions: \begin{equation} v_m^{+}=v_{\theta}^{m}(Z_t, I^r, A^r, C^{+}, t), \qquad v_m^{-}=v_{\theta}^{m}(Z_t, I^r, A^r, C^{-}, t), \end{equation} where C−C^{-} denotes negative text condition and m∈V,Am \in \\{V, A\\}. The modality-specific guided prediction is as follows: \begin{equation} \hat{v}_m=v_m^{-}+s_m\left(v_m^{+}-v_m^{-}\right), \end{equation} where sV=4.0s_V=4.0 and sA=3.0s_A=3.0 are the video and audio guidance scales.

In contrast, our contrastive regularizer is used only during training and compares vθ(Zt,Ir,C,t)v_{\theta}(Z_t,I^r,C,t) and vθ(Zt,ϕ,C,t)v_{\theta}(Z_t,\phi,C,t) while keeping the text condition fixed. It encourages the learned velocity field to respond distinctly to the reference condition, but no extrapolation between reference-conditioned and no-reference predictions is performed during inference. Conceptually, the contrastive regularizer internalizes stronger reference dependence into the learned velocity field, whereas CFG externally extrapolates the predictions at each sampling step.

All ablations in Tab. 1 use the same text-CFG scales. Adding the contrastive losses improves Sim-Arc from 0.480.48 to 0.600.60, Sim-Cur from 0.490.49 to 0.620.62, and Speaker-Sim from 0.380.38 to 0.510.51. Therefore, these improvements result from the training regularizer rather than a change in inference-time CFG.

We have added A4 to Sec. 7.

Q5: YouTube identities.

A5: 70 YouTube identities are unseen using the ArcFace similarity. We select those videos whose similarity to our dataset embeddings is lower than 0.10.

We have added A5 to Sec. 6.1.

Q6: LoRA

A6: We design modality-specific LoRAs, which serves as baseline for sync customization. Specifically, the image-reference adapter is restricted to self-attention in video branch, and the audio-reference adapter to audio branch. Here, audio-video Cross-Attention, Cross Attention With Text, and FFN are frozen. This preserves the original cross-modal alignment.

Besides, we use different timestep assignment for reference tokens and original audio-video tokens. We make the reference tokens as clean condition by assigning them time step t=0t=0. Here, tokens in the original audio-video branch are noisy. Otherwise, the reference will attend to noise, increasing the learning difficulty.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Score Changes and Discussions in Rebuttal

Dear Area Chairs,

For the convenience of your review, we have made the following summary for rebuttal.

This paper proposes OmniCustom, a powerful DiT-based sync audio-video customization framework that can synthesize a video following reference image identity, audio timbre, and text prompts all at once in a zero-shot manner. It achieves state-of-the-art customization performance while providing user-friendly inference runtime.

Before the system freezed, our submission had effectively increased from scores 222 (confidence 4,2,2) to scores 223 (confidence 4,2,2), with one reviewer Jo1Q explicitly confirming score increases. We summarize the concerns raised by Reviewer QEDR and Reviewer sCCi, along with our corresponding responses as follows:

Reviewer QEDR raises concerns regarding incremental contribution, an issue also raised by Reviewer Jo1Q. We address this matter from both methodological and data perspectives, and our clarification has fully resolved Reviewer Jo1Q’s concern. We have also supplied variance and significance analysis for Reviewer Jo1Q. The results demonstrate that our model achieves on-par audio fidelity and audio–visual synchronization compared with MiniMax-H3, while retaining clear advantages in visual subject consistency and inference latency. Analyses on robustness and generalization are in our original manuscript, and we additionally include text-timbre mismatch results in this rebuttal. Moreover, Reviewer QEDR assumed our baseline adopts off-the-shelf weights. In reality, the baseline, "Ours (+ Reference embeddings)", and "Ours (+ Contrastive losses)" share identical training data. This isolates the data factor from the methodological factor. We further supplement metrics for background sound generation.

Reviewer sCCi is concerned that our contrastive regularizer may resemble reference CFG. We first point out that our contrastive regularizer is a method for enhanced condition injection during training, whereas reference CFG is an inference-time enhancement technique, which we do not use by default. This ensures that the gains observed in our ablation studies stem entirely from our training-time design. Note that the two are orthogonal: we can additionally enable such CFG on top of OmniCustom. We have added ablation studies in the Appendix that test different reference CFG scales during inference for "Ours (+ Reference embeddings)" and "Ours (+ Contrastive losses)". It turns out that, under the setting of "Ours (+ Reference embeddings)", the improvements from reference CFG alone are inferior to those brought by our contrastive regularizer. Moreover, enabling such CFG introduces an extra 40% latency overhead, and inappropriate reference CFG scales quickly distort the customized outputs. In addition, we provide ablations on separate identity and timbre losses, dataset ablation, and the updated user study analysis.

We sincerely thank you for the additional time and effort devoted to reviewing and evaluating our submission and hope these clearly comments and score changes will be taken into consideration when making the final recommendation.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.