acceptodds
Under review as a conference paper at ICLR 2027

Omni-Customizer: End-to-End Multimodal Video Generation with Joint Identity and Audio Customization

Abstract

Despite recent advances in joint audio-video generation, precise appearance-and-voice customization remains challenging. We present Omni-Customizer, an end-to-end framework for human-centric joint audio-video customization. To establish cross-modal identity associations, we introduce Omni-Context Fusion (OCF) to integrate text embeddings with visual and audio references, together with Semantic-Anchored Multimodal RoPE (SA-MRoPE) to anchor paired references to their corresponding subjects. To reduce face duplication and timbre leakage under incomplete reference coverage, we further propose Subject-Aware Reference Routing (SARR), which controls where and how broadly each reference can directly influence generated content. We adopt a three-stage progressive training curriculum to achieve stable optimization and robust customization. We also develop a human-centric data curation pipeline to construct high-quality multimodal training data. Experiments on our proposed OC-Bench demonstrate strong video and audio quality, robust appearance-and-voice preservation, and faithful identity binding in complex multi-subject and incomplete-reference settings.

Primary area:
generative models
Keywords:
video generation, customization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KYTDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Omni-Customizer addresses joint appearance and voice customization for multi-subject audio-video generation, including cases with missing image or audio references. Built on Ovi, it introduces Omni-Context Fusion to enrich text conditioning with multimodal references, Semantic-Anchored Multimodal RoPE to associate references with subject descriptions, and Subject-Aware Reference Routing to constrain direct reference attention through subject scores and predicted coverage budgets. Training combines single to multi-subject adaptation, reference dropout, and subject reindexing.

Critical strengths

The paper addresses a consequential customization setting. Incomplete reference coverage introduces a practical challenge beyond preserving fully specified identities: available faces or voices should not spread to other characters. The distinction between subject association and selective reference access provides a useful formulation of this problem.

The reported evidence supports the usefulness of the complete system. Table 2 includes both incremental additions and removals from the full configuration. The user study also favors the proposed method for joint identity preservation and binding reliability.

Critical weaknesses

Leakage scores can improve simply because required characters or speech are omitted. Appendix C.5 defines character-presence metrics (A-CP and V-CP) to detect this problem, but does not report them. The identity-matching thresholds also lack calibration or sensitivity analysis, and the preference study does not directly validate these automatic measurements. It therefore remains unclear whether lower leakage reflects better identity separation while preserving all required characters and speech.

The progressive-training comparison combines curriculum ordering, reference dropout, and subject reindexing, leaving their individual effects unclear. Architectural comparisons are also limited: OCF lacks a capacity-matched baseline, SA-MRoPE is tested only through removal, and SARR is not compared with simpler routing or masking alternatives. The results support the complete system but do not establish how much its specific mechanisms contribute.

Questions for authors

Please report A-CP and V-CP alongside ILR for the compared methods and Table 2 routing ablations, separately for partial-modality and partial-subject cases. Do the leakage improvements persist when required-character and utterance presence are comparable?

How were the ArcFace and WavLM thresholds chosen, and what evidence shows that they reliably distinguish matching from non matching identities?

What exactly differs between the training configurations with and without “Prog. Train.” in Table 2? Are optimization steps, single-/multi-subject exposure, dropout, and reindexing matched? Can the authors isolate curriculum ordering from the two augmentation strategies?

Are controlled comparisons available against a capacity-matched conditioning module, simple subject-identity embeddings or positional indexing, and simpler supervised reference-routing alternatives? Such evidence would materially clarify the methodological contribution.

Reviewer_ro5qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies appearance and voice customization for joint audio-video generation. Omni-Customizer builds on Ovi and combines several technicial components including Omni-Context Fusion (OCF), Subject-Aware Reference Routing (SARR), SA-MRoPE. On OC-Bench, it reports better voice preservation and binding accuracy than the closest baselines

Critical strengths
  • The paper is self-contained and easy to follow.
  • The paper targets a concrete failure mode. Missing-reference cases make identity leakage easy to expose and give the proposed routing mechanism a clear purpose.
  • The ablation is clear. The authors separate identity association from reference routing and show that stronger routing can improve binding at a small cost to identity fidelity.
  • The evaluation includes new baselines (i.e., MiniMax-H3) and a user study.
Critical weaknesses
  • (main concern) The novelty is mostly in the combination. Multimodal fusion, positional anchoring, attention routing, and reference dropout are established ideas. The paper offers a useful task-specific design, but the methodological advance seems incremental.
  • The method is only tested on the older Ovi backbone. It is unclear whether the proposed modules would still help a stronger backbone such as H3, or whether they mainly address limitations specific to Ovi. Have the authors tried adding OCF, SA-MRoPE, and SARR to H3?
  • The coverage budget is fixed from the conditioning, while routing scores and selected positions are updated during denoising. If the actual face coverage or speaking duration differs from the conditioning, this budget may allocate too many or too few tokens to a subject. The paper does not directly evaluate this potential mismatch.
  • The routing only controls one path of reference influence. SARR masks direct attention, but identity information can still reach other subjects through the OCF-enriched text and audio–video cross-attention. It is unclear how much these indirect paths contribute to the remaining leakage.
  • The subject anchor is assumed to provide a reliable identity representation, but this is not directly validated. The pooled subject-token states may mix identity information with scene, action, and other subjects’ features, making their similarity scores with target tokens unreliable.
Questions for authors

Please refer to the Weaknesses section.

Reviewer_KsDAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Omni-Customizer, an end-to-end human-centric Joint Audio-Video Generation (JAVG) framework that addresses cross-modal binding errors, face duplication, and timbre leakage in multi-subject and partial-reference scenarios. Its core architecture comprises Omni-Context Fusion (OCF) for multimodal reference unification, Semantic-Anchored Multimodal RoPE (SA-MRoPE) for subject-level position alignment, and Subject-Aware Reference Routing (SARR) to restrict reference attention via predicted coverage budgets. Supported by a 3-stage training curriculum, a 1M-scale JAVG dataset, and the OC-Bench benchmark, Omni-Customizer achieves state-of-the-art identity preservation and binding accuracy with minimal computational overhead (+3.57% FLOPs).

Critical strengths

SARR offers an elegant architectural solution by explicitly constraining reference influences at the self-attention logit level via predicted coverage budgets, effectively eliminating face duplication and timbre leakage common in prior global cross-attention methods. Additionally, the proposed OC-Bench advances evaluation rigor in multi-subject customization by introducing Presence-corrected identity leakage metrics (V-CP, A-CP) to compensate for character omission artifacts and Active Speaker Detection-based Binding Accuracy.

Critical weaknesses
  • Baseline comparisons lack strict experimental control regarding training data advantages. Omni-Customizer is fine-tuned on a 1M-scale human-centric dataset on top of the Ovi backbone, whereas baseline models (e.g., Wan2.2-S2V, DreamID-Omni, LTX2.3) were evaluated zero-shot without access to this fine-tuning data. Consequently, it is difficult to isolate whether the reported quantitative gains stem purely from the proposed architectural components (SARR, OCF) or primarily from large-scale domain-specific data fine-tuning.

  • The sample size of the proposed OC-Bench is insufficient for robust evaluation. The benchmark consists of only 200 total scenarios, with key subsets containing just 50 evaluation cases each. Compared to standard open-domain generation benchmarks that feature over 1,000 diverse scenarios, this limited scale makes it hard to thoroughly demonstrate model robustness across varied real-world environments.

Questions for authors
  • To disentangle the impact of the 1M dataset from the architectural contributions, do you have experimental results comparing baseline open-source models after fine-tuning them on the same 1M dataset?

  • What is the specific motivation behind injecting only text position residuals into the backbone rather than passing the fused visual and audio reference tokens? Did full multi-modal token injection cause cross-modal interference or quality degradation in preliminary experiments?

  • While the qualitative comparisons provided in the submission are appreciated, do you plan to host a project/demo page featuring additional audio-video generation samples in the future?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.