acceptodds
Withdrawn from ICLR 2027

From Seeing to Hearing: Unified Audio-Video Generation with Progressive Adaptation and Modality Forcing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating temporally synchronized and semantically coherent audio and video within a unified model remains challenging, particularly when supporting both joint and directional generation. In this work, we propose a multi-stage adaptation framework that extends a pretrained video generator (Wan2.1) into a unified audio–video model. Our framework employs modality-specific branches with bidirectional cross-attention and progressively acquires multimodal generation capabilities through three stages. First, audio-only adaptation equips the pretrained backbone with audio generation capability. Second, joint audio–video fine-tuning establishes cross-modal interaction for synchronized audiovisual generation. Finally, we introduce , a task-dependent adaptation strategy that combines clean modality conditioning, directional information routing, and asymmetric optimization to enable video-to-audio and audio-to-video generation while retaining joint-generation capability. Extensive experiments demonstrate competitive joint audio–video generation, strong bidirectional conditional generation, and preservation of the pretrained video generation capability, highlighting the effectiveness of progressive adaptation and Modality Forcing for unified audiovisual generation.

Primary area:
generative models
Keywords:
Joint Audio-Video Generation, Multi-Stage Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QhqUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a progressive adaptation framework that extends Wan2.1 into a unified model supporting text-to-audio-video generation, video-to-audio generation, and audio-to-video generation. The architecture consists of modality-specific branches connected through bidirectional cross-attention, with temporally aligned rotary positional embeddings. Training proceeds through three stages: audio-only adaptation of a video-initialized backbone, joint audiovisual fine-tuning, and a final “Modality Forcing” stage. During directional training, the conditioning modality remains clean, its branch is frozen, the reverse cross-modal pathway is disabled, and only the target branch is optimized. Joint-generation training is interleaved to retain joint synthesis capability. The model is evaluated on JavisBench for joint generation and VGGSound for conditional generation. The paper reports competitive results, architecture and training ablations, transfer of Modality Forcing to JavisDiT, and a small human evaluation.

Critical strengths

1.A single model supports joint generation and both conditional directions with competitive performance. The evaluation covers three tasks rather than joint synthesis alone. On VGGSound V2A generation, the model achieves FAD of 5.5, compared with 7.0 for AudioX and 8.2 for HunyuanVideo-Foley, although these methods remain stronger on other metrics. This demonstrates practical value in consolidating multiple generation modes within one model.

2.The additional adaptation stage transfers to another architecture. Applying Modality Forcing to JavisDiT improves the reported metrics across joint generation, V2A, and A2V. For example, V2A FAD improves from 8.5 to 6.8, and A2V FVD improves from 415.8 to 352.5. This strengthens the evidence that the training recipe has utility beyond the proposed Wan-based implementation.

Critical weaknesses

1.The methodological novelty is limited and insufficiently distinguished from closely related work. Much of the framework follows established designs: twin modality-specific transformers, bidirectional cross-attention, scaled temporal RoPE, and audio training followed by audiovisual fusion training are already central ingredients of Ovi. Supporting joint and bidirectional conditional generation within a single model is also established in MoNL and UniForm. Consequently, the main potentially distinctive contribution is the particular combination of clean conditioning, reverse-path removal, and conditioning-branch freezing in Modality Forcing. These are plausible engineering choices, but the paper does not establish a substantial conceptual advance or demonstrate which combination is necessary. A controlled comparison against conventional clean-conditioned multitask fine-tuning and existing noise/masking-based approaches is needed to substantiate the claimed methodological contribution. My concern is insufficiently demonstrated novelty, rather than that these prior methods are identical to the proposed implementation.

2.The experimental controls do not establish that Modality Forcing itself explains the reported gains. The “No Modality-Forcing” ablation removes the entire final stage, which includes ten additional epochs on a filtered 355K-sample dataset. The comparison therefore changes training budget, data distribution, and training strategy simultaneously. Likewise, the JavisDiT transfer experiment lacks an equal-budget continued-training control. There are no separate ablations for clean conditioning, reverse-path removal, branch freezing, or joint-task replay. These omissions matter especially because the strongest novelty claim concerns their combination. The empirical case is further limited by the absence of unified conditional-generation baselines such as UniForm, the use of only TempoToken and TPoS for the main A2V comparison, and a human study covering only 20 prompts and five raters without uncertainty estimates.

Questions for authors

1.What benefit does Modality Forcing provide over ordinary multitask fine-tuning with clean conditioning and target-only loss, while retaining bidirectional attention? Please compare under the same initialization, dataset, task mixture, and compute budget.

2.Can you compare Modality Forcing with continued joint training on the same filtered 355K samples for the same number of updates? Can you separately ablate reverse-path removal, conditioning-branch freezing, and joint-task replay?

3.Why are MoNL/UniForm-style conditional training strategies not included as controlled baselines? A matched implementation on the same backbone would directly test the contribution beyond established approaches to unified joint and conditional generation.

4.What are the T2AV/V2A/A2V sampling probabilities, conditioning-branch timestep settings, LoRA configurations, and exact trainable parameter sets? Which vocoder is used? Also, can you provide a direct comparison with the original Wan2.1 checkpoint on a standard text-to-video evaluation to substantiate the claim of preserving its video-generation capability?

Reviewer_5vzyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors describe a progressive adaptation framework that turns a pretrained video generator (Wan2.1) into a unified audio-video model capable of joint generation as well as bidirectional conditional generation (V2A, A2V). Modality-specific audio and video branches are coupled through bidirectional cross-attention, and training proceeds in three stages — audio-only adaptation, joint audio-video fine-tuning, and "Modality Forcing," which freezes the conditioning branch, removes the reverse cross-attention pathway, and updates only the target branch while continuing to include joint samples.

Critical strengths
  1. The approach is practical and economical, exploiting Wan2.1's visual and temporal priors through LoRA-based adaptation rather than training from scratch, while claiming to retain the pretrained video-generation capability.
Critical weaknesses
  1. Novelty at the framework level is limited; the work is primarily an integration of established components. The dual-branch bidirectional cross-attention fusion, the strategy of extending a pretrained video generator into an audio-video model, and the audio-only hybrid LoRA adaptation all follow directly from prior work the paper itself cites, making the contribution closer to a well-executed assembly with empirical validation than to a new mechanism.
  2. The contribution, Modality Forcing, is incremental. Freezing the conditioning modality and training only the target branch under clean conditioning closely parallels modality-dropout or classifier-free-guidance-style conditioning, and the paper does not clearly distinguish it from the variable-noise conditioning (MoNL) and dynamic temporal masking (JavisDiT) it lists as related; a controlled comparison against these alternatives is absent.
  3. Joint generation is competitive only within the same-data, open-source regime rather than state of the art: proprietary-data models lead clearly (LTX-2: FVD 116.7, FAD 4.9, DeSync 0.569; likewise OVI and MOVA), JavisDiT++ surpasses the proposed model on FVD/FAD, and the margin over the same-data OVI baseline is modest.
  4. The baselines for directional generation are weak or dated. A2V is compared solely against cascaded TempoToken (627.5) and TPoS (916.8) with no strong recent A2V system, and V2A, although competitive, is not dominant (AudioX is better on KL/IS/DeSync and HunyuanVideo-Foley on IB-score), so the "strong bidirectional" claim rests partly on a straw-man comparison.
  5. Scale and evaluation rigor are constrained. All results are produced at 240p, 4 seconds, and 16 kHz with no high-resolution or long-duration evidence, and the user study involves only 5 participants over 20 prompts without variance or significance reporting.
Questions for authors

None

Reviewer_nLfbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends Wan2.1 to joint audio–video generation, V2A, and A2V through progressive adaptation and Modality Forcing. Results are competitive, but the benefits of Modality Forcing are not sufficiently separated from additional training and data filtering.

Critical strengths
  1. The framework supports three generation modes and achieves competitive V2A performance, including an FAD of 5.5. Architectural, fusion-direction, loss-weight, and audio-adaptation ablations provide useful supporting evidence.

  2. Modality Forcing also improves JavisDiT across all three tasks, suggesting that the strategy can transfer beyond the proposed architecture.

Critical weaknesses
  1. The contribution of Modality Forcing is insufficiently isolated. Table 3 removes the entire final adaptation stage, which adds ten epochs on a filtered 355K-sample dataset. The gains may therefore partly reflect additional training and data selection. Matched-budget continued training and component-wise ablations are needed to establish the benefits of directional routing and conditioning-branch freezing beyond ordinary clean-conditioned training or a MoNL-style baseline [1]. Neither the architectural fusion ablations nor the JavisDiT transfer experiment resolves this issue.

  2. Related-work comparisons and capability-preservation evidence are incomplete. The paper omits UniAVGen [3], which also uses dual-branch diffusion transformers with asymmetric cross-modal interactions and supports joint generation, V2A, and A2V. The additional contribution of Modality Forcing should be clarified. The A2V comparison includes only TempoToken and TPoS, omitting directly relevant approaches such as AV-Link [2]. Moreover, the absence of Stage 2 conditional-generation results limits assessment of Stage 3’s contribution, while joint audiovisual metrics alone do not substantiate preservation of Wan2.1’s original video-generation capability.

[1] Gwanghyun Kim et al. A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation. NeurIPS, 2024. [2] Moayed Haji-Ali et al. AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation. ICCV, 2025. [3] Guozhen Zhang et al. UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions. CVPR 2026.

Questions for authors
  1. Can Modality Forcing be compared with continued T2AV training on the same filtered data under matched training budgets? What are the separate contributions of clean conditioning, reverse-path removal, and conditioning-branch freezing?
  2. How does the method differ from UniAVGen [3] in cross-modal interaction and conditional training? Can comparisons with UniAVGen and AV-Link [2] be provided on mutually supported tasks under compatible settings?
  3. How does the Stage 2 checkpoint perform on V2A and A2V before Modality Forcing? Are text prompts and other conditioning inputs consistent across the compared methods?
  4. Can matched video-only evaluations of the original Wan2.1 and the adapted checkpoints substantiate the capability-preservation claim?
  5. Please clarify task-sampling probabilities, conditioning timesteps, LoRA sharing across tasks, and inference settings. Also resolve the BigVGAN/HiFi-GAN inconsistency and explain whether OVI⋆ uses random initialization or pretrained unimodal backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.