- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a progressive adaptation framework that extends Wan2.1 into a unified model supporting text-to-audio-video generation, video-to-audio generation, and audio-to-video generation. The architecture consists of modality-specific branches connected through bidirectional cross-attention, with temporally aligned rotary positional embeddings. Training proceeds through three stages: audio-only adaptation of a video-initialized backbone, joint audiovisual fine-tuning, and a final “Modality Forcing” stage. During directional training, the conditioning modality remains clean, its branch is frozen, the reverse cross-modal pathway is disabled, and only the target branch is optimized. Joint-generation training is interleaved to retain joint synthesis capability. The model is evaluated on JavisBench for joint generation and VGGSound for conditional generation. The paper reports competitive results, architecture and training ablations, transfer of Modality Forcing to JavisDiT, and a small human evaluation.
Critical strengths
1.A single model supports joint generation and both conditional directions with competitive performance. The evaluation covers three tasks rather than joint synthesis alone. On VGGSound V2A generation, the model achieves FAD of 5.5, compared with 7.0 for AudioX and 8.2 for HunyuanVideo-Foley, although these methods remain stronger on other metrics. This demonstrates practical value in consolidating multiple generation modes within one model.
2.The additional adaptation stage transfers to another architecture. Applying Modality Forcing to JavisDiT improves the reported metrics across joint generation, V2A, and A2V. For example, V2A FAD improves from 8.5 to 6.8, and A2V FVD improves from 415.8 to 352.5. This strengthens the evidence that the training recipe has utility beyond the proposed Wan-based implementation.
Critical weaknesses
1.The methodological novelty is limited and insufficiently distinguished from closely related work. Much of the framework follows established designs: twin modality-specific transformers, bidirectional cross-attention, scaled temporal RoPE, and audio training followed by audiovisual fusion training are already central ingredients of Ovi. Supporting joint and bidirectional conditional generation within a single model is also established in MoNL and UniForm. Consequently, the main potentially distinctive contribution is the particular combination of clean conditioning, reverse-path removal, and conditioning-branch freezing in Modality Forcing. These are plausible engineering choices, but the paper does not establish a substantial conceptual advance or demonstrate which combination is necessary. A controlled comparison against conventional clean-conditioned multitask fine-tuning and existing noise/masking-based approaches is needed to substantiate the claimed methodological contribution. My concern is insufficiently demonstrated novelty, rather than that these prior methods are identical to the proposed implementation.
2.The experimental controls do not establish that Modality Forcing itself explains the reported gains. The “No Modality-Forcing” ablation removes the entire final stage, which includes ten additional epochs on a filtered 355K-sample dataset. The comparison therefore changes training budget, data distribution, and training strategy simultaneously. Likewise, the JavisDiT transfer experiment lacks an equal-budget continued-training control. There are no separate ablations for clean conditioning, reverse-path removal, branch freezing, or joint-task replay. These omissions matter especially because the strongest novelty claim concerns their combination. The empirical case is further limited by the absence of unified conditional-generation baselines such as UniForm, the use of only TempoToken and TPoS for the main A2V comparison, and a human study covering only 20 prompts and five raters without uncertainty estimates.
Questions for authors
1.What benefit does Modality Forcing provide over ordinary multitask fine-tuning with clean conditioning and target-only loss, while retaining bidirectional attention? Please compare under the same initialization, dataset, task mixture, and compute budget.
2.Can you compare Modality Forcing with continued joint training on the same filtered 355K samples for the same number of updates? Can you separately ablate reverse-path removal, conditioning-branch freezing, and joint-task replay?
3.Why are MoNL/UniForm-style conditional training strategies not included as controlled baselines? A matched implementation on the same backbone would directly test the contribution beyond established approaches to unified joint and conditional generation.
4.What are the T2AV/V2A/A2V sampling probabilities, conditioning-branch timestep settings, LoRA configurations, and exact trainable parameter sets? Which vocoder is used? Also, can you provide a direct comparison with the original Wan2.1 checkpoint on a standard text-to-video evaluation to substantiate the claim of preserving its video-generation capability?