- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates sparse capacity allocation and training stability in MoE-based audio-visual diffusion models. It keeps the text-conditioning and audio branches dense, studies the activation ratio and expert granularity of the video branch, proposes rescaling the routed-FFN contribution relative to the shared FFN, and combines bias-based load balancing with gradient clipping to improve training stability. The final model achieves performance comparable to much larger dense baselines with only 4.7B active parameters.
Critical strengths
-
The authors systematically study the design and stable training of MoE-based audio-visual diffusion models, laying a solid foundation for further scaling of this class of models.
-
The distinction between benign, specialization-driven load variation and instability-driven collapse is well motivated, and is supported by routing diagnostics collected during large-scale training.
-
The proposed model achieves performance competitive with dense models in the 22B–33B range while activating only 4.7B parameters.
Critical weaknesses
-
Although the paper is framed as an MoE design study for audio-visual diffusion models, the MoE analysis is almost entirely confined to the video branch, and the text and audio branches introduce little that differs from established practice. As a result, the paper becomes a re-validation of MoE design strategies from LLMs and video generation models on the video backbone, including bias-based load balancing and gradient clipping for training stability. I acknowledge that the video branch is plausibly the component that benefits most from MoE scaling in audio-visual generation, and that systematically re-validating existing MoE strategies in this setting is itself valuable. However, the paper would be better with more justification for not extending MoE to the text and audio branches. Concrete negative experimental results would make this design decision verifiable.
-
The evaluation relies on a self-constructed set of only 87 prompts and includes no human study. Evaluating on a larger public benchmark such as VABench [1] or AVGen-Bench [2] and conducting a blinded pairwise human preference study would substantially strengthen the empirical claims.
[1] VABench: A Comprehensive Benchmark for Audio-Video Generation. CVPR 2026 [2] AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation. ICML 2026.
Questions for authors
The shared/routed FFN scaling rule proposed in Sec. 3.2.3 (Eq. 5) is supported by a reasonable norm-scale analysis, but no corresponding ablation is reported.