acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Mixture-of-Experts for Audio-Visual Diffusion Models

Abstract

This work systematically investigates how to design and train Mixture-of-Experts (MoE) architectures for large-scale audio-visual diffusion models. Audio-visual latent tokens form continuous representations with strong spatial, temporal, and cross-modal correlations, motivating a re-evaluation of existing MoE design choices. Through systematic study, we summarize sparse capacity design principles tailored to audio-visual generation. We strengthen text conditioning with richer semantic representations while keeping the text conditioning branch dense and allocating sparse capacity to the visual backbone. Meanwhile, we find that load variation persists under continuous balancing and can arise from either semantic specialization or optimization instability. We therefore combine routing regulation with targeted stabilization to preserve normal expert differentiation while suppressing instability-driven collapse. Finally, we validate these designs using millions of high-quality audio-visual samples in a large-scale distributed training environment, establishing architectural and training principles for large-scale audio-visual diffusion MoE models.

Primary area:
generative models
Keywords:
Audio-Visual Joint Generation, Mixture of Experts Model, Large-Scale Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WzpAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates sparse capacity allocation and training stability in MoE-based audio-visual diffusion models. It keeps the text-conditioning and audio branches dense, studies the activation ratio and expert granularity of the video branch, proposes rescaling the routed-FFN contribution relative to the shared FFN, and combines bias-based load balancing with gradient clipping to improve training stability. The final model achieves performance comparable to much larger dense baselines with only 4.7B active parameters.

Critical strengths
  1. The authors systematically study the design and stable training of MoE-based audio-visual diffusion models, laying a solid foundation for further scaling of this class of models.

  2. The distinction between benign, specialization-driven load variation and instability-driven collapse is well motivated, and is supported by routing diagnostics collected during large-scale training.

  3. The proposed model achieves performance competitive with dense models in the 22B–33B range while activating only 4.7B parameters.

Critical weaknesses
  1. Although the paper is framed as an MoE design study for audio-visual diffusion models, the MoE analysis is almost entirely confined to the video branch, and the text and audio branches introduce little that differs from established practice. As a result, the paper becomes a re-validation of MoE design strategies from LLMs and video generation models on the video backbone, including bias-based load balancing and gradient clipping for training stability. I acknowledge that the video branch is plausibly the component that benefits most from MoE scaling in audio-visual generation, and that systematically re-validating existing MoE strategies in this setting is itself valuable. However, the paper would be better with more justification for not extending MoE to the text and audio branches. Concrete negative experimental results would make this design decision verifiable.

  2. The evaluation relies on a self-constructed set of only 87 prompts and includes no human study. Evaluating on a larger public benchmark such as VABench [1] or AVGen-Bench [2] and conducting a blinded pairwise human preference study would substantially strengthen the empirical claims.

[1] VABench: A Comprehensive Benchmark for Audio-Video Generation. CVPR 2026 [2] AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation. ICML 2026.

Questions for authors

The shared/routed FFN scaling rule proposed in Sec. 3.2.3 (Eq. 5) is supported by a reasonable norm-scale analysis, but no corresponding ablation is reported.

Reviewer_JQTKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use MoE in audio-visual diffusion models. The authors make the visual feed-forward layers sparse, keep the pretrained audio branch and text branch dense, and use Qwen features with an adaptor for text conditioning. They study expert count and width, the relative scale of shared and routed expert outputs, and ways to keep training stable. Their development experiments suggest that adding more experts eventually brings smaller gains, and that neither very wide nor very narrow experts work best. The final model has 32B total and 4.7B reported active parameters, using 128 routed experts with top-8 selection. The evidence includes development loss and routing measurements, qualitative ablations, and a comparison with four systems on 87 prompts. The model scores well on Face and Adherence and has a similar Overall score to the strongest systems in the comparison, though its audio-visual synchronization is weaker.

Critical strengths
  1. The authors have trained a large sparse model with audio-visual generation scores close to the strongest systems in their comparison. The architecture and training schedule in Tables 3 and 5 give useful details for others trying to train similar models.

  2. Table 4 shows that the load controller helps in the reported run. Average load violation falls from 1.70 to 0.09, and expert utilization rises from 35% to 100%. The discussion of load changes is also useful. A change in the noise distribution can shift expert loads without causing routing failure, so uneven loads alone can be misleading.

Critical weaknesses
  1. I cannot tell which design choices improve the final model. Appendix A.3 says that the sweeps use development flow-matching loss and a single shared seed. Only the selected configuration is trained at full scale. For Figure 3, I could not find enough information about the development model sizes, training budgets, absolute losses, or dense models used to calculate loss-equivalent capacity. There are no numerical comparisons showing how rescaling the routed expert outputs or keeping text dense affects generation quality. Figures 6 and 7 show individual examples for the shift schedule and adaptor. The stability discussion also relies on a few routing illustrations and one training run continued after a gradient spike. The final comparison cannot separate the benefits of these choices from those of the conditioning, pretrained audio, data, and training schedule. A few controlled comparisons at the same smaller training budget would help.

  2. The final comparison is hard to judge without more evaluation details. Section 4 uses 87 prompts and six automatic metrics, but does not provide the prompts, results from repeated generations, uncertainty estimates, or enough detail about the judges, score transformations, and baseline inference settings. The Overall scores are close, at 71.24 for this model, 71.82 for MiniMax H3, and 70.26 for LTX-2.5. I cannot tell how robust those differences are. The model's AV score is much lower, at 61.36 compared with 75.85 for MiniMax H3 and 82.56 for LTX-2.5. The paper acknowledges this gap, which is important for audio-visual generation. I would also like to see measured compute or runtime and a clear account of what the 4.7B active count includes. Since the code and checkpoints are unavailable, the evaluation needs enough detail to be checked independently.

Questions for authors
  1. Could you give the model configurations, training budgets, absolute losses, and sizes of the dense models behind Figure 3? How do you calculate loss-equivalent capacity, and how do expert count and top-K change across the granularity sweep to keep the budget fixed?

  2. Do you have controlled comparisons showing how rescaling the routed expert outputs and keeping text dense affect generation quality, using the same conditioning, data, and training budget?

  3. What triggers gradient clipping for the experts? Do you have training traces from the same checkpoint, with and without clipping, that show whether clipping caused the recovery? Do the traces support the proposed feedback between gradients, expert outputs, and routing?

  4. Could you provide the 87 prompts, judge and scoring settings, baseline inference settings, and uncertainty across prompts and generated samples? What does the 4.7B active count include, and how do total FLOPs or runtime compare at the same output settings?

Reviewer_fDpTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the design and training of MoE architectures for audio-visual diffusion models. The authors investigate activation ratio, expert granularity, shared/routed expert scaling, and routing stability. MoE is applied to the video backbone, while the audio and text branches remain dense. The paper combines development-scale experiments with a large-scale training recipe and comparisons against existing audio-visual generation models.

Critical strengths
  1. The paper studies a useful set of design choices for diffusion MoE, rather than simply demonstrating that a larger sparse model can be trained. The development-scale experiments on activation ratio and expert granularity provide practical guidance.

  2. The discussion of timestep-dependent load variation versus optimization-induced collapse is valuable. The training schedule and stability diagnostics also provide useful context for applying the recipe in practice.

Critical weaknesses
  1. My main concern is that the experiments supporting the design recommendations are not described in enough detail. Appendix A.3 explains the scope of the development-scale study, but the dense-model calibration, raw losses, and full configurations behind Figure 3 are missing. The proposed scaling in Eq. (5) also lacks a direct ablation or measurements of shared/routed output norms. This makes it difficult to assess how strongly the recommended settings are supported.

  2. The final evaluation is promising but under-specified. It uses 87 prompts, without reporting the number of samples per prompt, detailed inference settings, the judge configuration, or uncertainty estimates. The model achieves a competitive Overall score but lower AV synchronization scores than the strongest baselines. A clearer evaluation protocol is needed to interpret these trade-offs and the small differences in Overall score.

Questions for authors
  1. Could you provide the dense-model calibration and raw losses used to compute EL, together with the model sizes, training budgets, and configurations in Figure 3? Were these losses measured on held-out data?

  2. Could you provide a development-scale ablation of Eq. (5), along with measured shared/routed output norms? A full-scale sweep would not be necessary to address this concern.

  3. Could you clarify the evaluation settings, including samples per prompt, inference configurations, and the Gemini judge setup? Per-prompt results or uncertainty estimates would help interpret Table 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.