acceptodds
Withdrawn from ICLR 2027

Polarization-Aware Experts for Unified Video-Text-to-Audio Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified video-text-to-audio generation supports Text-to-Audio (T2A), Video-to-Audio (V2A), and joint Video-Text-to-Audio (VT2A) within a single model. However, joint optimization introduces cross-task competition due to their heterogeneous conditioning requirements, as well as intra-task competition in VT2A, where excessive reliance on one modality can weaken alignment with the other. Existing systems primarily rely on shared dense backbones and progressive training, without explicitly preserving task-associated computation or directly regulating condition dominance in VT2A samples. We propose a two-stage Mixture-of-Experts framework that independently specializes T2A- and V2A-associated expert pools before composing them for unified multi-task training. Task-conditioned candidate restriction separates the computation paths of the single-condition tasks while allowing VT2A to access both pools. We further characterize excessive routing preference for one pool as expert-pool polarization and introduce a training-only intervention that reserves one routing slot for the under-selected pool while leaving the other unrestricted. Experiments demonstrate competitive T2A performance and strong V2A and VT2A results. Ablations validate the proposed specialization and candidate restriction, while the intervention improves targeted VT2A cases and mitigates inference-time expert-pool polarization. Our code will be publicly released.

Primary area:
generative models
Keywords:
Unified Video-Text-to-Audio Generation, Audio-Visual & Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iJrDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a unified MoE framework built on MMAudio, combining separately trained T2A and V2A expert pools through task-conditioned routing. For VT2A, a training-only intervention stochastically reserves one top-2 slot for the under-selected pool when routing is polarized. Experiments report competitive performance across the three tasks and intervention gains on polarized and off-screen VT2A cases.

Critical strengths

S1. Combining task-specific expert pools with a selective intervention is a good fit for the condition-competition problem in unified audio generation. T2A and V2A each use only their own pool, while VT2A can use both, which is a clean setup. The intervention only applies to samples flagged as polarized and always keeps the original top expert, so it gives the other pool a chance to learn without overriding per-sample preferences much. The candidate-restriction ablation in Table 2 supports this design, and I also like that the intervention is not enforced at inference.

S2. Results are strong across the three tasks and several evaluation settings. In Table 1, against MMAudio, AudioX, Omni2Sound and others, T2A is competitive and V2A and VT2A are strong. Table 3 compares the intervention on the same polarized subset, and Table 4(b) shows gains in the off-screen setting, where the text carries information the video does not. The paper reports both objective metrics and a human study, and the parameter increase is only about 7.7% (excluding the condition encoders).

Critical weaknesses

W1. The added benefit of polarization-based sample selection over generic cross-pool exploration has not been isolated. Table 3 and the polarized-subset results show that the intervention helps. But it is unclear whether this comes from targeting polarized samples or simply from making the model try the other pool more often. There is no control that applies the same amount of intervention to random samples. The threshold sensitivity study changes both which samples are targeted and how many, so it does not answer this either. Since selective, state-dependent intervention is the main distinguishing point of the method, the benefit of the polarization detection itself should be shown directly.

W2. There is no direct analysis of whether the routing-probability changes in Figure 2 translate into actual expert usage. Figure 2 looks at pre-selection routing probabilities, but the MoE output is computed with the renormalized weights of the two selected experts. The polarization measures can drop just because the scores of unselected experts change, while the actual top-2 and their weights stay the same. Also, if the expert in the reserved slot gets a small weight, its contribution to the output may be small, and the expert-to-shared lower-bound loss only looks at expert output norms before the routing weights are applied. More evidence is needed to clarify how the routing-probability changes relate to actual expert contributions and performance gains.

Questions for authors

Q1 (W1). Can the authors clarify the additional benefit of polarization-based targeting beyond increased exploration of the other pool? From the current results, I cannot tell whether the gain comes from finding polarized cases or simply from more exploration of the other pool. It would help to show, through additional experiments or analysis, which cases benefit from the intervention and which do not, and how well the proposed criterion captures that difference. For example, a comparison with random-target intervention, with the number of actual expert swaps matched as closely as possible, would be the most direct evidence.

Q2 (W2). Does the reduced polarization lead to changes in actual expert selection and output contribution? Analyses such as the fraction of tokens whose top-2 changes under the intervention, per-pool selection frequency and routing weights at inference, and weight-scaled expert output norms would show how the actual computation changes. Showing how these changes relate to the samples where performance improves would also help.

For now, I am giving a tentative score of 2, but I will consider raising the score if both are adequately addressed in the rebuttal.

Reviewer_NNeROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds upon MMAudio by replacing selected FFN layers with task-specific MoE layers. It employs a two-stage training strategy and a polarization-aware expert routing mechanism to balance MoE optimization across different tasks. The proposed method achieves performance improvements on V2A, T2A, and VT2A tasks.

Critical strengths
  1. The paper is well-structured and easy to follow.
  2. The method is well-motivated. To address cross-task competition and intra-task conditioning competition in unified audio generation, the authors introduce task-associated expert pools, task-conditioned candidate restriction, and polarization-aware routing intervention. The overall design is reasonable.
Critical weaknesses

My main concern is the performance gains of the proposed method.

  1. As shown in Table 2, the full method achieves only limited improvements over the standard single-stage training baseline. Considering that the proposed method introduces an additional task-specific specialization stage, the trade-off between performance gains and additional training costs needs further justification. Moreover, it is unclear whether the two settings use comparable total training budgets.
  2. The evaluation of off-screen audio generation is not sufficiently convincing. IB measures semantic correspondence between video and audio, but off-screen sound sources are not directly visible in the video. Therefore, this metric may not accurately reflect whether the specified off-screen sounds are correctly generated. Although CLAP measures text–audio alignment, its global similarity score may not reliably capture whether specific sound sources are missing, clearly audible, or consistent with the detailed text descriptions. The existing subjective evaluation does not separately report results for off-screen scenarios. I suggest conducting an additional user study specifically for off-screen audio generation, focusing on text-following accuracy and the correctness of the generated sound events.
Questions for authors

Instead of applying the polarization-aware intervention, could the model simply select one expert from each pool for every VT2A token, while retaining learned routing weights? Would this simpler strategy achieve comparable performance to the proposed method?

Reviewer_DFxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-stage mixture-of-experts framework for unified text-to-audio, video-to-audio, and video-text-to-audio generation. It first trains separate low-rank expert pools for the single-condition tasks, then combines them through task-conditioned routing. A training-only polarization intervention encourages participation from the under-selected pool when routing becomes overly concentrated. Experiments evaluate generation performance across the three tasks, alongside routing analysis, ablations, and human evaluation.

Critical strengths
  1. The paper addresses a meaningful problem: supporting text-only, video-only, and joint conditioning within one audio generation model. Combining task-specialized experts with a shared backbone offers a practical approach to accommodating these different settings.
  2. The complete system reports improvements over MMAudio on several metrics across the three tasks. The architectural and training ablations provide useful evidence about individual design choices, and the polarization intervention requires no forced cross-pool selection at inference.
Critical weaknesses
  1. The novelty beyond a mixture of LoRA experts appears limited. The polarization intervention is the main distinguishing component, but its gains on the full evaluation set are modest, leaving its practical contribution insufficiently established.
  2. The improvements over MMAudio are difficult to separate from the effects of additional data and training. Comparisons with MMAudio continued training on the same data, or separate task-specific LoRA adapters under comparable training budgets, would help establish the benefit of the proposed framework.
  3. The paper does not directly demonstrate that modality competition occurs or that the proposed intervention alleviates it. Preference for one expert pool does not necessarily imply neglect of the other modality, since both pools process fused representations. More balanced routing alone is therefore insufficient to support this explanation.
  4. The evaluation provides limited evidence of statistical reliability. The human study includes only 10 conditioning instances per task and 12 raters, while the modest gains from the polarization intervention are reported without uncertainty estimates. The size and precise selection procedure of the polarized subset also remain unclear.
Questions for authors
  1. Could the authors clarify the main contribution beyond existing mixtures of LoRA experts? Does the polarization intervention outperform a simpler strategy that encourages cross-pool participation without detecting polarization?
  2. Could the authors compare against continued training of MMAudio and separate task-specific LoRA adapters using the same training data and comparable budgets? How much of the improvement comes from the additional IF-Caps data?
  3. What direct evidence shows that polarized routing corresponds to neglecting one modality, and that the intervention restores its use? Could controlled removal or modification of text and video conditions demonstrate this relationship?
  4. Could the authors provide uncertainty estimates for the intervention gains and human evaluation, and clarify the size and selection procedure of the polarized subset? Would the human preferences remain consistent across a larger set of conditioning instances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.