acceptodds
Withdrawn from ICLR 2027

EVA: Efficient Training for Joint Video-Audio Generation via Selective Cross-Modal Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

18 views

Abstract

Joint video–audio generation has advanced rapidly, with most recent approaches building upon pretrained text-to-video and text-to-audio models to construct joint generators. However, dense cross-modal interaction throughout training can introduce redundant or noisy signals in randomly initialized interaction modules, impairing both training efficiency and generation quality. Thus, we present an Efficient training framework for joint Video-Audio generation, namely EVA, for fast adaptation from pretrained uni-modal generators through multi-granularity selective cross-modal interaction: (1) at the task level, EVA selectively enables audio-to-video, video-to-audio, or bidirectional interaction according to the dominant cross-modal dependency of each sample; (2) at the model level, it restricts cross-modal interaction to deeper layers to reduce unnecessary computation and avoid premature interference with uni-modal representations; (3) at the token level, adaptive gating modulates cross-modal attention to emphasize informative visual–audio correspondences and ignore irrelative interactions, improving synchronization while preserving uni-modal generation quality. On Verse-Bench, our 7B model achieves competitive visual, audio, and speech quality with substantially lower training cost. It also obtains the lowest DeSync of 0.1902 and the highest LSE-C of 7.8018, demonstrating strong audio–visual synchronization. By reducing the cost of cross-modal adaptation, EVA can free training budget for scaling model capacity or further optimization. Code and weights will be released soon.

Primary area:
generative models
Keywords:
Diffusion, Joint Audio Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_n4AUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EVA proposes an efficient adaptation framework for joint video–audio generation by selectively controlling cross-modal interaction at three levels. At the sample level, training probabilistically selects audio-to-video, video-to-audio, or bidirectional interaction based on content-dependent assignments, together with directional noise scheduling and gradient routing. At the model level, cross-modal attention is restricted to the later halves of the backbones, with early-layer freezing during initial adaptation. At the token level, learned gates regulate cross-modal feature injection. The authors construct a roughly 7B model from Wan2.2-TI2V-5B and MOVA’s audio generator and train it using a three-stage schedule totaling approximately 1,280 H100 GPU-days. On the 600-prompt Verse-Bench benchmark, EVA achieves the best DeSync and LSE-C scores among the main-table baselines while maintaining competitive modality-specific quality. Validation includes component ablations, training curves, human preference studies, comparisons with the pretrained backbones, and an additional MiniMax-H3 comparison.

Critical strengths

Strong synchronization results supported by complementary evaluations. EVA achieves DeSync of 0.1902, compared with 0.2342 for the next-best main-table baseline, while maintaining competitive video quality and speech accuracy. Human evaluations also report more wins than losses in audio–visual alignment against all six main baselines. Agreement between automatic and perceptual evaluations strengthens the evidence that the model improves cross-modal coordination.

A practical adaptation design with measurable component benefits. The framework combines pretrained generators while largely preserving their modality-specific quality, as examined in Appendix D. The controlled layer-placement comparison reports a reduction from 41 to 27 seconds per iteration, with comparable or better reported quality and synchronization. This is concrete evidence that restricting interaction depth can provide a useful computational benefit.

Critical weaknesses

The approximately 3% figure compares EVA’s nominal GPU-days with those of MOVA, despite differences in model capacity, initialization, data, and potentially hardware and training workloads. EVA also inherits a pretrained MOVA audio branch; Appendix B.5 appropriately excludes backbone pretraining, but the comparison still does not isolate the savings attributable to selective interaction. Figure 7 compares methods at equal optimization steps rather than equal compute, while Table 3 measures a specific 10K-step experiment on H20 GPUs rather than the full H100 training schedule. These experiments support individual benefits, but do not establish the end-to-end cost reduction of the proposed framework relative to dense adaptation from identical backbones. This distinction is central to a paper whose main contribution is training efficiency.

Questions for authors

1.Matched efficiency comparison: Starting from identical pretrained backbones and using the same data and hardware, how does EVA compare with dense bidirectional adaptation in synchronization versus GPU-hours? Can you provide the compute breakdown across all three training stages?

2.Attribution of sample-level gains: Does content-dependent routing outperform content-independent routing with the same overall A2V/V2A/Joint frequencies? How much of the improvement remains when noise scheduling and stop-gradient are controlled separately?

3.Training and inference specification: What distributions generate the paired noise levels in directional training? At inference, are both cross-modal directions always enabled, and do the modalities follow identical noise schedules? Please also specify output duration, resolution, sampling steps, guidance settings, and whether benchmark images are supplied consistently to the compared models.

4.Ablation interpretation and reliability: What checkpoints and trainable components initialize the 10K-step layer and gating ablations? Why are their LSE-C values around 1.3–1.4, substantially below the final model’s 7.8018? Please clarify the evaluation subset and provide uncertainty estimates for the main synchronization comparisons, particularly the relatively small LSE-C margin over NAVA.

Reviewer_QD58OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EVA, an efficient framework for joint audio–video generation using pretrained video and audio models. It selectively controls cross-modal interactions at the sample, layer, and token levels. Experiments show competitive generation quality and strong audio–visual synchronization with a limited training budget.

Critical strengths
  1. Joint audio–video generation is an important research topic, and achieving high-quality generation with precise cross-modal synchronization is a meaningful objective.

  2. The proposed training strategy achieves competitive generation quality and strong synchronization with a relatively modest training budget, offering practical value for resource-efficient audio–video generation.

Critical weaknesses

Major Weakness:

  1. Leveraging pretrained video and audio generators and introducing cross-modal interaction is already well explored in works such as MiniMax-H3 [1], MOVA [2], LTX-2 [3], UniAVGen [4], JointDiT [5], BridgeDiT [6], and the JavisDiT series [7, 8]. These methods differ in their architectures and training strategies, but the manuscript does not sufficiently explain how EVA advances beyond their cross-modal interaction designs. EVA’s potential contribution lies in its selective interaction strategy rather than the general integration of pretrained generators. A more explicit comparison of interaction placement, directional conditioning, gradient routing, and feature modulation would help establish the technical distinction and significance of the proposed approach.
  2. The main comparisons do not isolate the contribution of the proposed architecture. The compared systems use different pretrained backbones, model sizes, training data, and optimization schedules. Although these comparisons are useful for assessing overall system performance, they do not establish whether EVA’s gains arise from its selective interaction design or from other differences in the training setup. The existing ablations provide partial evidence, but a controlled comparison using identical backbones, initialization, data, and training budgets would be needed to substantiate the architectural advantages. In particular, representative dense and selective interaction designs should be evaluated under the same experimental conditions.

References

[1] MiniMax-AI. MiniMax-H3.

[2] MOVA: Towards Scalable and Synchronized Video–Audio Generation. 2026.

[3] LTX-2: Efficient Joint Audio-Visual Foundation Model. 2026.

[4] UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions. CVPR 2026.

[5] Animate and Sound an Image. CVPR 2025.

[6] Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction. ECCV 2026.

[7] JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization.

[8] JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation. 2026.

Minor Weakness:

  1. The training-efficiency claim needs stronger evidence. EVA’s reported training cost is approximately 3% of MOVA’s, but the models differ in size and training setup, making it unclear how much the proposed method actually saves. The reduction from 41 to 27 seconds per iteration is useful, but the authors should also compare the total training time needed to reach similar generation quality and synchronization under the same settings.

  2. The generation setting is unclear. The manuscript does not clearly state whether EVA generates audio and video from text alone or requires an input image. The presented examples appear to use image conditioning. The authors should clarify whether text-only generation is supported, which setting is used for the reported results, and whether all baselines receive the same conditioning inputs.

  3. The LTX-2 results in the supplementary material raise concerns about the evaluation. All LTX-2 MP4 examples I inspected appear static, with no visible motion. This raises the possibility of an issue in inference or video export, which could affect the validity of the comparison. The authors should verify the generation pipeline and clarify whether the quantitative results were computed from the same outputs. If an implementation issue is identified, the affected results should be corrected and reevaluated.

Questions for authors
  1. I would encourage the authors to discuss the related works listed above and provide comparisons with the most relevant methods.
  2. Controlled experiments using the same backbones, training data, conditioning inputs, and computational budgets would help establish the benefits of the proposed approach.
  3. The authors should investigate why the LTX-2 examples appear static and verify whether this issue affects the reported evaluation results.

Substantive evidence addressing these points would provide a basis for reconsidering my score.

Reviewer_WhSWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper proposes EVA, a computationally efficient framework for constructing a joint audio–video generation model from pretrained text/image-to-video and audio generation models. EVA controls cross-modal interactions at three levels: samples, layers, and tokens. Specifically, it combines content-dependent selection of audio-to-video, video-to-audio, or bidirectional training; placement of cross-modal attention modules only in the later layers; and learnable gates that regulate the injected cross-modal features. Direction-specific training also employs different noise levels across modalities and stop-gradient operations.
  • EVA is using Wan2.2-TI2V-5B and MOVA’s 1.3B audio model. Training follows three stages: adaptation using later-layer LoRA adapters and newly introduced modules, full-model optimization, and finetuning on high-quality data. The reported additional training budget is 1,280 H100 GPU-days, with approximately 5M cumulative training samples* The evidence for a broad performance advantage is limited, and the methodological novelty is not sufficiently compelling. Although EVA achieves strong LSE-C and DeSync scores, its performance on other metrics is less competitive. The human evaluation also does not establish a clear overall advantage, particularly when ties are taken into account. Furthermore, the general strategy of combining pretrained video and audio generators and updating only cross-attention modules during joint adaptation has already been explored in prior work such as JavisDiT. While EVA introduces additional mechanisms for selective interaction, I am not yet convinced that these methodological advances, together with the reported empirical gains, constitute a sufficiently substantial contribution.
  • The provenance of the audio initialization and its implications for the reported training efficiency remain unclear. EVA uses Wan2.2-TI2V-5B and MOVA’s Audio DiT as its backbones. MOVA itself is trained for joint audio–video generation using Wan2.2-I2V-A14B and an audio generator. If EVA uses the audio checkpoint obtained after MOVA’s joint training, it may inherit representations already adapted to audio–visual generation. Combining this checkpoint with another video backbone from the Wan2.2 family could provide a favorable initialization, although the two video backbones differ in size and model variant. The paper should clarify whether the audio checkpoint was obtained before or after MOVA’s joint training. It also remains unclear whether the proposed recipe would achieve similar training efficiency and competitive performance when applied to other independently pretrained, standalone video and audio generators..
  • The paper evaluates EVA on the existing Verse-Bench benchmark using automatic metrics and human evaluation. It reports the best DeSync and LSE-C scores among the models compared in Table 1.
Critical strengths
  • The paper addresses a practically important problem with concrete design choices and controlled experiments. Training audio–visual joint generation models at low computational cost is a meaningful topic. Figure 7 shows improvements in synchronization metrics under matched initialization and training data. Table 3 further demonstrates that restricting cross-modal interactions to the later layers reduces training time per iteration from 41 to 27 seconds while broadly maintaining generation quality.
Critical weaknesses
  • The evidence for a broad performance advantage is limited. Although EVA achieves strong LSE-C and DeSync scores, its performance on other metrics is less competitive. The human evaluation also does not establish a clear overall advantage, particularly when ties are taken into account. Furthermore, the general strategy of combining pretrained video and audio generators and updating only cross-attention modules during joint adaptation has already been explored in prior work such as JavisDiT. While EVA introduces additional mechanisms for selective interaction, I am not yet convinced that these methodological advances, together with the reported empirical gains, constitute a sufficiently substantial contribution.
  • The provenance of the audio initialization and its implications for the reported training efficiency remain unclear. EVA uses Wan2.2-TI2V-5B and MOVA’s Audio DiT as its backbones. MOVA itself is trained for joint audio–video generation using Wan2.2-I2V-A14B and an audio generator. If EVA uses the audio checkpoint obtained after MOVA’s joint training, it may inherit representations already adapted to audio–visual generation. Combining this checkpoint with another video backbone from the Wan2.2 family could provide a favorable initialization, although the two video backbones differ in size and model variant. It remains unclear whether the proposed recipe would achieve similar training efficiency and competitive performance when applied to other independently pretrained, standalone video and audio generators.
Questions for authors
  1. Generalization to alternative pretrained generators. Could you demonstrate, even at a smaller scale, whether the proposed training recipe provides similar efficiency benefits and competitive generation quality when using a different pair of completely independently pretrained, standalone video and audio generation models?
  2. Validity of DeSync for joint audio–video generation. Please explain what evidence supports the use of DeSync to evaluate audio–visual alignment in joint audio–video generation. Although it has been used to assess temporal synchronization in video-to-audio generation, its applicability when both modalities are generated requires clarification. In particular, how does it respond to video degradation, audio degradation, temporal shifts, and semantic misalignment? This could be addressed through additional analysis or references to existing studies that directly validate these properties.
  3. Evaluation beyond temporal synchronization and lip synchronization. The current evaluation of audio–visual alignment primarily relies on DeSync for temporal synchronization and LSE-C for lip synchronization. Please complement these results with evaluations using appropriately validated metrics for audio–visual semantic alignment and for the alignment of both generated audio and generated video with the input text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.