acceptodds
Under review as a conference paper at ICLR 2027

OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

Abstract

Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These results demonstrate the importance of aligning modalities at the representation stage for omni-modal generation.

Primary area:
generative models
Keywords:
Multimodal Representation Learning, Audio-Video Generation, Cross-Modal Alignment, Variational Autoencoders, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_YqcJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniVAE, a jointly trained audio-video VAE framework whose audio and video latent spaces are encouraged to be semantically meaningful and temporally aligned. In addition to standard modality-specific reconstruction objectives, the method uses modality-specific semantic distillation from frozen Qwen3-Omni encoders and a segment-level bidirectional audio-video contrastive loss. The resulting VAEs are then used as frozen tokenizers for downstream text-to-audio-video generation, where the authors report improved audio quality and cross-modal synchronization while largely preserving reconstruction and video quality. The paper also introduces an audio-video desynchronization probing task to evaluate fine-grained alignment directly on frozen latents.

Critical strengths
  1. This paper addresses a question about what audio and video representations should be like for audio-video joint latent generation. The problem is interesting, and the community can benefit from this topic.
  2. Most parts of the framework design are reasonable, and the experimental results demonstrate its effectiveness.
Critical weaknesses
  1. A key concern is the interpretation of the semantic distillation objective and its dependence on Qwen3-Omni. The paper frames the distillation objective as purely intra-modal, because the video VAE is distilled from a visual encoder and the audio VAE from an audio encoder. While this is true at the level of the loss formulation, the teachers are taken from Qwen3-Omni, a unified multimodal model. Therefore, their intermediate representations may already contain cross-modal structure, or at least may be shaped to be compatible with a shared multimodal system. This makes it unclear whether the benefit of distillation should be attributed to modality-specific semantic enrichment, implicit cross-modal structure in the teacher encoders, or a combination of both. This ambiguity matters because one of the paper’s central claims is about the complementary roles of intra-modal semantic distillation and cross-modal contrastive alignment. Without an ablation using independently trained modality-specific teachers, e.g., a video encoder such as V-JEPA 2.1 and a separately trained audio encoder, it is difficult to tell whether the observed benefit comes from general semantic enrichment or from the particular cross-modal structure induced by Qwen3-Omni. Is the use of mutually compatible encoders from a unified multimodal model important, or would independent modality-specific teachers work equally well? The current manuscript does not clarify this requirement.

  2. Some methodological details are underspecified, and the current validation does not fully support a clean attribution of the reported gains. This makes it difficult to fully assess whether the paper’s central claims are supported by the evidence.

    • The hierarchical negative construction would benefit from a more formal description. The paper states that sibling-clip negatives are sampled from temporally disjoint clips of the same source video, but the exact sampling procedure is unclear: how the sibling clip is selected, whether multiple sibling clips are used, how temporal overlap is avoided, and how false negatives are handled when similar events recur in the same source video. A diagram and a mathematical definition of the three negative types would substantially improve the paper’s clarity.
    • The effect of the Stage-3 audio decoder fine-tuning is not fully isolated. According to the training recipe, in Stage 3, the audio encoder is frozen, and only the decoder is refined with adversarial and feature-matching losses. This affects waveform-level audio quality in downstream T2AV generation. Since audio quality is one of the main reported benefits, an ablation comparing the models before and after Stage 3 would help separate improvements due to a more learnable or better aligned latent space from improvements due to the final decoder refinement.
    • The downstream T2AV generator is used to demonstrate that OmniVAE latents improve generation quality and cross-modal synchronization. However, its architecture and training recipe are described only briefly. Since the reported improvements are measured after training a nontrivial joint generator on top of the proposed VAE, more details are needed to understand whether the gains come from the tokenizer alone or from interactions with the particular generator design.
Questions for authors
  1. Related to Weakness 1, I would appreciate it if the authors could answer what kind of information the authors expect each of the Qwen3-Omni encoders to offer to each modality's representation in the OmniVAE latent space, more specifically.
  2. Related to Weakness 2, I would appreciate it if the authors could clarify how "Sibling-clip negatives" are constructed in detail.
  3. In Equation (2), −log⁡σ(cos⁡(z~j,fj))-\log\sigma(\cos(\tilde{z}_j, f_j)) is used for aligning the latent representation z~j\tilde{z}_j with fjf_j. Why did the authors not use just −cos⁡(z~j,fj)-\cos(\tilde{z}_j, f_j) or −log⁡σ(cos⁡(z~j,fj)/γ)-\log\sigma(\cos(\tilde{z}_j, f_j)/\gamma) (γ\gamma: temperature)? Explaining the design choice would be appreciated.
  4. L.287 says "layer 27 of the Qwen3-Omni (Xu et al., 2025) visual encoder and layer 18 of its audio encoder as supervision" are used, but the reason why those layers are selected is not explained. Were they selected empirically? Or is there any previous work suggesting them? If they were empirically chosen, ablation studies and discussions would be appreciated.
  5. Could the authors report an ablation of Stage 3? Since Stage 3 fine-tunes only the audio decoder with adversarial and feature-matching losses, evaluating the model after Stages 1 and 2 but before Stage 3 would help separate improvements due to the aligned latent space from those due to the final decoder refinement.
7 comments
AuthorsOpenReview ↗

Summarization of Review and Overview of Our Responses

We thank the reviewer for the detailed assessment and concrete suggestions. We appreciate your recognition of the value of studying audio-video representations for joint generation, your assessment that most of the framework design is reasonable, and your observation that the experiments demonstrate its effectiveness. Your discussion of teacher provenance, sampling, and implementation details helps us make the methodological description and experimental interpretation more precise.

We first clarify the roles and terminology of the two additional objectives. Distillation supplies teacher-feature supervision separately to the visual and audio latents, while AVCLIP explicitly learns correspondence between audio-video segments. We will replace “uni-modality semantic distillation” with “modality-specific semantic distillation” to avoid implying that the Qwen3-Omni teachers were pretrained independently of other modalities. We use the distillation-only, contrastive-only, and combined configurations to explain their practical contributions.

We organize your main concerns and minor comments as follows:

  1. Teacher information and dependence on a unified multimodal teacher (Q1). We explain the intended role of distillation, teacher provenance, and the conclusions supported by the existing component comparisons.
  2. Sibling-clip sampling and repeated events (Q2). We clarify random candidate selection, non-overlapping clip construction, and the interpretation of hard negatives for temporal-instance correspondence.
  3. Loss design, teacher layers, and projection modules (Q3–Q4). We explain the connection between sigmoid-cosine and adaptive loss-magnitude balancing, correct the teacher-layer indices, and describe convolutional projection and feature-resolution matching.
  4. Stage 3 and generator implementation (Q5–Q6). We provide before/after decoder-refinement measurements and clarify its role. We will expand the generator description in the main text and provide complete training configurations in the appendix.
  5. Reproducibility and related work (Q7). We specify downsampling factors, tensor layouts, aggregation and pooling operations, and architecture diagrams, and discuss Diff-Foley/CAVP and Sequential Contrastive Audio-Visual Learning.

The following responses provide the results and explanations point by point, with corresponding additions to the main text and appendix.

AuthorsOpenReview ↗

Q1. What does semantic distillation transfer, and does it require a unified multimodal teacher?

Thank you for identifying this ambiguity. Although the two distillation losses are applied separately to the visual and audio branches, their teachers come from Qwen3-Omni, whose pretraining may already impart cross-modal structure. We will replace “uni-modality semantic distillation” with “modality-specific semantic distillation.” This describes where supervision is applied without implying that the teachers were pretrained independently of other modalities.

The visual supervision transfers object-, scene-, and local spatial information into visual latents. The audio supervision transfers sound-event and speech-related features and their temporal variation. These feature targets provide additional structure beyond pixel- or waveform-level reconstruction.

Computationally, each student is matched to its corresponding teacher, without directly comparing audio and video latents in the distillation loss. AVCLIP explicitly compares paired audio-video segment features to learn temporal-instance correspondence. The two objectives therefore provide complementary feature-level semantic supervision and explicit cross-modal alignment.

Our experiments include distillation alone, AVCLIP alone, and their combination. The following results summarize the frozen probe in Table 3 and downstream generation in Table 4:

ConfigurationProbe A@1 ↑DeSync ↓LSE-C ↑MusiQ ↑ManiQA ↑Audio IS ↑Audio FD ↓WER ↓
Recon6.40.8841.4790.5030.3383.0411.0290.243
Recon + Distill7.10.8141.4500.5230.3593.6040.9590.205
Recon + AVCLIP18.10.5761.9700.5030.3413.5980.9320.172
OmniVAE20.20.5702.0930.5150.3554.0010.8750.168

Distillation improves several modality-quality metrics, including MusiQ, ManiQA, audio IS, FD, and WER. AVCLIP provides the main temporal-alignment gains: probe A@1 increases from 6.4% to 18.1%, and DeSync decreases from 0.884 to 0.576, alongside improved audio generation. Their combination obtains the best synchronization and audio results among these configurations, demonstrating their complementary contributions.

The formulation requires position-matched features from each teacher; it does not require shared teacher parameters, equal feature dimensions, or joint multimodal pretraining. In our implementation, Qwen3-Omni provides strong visual and audio semantic targets within one unified teacher family, and the ablations above show the distinct gains from this modality-semantic supervision and from AVCLIP.

AuthorsOpenReview ↗

Q2. How are sibling-clip negatives sampled, and how are repeated events treated?

We will clarify the sampling procedure and formal definition. Preprocessing partitions each source video into temporally non-overlapping clips and retains its source-video identity. Data organization aims to make at least one additional clip from the same source available for each anchor in the candidate pool; selection among available siblings is random. The negative sampler then samples segments uniformly without replacement from all eligible same-source, different-clip candidates. Consequently, selected sibling segments may come from one or multiple sibling clips.

Let g(i)g(i) denote clip ii's source-video identity, IiI_i its temporal interval in the source, and B\mathcal{B} the distributed candidate pool. For anchor (i,s)(i,s), the positive is (i,s)(i,s) in the other modality. The three negative candidate sets are:

Ni,sintra={(i,t)∣t≠s}.\mathcal{N}^{\mathrm{intra}}_{i,s}=\{(i,t)\mid t\neq s\}.

Cisib={(j,t)∣j∈B, j≠i, g(j)=g(i), Ij∩Ii=∅}.\mathcal{C}^{\mathrm{sib}}_i=\{(j,t)\mid j\in\mathcal{B},\ j\neq i,\ g(j)=g(i),\ I_j\cap I_i=\varnothing\}.

Cicross={(j,t)∣j∈B, g(j)≠g(i)}.\mathcal{C}^{\mathrm{cross}}_i=\{(j,t)\mid j\in\mathcal{B},\ g(j)\neq g(i)\}.

The default setting retains all 47 intra-clip negatives and randomly selects 48 sibling and 24 cross-video segments. Non-overlap is guaranteed by preprocessing rather than by random sampling itself, and the implementation identifies siblings through source identity and distinct clip indices.

For repeated events, semantic category identity should be distinguished from temporal-instance correspondence. In synchronized data, the audio and video from the same temporal window form the ground-truth pair. Another window containing the same event category, such as drumming, may differ in impact timing, rhythmic phase, or action progression. It can therefore remain a valid hard negative. Removing all such examples could let the model match the category “drum” without learning when an impact occurs.

We therefore use repeated events with different impact timings, rhythmic phases, or action progressions as hard negatives for temporal alignment, encouraging the model to learn correspondence between specific event instances.

Reviewer_EGMYOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OmniVAE, designed to address the lack of cross-modal correspondence in the latent space, a consequence of training audio and video VAEs independently, which forces downstream generative models to relearn audio-visual synchronization. The proposed method retains separate encoders and decoders for audio and video while incorporating two types of supervision beyond the standard reconstruction objective: first, it aggregates audio and video latent variables into temporally aligned segment features, establishing fine-grained audio-visual associations through bidirectional contrastive learning; second, it employs frozen Qwen3-Omni vision and audio encoders to perform unimodal semantic distillation.

Critical strengths

Existing approaches to joint audio-visual generation typically learn cross-modal associations at the generator level. This paper, by contrast, focuses on the representation quality of the underlying tokenizers. Incorporating reconstructability, unimodal semantics, and cross-modal synchronization into the training of a Variational Autoencoder (VAE) is a promising direction for further research.

Critical weaknesses
  1. Insufficient distinction from prior work. ProAV-DiT [1] already learns aligned audio-visual latents through a dual-stream autoencoder for joint generation. The paper neither discusses this closely related work nor demonstrates advantages over fusion-based autoencoders, leaving its “first” claim insufficiently justified.
  2. Potential confounding in sync probing. Aggregator initialization and pretraining are unclear across configurations. If AVCLIP variants benefit from pretrained aggregators, probing gains cannot be attributed solely to VAE latents. Identically initialized probes with matched training budgets are needed.
  3. Unmatched training protocols. Configurations differ in training steps (250k vs. 82k), clip lengths (121 vs. 193 frames), checkpoint initialization, and decoder fine-tuning. Without matching cumulative data exposure and compute, the contributions of the proposed objectives remain difficult to isolate.
  4. Understated quality trade-offs. OmniVAE reduces Panda-70M reconstruction PSNR (36.56→35.24), audio ViSQOL (4.55→4.08), and generation Motion Score (0.556→0.456). These declines weaken claims of preserved quality and warrant human evaluation and a clearer reconstruction–synchronization trade-off analysis.
  5. Limited downstream validation. Evaluation covers one generator at 256×256 and lacks comparisons with generator-level semantic or synchronization supervision. Broader experiments are needed to establish generalizability and distinguish improved learnability from faster convergence.

[1] ProAV-DiT: A Projected Latent Diffusion Transformer for Efficient Synchronized Audio-Video Generation

Questions for authors

Please see the weaknesses.

11 comments
AuthorsOpenReview ↗

Summarization of Review and Overview of Our Responses

We thank the reviewer for the detailed assessment and the specific questions about related work, attribution, and quality trade-offs. We appreciate your recognition of the value of studying joint generation through the quality of the underlying tokenizer representations, and of combining reconstructability, modality semantics, and cross-modal synchronization in VAE training as a promising research direction.

We will sharpen the contribution as improving the latent space used for downstream generation through training-time semantic and alignment supervision, while preserving independent audio and video VAE encoding and decoding. We will add the omitted ProAV-DiT work, remove the “first” claim, and clarify differences in architecture, alignment mechanisms, objectives, and inference dependencies. We will also distinguish the roles of latents and aggregators, encoders and decoders, and reconstruction fidelity and downstream generation quality more clearly.

We address your main concerns in the following groups:

  1. Related work and methodological distinctions (Q1). We discuss ProAV-DiT and explain our specific design and evaluation scope.
  2. Attribution in synchronization probing and training protocols (Q2–Q3). We explain aggregator pretraining and the Frozen/+ft protocols, clarify stage-wise training histories, and provide additional comparisons using 5-second and 8-second clips.
  3. Decoder refinement and quality trade-offs (Q4–Q5). We report before/after Stage-3 measurements and additional refinement experiments on reconstruction-only models, explain the restoration of audio decoding quality, and examine reconstruction, Motion Score, and downstream metrics together.
  4. Validation scope and the relation to generator supervision (Q6–Q7). We explain the resolution and generator choices and clarify the complementary roles of tokenizer alignment and generator-level supervision.
  5. Learning efficiency and late-training performance (Q8). We use the full learning curves and late-training results to distinguish earlier attainment of better performance from improved outcomes at the end of the evaluated budget.

The following responses present the detailed explanations, additional experiments, and corresponding revision plans.

AuthorsOpenReview ↗

Q1. Relation to ProAV-DiT and methodological distinction

We thank the reviewer for pointing out this important related work and apologize for overlooking it in our literature review. ProAV-DiT also addresses alignment at the representation stage, using cross-modal attention within its autoencoder to improve downstream audio-video generation. We will add it to the related work and remove our “first” claims.

Methodological distinction. The methods differ along four dimensions.

  1. Architecture. ProAV-DiT uses a Multi-scale Dual-stream Spatio-Temporal Autoencoder (MDSA): frame-wise audio chunks become a Mel-spectrogram sequence, and TimeSformer-based encoding with orthogonal projection represents each modality as three 2D latent planes (HW, TW, and TH). OmniVAE uses a Wan2.2 video VAE and a HunyuanVideo-Foley-based, DAC-style continuous audio VAE, encoding video into spatiotemporal latents and raw audio waveforms into a 1D temporal latent sequence. It preserves modality-specific representation structures.
  2. Alignment mechanism. ProAV-DiT exchanges features through Group Cross-Modal Attention and further fuses the modalities through Bidirectional Block Cross-Attention during decoding. OmniVAE separately aggregates and projects audio/video latents, then applies segment-level bidirectional InfoNCE. Temporally matched features form positive pairs; other segments within the clip, sibling clips, and unrelated source videos supply negatives. This explicitly supervises fine-grained temporal correspondence through gradients to both encoders, without introducing cross-modal feature fusion into the VAE encoding or decoding paths.
  3. Training objectives. ProAV-DiT follows a conventional autoencoder reconstruction recipe, including reconstruction, perceptual, adversarial, feature-matching, and KL losses; its audio-video alignment is mainly implemented through the cross-modal attention modules described above. Beyond the standard reconstruction training, OmniVAE introduces two additional representation-learning objectives: modality-specific semantic distillation from frozen Qwen3-Omni visual and audio encoders, which improves the semantic structure of each latent space, and segment-level bidirectional audio-video contrastive learning, which explicitly supervises fine-grained cross-modal semantic and temporal correspondence. Both auxiliary objectives are used only during training. Stage 3 freezes the audio encoder and restores adversarial and feature-matching supervision only to recover audio decoding quality, without further changing the learned latents.
  4. Cross-modal dependence at inference. ProAV-DiT retains cross-modal attention in latent processing and decoding, so these operations involve features from both modalities. During generation, these are generated latent features; this does not imply a requirement for ground-truth input from the other modality. OmniVAE discards the contrastive aggregators and distillation projectors after training. Each encoder accepts only its own modality, and each decoder uses only its own modality's latent, enabling independent encoding and decoding without additional VAE inference cost from alignment. This independence concerns the tokenizer; the downstream joint generator still uses cross-modal attention.

For ProAV-DiT's architecture and objectives, see Sec. 3.1 and Appendix 5.2.

Experimental scope. ProAV-DiT reports 16-frame inputs and an open-domain experiment on 100k AudioSet videos (Secs. 4.1–4.2). Our corpus contains approximately 23M audio-video clips, and joint VAE training uses 193-frame clips at 24 fps, covering approximately eight seconds. We additionally study tokenizer behavior through reconstruction evaluation, latent synchronization probing, and downstream T2AV learning curves. Our generation evaluation covers temporal synchronization, lip synchronization, video–audio and text–audio semantic consistency, visual quality, non-speech audio quality, speech intelligibility, and audio aesthetics. These experiments provide evidence for the benefits and trade-offs of explicit tokenizer-level supervision at a substantially larger data scale and with longer frame sequences.

We will revise the paper to emphasize this methodological distinction and empirical scope.

AuthorsOpenReview ↗

Q2. Effect of the aggregators in synchronization probing

We thank the reviewer for raising the role of the aggregator. The AVCLIP aggregators are training modules that introduce additional cross-modal semantic and temporal supervision into the VAE representations. During contrastive training, they learn to extract temporally corresponding segment features from heterogeneous audio and video latents. We therefore report both frozen and trainable-aggregator protocols to evaluate the representation under fixed-readout and adapted-readout settings.

To reduce this asymmetry, the paper also reports a +ft protocol. The VAE encoders and decoders remain frozen, while the audio/video aggregators are unfrozen and trained jointly with the offset-classification head on the same synchronization-probing task. This allows the initially non-contrastive aggregators of Recon and Recon + Distill to learn from the probing data. The two protocols give:

VAE configurationA@1 FrozenA@1 +ftA@5 FrozenA@5 +ftA@1tol FrozenA@1tol +ft
Recon6.47.031.532.416.217.1
Recon + Distill7.17.833.033.917.518.4
Recon + AVCLIP18.121.847.852.434.539.6
OmniVAE20.222.350.153.237.240.4

Unfreezing the aggregators improves all configurations, while leaving the ordering and overall conclusion essentially unchanged: Recon and Recon + Distill remain close, both AVCLIP configurations remain substantially stronger, and OmniVAE performs best under both protocols. For example, OmniVAE exceeds Recon in A@1 by 13.8 points with frozen aggregators and 15.3 points with trainable aggregators; the corresponding A@1tol gaps are 21.0 and 23.3 points. Thus, the result is not explained simply by preventing Recon from updating its non-pretrained aggregator.

We will distinguish the protocols more precisely in the revision. Frozen probing evaluates the usability of the learned alignment representation together with its trained aggregator, whereas +ft allows every configuration's aggregator to adapt to the same probing task. Both protocols show that the cross-modal structure learned through AVCLIP consistently translates into stronger synchronization discrimination.

Reviewer_PyA2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniVAE, a jointly trained audio-video VAE that improves cross-modal alignment and latent representation quality for downstream joint generation. The method combines segment-level contrastive learning with modality-specific semantic distillation from pretrained Qwen3-Omni encoders. Experiments demonstrate improvements in audio-video synchronization and generation quality while largely preserving reconstruction performance.

Critical strengths
  1. Well-motivated problem. The paper addresses an important challenge in joint audio-video generation by incorporating cross-modal alignment directly into VAE training.

  2. Clear presentation. The paper is generally well-organized, and the overall framework is easy to understand.

  3. Extensive experiments. The paper evaluates reconstruction quality, audio-video synchronization, and downstream generation performance across multiple benchmarks.

Critical weaknesses

1.Insufficient evidence of OmniVAE's effectiveness. I am not fully convinced of OmniVAE's overall effectiveness. As shown in Table 1, the reconstruction-only baseline (Recon) outperforms OmniVAE on many reconstruction metrics, particularly for audio. Although OmniVAE improves downstream generation performance, the paper does not sufficiently investigate the trade-off between reconstruction fidelity and generation quality. More convincing evidence and analysis are needed to justify the benefits of OmniVAE.

2.Insufficient comparisons with existing methods. The text-to-audio-video generation experiments only compare different variants of the proposed framework, without including representative existing audio-video generation methods or alternative VAE baselines. Such comparisons are important for demonstrating the effectiveness and competitiveness of OmniVAE.

3.Limited technical novelty and insufficient analysis of individual contributions. The main contribution appears to be the integration of existing techniques for jointly training audio and video VAEs, followed by downstream generation. However, the individual contributions of the proposed components are not sufficiently clear. Although some ablation studies are provided, more comprehensive experiments are needed to isolate the effects of the hierarchical negative sampling strategy, semantic distillation, and cross-modal alignment. Overall, the work appears primarily to be an engineering integration of existing techniques, without sufficiently insightful analysis or novel findings.

Questions for authors
  1. How do the authors justify the reconstruction quality degradation of OmniVAE compared with Recon?

  2. Can the authors provide downstream comparisons with existing audio-video generation methods and alternative VAEs?

  3. What are the specific benefits of jointly training audio and video VAEs over independent training?

  4. How much does each component of the hierarchical negative sampling strategy contribute to performance?

5 comments
AuthorsOpenReview ↗

Summarization of Review and Overview of Our Responses

We thank the reviewer for the assessment and suggestions. We appreciate your recognition of the motivation for representation-level alignment in joint audio-video generation, the clear organization and understandable framework, and the experimental coverage across reconstruction, synchronization probing, and downstream generation.

The paper focuses on representation learning for generative tokenizers and its value for downstream joint generation. Our four configurations—Recon, Recon + Distill, Recon + AVCLIP, and full OmniVAE—examine the individual and complementary effects of the two additional objectives. The clearest reported benefits concern audio-video synchronization and generated audio quality. We will describe these benefits and their reconstruction-fidelity trade-offs more precisely, and clarify the question addressed by each comparison.

Our responses address your main concerns in four parts:

  1. Reconstruction–generation trade-offs (Q1). We distinguish reconstruction changes across speech, general audio, and music, and relate them to downstream synchronization, semantic consistency, and audio-generation metrics.
  2. Comparison scope and experimental positioning (Q2). We explain the role of tokenizer comparisons within a common generation framework and the positioning of existing generators and external VAEs, clarifying the relationship between our representation-learning question and end-to-end system comparisons.
  3. Joint training and component contributions (Q3). We compare the effect of adding AVCLIP both with and without semantic distillation, reporting the resulting changes in cross-modal alignment and audio generation.
  4. Contributions of negative sources (Q4). We provide a controlled experiment that changes negative-source composition while fixing the total negative budget, and report intra-clip and mixed retrieval results.

The following responses provide the experimental evidence and corresponding manuscript revisions for each point.

AuthorsOpenReview ↗

Q1. Reconstruction–generation trade-off

We thank the reviewer for raising this trade-off. OmniVAE should be assessed through both reconstruction fidelity and the downstream generation enabled by its latent space. Reconstruction alone does not determine how effectively that space can be modeled; our results show substantial generation benefits from improved semantic structure and cross-modal correspondence.

1. Reconstruction degradation varies substantially across domains. For speech, the changes are small in absolute terms: STOI decreases from 0.9981 to 0.9948, PESQ-NB from 4.4916 to 4.4400, and PESQ-WB from 4.5557 to 4.4085. The decrease is more apparent for general audio and music: ViSQOL changes from 4.55 to 4.08 on AudioSet and from 4.60 to 4.31 on MUSDB18. Spectral reconstruction errors also increase, with Mel changing from 0.44 to 0.74 and from 0.50 to 0.73, respectively. We acknowledge these losses and will distinguish the domains explicitly rather than describing all reconstruction changes as uniformly slight.

The VAE provides the representation foundation for downstream generation. Our experiments show substantial improvements in both generated audio and audio-visual alignment.

2. Both semantic objectives improve all four primary audio-generation metrics, and their combination gives the best results. Table 4 compares the same downstream generator design across tokenizer variants, averaging the same final three checkpoints (180k, 190k, and 200k) at CFG = 5:

TokenizerIS ↑KL ↓FD ↓Speech WER ↓
Recon3.0411.2791.0290.243
Recon + Distill3.6041.2580.9590.205
Recon + AVCLIP3.5981.2310.9320.172
OmniVAE4.0011.1400.8750.168

Relative to Recon, OmniVAE increases IS by 31.6% and reduces KL, FD, and WER by 10.9%, 15.0%, and 30.9%. Both objectives individually improve every metric, and their combination is best on all four. The WER gain also shows substantially more intelligible generated speech despite the small reconstruction-score change.

3. The benefits extend to fine-grained synchronization and overall semantic correspondence.

TokenizerDeSync ↓Lip-sync LSE-C ↑Video–Audio similarity ↑Text–Audio similarity ↑
Recon0.8841.4790.2540.201
Recon + Distill0.8141.4500.2560.227
Recon + AVCLIP0.5761.9700.2570.225
OmniVAE0.5702.0930.2740.246

Compared with Recon, OmniVAE reduces DeSync by 35.5%, increases LSE-C by 41.5%, raises video–audio similarity from 0.254 to 0.274, and raises text–audio similarity from 0.201 to 0.246. The gains cover temporal and lip synchronization and broader semantic consistency. Figure 2 shows that they emerge earlier and remain through the final evaluated checkpoints.

OmniVAE also remains competitive with external VAEs: it outperforms Stable Audio 3 SAME-S and SAME-L on all reported speech and AudioSet/MUSDB18 reconstruction metrics. We will revise the discussion to state the reconstruction changes precisely and ground the effectiveness claim in the measured downstream gains above.

AuthorsOpenReview ↗

Q2. Comparisons with existing generators and alternative VAEs

Our goal is to evaluate the proposed representation-learning approach itself, rather than to pursue state-of-the-art performance in end-to-end audio-video generation. Specifically, we examine how modality-specific semantic distillation and audio-video contrastive learning improve generative representations relative to reconstruction-only VAE training. We therefore compare the reconstruction-only baseline with variants incorporating either objective individually or both jointly, within a common downstream generation setting. This comparison focuses on the effectiveness of the proposed training objectives and their complementary contributions.

Direct comparisons with independently developed audio-video generators, including open-source systems, are difficult to interpret because their training data and training budgets are difficult to match. Their training settings also differ in video resolution, clip duration, and audio sampling rate. Consequently, differences in downstream performance cannot readily be attributed to the tokenizer or its training objectives. This motivates evaluating our approach within a common downstream setting that makes the contributions of the two semantic objectives easier to assess.

We therefore use comparisons within the same downstream generation framework as the primary evidence for the proposed representation-learning approach. We evaluate four tokenizer configurations: (i) reconstruction only, (ii) reconstruction plus modality-specific semantic distillation, (iii) reconstruction plus audio-video contrastive learning, and (iv) reconstruction plus both semantic objectives. These variants use the same VAE backbones and are evaluated with the same downstream generator design, training data, and evaluation protocol, directly measuring how each tokenizer objective changes downstream learnability.

The results reveal distinct contributions from the two objectives:

  • Audio-video contrastive learning provides the main improvement in fine-grained synchronization. Relative to Recon, Recon + AVCLIP reduces DeSync from 0.884 to 0.576 and increases lip-sync LSE-C from 1.479 to 1.970. Distillation alone has a much smaller effect on DeSync (0.814) and does not improve LSE-C (1.450). Thus, generic semantic enrichment alone does not reproduce the synchronization gains obtained with explicit cross-modal contrastive supervision.
  • Both objectives improve text–audio semantic consistency and audio generation. Text–Audio similarity increases from 0.201 to 0.227 with distillation and 0.225 with contrastive learning, reaching 0.246 when both are combined. Both objectives individually improve IS, KL, FD, and WER, and their combination gives the best result on all four metrics. For example, FD changes from 1.029 to 0.959 / 0.932 / 0.875, and WER from 0.243 to 0.205 / 0.172 / 0.168, for distillation, contrastive learning, and the full model, respectively.
  • Their combination gives the strongest overall cross-modal alignment. Video–Audio similarity improves modestly with either objective alone (0.254 → 0.256 / 0.257) and more substantially with both (0.274). The full model also obtains the best DeSync (0.570) and LSE-C (2.093) among the four variants.

These comparisons provide evidence for the usefulness of each objective and their complementary effects on downstream generation. They are not intended to establish superiority over all existing audio-video systems.

Regarding alternative VAEs, the reconstruction-only configuration provides a baseline with the same backbone and latent dimensionality, directly isolating the effect of tokenizer training. Table 2 additionally compares reconstruction against external VAEs. Together, these experiments evaluate controlled downstream learnability within a fixed architecture and reconstruction competitiveness against established tokenizers.

Reviewer_zdQYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OmniVAE, an audio-video VAE framework designed to improve cross-modal alignment in joint audio-video generation.

Instead of independently training audio and video VAEs solely with reconstruction objectives, the authors introduce two additional training objectives. First, segment-level audio-video contrastive learning (AVCLIP) aligns temporally corresponding audio and video latent representations using a bidirectional InfoNCE objective with hierarchical negative sampling. Second, modality-specific semantic distillation transfers semantic features from pretrained Qwen3-Omni audio and vision encoders into the corresponding VAE latent spaces.

The proposed framework retains separate audio and video VAEs, while the contrastive and distillation modules are discarded during inference. The authors evaluate the approach through reconstruction experiments, audio-video synchronization probing, and downstream text-to-audio-video generation.

The experimental results suggest that contrastive alignment provides substantial improvements in temporal synchronization, while semantic distillation and contrastive learning together improve several downstream audio generation metrics. The paper also reports relatively small degradation in reconstruction quality.

Overall, the work investigates an important problem in multimodal generative modeling: incorporating cross-modal correspondence directly into the latent representation rather than relying entirely on downstream generative models to learn such relationships.

Critical strengths
  1. The paper addresses an important problem in joint audio-video generation. Learning cross-modal correspondence directly at the representation level is a meaningful direction, as it may reduce the burden on downstream generative models to learn temporal synchronization from scratch.

  2. The proposed segment-level contrastive learning objective is technically reasonable. In particular, the hierarchical negative sampling strategy, which combines intra-clip, sibling-clip, and cross-video negatives, provides a useful mechanism for learning fine-grained temporal correspondence. The synchronization probing results show substantial improvements over reconstruction-only training, supporting the effectiveness of explicit cross-modal alignment.

Critical weaknesses

1. The contribution beyond established alignment objectives is not sufficiently isolated.

The paper makes a meaningful contribution by incorporating cross-modal alignment into reconstructive audio and video VAEs, rather than relying entirely on the downstream generator to learn correspondence. However, its segment-level bidirectional contrastive objective closely follows the formulation in Synchformer [1], which the authors acknowledge. The contribution therefore lies primarily in adapting and integrating alignment into the tokenizer, rather than introducing a fundamentally new contrastive objective.

While the objective-level ablations support the usefulness of this integration, they do not isolate the benefit of the proposed three-level negative construction over a simpler segment-level contrastive baseline. In particular, the ablations on clip duration and temporal granularity in Section 4.5.2 do not establish whether sibling-clip negatives provide an advantage beyond increasing the negative pool or changing temporal context. A comparison that controls the number of negatives while varying their sources would help establish the specific contribution of this design. This matters to my assessment because the current evidence more clearly supports the value of adding contrastive supervision than the advantage of the particular formulation proposed here.

2. Training differences and limited robustness analysis weaken the attribution and scope of the conclusions.

The training configurations differ in ways that complicate attribution of the gains to the proposed objectives. Section 4.1 and Appendix B report 250k steps on 121-frame clips for the modality-specific variants and 82k steps on 193-frame clips for the joint variants. Moreover, the full model is initialized from the Recon+Distill checkpoints. These differences do not, by themselves, make the comparison unfair, but the cumulative training histories and the extent to which the comparisons control for additional optimization and temporal context require clarification. A matched continuation baseline, using comparable initialization, clip length, and training budget but without the contrastive objective, would strengthen the evidence that the gains arise specifically from cross-modal alignment.

The scope of the alignment results also needs clearer characterization. OmniVAE treats temporally matched segments as positive pairs after feature aggregation and projection. I recognize that this does not require the complete audio and video latents to become identical, and that reconstruction losses and the filtering pipeline help preserve modality-specific information and improve correspondence. Nevertheless, the paper does not directly characterize performance on segments with weak or incomplete audio-visual correspondence, such as background audio or off-screen events. Selective-alignment work such as [2] provides relevant motivation for examining this limitation, although it does not establish that OmniVAE suffers from the same failure mode. A targeted analysis of such cases would clarify the generality of the reported benefits. My concern is the missing characterization of robustness, rather than a demonstrated failure of the proposed objective.

References

[1] Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Synchformer: Efficient Synchronization From Sparse Cues. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 5325–5329.

[2] Jiyang Zheng, Siqi Pan, Yu Yao, Zhaoqing Wang, Dadong Wang, and Tongliang Liu. Aligning What Matters: Masked Latent Adaptation for Text-to-Audio-Video Generation. Advances in Neural Information Processing Systems (NeurIPS), 2025.

Questions for authors

Q1. Methodological Novelty

How does the proposed AVCLIP objective fundamentally differ from existing segment-level audio-video contrastive learning methods, particularly Synchformer [1]? While OmniVAE integrates contrastive alignment into jointly trained reconstructive VAEs, it remains unclear what specific methodological advances are introduced beyond adapting existing contrastive learning mechanisms to VAE latent representations. Could the authors clarify the fundamental differences and demonstrate the advantages of their formulation?

Q2. Experimental Isolation

Could the authors provide more controlled experiments to isolate the contributions of semantic distillation, audio-video contrastive learning, and joint optimization? In particular, the modality-specific configurations are trained for 250k steps, whereas the joint configurations are trained for 82k steps, with different video clip lengths (121 versus 193 frames). How much of the observed improvement can be attributed specifically to the proposed objectives rather than these differences in training configurations?

Q3. Robustness of Cross-Modal Alignment

The proposed contrastive objective aligns temporally corresponding audio and video segments as positive pairs. However, temporal correspondence does not necessarily imply semantic correspondence. Real-world audio-video data may contain background sounds, off-screen events, and modality-specific information. Recent work on selective cross-modal alignment [2] explicitly addresses incomplete and asymmetric audio-visual correspondence. How does OmniVAE address this issue, and can the authors provide evidence that enforcing alignment across all temporally corresponding segments does not introduce semantic noise or negatively affect modality-specific representations?

References

[1] Iashin et al. Synchformer: Efficient Synchronization From Sparse Cues. ICASSP 2024.

[2] Aligning What Matters: Masked Latent Adaptation for Text-to-Audio-Video Generation. NeurIPS 2025.

6 comments
AuthorsOpenReview ↗

Summarization of Review and Overview of Our Responses

We thank the reviewer for the careful and specific assessment. We appreciate your recognition of the importance of incorporating audio-video correspondence at the representation level, the technical rationale of segment-level contrastive learning, and the synchronization-probing gains as evidence for explicit cross-modal alignment. Your discussion of negative construction also helps us distinguish the benefit of the overall training objective from the contribution of a particular sampling design.

We first clarify the central question of the paper: how to make reconstructive audio-video latents better suited to downstream joint generation. OmniVAE incorporates modality-specific semantics and fine-grained cross-modal correspondence into VAE training and evaluates their effects on synchronization and generation quality. The VAEs retain independent encoding and decoding paths, and the auxiliary aggregators and projectors are training-only, so these objectives introduce no additional VAE inference cost.

We organize your main concerns into three groups and address them below:

  1. Methodological contribution and negative construction (Q1). We clarify the relationship to Synchformer and the contribution of alignment supervision within decodable tokenizers. We provide a fixed-budget comparison of negative-source compositions and report changes in intra-clip and mixed retrieval.
  2. Training protocols and attribution (Q2). We explain initialization, stage-wise training histories, and the common downstream generation setting, and provide reconstruction and synchronization-probing comparisons for VAEs trained on 5-second versus 8-second clips.
  3. Behavior under incomplete correspondence (Q3). We explain data filtering and representation design, and provide strong/weak-correspondence analyses, mismatched-audio controls, and reconstruction measurements to characterize off-screen speech, background music, and related cases.

The following responses present the experimental results, implementation details, and corresponding manuscript revisions in this order.

AuthorsOpenReview ↗

Q1. Methodological novelty

Our central contribution is to improve the learnability of the audio and video representations used by downstream generative models. Reconstruction-oriented VAEs learn to preserve signals, but reconstruction alone does not ensure that their latent spaces have the semantic structure and cross-modal correspondence needed for effective joint generation. OmniVAE addresses this representation-learning problem by incorporating modality-specific semantics and fine-grained audio-video correspondence directly into VAE training.

In research on improving generative tokenizers, semantic supervision commonly draws on representations or learning objectives developed for self-supervised representation learning and discriminative tasks, including cross-modal alignment. The key contribution lies in how this supervision improves the structure and learnability of decodable latent spaces, and whether those changes benefit downstream generation; it does not depend on introducing a new standalone semantic loss. OmniVAE follows this direction and extends it to jointly learned audio-video representations. Its segment-level contrastive formulation draws on Synchformer, while addressing the generative requirement that the aligned representations remain decodable and useful for joint distribution modeling.

OmniVAE balances these requirements through separate reconstruction pathways and training-only alignment and distillation heads. At inference, the two VAEs remain independent and the auxiliary heads are discarded, so alignment adds no VAE inference cost.

Our downstream experiments show improvements across three complementary aspects: fine-grained audio-video alignment, global semantic consistency across text, audio, and video, and the quality of the generated audio itself. Specifically, Table 4 compares the full OmniVAE with the reconstruction-only baseline:

  • Fine-grained alignment: DeSync decreases from 0.884 to 0.570, and lip-sync LSE-C increases from 1.479 to 2.093.
  • Global cross-modal semantic consistency: ImageBind video–audio similarity increases from 0.254 to 0.274, and CLAP text–audio similarity increases from 0.201 to 0.246.
  • Audio generation quality: IS increases from 3.041 to 4.001, while FD decreases from 1.029 to 0.875.

These results show that the learned VAE representations improve synchronization, semantic correspondence, and generated-audio quality. We will emphasize this generative representation-learning contribution in the revision.

Negative-source comparison. We compare three configurations after 30k steps, using the same reconstruction-only initialization, data, 193-frame clips, optimization, and freezing schedule. Semantic distillation is disabled. Training uses 1,662,760 entries from FineVideo and Vript that can provide multiple non-overlapping clips per source video. Runs use 32 GPUs with per-GPU batch size 1; the grouped sampler forms four source-video blocks per global step, totaling 16 paired clips.

The first configuration uses only sibling-clip negatives. The other two retain them and compare source composition under a fixed 144-negative budget:

Training negatives: intra / sibling / cross-videoTotal negativesIntra-clip retrievalSibling-clip retrievalCross-video retrievalMixed retrieval
0 / 48 / 0480.2170.3020.2310.131
0 / 48 / 961440.2560.3230.3160.184
47 / 48 / 491440.2910.3240.3220.206

The sibling-only row has 48 negatives because the number of non-overlapping clips available from the same source video is limited by training-video duration; this source cannot be expanded to 144 while keeping the sampling definition unchanged. Evaluation candidates are identical across models. Intra-clip and cross-video evaluation covers 2,062 clips; sibling and mixed evaluation covers 1,417 clips.

Sibling-only training already learns correspondence. Adding cross-video negatives improves all four retrieval tasks and raises mixed retrieval from 0.131 to 0.184. Under the fixed 144-negative budget, adding intra-clip negatives further raises intra-clip retrieval from 0.256 to 0.291 and mixed retrieval from 0.184 to 0.206. The three sources provide complementary same-source, semantic, and temporal supervision, and their combination performs best overall.

AuthorsOpenReview ↗

Q2. Experimental isolation

All downstream T2AV models are trained on 5-second clips, matching the duration used to train the reconstruction-only VAEs. Under this common downstream setting, OmniVAE achieves substantially better audio generation quality and cross-modal alignment than Recon. We clarify the complete VAE training histories and provide additional 5-second versus 8-second controls below.

1. Training duration and convergence. We continued training the reconstruction-only VAEs on our dataset and observed that reconstruction initially improved and then reached a stable plateau, with no meaningful gains under extended optimization. For Recon + Distill, we additionally monitored the semantic distillation loss and continued training until it remained stable. Thus, both baselines were sufficiently optimized before introducing the next objective.

2. Initialization and progressive optimization. The full model is initialized from the converged Recon + Distill checkpoints before introducing cross-modal contrastive alignment. Its 82k-step joint stage is therefore additional optimization rather than its entire training history. The different stage durations reflect the convergence rates of reconstruction, distillation, and alignment. We will make this initialization and stage-wise history explicit in the revision.

3. Clip duration. We conducted additional reconstruction-only runs using identical initialization and modality-specific training recipes while changing clip length from approximately 5 to 8 seconds. Video models are compared at 80k steps and audio models at 60k steps. These controls use the same reproduction corpus within each comparison.

Video reconstruction PSNR (dB; higher is better):

Training lengthPanda-121Panda-193UCF-121UCF-193
121 frames (~5 s)35.72735.68636.79136.791
193 frames (~8 s)35.70035.66936.78836.790

Audio reconstruction on 5-second inputs:

Training lengthSTOI ↑PESQ-WB ↑Mel ↓STFT ↓ViSQOL ↑
5 s0.99794.5470.5212.0214.501
8 s0.99794.5470.5192.0204.503

At these endpoints, the maximum video PSNR difference is 0.028 dB, and the audio metrics are nearly identical. Across eight matched video checkpoints from 10k to 80k, neither clip length consistently leads; individual PSNR differences range from −0.188 to +0.302 dB. Longer VAE training clips therefore show no consistent reconstruction advantage.

We also compare synchronization probes on the frozen 5-second and 8-second reconstruction-only VAEs, using the same probe architecture, batch size (128), and learning rate (5e-5). For a matched training interval, both runs are restricted to epochs 0–51, and we average the final five common epochs (47–51). Validation accuracies are:

VAE training lengthBest A@1, epochs 0–51Mean A@1, epochs 47–51Mean A@5, epochs 47–51Mean A@1tol, epochs 47–51
5 s6.3155.97728.40216.053
8 s6.3395.82128.47215.002

Best A@1 differs by only 0.024 percentage points, while mean A@1 is slightly higher for the 5-second model. Increasing VAE training length therefore provides no consistent synchronization advantage.

Together, these results show that Recon reached a stable plateau and that extending reconstruction-only VAE clips from 5 to 8 seconds does not consistently improve reconstruction or synchronization probing. Meanwhile, OmniVAE substantially improves downstream generation and alignment when every generator uses the same 5-second training setting. We will include these controls and clarify the full stage-wise training history.

Comments on the paper

2 comments

We sincerely thank the reviewer for the thoughtful follow-up and constructive feedback. We have incorporated the additional experiments and discussions from our rebuttal into the revised manuscript and uploaded the updated PDF. We would like to further address the three remaining concerns.

1. Methodological Novelty and Negative Sampling

We appreciate the reviewer's suggestion regarding a more exhaustive isolation of individual negative sources. Our additional experiments demonstrate the benefits of combining negatives with different temporal and semantic relationships, while further ablations could better characterize their individual contributions.

We have also revised the manuscript to reduce the emphasis on negative sampling as a methodological contribution in the main text, while providing a more detailed explanation of the sampling design and its rationale in the appendix.

We would like to emphasize that hierarchical negative sampling is a practical implementation choice rather than the central contribution of OmniVAE. Our main contribution is a generative representation-learning framework that integrates reconstruction, per-modality semantic distillation, and segment-level audio-video contrastive learning, and, more importantly, reveals the complementary roles of holistic semantic alignment and fine-grained temporal alignment in joint audio-video generation.

This research direction is also reflected in works such as SpeechTokenizer (ICLR 2024), VA-VAE (CVPR 2025), MAETok (ICML 2025), RAE (ICLR 2026), and VTP (ECCV 2026). These studies explore how semantic information and representation structure can improve the learnability and performance of downstream generative models, rather than necessarily proposing fundamentally new semantic learning objectives. Their contributions lie in designing generation-friendly representations and demonstrating their benefits for downstream modeling.

Similarly, our central finding is that joint audio-video generation benefits from two complementary forms of semantic supervision: fine-grained temporal correspondence between audio and video, and holistic semantic consistency across modalities. Their combination leads to the strongest overall generation performance. Exploring more advanced implementations of each semantic objective remains an interesting direction for future work.

2. Experimental Attribution and the Four-Way Ablation

We would like to highlight an important aspect of our experimental design: we evaluate four VAE configurations—Recon, Recon + Distill, Recon + AVCLIP, and Recon + Distill + AVCLIP (OmniVAE). These comparisons allow us to examine the individual and complementary effects of the two semantic objectives across different dimensions of downstream generation.

As shown in Figure 2, the two objectives exhibit distinctly different effects.

Fine-grained temporal synchronization: For DeSync and LSE-C, the four configurations form two clear groups:

  • Without AVCLIP: Recon (black) and Recon + Distill (blue) show similar synchronization performance, with DeSync scores of 0.884 vs. 0.814 (↓) and LSE-C scores of 1.479 vs. 1.450 (↑).
  • With AVCLIP: Recon + AVCLIP (orange) and OmniVAE (green) achieve similarly strong synchronization performance, with DeSync scores of 0.576 vs. 0.570 (↓) and LSE-C scores of 1.970 vs. 2.093 (↑).

These results clearly indicate that the improvements in fine-grained audio-video synchronization primarily come from AVCLIP, while semantic distillation alone has limited effects on these metrics.

Holistic semantic consistency and generation quality: Both objectives improve audio generation quality, aesthetic quality, ImageBind (audio-video similarity), and CLAP (text-audio similarity). Combining them generally achieves the strongest overall results.

These differences are reflected throughout the training curves in Figure 2, rather than only at the final checkpoint. They demonstrate that the two objectives contribute differently to downstream generation: semantic distillation primarily improves holistic semantic consistency and per-modality generation quality, whereas AVCLIP primarily improves fine-grained temporal synchronization.

We appreciate the suggestion of a matched continuation baseline as an additional control for optimization history. Meanwhile, our four-way ablation provides important evidence beyond a simple comparison between models with different training durations: the two semantic objectives exhibit distinct and complementary effects across evaluation metrics, rather than merely producing uniform improvements.

3. Robustness Under Incomplete Cross-modal Correspondence

We would also like to clarify the intended objective of segment-level audio-video contrastive learning.

For weakly correlated audio-video content, such as entertainment videos accompanied by background music (BGM), there may be no meaningful fine-grained correspondence between individual audio and video segments. Their relationship often exists at a more holistic level, such as overall mood, style, or longer-term semantic consistency, rather than precise event-level synchronization.

Consequently, segment-level retrieval on such data does not necessarily reflect the quality of the representations for their intended purpose. The substantially larger retrieval improvement under strong correspondence (+15.9 points) than weak correspondence (+1.6 points) is therefore consistent with the nature of the supervision: segment-level contrastive learning is most informative when audio and video contain meaningful local temporal correspondences.

For this reason, our training pipeline filters audio-video data based on semantic similarity and synchronization scores to prioritize reliable cross-modal correspondence for AVCLIP training.

Importantly, our ultimate objective is not to maximize segment-level retrieval accuracy across all types of audio-video content, but to improve the learnability of audio-video synchronization in downstream joint generative models. Our downstream experiments demonstrate substantial synchronization improvements from this training strategy.

For weakly correlated content, holistic semantic alignment is often more appropriate than enforcing precise segment-level correspondence. This is also why OmniVAE incorporates per-modality semantic distillation alongside AVCLIP: the two objectives address different levels of audio-video correspondence and complement each other.

We agree that more adaptive correspondence-aware training strategies could be explored in the future. However, the current results support our central motivation of combining holistic semantic supervision with fine-grained temporal alignment for joint audio-video generation.

Overall, we sincerely appreciate the reviewer's constructive discussion. Our core contribution lies not in introducing a fundamentally new contrastive objective or negative sampling algorithm, but in demonstrating how holistic semantic alignment and fine-grained temporal alignment complement each other in learning generation-friendly audio-video representations.

Our four-way ablation and downstream generation results provide evidence that these two objectives improve different aspects of generation, and their combination achieves the strongest overall performance.

We thank the reviewer again for the valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.