- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OmniVAE, a jointly trained audio-video VAE framework whose audio and video latent spaces are encouraged to be semantically meaningful and temporally aligned. In addition to standard modality-specific reconstruction objectives, the method uses modality-specific semantic distillation from frozen Qwen3-Omni encoders and a segment-level bidirectional audio-video contrastive loss. The resulting VAEs are then used as frozen tokenizers for downstream text-to-audio-video generation, where the authors report improved audio quality and cross-modal synchronization while largely preserving reconstruction and video quality. The paper also introduces an audio-video desynchronization probing task to evaluate fine-grained alignment directly on frozen latents.
Critical strengths
- This paper addresses a question about what audio and video representations should be like for audio-video joint latent generation. The problem is interesting, and the community can benefit from this topic.
- Most parts of the framework design are reasonable, and the experimental results demonstrate its effectiveness.
Critical weaknesses
-
A key concern is the interpretation of the semantic distillation objective and its dependence on Qwen3-Omni. The paper frames the distillation objective as purely intra-modal, because the video VAE is distilled from a visual encoder and the audio VAE from an audio encoder. While this is true at the level of the loss formulation, the teachers are taken from Qwen3-Omni, a unified multimodal model. Therefore, their intermediate representations may already contain cross-modal structure, or at least may be shaped to be compatible with a shared multimodal system. This makes it unclear whether the benefit of distillation should be attributed to modality-specific semantic enrichment, implicit cross-modal structure in the teacher encoders, or a combination of both. This ambiguity matters because one of the paper’s central claims is about the complementary roles of intra-modal semantic distillation and cross-modal contrastive alignment. Without an ablation using independently trained modality-specific teachers, e.g., a video encoder such as V-JEPA 2.1 and a separately trained audio encoder, it is difficult to tell whether the observed benefit comes from general semantic enrichment or from the particular cross-modal structure induced by Qwen3-Omni. Is the use of mutually compatible encoders from a unified multimodal model important, or would independent modality-specific teachers work equally well? The current manuscript does not clarify this requirement.
-
Some methodological details are underspecified, and the current validation does not fully support a clean attribution of the reported gains. This makes it difficult to fully assess whether the paper’s central claims are supported by the evidence.
- The hierarchical negative construction would benefit from a more formal description. The paper states that sibling-clip negatives are sampled from temporally disjoint clips of the same source video, but the exact sampling procedure is unclear: how the sibling clip is selected, whether multiple sibling clips are used, how temporal overlap is avoided, and how false negatives are handled when similar events recur in the same source video. A diagram and a mathematical definition of the three negative types would substantially improve the paper’s clarity.
- The effect of the Stage-3 audio decoder fine-tuning is not fully isolated. According to the training recipe, in Stage 3, the audio encoder is frozen, and only the decoder is refined with adversarial and feature-matching losses. This affects waveform-level audio quality in downstream T2AV generation. Since audio quality is one of the main reported benefits, an ablation comparing the models before and after Stage 3 would help separate improvements due to a more learnable or better aligned latent space from improvements due to the final decoder refinement.
- The downstream T2AV generator is used to demonstrate that OmniVAE latents improve generation quality and cross-modal synchronization. However, its architecture and training recipe are described only briefly. Since the reported improvements are measured after training a nontrivial joint generator on top of the proposed VAE, more details are needed to understand whether the gains come from the tokenizer alone or from interactions with the particular generator design.
Questions for authors
- Related to Weakness 1, I would appreciate it if the authors could answer what kind of information the authors expect each of the Qwen3-Omni encoders to offer to each modality's representation in the OmniVAE latent space, more specifically.
- Related to Weakness 2, I would appreciate it if the authors could clarify how "Sibling-clip negatives" are constructed in detail.
- In Equation (2), is used for aligning the latent representation with . Why did the authors not use just or (: temperature)? Explaining the design choice would be appreciated.
- L.287 says "layer 27 of the Qwen3-Omni (Xu et al., 2025) visual encoder and layer 18 of its audio encoder as supervision" are used, but the reason why those layers are selected is not explained. Were they selected empirically? Or is there any previous work suggesting them? If they were empirically chosen, ablation studies and discussions would be appreciated.
- Could the authors report an ablation of Stage 3? Since Stage 3 fine-tunes only the audio decoder with adversarial and feature-matching losses, evaluating the model after Stages 1 and 2 but before Stage 3 would help separate improvements due to the aligned latent space from those due to the final decoder refinement.
7 comments
Q1. What does semantic distillation transfer, and does it require a unified multimodal teacher?
Thank you for identifying this ambiguity. Although the two distillation losses are applied separately to the visual and audio branches, their teachers come from Qwen3-Omni, whose pretraining may already impart cross-modal structure. We will replace “uni-modality semantic distillation” with “modality-specific semantic distillation.” This describes where supervision is applied without implying that the teachers were pretrained independently of other modalities.
The visual supervision transfers object-, scene-, and local spatial information into visual latents. The audio supervision transfers sound-event and speech-related features and their temporal variation. These feature targets provide additional structure beyond pixel- or waveform-level reconstruction.
Computationally, each student is matched to its corresponding teacher, without directly comparing audio and video latents in the distillation loss. AVCLIP explicitly compares paired audio-video segment features to learn temporal-instance correspondence. The two objectives therefore provide complementary feature-level semantic supervision and explicit cross-modal alignment.
Our experiments include distillation alone, AVCLIP alone, and their combination. The following results summarize the frozen probe in Table 3 and downstream generation in Table 4:
| Configuration | Probe A@1 ↑ | DeSync ↓ | LSE-C ↑ | MusiQ ↑ | ManiQA ↑ | Audio IS ↑ | Audio FD ↓ | WER ↓ |
|---|---|---|---|---|---|---|---|---|
| Recon | 6.4 | 0.884 | 1.479 | 0.503 | 0.338 | 3.041 | 1.029 | 0.243 |
| Recon + Distill | 7.1 | 0.814 | 1.450 | 0.523 | 0.359 | 3.604 | 0.959 | 0.205 |
| Recon + AVCLIP | 18.1 | 0.576 | 1.970 | 0.503 | 0.341 | 3.598 | 0.932 | 0.172 |
| OmniVAE | 20.2 | 0.570 | 2.093 | 0.515 | 0.355 | 4.001 | 0.875 | 0.168 |
Distillation improves several modality-quality metrics, including MusiQ, ManiQA, audio IS, FD, and WER. AVCLIP provides the main temporal-alignment gains: probe A@1 increases from 6.4% to 18.1%, and DeSync decreases from 0.884 to 0.576, alongside improved audio generation. Their combination obtains the best synchronization and audio results among these configurations, demonstrating their complementary contributions.
The formulation requires position-matched features from each teacher; it does not require shared teacher parameters, equal feature dimensions, or joint multimodal pretraining. In our implementation, Qwen3-Omni provides strong visual and audio semantic targets within one unified teacher family, and the ablations above show the distinct gains from this modality-semantic supervision and from AVCLIP.
Q2. How are sibling-clip negatives sampled, and how are repeated events treated?
We will clarify the sampling procedure and formal definition. Preprocessing partitions each source video into temporally non-overlapping clips and retains its source-video identity. Data organization aims to make at least one additional clip from the same source available for each anchor in the candidate pool; selection among available siblings is random. The negative sampler then samples segments uniformly without replacement from all eligible same-source, different-clip candidates. Consequently, selected sibling segments may come from one or multiple sibling clips.
Let denote clip 's source-video identity, its temporal interval in the source, and the distributed candidate pool. For anchor , the positive is in the other modality. The three negative candidate sets are:
The default setting retains all 47 intra-clip negatives and randomly selects 48 sibling and 24 cross-video segments. Non-overlap is guaranteed by preprocessing rather than by random sampling itself, and the implementation identifies siblings through source identity and distinct clip indices.
For repeated events, semantic category identity should be distinguished from temporal-instance correspondence. In synchronized data, the audio and video from the same temporal window form the ground-truth pair. Another window containing the same event category, such as drumming, may differ in impact timing, rhythmic phase, or action progression. It can therefore remain a valid hard negative. Removing all such examples could let the model match the category “drum” without learning when an impact occurs.
We therefore use repeated events with different impact timings, rhythmic phases, or action progressions as hard negatives for temporal alignment, encouraging the model to learn correspondence between specific event instances.
Summarization of Review and Overview of Our Responses
We thank the reviewer for the detailed assessment and concrete suggestions. We appreciate your recognition of the value of studying audio-video representations for joint generation, your assessment that most of the framework design is reasonable, and your observation that the experiments demonstrate its effectiveness. Your discussion of teacher provenance, sampling, and implementation details helps us make the methodological description and experimental interpretation more precise.
We first clarify the roles and terminology of the two additional objectives. Distillation supplies teacher-feature supervision separately to the visual and audio latents, while AVCLIP explicitly learns correspondence between audio-video segments. We will replace “uni-modality semantic distillation” with “modality-specific semantic distillation” to avoid implying that the Qwen3-Omni teachers were pretrained independently of other modalities. We use the distillation-only, contrastive-only, and combined configurations to explain their practical contributions.
We organize your main concerns and minor comments as follows:
The following responses provide the results and explanations point by point, with corresponding additions to the main text and appendix.