- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses optimization conflicts between audio and video objectives when RL-based post-training a joint audio-video generation model. It trains separate specialized audio and video RL teachers and distills them into a shared student along student-generated trajectories. OmniOPD augments audio distillation with a lower-noise video condition for the audio teacher, multiple student rollouts per prompt, and a timestep-dependent weight based on teacher–student transition divergence. Experiments on an LTX-based model use JavisBench and VBench, compare against RL, cascade, and vanilla OPD baselines proved the effectiveness of the proposed method.
Critical strengths
- The paper conduct a comprehensive analysis of the audio-video coupling and the substantial token imbalance. The comparison of audio-only reward optimization, joint-reward optimization, and sequential video-to-audio optimization gives a concrete motivation for stabilizing audio supervision during joint learning.
- The modality-wise on-policy distillation formulation is clear: the frozen audio and video teachers supervise their respective modalities along student-generated joint trajectories. In privileged visual guidance, the audio teacher shares the student's audio state while receiving cleaner video context, directly reflecting the paper's motivation that visual context supports audio learning.
- Relative to Vanilla OPD in Table 2, OmniOPD improves VQ, AQ, TA-IB, and DeSync. These results provide evidence that the combined objective can improve several JavisBench measures.
- The supplementary videos offer qualitative examples of improved voice–character alignment in multi-character scenes. These properties matter for joint audio-video generation.
Critical weaknesses
- The benefit of grouped multi-noise sampling is not separated from its additional computation. Equation (11) averages independently sampled joint audio-video trajectories for each prompt, so reduced sampling variance is expected as increases. The paper does not give a comparison at equal total rollouts or teacher evaluations.
- The reported results do not fully establish the claimed balanced gains in audio quality, video quality, and audio-video synchronization. In Table 2, Cascade exceeds OmniOPD on JavisScore, AV-IB, AVHScore, and DeSync. No user study is reported, leaving modest metric differences and the qualitative claims about voice–character alignment and lip synchronization insufficiently validated.
Questions for authors
- At equal total student-generated joint trajectories, teacher evaluations, and optimizer updates, does grouped multi-noise sampling still outperform Vanilla OPD or a larger ordinary batch?
- Because audio-video quality, voice–character alignment, correct speaker attribution, and lip synchronization are partly perceptual, could the authors provide a user study on the same prompts comparing OmniOPD with Vanilla OPD and Cascade?
We thank the reviewer for the insightful comments on grouped sampling and perceptual validation, which helped us strengthen the experimental evidence.
W1,Q1: Grouped sampling under matched budgets.
We therefore ran a strictly budget-matched comparison. All variants are built on the baseline with identical settings, and use the same numbers of student trajectories (6 per GPU per epoch), teacher evaluations, and optimizer updates. The only difference is how the budget is allocated:
(i) Larger batch: 3 video + 3 audio trajectories from distinct prompts;
(ii) Larger audio batch: 2 video + 4 audio trajectories from distinct prompts;
(iii) Grouped: 2 video + 2 audio prompts × 2 noises.
As shown in Table, under the same budget, grouping is best on 6 of 10 metrics, including VQ, CLAP, AV-IB, AVH, and JavisScore, while DeSync is slightly worse. This shows that the benefit comes from multiple noises per prompt rather than from extra compute.
W2,Q2: User study.
We conducted a blind pairwise user study and have added full details to Appendix A.2. We use 104 test prompts covering multi-character dialogue, natural scenery, film/TV scenes, animation, and music performance, and generate all videos under identical settings. In each trial, 10 participants compare our result against LTX-2.3, Cascade, Vanilla OPD, or OmniNFT (randomized placement, anonymized files) on visual quality, audio quality, lip-sync, and voice-character alignment. The last two aspects allow N/A when no speaking character appears. In total, the study comprises 300 judgments, balanced across competitors and scenarios. Our method is preferred over all competitors on all four aspects (p<0.01, two-sided sign test with ties discarded), with an 85.7% overall win rate on visual quality and only 3.3% / 2.7% losses on lip-sync / voice-character alignment.