acceptodds
Under review as a conference paper at ICLR 2027

OmniOPD: Stable On-Policy Distillation for Joint Audio-Video Generation

Abstract

Reinforcement learning (RL) elevates joint audio-video generation, but optimizing heterogeneous, modality-specific audio and video rewards often introduces cross-modal interference. On-policy distillation (OPD) alleviates this issue by transferring the capabilities of specialized audio and video RL teachers into a single student model. However, a significant imbalance in cross-modal token counts leaves audio with sparse supervision, causing unstable distillation. We observe that naively increasing the audio loss weight fails to solve this problem since audio learning relies heavily on visual context, indicating richer visual cues are crucial to offset sparse audio signals. To address this, we propose OmniOPD, a modality-wise on-policy distillation framework designed for balanced and stable audio-video generation. First, we construct privileged visual information using asymmetric noise scheduling, allowing the audio teacher to generate more informative targets from cleaner video context. Second, we introduce grouped multi-noise sampling to expand audio supervision coverage and stabilize training. Third, we introduce timestep-wise importance weighting based on teacher-student transition divergence, preserving reliable supervision while down-weighting unreliable learning signals, to stabilize fluctuations in audio learning. Evaluated on JavisBench and VBench demonstrate OmniOPD preserves audio quality while significantly enhancing video generation, achieving effective joint distillation.

Primary area:
generative models
Keywords:
Joint Audio-Video Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dd37OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses optimization conflicts between audio and video objectives when RL-based post-training a joint audio-video generation model. It trains separate specialized audio and video RL teachers and distills them into a shared student along student-generated trajectories. OmniOPD augments audio distillation with a lower-noise video condition for the audio teacher, multiple student rollouts per prompt, and a timestep-dependent weight based on teacher–student transition divergence. Experiments on an LTX-based model use JavisBench and VBench, compare against RL, cascade, and vanilla OPD baselines proved the effectiveness of the proposed method.

Critical strengths
  1. The paper conduct a comprehensive analysis of the audio-video coupling and the substantial token imbalance. The comparison of audio-only reward optimization, joint-reward optimization, and sequential video-to-audio optimization gives a concrete motivation for stabilizing audio supervision during joint learning.
  2. The modality-wise on-policy distillation formulation is clear: the frozen audio and video teachers supervise their respective modalities along student-generated joint trajectories. In privileged visual guidance, the audio teacher shares the student's audio state while receiving cleaner video context, directly reflecting the paper's motivation that visual context supports audio learning.
  3. Relative to Vanilla OPD in Table 2, OmniOPD improves VQ, AQ, TA-IB, and DeSync. These results provide evidence that the combined objective can improve several JavisBench measures.
  4. The supplementary videos offer qualitative examples of improved voice–character alignment in multi-character scenes. These properties matter for joint audio-video generation.
Critical weaknesses
  1. The benefit of grouped multi-noise sampling is not separated from its additional computation. Equation (11) averages GG independently sampled joint audio-video trajectories for each prompt, so reduced sampling variance is expected as GG increases. The paper does not give a comparison at equal total rollouts or teacher evaluations.
  2. The reported results do not fully establish the claimed balanced gains in audio quality, video quality, and audio-video synchronization. In Table 2, Cascade exceeds OmniOPD on JavisScore, AV-IB, AVHScore, and DeSync. No user study is reported, leaving modest metric differences and the qualitative claims about voice–character alignment and lip synchronization insufficiently validated.
Questions for authors
  1. At equal total student-generated joint trajectories, teacher evaluations, and optimizer updates, does grouped multi-noise sampling still outperform Vanilla OPD or a larger ordinary batch?
  2. Because audio-video quality, voice–character alignment, correct speaker attribution, and lip synchronization are partly perceptual, could the authors provide a user study on the same prompts comparing OmniOPD with Vanilla OPD and Cascade?
1 comment

We thank the reviewer for the insightful comments on grouped sampling and perceptual validation, which helped us strengthen the experimental evidence.

W1,Q1: Grouped sampling under matched budgets.

We therefore ran a strictly budget-matched comparison. All variants are built on the baseline with identical settings, and use the same numbers of student trajectories (6 per GPU per epoch), teacher evaluations, and optimizer updates. The only difference is how the budget is allocated:

(i) Larger batch: 3 video + 3 audio trajectories from distinct prompts;

(ii) Larger audio batch: 2 video + 4 audio trajectories from distinct prompts;

(iii) Grouped: 2 video + 2 audio prompts ×\times 2 noises.

As shown in Table, under the same budget, grouping is best on 6 of 10 metrics, including VQ, CLAP, AV-IB, AVH, and JavisScore, while DeSync is slightly worse. This shows that the benefit comes from multiple noises per prompt rather than from extra compute.

Setting (100 updates)VQ↑AQ↑TV-IB↑TA-IB↑CLIP↑CLAP↑AV-IB↑AVH↑Javis↑DeSync↓
Larger batch3.3355.460.2734.1908.3098.4449.2596.2550.2305.4734
Larger audio batch3.3345.476.2744.1915.3101.4432.2601.2557.2305.4516
Grouped3.3645.469.2734.1916.3098.4466.2616.2575.2329.4766

W2,Q2: User study.

We conducted a blind pairwise user study and have added full details to Appendix A.2. We use 104 test prompts covering multi-character dialogue, natural scenery, film/TV scenes, animation, and music performance, and generate all videos under identical settings. In each trial, 10 participants compare our result against LTX-2.3, Cascade, Vanilla OPD, or OmniNFT (randomized placement, anonymized files) on visual quality, audio quality, lip-sync, and voice-character alignment. The last two aspects allow N/A when no speaking character appears. In total, the study comprises 300 judgments, balanced across competitors and scenarios. Our method is preferred over all competitors on all four aspects (p<0.01, two-sided sign test with ties discarded), with an 85.7% overall win rate on visual quality and only 3.3% / 2.7% losses on lip-sync / voice-character alignment.

Ours vs.Visual Quality (W / T / L)Audio Quality (W / T / L)Lip-Sync (W / T / L)Voice–Character Align. (W / T / L)
LTX-2.384.4 / 6.5 / 9.146.8 / 37.7 / 15.659.4 / 31.3 / 9.443.8 / 50.0 / 6.3
Cascade87.8 / 4.1 / 8.159.5 / 24.3 / 16.248.9 / 51.1 / 0.048.9 / 46.8 / 4.3
Vanilla OPD86.5 / 6.8 / 6.851.4 / 37.8 / 10.857.6 / 39.4 / 3.048.5 / 51.5 / 0.0
OmniNFT84.0 / 8.0 / 8.056.0 / 33.3 / 10.750.0 / 47.4 / 2.647.2 / 52.8 / 0.0
Overall85.7 / 6.3 / 8.053.3 / 33.3 / 13.353.3 / 43.3 / 3.347.3 / 50.0 / 2.7
Reviewer_QTr9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work systematically analyzes the key challenges in joint audio-video generation and addresses them with on-policy distillation (OPD), which informs both the training procedure and the model design.

Critical strengths

The authors systematically analyse the key challenges in joint audio-video generation and their underlying causes, and this analysis naturally motivates an on-policy distillation (OPD)-based approach to address them.

Critical weaknesses
  1. The experimental design is incomplete: it lacks comparisons with off-policy distillation baselines and a user study to validate the perceptual improvements.
  2. On-policy training assumes the teacher provides reliable guidance on student-visited states. Early in training, however, student samples may lie far outside the teacher’s training distribution, making its scores unreliable. This is especially true for the audio teacher, which is trained on less data and generalizes worse out of distribution. The resulting erroneous signals can destabilize early training or steer the student in the wrong direction.
Questions for authors
  1. The paper lacks key implementation details, such as hyperparameters, needed to reproduce and verify the results.
  2. The paper does not compare against off-policy distillation, which is necessary to justify the on-policy design. The authors should also report GPU memory and training overhead, and ideally compare methods under matched computational budgets.
  3. No user study is provided. Since automatic metrics for audio quality and audio-visual synchronization correlate poorly with human perception, a user study would strengthen the claimed improvements.
  4. In Table 1, the student outperforms both the audio and video teachers in text consistency. The authors should explain this result, for example by checking whether it stems from differences in CFG scale or evaluation settings, or from reduced sample diversity.
  5. How does the proposed multi-teacher design differ from existing multi-teacher distillation and expert-stitching approaches (e.g., UniVerse-1), and what aspects of it are specific to joint audio-video generation?
3 comments

We thank the reviewer for the constructive and helpful comments.

W1, Q2: Off-policy distillation and overhead.

We fine-tune the base model with the standard flow-matching loss on teacher samples, each prompt generated by either the video or the audio teacher, using the same LoRA setting as OmniOPD.

ModelVQ↑AQ↑TV-IB↑TA-IB↑CLIP↑CLAP↑AV-IB↑AVHScore↑JavisScore↑DeSync↓
LTX-2.32.7145.2520.2790.1780.3080.4290.2560.2510.2220.584
Vanilla OPD3.1745.3420.2750.1800.3090.4330.2560.2500.2260.583
SFT3.4445.2200.2770.1900.3140.4230.2450.2410.2140.480
OmniOPD3.3475.4490.2750.1920.3100.4470.2620.2580.2320.457

SFT (800 steps) improves visual quality but drops AQ, AV-IB, AVHScore, and JavisScore below the base model. With 100 updates, OmniOPD is best on 7 of 10 metrics, showing that on-policy distillation transfers joint audio-video capabilities more effectively than off-policy imitation.

Each teacher takes 800 steps / 40 h (320 GPU-h, ~130 GB/GPU, and OmniOPD takes 100 steps / 11 h (88 GPU-h, ~100 GB/GPU). Both teachers are LoRA adapters on one frozen base, so distillation needs no extra base-model copy. SFT is trained for 800 steps, which takes about 8 hours (64 GPU-hours) with a peak memory of 90 GB per GPU.

W1, Q3: User study.

In a blind pairwise study (10 participants, 300 comparisons, 104 prompts) against LTX-2.3, Cascade, Vanilla OPD, and OmniNFT, OmniOPD is preferred on all four aspects (p<0.01), with an 85.7% win rate on visual quality and only 3.3%/2.7% losses on lip-sync/voice-character alignment. More details are shown in Table 4 and Appendix A.2.

Ours vs.Visual Quality (W / T / L)Audio Quality (W / T / L)Lip-Sync (W / T / L)Voice–Character Align. (W / T / L)
LTX-2.384.4 / 6.5 / 9.146.8 / 37.7 / 15.659.4 / 31.3 / 9.443.8 / 50.0 / 6.3
Cascade87.8 / 4.1 / 8.159.5 / 24.3 / 16.248.9 / 51.1 / 0.048.9 / 46.8 / 4.3
Vanilla OPD86.5 / 6.8 / 6.851.4 / 37.8 / 10.857.6 / 39.4 / 3.048.5 / 51.5 / 0.0
OmniNFT84.0 / 8.0 / 8.056.0 / 33.3 / 10.750.0 / 47.4 / 2.647.2 / 52.8 / 0.0
Overall85.7 / 6.3 / 8.053.3 / 33.3 / 13.353.3 / 43.3 / 3.347.3 / 50.0 / 2.7

W2: Teacher reliability on student states.

We agree and discuss this in the Limitations section. Teachers may give unreliable supervision on student states outside their training distribution. This is an open problem also reported for LLM OPD (e.g., TrOPD) but little studied in joint audio-video generation. Our design partly mitigates it: timestep-wise importance weighting down-weights steps with large teacher-student mismatch, and grouped multi-noise sampling averages supervision over multiple trajectories. These measures do not guarantee correct supervision; a systematic analysis, especially for audio, is left to future work.

Q1: Implementation details.

We have added a full hyperparameter table (Table 5) and implementation details in Appendix A.3. Key settings: All models use bf16 and FSDP.

Video TeacherAudio TeacherOmniOPD
LoRA rank / alpha32 / 6432 / 6432 / 64
OptimizerAdamW, lr 3e-5, wd 1e-4, grad clip 1.0samesame, EMA 0.9
Steps / GPUs800 / 8800 / 8100 / 8
Trajectories per update161616
Reward weightsVideoAlign 1.0, HPSv3 1.5, Sync 1.0Audiobox 1.0, CLAP 1.0, Sync 1.0–
Advantage clipping[-5, 5][-5, 5]–
CFG (video / audio)3.0 / 7.03.0 / 7.03.0 / 7.0

Q4: Student text consistency vs. teachers.

All models in Table 1 use identical evaluation settings (same prompts, pipeline, 30 sampling steps, CFG 3.0/7.0). The student exceeds both teachers only on TA-IB and CLIP; it is slightly lower on TV-IB and below the audio teacher on CLAP. A plausible cause is reduced sample diversity: distilling CFG-guided teacher targets can sharpen the student toward prompt-aligned modes.

Q5: Relation to expert stitching and multi-teacher distillation.

UniVerse-1 stitches pretrained unimodal experts (Wan2.1, Ace-step) at the block level and trains them with supervised flow matching on paired data. OmniOPD is post-training: it keeps one joint architecture, needs no paired data, and distills reward-tuned teachers along student rollouts, so the two are complementary.

Unlike conventional multi-teacher distillation, where teachers cover different tasks and each supervises the full output, our teachers supervise the same joint sample, each primarily on its own modality. Because the streams are coupled through cross-attention, the audio loss also updates parameters the video depends on, causing the interference analyzed in Sec. 3.

thanks for the response. My concerns are addressed.

Thank you again for your time and for helping us improve this work!

Reviewer_UhYMOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

OmniOPD integrates separately optimized audio and video teachers into a joint audio-video student through modality-wise on-policy distillation. Motivated by unstable audio optimization and an imbalance in modality token counts, the method combines three components: cleaner visual conditioning for the audio teacher through asymmetric noise scheduling, grouped multi-noise sampling, and timestep-wise weighting based on teacher–student transition divergence. Experiments on JavisBench and VBench compare the method with joint RL, sequential optimization, and vanilla OPD, and investigate its individual components.

Critical strengths
  1. The paper addresses interference and unstable audio learning when integrating modality-specialized teachers.
  2. Privileged visual guidance, grouped sampling, and reliability weighting target complementary aspects of audio supervision.
  3. OmniOPD improves quality and alignment over vanilla OPD, with supporting ablations and complementary VBench results.
Critical weaknesses
  1. The synchronization reward is insufficiently specified. Section 6.1 identifies DeSync as its source, but the exact transformation into a reward remains unclear. Please specify its sign, normalization or clipping, temporal sampling, and aggregation, together with its weighting during teacher training. These details matter because synchronization is both an optimization target and an evaluation criterion.
  2. The comparison with Cascade needs an explicit trade-off analysis. In Table 2, OmniOPD has lower AV-IB (0.262 vs. 0.274), AVHScore (0.258 vs. 0.269), and JavisScore (0.232 vs. 0.241), and worse DeSync (0.457 vs. 0.432; lower is better). It improves several quality measures, but loses consistency and synchrony to Cascade. Please explain these losses and qualify the claim of the best overall audio-video balance accordingly.
  3. The abstract's causal conclusion needs qualification. Video-first optimization changes initialization and shared parameters, so its stabilizing effect does not isolate visual conditioning alone. The noise-shift ablation supports cleaner teacher-side visual context, but does not establish that richer visual cues specifically compensate for sparse audio tokens. Please distinguish these observations from the stronger claim in the abstract that such cues are crucial. A more targeted control or a narrower interpretation would resolve this concern.
Questions for authors

please see weaknesses

3 comments

We thank the reviewer for the constructive comments on the synchronization reward, the trade-off with Cascade, and our causal claims, which helped us clarify and qualify the paper.

W1: Implementation details of the synchronization reward.

We thank the reviewer and will add the following details to the appendix. Our synchronization reward reimplements the DeSync protocol of JavisBench and uses the same pretrained Synchformer.

Synchformer classifies the audio-video offset into 21 classes over [-2, 2] s, with 0.2 s spacing.

Sign/transform: For each block j, we take the absolute value of the top-1 offset, ∣δj∣|\delta_j|, and average over blocks to obtain dd. The reward is rsync=1/(1+d)r_{sync} = 1/(1+d), which lies in [1/3,1][1/3, 1]; higher values mean better synchronization. Table 2 reports dd itself (DeSync, lower is better).

Temporal sampling: Generated clips (121 frames at 24 fps, ~5 s) are scored in memory without resampling. Video is resized to 224x224 and audio is converted to 16 kHz mono; both are zero-padded to 8 s. Video segments are 16 frames with a stride of 8, and audio segments are 0.64 s (128-bin log-mel) with a stride of 0.32 s.

Aggregation: Every 14 paired segments form one block, an incomplete final block uses the last 14 segments, and block offsets are averaged uniformly.

Normalization/clipping: rsyncr_{sync} is not clipped. Each reward is normalized separately (prompt-group mean, batch std + 1e-4), and only the weighted sum of advantages is clipped to [-5, 5].

Weights: Sync has weight 1.0 in both teachers. The full weights are VideoAlign:HPSv3:sync = 1:1.5:1 for the video teacher and AudioBox:CLAP:sync = 1:1:1 for the audio teacher.

W2: Trade-off with Cascade.

We agree that the "best overall balance" claim is too strong. Cascade leads on AV-IB, AVHScore, JavisScore, and DeSync, whereas OmniOPD improves VQ (3.347 vs. 3.131), AQ (5.449 vs. 5.235), and TA-IB (0.192 vs. 0.179). Cascade directly optimizes the full model with the synchronization reward, which is closely tied to these metrics, at the cost of audio quality (its AQ is below the base model's 5.252). OmniOPD receives such signals only through distillation, and its consistency remains below that of its teachers (AV-IB 0.262 vs. 0.275/0.282).

We conducted a blind pairwise user study and have added full details to Appendix A.2. We use 104 test prompts covering multi-character dialogue, natural scenery, film/TV scenes, animation, and music performance, and generate all videos under identical settings. In each trial, 10 participants compare our result against LTX-2.3, Cascade, Vanilla OPD, or OmniNFT (randomized placement, anonymized files) on visual quality, audio quality, lip-sync, and voice-character alignment. In total, the study comprises 300 judgments, balanced across competitors and scenarios. For lip-sync and voice-character alignment, raters could select N/A for clips without speech.

Ours vs.Visual Quality (W / T / L)Audio Quality (W / T / L)Lip-Sync (W / T / L)Voice–Character Align. (W / T / L)
LTX-2.384.4 / 6.5 / 9.146.8 / 37.7 / 15.659.4 / 31.3 / 9.443.8 / 50.0 / 6.3
Cascade87.8 / 4.1 / 8.159.5 / 24.3 / 16.248.9 / 51.1 / 0.048.9 / 46.8 / 4.3
Vanilla OPD86.5 / 6.8 / 6.851.4 / 37.8 / 10.857.6 / 39.4 / 3.048.5 / 51.5 / 0.0
OmniNFT84.0 / 8.0 / 8.056.0 / 33.3 / 10.750.0 / 47.4 / 2.647.2 / 52.8 / 0.0
Overall85.7 / 6.3 / 8.053.3 / 33.3 / 13.353.3 / 43.3 / 3.347.3 / 50.0 / 2.7

All differences are significant (p < 0.01, sign test). Against Cascade, OmniOPD wins 87.8% on visual quality and has no losses on lip-sync. We will revise the claim to: OmniOPD achieves the highest visual quality and improves audio quality over the base and distillation baselines, with lip-sync preferred or tied by human raters.

W3. Causal claims.

We thank the reviewer and clarify the intended roles. We agree that video-first optimization changes the initialization and shared parameters, so we use it only as motivation, not as causal evidence. The direct support is the noise-shift ablation (Table 3a): with all else fixed, cleaner teacher-side video improves AQ (5.342 to 5.376) and TA-IB (0.180 to 0.189) over matched noise.

We will state this narrower, ablation-backed claim. Privileged visual guidance is designed to stabilize audio learning by giving the audio teacher cleaner visual context, which yields more informative targets. Sparse audio supervision is addressed by grouped multi-noise sampling, which expands audio supervision coverage (Sec. 4.2). The abstract sentence "richer visual cues are crucial to offset sparse audio signals" conflated these two roles. We have revised it to "indicating that richer visual cues are crucial for stabilizing audio learning."

Thanks for the response. I have no further questions.

Thank you again for your time and for helping us improve this work!

Comments on the paper

2 comments

We thank you and all reviewers for the constructive feedback. Below we summarize the new experiments and revisions. All changes are highlighted in the revised PDF.

New experiments

  1. Off-policy distillation baseline (QTr9). We add an SFT baseline that fine-tunes LTX-2.3 on teacher-generated samples with the standard flow-matching loss (Table 2). SFT improves visual quality but degrades audio quality and audio-video consistency below the base model, while OmniOPD outperforms SFT on 7 of 10 metrics. This supports the on-policy design.

  2. Grouped sampling under matched budgets (dd37). We compare grouped multi-noise sampling against a larger batch and a larger audio batch with the same numbers of student trajectories, teacher evaluations, and optimizer updates (Appendix A.4, Table 6). Grouped sampling is best on 6 of 10 metrics, showing that its benefit comes from multiple noises per prompt rather than from extra compute.

  3. Human evaluation (UhYM, QTr9, dd37). We conduct a blind pairwise user study (10 participants, 300 comparisons, 104 prompts) against LTX-2.3, Cascade, Vanilla OPD, and OmniNFT (Table 4, Appendix A.2). OmniOPD is preferred on all four aspects (p<0.01), with an 85.7% win rate on visual quality and only 3.3%/2.7% losses on lip-sync/voice-character alignment.

Revisions

  1. Qualified claims (UhYM). We remove the "best overall audio-video balance" claim and explicitly state that Cascade retains an advantage on embedding-based consistency and DeSync (Sec. 6.2). We also revise the abstract to separate the roles of the two components: richer visual context stabilizes audio learning, while grouped multi-noise sampling expands sparse audio supervision. The video-first analysis is presented as motivation, and the noise-shift ablation (Table 3a) is cited as direct evidence.

  2. Reproducibility (UhYM, QTr9). We add a full hyperparameter table for both teachers and the student (Table 5), the synchronization reward definition, reward normalization and clipping, the OmniOPD component settings (s, G, p), and training cost and GPU memory (Appendix A.3).

  3. Limitations (QTr9). We add a Limitations section (Sec. 8) on teacher reliability on out-of-distribution student states, and discuss how timestep-wise importance weighting and grouped sampling partly mitigate it.

  4. Related work (QTr9). We add a discussion of expert stitching (UniVerse-1), and multi-teacher distillation (MOPD, Open-MOPD) (Appendix A.6).

  5. Clarifications in the responses. We clarify that all models in Table 1 use identical evaluation settings, and discuss reduced sample diversity as a plausible cause of the student's higher text consistency.

We hope these revisions address the reviewers' concerns.

AuthorsOpenReview ↗

Summary of Contributions and Revisions in Response to the Reviews

We sincerely thank all reviewers for their constructive feedback, which has helped us strengthen both the experimental evidence and the presentation of our work. We would like to summarize the main contributions of OmniOPD:

  1. Analysis of joint audio–video post-training. We systematically investigate optimization interference between audio and video objectives and the instability of audio optimization. These observations motivate a training strategy that preserves modality-specific improvements while accounting for interactions.

  2. Modality-specific on-policy distillation. OmniOPD integrates separately optimized audio and video teachers into a single joint student through supervision on student-generated trajectories. Privileged visual conditioning, grouped multi-noise sampling, and timestep-dependent reliability weighting address complementary challenges in transferring audio supervision within joint audio–video generation.

  3. Empirical validation with additional controlled experiments. Our evaluations on JavisBench and VBench are complemented by an off-policy SFT baseline, comparisons with larger batches under matched rollout and teacher-evaluation budgets, and a blinded human evaluation.

In response to the reviews, we have provided point-by-point answers, incorporated the additional experiments, expanded implementation and computational-cost details, and clarified the interpretation of our findings. The corresponding revisions are highlighted in the updated manuscript.

We believe that OmniOPD offers a useful methodological contribution to joint audio–video post-training, supported by the strengthened experimental evidence.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.