acceptodds
Under review as a conference paper at ICLR 2027

Horses for Courses: Breaking Modality Symmetry in Few-Step Audio-Video Diffusion Distillation

Abstract

Unified audio-video diffusion transformers generate synchronized video and audio, but the prohibitive cost of iterative sampling necessitates aggressive few-step distillation. Prevailing distillation practice inherits a modality-symmetric recipe, indiscriminately applying identical distribution matching (DM) objectives and synchronous step schedules across both modalities. In this work, we reveal that this symmetric recipe systematically undermines the relational temporal structure of audio. Through a Fisher information analysis of Gaussian data along the diffusion path, we prove that re-noising suppresses information about cross-time correlations at a quadratic rate in the signal fraction , whereas information about first-order statistics decays only linearly, . Consequently, weak long-range acoustic dependencies receive an attenuated distributional gradient that is readily overwhelmed by the bias of the learned fake score. Teacher re-noising diagnostics confirm that audio loses disproportionately more structure than video at the same signal-to-noise ratio. To address this modality asymmetry, we propose Modality-Aware Asynchronous Distillation (MAAD), which breaks distillation homogeneity by decoupling optimization objectives and step schedules within a single shared student. Exploiting the substantial token-count disparity between modalities, asynchronous denoising inserts audio-only sub-steps that reuse the visual key-value cache, affording dense acoustic refinement at merely 6% additional computation. Student-context trajectory alignment (SCTA) evaluates the teacher's audio velocity directly under the student's detached visual cache, removing the cross-modal conditioning mismatch between the visual context of the target and that of the student. On MiniMax-H3 (33B), MAAD outperforms compared methods on all nine singing metrics, reducing MuQ FAD from 11.37 to 4.58, and matches or exceeds the 50-step teacher on four of six song-quality scores.

Primary area:
generative models
Keywords:
Joint Video-Audio Generation, Distillation Model, Multi-Modal Representation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MyshOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses few-step distillation of a unified audio-video diffusion model that jointly generates synchronized video and audio from one shared model. The authors argue that the prevailing modality-symmetric recipe undermines the temporal structure of audio, and propose Modality-Aware Asynchronous Distillation (MAAD), which decouples the objective and step schedule of each modality within one shared student through asynchronous denoising and student-context trajectory alignment (SCTA). On MiniMax-H3 (33B), MAAD outperforms released four-step distillations on all nine singing metrics, improving MuQ FAD from 11.37 to 4.58, and matches or exceeds the 50-step teacher on four of six song-quality scores.

Critical strengths
  1. Reasonable motivation: The motivation for treating the two modalities differently is sensible to me, and it provides a coherent starting point for the proposed design.

  2. Strong four-step performance on singing metrics: On the 33B MiniMax-H3 backbone, MAAD is the best four-step model across all nine Sing350 singing metrics, improving MuQ FAD from 11.37 to 4.58, and it matches or exceeds the 50-step teacher on four of six song-quality scores. Achieving this quality at four steps is the main consideration supporting acceptance.

  3. Low additional cost of asynchronous denoising: The asynchronous denoising scheme adds only about 6% extra computation, so the gains in audio quality do not come at a large efficiency price.

  4. Helpful demo: The demo video and website are helpful and support the credibility of the reported results.

Critical weaknesses

Major weaknesses

W1. The ablation may attribute the gain to extra audio steps rather than to the claimed mechanism: In Table 4, the asynchronous component alone raises the SongBench score from 4.784 to 4.926, whereas adding the remaining components of MAAD raises it only from 4.926 to 4.977, a much smaller increment than that of async-only over DM-only. This raises the question of whether the main improvement comes from giving audio more steps rather than from the claimed effect that the symmetric recipe erodes the temporal structure of audio. Because it bears directly on the paper's central claim, this is the single consideration that most affects my assessment.

Minor weaknesses

W2. Limited backbone coverage: The method is validated on a single backbone. Additional backbones would make the effectiveness of the proposed method more convincing.

W3. Reporting of fine-grained metrics may be selective: Table 2 highlights only the dimensions on which MAAD is strong (Coherence and Melody), whereas Table 7 shows that for the Instrumental dimension the Traj./DM setting is the highest, at 5.1538 versus 5.0940 for MAAD. The metric selection is therefore not fully comprehensive.

W4. Outdated references: Some arXiv references should be updated to their published versions, for example MusicRL (ICML).

Questions for authors

I am overall positive about this paper. My specific questions follow from the weaknesses listed above. The main one is whether the improvement in Table 4 should be attributed to the additional audio steps rather than to the claimed erosion of the temporal structure of audio under the symmetric recipe.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Mysh (1/2)

We thank the reviewer for the positive assessment, and for finding the motivation sensible and the four-step singing results strong.

W1. Is the gain due to extra audio steps rather than the claimed mechanism? Thank you for this key question, which bears directly on our central claim. Table 4 (Table 13 in new version) compares the two parts only through mean scores, which can hide gains concentrated in a few dimensions. We therefore compare the three Sing350 students dimension by dimension: DM/DM 4/4 → 4/20 (DM only → async only in Table 4 (Table 13 in new version)) changes only the steps, and DM/DM 4/20 → MAAD 4/20 changes only the objective. Parentheses: relative change from the row above.

SongBenchVocalInstr.MelodyStruct.Arr.MixingMusic.Mean
DM/DM 4/44.87265.04944.52874.38614.64575.02644.98154.7843
DM/DM 4/205.1263 (+5.21%)5.1321 (+1.64%)4.6869 (+3.49%)4.4569 (+1.62%)4.6739 (+0.61%)5.1548 (+2.56%)5.2477 (+5.34%)4.9255 (+2.95%)
MAAD 4/205.0907 (−0.69%)5.0940 (−0.74%)4.7688 (+1.75%)4.6235 (+3.74%)4.7342 (+1.29%)5.2143 (+1.15%)5.3158 (+1.30%)4.9773 (+1.05%)
SongEvalCoherenceMusicalityMemorabilityClarityNaturalnessMean
DM/DM 4/43.74963.77393.85143.69103.49953.7131
DM/DM 4/203.8386 (+2.37%)3.8602 (+2.29%)3.9233 (+1.87%)3.7713 (+2.18%)3.5647 (+1.86%)3.7916 (+2.11%)
MAAD 4/203.8755 (+0.96%)3.9090 (+1.26%)3.9617 (+0.98%)3.7912 (+0.53%)3.5905 (+0.72%)3.8256 (+0.90%)

The results show that both contribute, to different aspects: additional audio steps notably improve vocal clarity, while trajectory supervision improves melody and musical structure, the long-range dependencies our mechanism concerns. The limited audio quality of existing four-step models led us to give audio more steps, and the extra steps improve every SongBench dimension, including Vocal (+5.21%). At equal steps, DM/DM 4/20 keeps slightly clearer vocals, but MAAD improves Melody, Structure, Arrangement, Mixing and Musicality, and all five SongEval dimensions. Its Structure (+0.1666) and Arrangement (+0.0603) gains exceed those from extra steps (+0.0708 and +0.0282), so the small mean gain (+0.0518) combines larger gains in these musical dimensions with slight decreases in Vocal and Instrumental. The objective also raises Dynamic Degree from 8.9% to 51.1% (teacher 43.7%).

Does the objective's gain follow long-range structure? If the symmetric objective erodes the temporal structure of audio, replacing it should help most on prompts whose outputs depend most on long-range order. To measure this dependence, we cut the first 5 s of each teacher output into five 1 s blocks, randomly reorder them (eight random orders), and record how much the teacher's score drops (SongEval Coherence for audio, AMT for video; audio crossfades are matched so that only the order differs). We then split the 350 prompts into thirds by this drop (Low/Middle/High, separately for audio and video) and, within each third, compare MAAD with DM/DM at equal steps (4 video / 20 audio updates).

MAAD − DM/DMLow (n = 117)Middle (n = 116)High (n = 117)High − Low
SongBench Melody+0.0087+0.0943+0.1429+0.1342
SongEval Coherence+0.0306+0.0386+0.0415+0.0109
Video motion smoothness (AMT)−0.0022−0.0025−0.0030−0.0008
Video Visual Reality+0.0000+0.0000−0.0342−0.0342

The results show that, from low to high sensitivity, MAAD's Melody gain rises from +0.0087 to +0.1429 (95% CI of the High − Low difference: [+0.013, +0.243]) and its Coherence gain from +0.0306 to +0.0415, while video differences stay close to zero. Since both models use the same steps, this trend reflects the audio objective, consistent with the mechanism. Finally, the two parts are complementary: without extra steps, trajectory supervision lowers SongBench/SongEval to 4.32/3.59 (Traj. only in Table 4 (Table 13 in new version)), which is why MAAD pairs it with audio-only steps.

AuthorsOpenReview ↗

Response to Reviewer Mysh (2/2)

W2. Does MAAD transfer to another backbone? We appreciate this suggestion. We agree that a second backbone, especially one with a different architecture, makes the evidence more convincing. We therefore applied MAAD to LTX-2.5, which has separate video and audio streams coupled by cross-attention, whereas H3 is single-stream. An audio-only step on LTX-2.5 runs only the audio stream, which reads the cached video keys and values. We used the same schedule as on H3 (4 video / 20 audio updates; 4.12 token-normalized NFE) and compared with a symmetric DM/DM 4/4 student trained under the same setup.

VABenchVisual Reality↑Expressiveness↑Visual QA↑Alignment↑DeSync↓ViCLIP↑CLAP↑AudioBox↑DNSMOS↑
Teacher4.4704.3330.6244.4360.5630.1930.3243.5791.484
DM/DM 4/44.1824.3150.6464.2910.8700.1870.2913.4001.378
MAAD4.1514.3500.6424.3470.6120.1980.3243.4681.508
Sing350MuQ FAD↓CLAP FAD↓Onset F1↑Vocal↑Melody↑Instr.↑Musicality↑Coherence↑Clarity↑
Teacher–––4.9504.6015.1473.7453.8153.664
DM/DM 4/421.740.2450.6513.9414.8034.3533.2943.2813.188
MAAD17.820.1960.7114.6884.9354.5723.4913.4233.332

The results show that MAAD transfers to LTX-2.5. On Sing350, it improves all nine metrics, e.g., Vocal 3.941 → 4.688 and MuQ FAD 21.74 → 17.82. On VABench it improves seven of nine, including DeSync (0.870 → 0.612 s), while Visual Reality and Visual QA are lower (4.151 vs. 4.182; 0.642 vs. 0.646). Both students are further from the teacher than on H3, likely because the reduction in forward passes is larger: the guided LTX-2.5 teacher uses 120 passes, against 50 for H3.

W3. Is the reporting of fine-grained metrics selective? Thank you for the careful reading of Tables 7–8 (Tables 16–17 in new version). We agree that the paper should show the full picture. Table 2 (Table 4 in new version) reports the means over all dimensions plus Coherence and Melody, the dimensions most tied to long-range structure; all twelve dimensions are in Tables 7–8 (Tables 16–17 in new version). The full picture is a trade-off that is consistent with our analysis: DM favors component-level audio quality, whereas trajectory supervision better preserves long-range musical organization. DM on audio gives the best Vocal and Instrumental scores (Traj./DM first, DM/DM second). Trajectory supervision (MAAD) gives the best Melody, Structure, Arrangement, Mixing and Musicality, and all five SongEval dimensions. Relative to Traj./DM, MAAD is lower on Vocal (−0.0709) and Instrumental (−0.0598) and higher on Structure (+0.2780) and Melody (+0.1610). With audio-only steps, trajectory supervision keeps component-level quality close to DM and avoids the large drop of Traj. only in Table 4 (Table 13 in new version). Future work will investigate improved trajectory optimization to match DM on these dimensions while retaining the benefits for long-range musical organization. We have stated this trade-off in Sec. 5.4 (Sec. 5.4 in new version) and refer to Tables 17–18 (Tables 16–17 in new version) in the caption of Table 4 (Table 4 in new version).

W4. Outdated references. Thank you for pointing this out. We have cited MusicRL (ICML 2024) in Related Work (Sec. 2 in new version), alongside other audio generators trained with objectives beyond denoising (e.g., DMOSpeech and Presto), and update the arXiv references that have since been published.

We thank the reviewer again and hope the equal-step analyses address the main concern.

Dear Reviewer Mysh,

Thank you again for your constructive feedback. We have incorporated the following updates into the revised manuscript, with changes marked in red:

  1. Mechanism: Matched-step ablations show that the audio objective improves musical structure, with larger Melody gains on prompts with stronger long-range dependencies (Tables 3 and 5).
  2. Transfer: On LTX-2.5, MAAD improves all nine Sing350 metrics and seven of nine VABench metrics over DM/DM 4/4 (Table 11).
  3. Reporting: We explicitly discuss the Vocal/Instrumental versus musical-structure trade-off across all twelve dimensions, added MusicRL, and updated published references.

We hope these results address your concerns about the mechanism, transferability, and reporting,, and we would appreciate it if you could reconsider the score.

Reviewer_UrUhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at few-step distillation of a joint audio-video diffusion transformer. The authors argue that the usual recipe, which applies the same distribution matching (DM) objective and step schedule to both modalities, hurts audio. The reason is that re-noising wipes out information about temporal correlations faster than information about simpler statistics. Based on this, they propose MAAD. Video is trained with DM and audio with trajectory supervision. Audio also gets extra audio-only steps that reuse the cached video keys and values. The teacher's audio target is computed under the student's own visual cache (SCTA).

Critical strengths
  1. The problem is clearly identified and well motivated, and the proposed solution is also practical. Because audio has far fewer tokens than video, the extra audio-only steps cost only a little more compute.
Critical weaknesses
  1. Prop 1 holds for any data, so the result that correlation information decays faster than mean information applies to video too; the only thing that ties the result to audio is the intuitive claim that audio quality relies more on such weak long-range relations, and the paper neither proves nor measures this. And Fig. 1 compares NMSE across two different latent spaces with very different compression rates (414 audio tokens vs. 37K video tokens); therefore, audio may lose more because each token is more compressed.

  2. Most of the gain seems to come from giving audio more steps, not from switching audio to trajectory supervision, e.g., in Table 4, "async only" already improves SongBench/SongEval from 4.78/3.71 to 4.93/3.79, and MAAD adds only another 0.05/0.04.

  3. The evaluation favors outputs that are close to the teacher. The metrics, e.g., MuQ FAD (11.37 → 4.58), CLAP FAD, and Onset F1, are all measured against the teacher's outputs. A method that regresses the teacher's trajectory will naturally do well on them, but that doesn't mean better audio quality.

  4. The DM/DM variant in Table 2 already beats all released baselines on every Sing350 metric and is close to the teacher. The training data and budget are not described, so it's hard to tell how much of the Table 1 improvement is due to MAAD itself. This raises a concern that the gap to baselines seems to come from the authors' own training setup, not from the proposed objective assignment. It would help to report the symmetric DM-only (4/4) baseline on the same metrics as Table 1, and to describe the training data.

Questions for authors
  1. Is there direct evidence that audio relies more on weak long-range correlations than video, for example by measuring cross-time correlation in the latents of both modalities? How much of the gap in Fig. 1a remains after accounting for the different compression rates of the two latent spaces?

  2. What training data (prompts) and training budget were used? Does the training data include many singing-style prompts similar to Sing350?

  3. Could the authors report the symmetric DM-only (4/4) baseline on the same metrics as Table 1, including FAD and Onset F1? This would show whether your own training setup already accounts for most of the gap to the released models.

  4. Could the authors add teacher-free quality metrics, such as FAD against real data or a MOS test focused on audio, to separate actual quality from closeness to the teacher?

  5. DeSync gets worse with MAAD (0.045 to 0.076 in Table 4). Could the authors explain this? Is there a way to keep the audio gains without hurting sync? or at least mention this in the limitations?

6 comments
AuthorsOpenReview ↗

Response to Reviewer UrUh (1/3)

We thank the reviewer for the careful and technical review, and for finding the problem clearly identified and the solution practical. We number the weaknesses W1–W4 in the order of the review and answer each question together with the weakness it concerns.

W1 & Q1. Prop. 1 is modality-agnostic, and Fig. 1a compares different latent spaces. Thank you for this insightful point, which goes to the core of our argument. We agree that Prop. 1 holds for any data and that Fig. 1a compares latent spaces with different compression; cross-time correlations measured in the latents would inherit the same confound (Q1). Prop. 1 therefore affects audio more than video only if audio quality depends more on long-range temporal structure. To test this premise without the latent-compression confound, we ran two analyses on decoded audio and video.

Does the teacher's quality depend more on temporal order in audio than in video? For each of the 350 Sing350 teacher outputs, we split the first 5 s into twenty 0.25 s blocks and swap adjacent pairs so that 20%, 40% or 80% of the blocks move (two permutation seeds per prompt). Changes are measured against a sham that applies the same splitting and crossfades without reordering. Audio is scored with SongEval Coherence and SongBench Melody; video with motion smoothness (AMT) and VABench Visual Reality (VR), whose rubric covers object permanence, plausible motion and temporal coherence.

MetricSham20% moved40% moved80% moved
SongEval Coherence (audio)3.8369−0.2080 (−5.4%)−0.3568 (−9.3%)−0.5570 (−14.5%)
SongBench Melody (audio)4.6733−0.1808 (−3.9%)−0.2768 (−5.9%)−0.3513 (−7.5%)
AMT (video)0.9922−0.0047 (−0.5%)−0.0091 (−0.9%)−0.0168 (−1.7%)
VR (video)4.9971+0.0000 (0.0%)−0.0071 (−0.1%)−0.0057 (−0.1%)

The results show that stronger reordering steadily lowers both audio scores, by up to 14.5%, whereas the video scores change by at most 1.7%. Since all scores are computed on decoded audio and video, where latent compression plays no role, this indicates that audio quality depends more on long-range temporal information than video quality.

Do MAAD's gains follow this premise? Under this premise, replacing DM with trajectory supervision for audio should help most on prompts whose outputs depend most on long-range order. To measure this dependence, we cut the first 5 s of each teacher output into five 1 s blocks, randomly reorder them (eight random orders), and record how much the teacher's score drops (SongEval Coherence for audio, AMT for video; audio crossfades are matched so that only the order differs). We then split the 350 prompts into thirds by this drop (Low/Middle/High, separately for audio and video) and, within each third, compare MAAD with DM/DM at equal steps (4 video / 20 audio updates).

MAAD − DM/DMLow (n = 117)Middle (n = 116)High (n = 117)High − Low
SongBench Melody+0.0087+0.0943+0.1429+0.1342
SongEval Coherence+0.0306+0.0386+0.0415+0.0109
Video AMT−0.0022−0.0025−0.0030−0.0008
Video VR+0.0000+0.0000−0.0342−0.0342

The results show that, from low to high sensitivity, MAAD's Melody gain rises from +0.0087 to +0.1429 (95% CI of the High − Low difference: [+0.013, +0.243]) and its Coherence gain from +0.0306 to +0.0415. AMT differences are slightly negative throughout (−0.0022 to −0.0030), and VR differences are 0, 0 and −0.0342; VR gives its maximum score to 99.03% of all evaluated inputs, which limits its discrimination. The audio trends support our long-range motivation, and the video metrics show that MAAD keeps video performance comparable to DM/DM.

AuthorsOpenReview ↗

Response to Reviewer UrUh (2/3)

W2. Is the gain from more audio steps rather than from trajectory supervision? Thank you for this question; separating the two parts is essential to our claim. Table 4 (Table 13 in new version) compares them only through mean scores, which can hide gains concentrated in a few dimensions. We therefore compare the three Sing350 students dimension by dimension: DM/DM 4/4 → 4/20 (DM only → async only in Table 4 (Table 13 in new version)) changes only the steps, and DM/DM 4/20 → MAAD 4/20 changes only the objective. Parentheses: relative change from the row above.

SongBenchVocalInstr.MelodyStruct.Arr.MixingMusic.Mean
DM/DM 4/44.87265.04944.52874.38614.64575.02644.98154.7843
DM/DM 4/205.1263 (+5.21%)5.1321 (+1.64%)4.6869 (+3.49%)4.4569 (+1.62%)4.6739 (+0.61%)5.1548 (+2.56%)5.2477 (+5.34%)4.9255 (+2.95%)
MAAD 4/205.0907 (−0.69%)5.0940 (−0.74%)4.7688 (+1.75%)4.6235 (+3.74%)4.7342 (+1.29%)5.2143 (+1.15%)5.3158 (+1.30%)4.9773 (+1.05%)
SongEvalCoherenceMusicalityMemorabilityClarityNaturalnessMean
DM/DM 4/43.74963.77393.85143.69103.49953.7131
DM/DM 4/203.8386 (+2.37%)3.8602 (+2.29%)3.9233 (+1.87%)3.7713 (+2.18%)3.5647 (+1.86%)3.7916 (+2.11%)
MAAD 4/203.8755 (+0.96%)3.9090 (+1.26%)3.9617 (+0.98%)3.7912 (+0.53%)3.5905 (+0.72%)3.8256 (+0.90%)

The results show that both contribute, to different aspects: additional audio steps notably improve vocal clarity, while trajectory supervision preserves the long-range dependencies underlying melody and musical structure. The limited audio quality of existing four-step models led us to give audio more steps, and the extra steps improve every SongBench dimension, including Vocal (+5.21%). At equal steps, DM/DM 4/20 keeps slightly clearer vocals, but MAAD improves Melody, Structure, Arrangement, Mixing and Musicality, and all five SongEval dimensions. Its Structure (+0.1666) and Arrangement (+0.0603) gains exceed those from extra steps (+0.0708 and +0.0282), so the small mean gain (+0.0518) combines larger gains in these musical dimensions with slight decreases in Vocal and Instrumental. The objective also raises Dynamic Degree from 8.9% to 51.1% (teacher 43.7%), and its Melody gain is largest on the prompts that depend most on long-range order (+0.1429, vs. +0.0087 on the least dependent third). Finally, the two parts are complementary: without extra steps, trajectory supervision lowers SongBench/SongEval to 4.32/3.59 (Traj. only in Table 4 (Table 13 in new version).

W3 & Q4. Metrics measured against the teacher. We agree with this important point: closeness to the teacher alone does not establish better audio quality, and we therefore report fidelity to the teacher alongside reference-free song-quality scores (SongBench, SongEval). The variant with trajectory supervision on both modalities (Traj./Traj.) illustrates the difference: it is the second closest to the teacher on both MuQ FAD (5.37) and CLAP FAD (0.027), yet its SongBench/SongEval scores are lower than MAAD's (4.74/3.70 vs. 4.98/3.83), and its Visual Reality is the lowest of the four objective assignments (4.24 vs. 4.70 for MAAD). These results show that closeness to the teacher and teacher-free quality can diverge, and that trajectory supervision on video lowers visual quality; this motivates using DM for video and trajectory supervision for audio in MAAD.

For a teacher-free perceptual check (Q4), our user study ((App. E (App. E in new version); 46 raters, 1,840 ratings; per method, 5 clips from Sing350 and 5 from VABench) gives MAAD the highest audio-quality score among the four-step models: 4.22, against 3.81, 2.99 and 2.69 for Larry Turbo, Sol-H3 and LightX2V Turbo.

AuthorsOpenReview ↗

Response to Reviewer UrUh (3/3)

W4, Q2 & Q3. Is the gap to the released models due to our training setup? We appreciate this suggestion; the symmetric baseline is a direct control for separating our training setup from MAAD. The DM/DM variant in Table 2 (Table 4 in new version) already uses 20 audio updates (async only in Table 4 (Table 13 in new version)), so it already includes part of MAAD's design. To answer the question, we describe our training data and budget (Q2) and report the fully symmetric DM/DM 4/4 on all Table 1 (Table 2 in new version) metrics (Q3).

Training data and budget (Q2). Training uses text prompts only, with no audio or video. The prompts are captions of real videos generated by Qwen-Omni, mostly film and TV (51.81%) and speech or dialogue (16.95%); music, including singing, makes up 4.96%. Sing350 prompts are composed differently: each combines a MusicCaps description, a SongBench lyric excerpt and a ChatGPT-written description of the scene and camera. Training and test prompts therefore differ in both source and annotation style, and no training prompt comes from MusicCaps, SongBench, VABench or T2AV-Compass. We run 3,000 iterations at batch size 1 and update the student every 5 iterations, so the student receives 600 updates, of which about 30 use a music prompt in expectation. Training takes about 200 H800 GPU-hours (24.9 h on 8 GPUs), and all variants in Tables 2, 4 and 5 (Tables 4, 13 and 14 in new version) share this setup.

Symmetric baseline (Q3).

VABenchNFEVisual Reality↑Expressiveness↑Visual QA↑Alignment↑DeSync↓ViCLIP↑CLAP↑AudioBox↑DNSMOS↑
Larry Turbo44.4954.2940.7164.4310.5500.2190.3123.4441.346
DM/DM 4/4 (ours)44.4584.2870.7334.454‾\underline{4.454}0.4820.220‾\underline{0.220}0.330‾\underline{0.330}3.489‾\underline{3.489}1.398‾\underline{1.398}
MAAD 4/204.254.476‾\underline{4.476}4.293‾\underline{4.293}0.729‾\underline{0.729}4.4780.501‾\underline{0.501}0.2230.3613.5441.418
Sing350MuQ FAD↓CLAP FAD↓Onset F1↑Vocal↑Melody↑Instr.↑Musicality↑Coherence↑Clarity↑
Larry Turbo11.370.070‾\underline{0.070}0.7274.8224.565‾\underline{4.565}4.8253.799‾\underline{3.799}3.7443.678
DM/DM 4/4 (ours)9.26‾\underline{9.26}0.0740.750‾\underline{0.750}4.873‾\underline{4.873}4.5295.049‾\underline{5.049}3.7743.750‾\underline{3.750}3.691‾\underline{3.691}
MAAD 4/204.580.0190.8035.0914.7695.0943.9093.8753.791

The results show that, with our data and budget, the symmetric DM/DM 4/4 student is about as close to the teacher as the released Larry Turbo (MuQ FAD 9.26 vs. 11.37, CLAP FAD 0.074 vs. 0.070, Onset F1 0.750 vs. 0.727) and has comparable song-quality scores. With the same data and budget, MAAD brings the student closer to the teacher (MuQ FAD 4.58, CLAP FAD 0.019, Onset F1 0.803) and improves all six song-quality scores. This indicates that the improvement over the released models is mainly attributable to MAAD rather than to our training data.

Q5. DeSync on Sing350 (0.045 → 0.076 s, Table 4 (Table 13 in new version)). Thank you for pointing this out. To locate the source of the increase, we compare DeSync across step schedules and objectives on Sing350: it is 0.045 s with 4/4 DM steps and 0.084 s with 4/20 DM steps, and switching the audio objective to trajectory supervision with SCTA lowers it to 0.076 s, the lowest among the four 4/20 objective assignments in Table 2 (Table 4 in new version). These comparisons indicate that the increase comes mainly from the extra audio-only steps rather than from the audio objective. A likely reason is that the audio-only steps of a stage attend to the video context cached at the start of that stage. In our user study, raters gave MAAD the highest audio–visual alignment score among the four-step models (4.41, vs. 4.15, 3.78 and 3.52), and on VABench, MAAD's DeSync (0.501 s) is lower than the teacher's (0.569 s). We have stated this in the limitations of the paper (Sec. 5.5 in new version). In future work, synchronization could be tightened by refreshing the cached video context within a stage or by adding a synchronization loss on the audio-only steps.

We thank the reviewer again for the detailed and technical comments, which helped us sharpen the analysis.

Reviewer_k3FSOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

This paper argues that a modality-symmetric distillation recipe is poorly suited to joint audio-video generation. Its analysis shows that re-noising suppresses information about correlation structure faster than information about first-order statistics, motivating different objectives for video and audio. The proposed MAAD framework uses distribution matching for video, trajectory supervision for audio, additional audio-only denoising steps with cached visual keys and values, and student-context trajectory alignment. Experiments distill MiniMax-H3 into a student with four video updates and twenty audio updates, evaluated on VABench and the proposed Sing350 prompt set. Presentation

Critical strengths
  1. The analysis and teacher diagnostics provide a useful explanation for treating audio and video differently during distillation.
  2. Audio-only updates and student-context trajectory alignment address concrete computation and conditioning issues.
  3. On MiniMax-H3, MAAD improves all nine singing metrics over the compared released four-step models, with informative component ablations.
Critical weaknesses
  1. Limited evaluation. General generation is evaluated on VABench only. Another established suite, such as T2AV-Compass, would test whether the improvements persist under a different prompt taxonomy and evaluation protocol. This would strengthen the evidence for the broader modality-specific distillation claim.
  2. The controlled studies focus on singing. Sing350 is a reasonable stress test for rhythm and musical structure, but the paper's motivation concerns temporal audio structure more generally. It remains unclear whether the same objective assignment benefits dialogue, intermittent impacts, footsteps, machinery, or changing ambience. Please explain the choice of singing and provide category-level evidence or a compact non-musical ablation where feasible.
  3. The practical efficiency evidence could be stronger. The approximately 6% overhead is based on token-normalized computation rather than measured end-to-end latency. As Appendix E notes, realized speed also depends on cache transfers, kernels, and numerical precision. Matched-hardware latency and peak-memory measurements would clarify the deployment cost of the additional audio updates.
Questions for authors

please see weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer k3FS (1/2)

We thank the reviewer for the constructive review and for finding our analysis and teacher diagnostics useful. We number the weaknesses W1–W3 in the order of the review.

W1. Do the improvements hold on another benchmark? Thank you for suggesting T2AV-Compass. We agree that a benchmark with a different prompt taxonomy and evaluation protocol is a stronger test of the modality-specific claim. We therefore evaluated MAAD and the three released four-step distillations of H3 on all 500 T2AV-Compass prompts with its official metrics.

ModelVA↑VT↑AA↑SQ↑T-V↑T-A↑A-V↑DeSync (s)↓LS↑
Teacher (50 steps)5.23720.12676.83591.71060.01450.16470.23650.71761.6508
Larry Turbo4.94520.11856.8270‾\underline{6.8270}1.73150.0190‾\underline{0.0190}0.1609‾\underline{0.1609}0.23960.72722.0997
Sol-H35.04160.12486.56961.75240.02370.15700.20090.67602.0205‾\underline{2.0205}
LightX2V Turbo5.1208‾\underline{5.1208}0.12606.60601.70200.01480.15740.21070.6952‾\underline{0.6952}1.8848
MAAD5.12600.1252‾\underline{0.1252}6.82991.7353‾\underline{1.7353}0.01880.17160.2333‾\underline{0.2333}0.78641.8839

The results show that the audio-side gains carry over to this benchmark. Among the four-step models, MAAD has the best audio aesthetics (AA) and text–audio alignment (T-A), as well as the best video aesthetics (VA), and it ranks second on VT, SQ and A-V.

W2. Why singing, and does the benefit extend beyond music? We appreciate this question. We agree that our motivation concerns temporal audio structure in general, not only singing. We chose singing as the main testbed for two reasons. First, it concentrates the structure our analysis concerns: a sung line sounds right only if pitch, syllable timing, beat and the voice–accompaniment relation stay consistent over seconds. Second, it is where we observed the clearest failure of existing four-step distillations, which often make sung vocals sound like speech; this observation motivated Prop. 1 and our use of trajectory supervision for audio. Our training is not tailored to singing: music, including singing, makes up 4.96% of the training prompts.

To test whether the benefit extends to other sounds, we removed the 66 music clips from T2AV-Compass and grouped the remaining 434 clips by their official audio prompts into speech (145), machinery (49), actions (150) and ambience (90). The results follow in the next comment.

AuthorsOpenReview ↗

Response to Reviewer k3FS (2/2)

W2 (continued). Results by sound category on T2AV-Compass. Avg.: pooled over all 434 clips (426 valid for T-A).

AA↑SpeechMach.ActionsAmb.Avg.
Teacher (50 steps)7.056.176.816.496.75
Larry Turbo7.07‾\underline{7.07}6.256.75‾\underline{6.75}6.496.75‾\underline{6.75}
Sol-H36.756.026.586.136.48
LightX2V Turbo6.466.23‾\underline{6.23}6.736.426.52
MAAD7.196.146.776.44‾\underline{6.44}6.77
T-A↑SpeechMach.ActionsAmb.Avg.
Teacher (50 steps)0.1090.1670.1770.1700.153
Larry Turbo0.106‾\underline{0.106}0.1660.1810.163‾\underline{0.163}0.151‾\underline{0.151}
Sol-H30.0950.1810.1730.1500.144
LightX2V Turbo0.0980.1630.1770.1540.145
MAAD0.1210.167‾\underline{0.167}0.179‾\underline{0.179}0.1810.160
A-V↑SpeechMach.ActionsAmb.Avg.
Teacher (50 steps)0.2620.2160.2420.1960.236
Larry Turbo0.264‾\underline{0.264}0.2340.2440.186‾\underline{0.186}0.238
Sol-H30.2260.1890.2050.1500.199
LightX2V Turbo0.2280.1880.2250.1720.211
MAAD0.2710.200‾\underline{0.200}0.234‾\underline{0.234}0.1970.235‾\underline{0.235}

The results suggest that the benefit extends beyond music. Across the 12 category–metric pairs, MAAD is first among four-step models on 6 and in the top two on 11. Averaged over all 434 clips, it is first on AA and T-A and second on A-V, 0.003 below the best. The gains are largest for speech, where it is first on all three metrics, and for ambience. For machinery and action sounds, its A-V alignment trails Larry Turbo; broader coverage of such sounds in training may help.

W3. What do the audio-only steps cost in practice? Thank you for this practical suggestion. We agree that token-normalized compute does not capture cache transfers, kernels or numerical precision. We therefore timed full generation and recorded peak memory on 2×H100 80GB at the paper's setting (768×1344, 124 frames, 32 kHz stereo). All samplers use the same BF16 mixed precision: transformer, LoRA and VAE weights are in BF16 with BF16 autocast, and a few root modules are kept in FP32.

SamplerSteps (joint + audio-only)Joint step (s)Audio-only step (ms)End-to-end (s)Speed-upPeak memory / GPU (GiB)
Teacher509.925–515.381.0×40.16
DM/DM 4/4410.676–61.718.4×40.81
MAAD4 + 1610.881239.966.527.7×66.44

The results show that MAAD takes 7.8% more time and 25.6 GiB more peak memory per GPU than a symmetric four-step student (DM/DM 4/4), and remains 7.7× faster than the teacher. About 19 s of each end-to-end time is spent outside the denoising loop and is the same for all three samplers. The 16 audio-only steps add 3.84 s, 8.8% of the time of the four joint steps, and each joint step is 1.9% slower, likely because it also writes the visual cache. Overall, the measured cost (4.35 NFE-equivalent) is close to the token-based estimate of 4.25. Memory is the larger cost: keeping the visual keys and values of all layers on the GPU raises peak memory from 40.81 to 66.44 GiB. This fits on 80 GB GPUs; offloading the cache or storing it at lower precision could reduce it, which we have not yet evaluated.

We thank the reviewer again for the helpful suggestions. The new benchmark, the category-level results and the measurements have broadened our evaluation and clarified the practical cost of the method.

Dear Reviewer k3FS, Thank you again for your helpful suggestions. We have incorporated the additional evaluations into the revised manuscript, with changes marked in red:

  1. T2AV-Compass: On 500 prompts, MAAD leads four-step models in audio aesthetics, text–audio alignment, and video aesthetics (Table 9).
  2. Beyond singing: On 434 non-musical prompts, MAAD leads in pooled audio aesthetics and text–audio alignment, with the strongest gains in speech (Table 10).
  3. Measured cost: Compared with DM/DM 4/4, MAAD adds 7.8% latency in end-to-end inference compared, while remaining 7.7× faster than the teacher (Table 12).

We hope these results address your concerns about generalization and practical efficiency, and we would appreciate it if you could reconsider the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the reviewers for their careful and constructive reviews. They found the motivation for treating audio and video differently well identified and sensible (k3FS, UrUh, Mysh), the audio-only steps and SCTA practical at little extra cost (k3FS, UrUh, Mysh), and the four-step singing results strong (k3FS, Mysh). Below we list the new results and then address the two questions raised by two reviewers each.

Reviewer questionNew result
R1Do the gains hold on another general benchmark? (k3FS W1)T2AV-Compass, 500 prompts
R2Do they extend beyond singing? (k3FS W2)Speech, machinery, actions and ambience on T2AV-Compass
R3What is the measured cost? (k3FS W3)End-to-end latency and peak memory
R4Does audio rely more on temporal order than video? (UrUh W1/Q1)Teacher sensitivity to temporal reordering in decoded audio and video
R5Extra audio steps or audio objective? (UrUh W2, Mysh W1)Split of the Sing350 gain into the two parts at equal steps
R6Does the objective's gain follow long-range structure? (UrUh W1/Q1, Mysh W1)MAAD vs. DM/DM at equal steps, by the prompt's dependence on long-range order
R7Is the gap to released models due to the training setup? (UrUh W4/Q2/Q3)Symmetric DM/DM 4/4 on all Table 1 (Table 2 in new version) metrics; training data and budget
R8Does MAAD transfer to another backbone? (Mysh W2)LTX-2.5, a dual-stream model

Is the gain from extra audio steps or from the audio objective? (UrUh W2, Mysh W1) MAAD gives audio both extra steps and a different objective . We understand the concern: in Table 4 (Table 13 in new version), the extra steps alone (async only) bring a larger gain than the change of objective. To identify the role of each part, we compare the three students dimension by dimension: DM/DM 4/4 → 4/20 changes only the steps, and DM/DM 4/20 → MAAD 4/20 changes only the objective. Parentheses: relative change from the row above.

SongBenchVocalInstr.MelodyStruct.Arr.MixingMusic.Mean
DM/DM 4/44.87265.04944.52874.38614.64575.02644.98154.7843
DM/DM 4/205.1263 (+5.21%)5.1321 (+1.64%)4.6869 (+3.49%)4.4569 (+1.62%)4.6739 (+0.61%)5.1548 (+2.56%)5.2477 (+5.34%)4.9255 (+2.95%)
MAAD 4/205.0907 (−0.69%)5.0940 (−0.74%)4.7688 (+1.75%)4.6235 (+3.74%)4.7342 (+1.29%)5.2143 (+1.15%)5.3158 (+1.30%)4.9773 (+1.05%)
SongEvalCoherenceMusicalityMemorabilityClarityNaturalnessMean
DM/DM 4/43.74963.77393.85143.69103.49953.7131
DM/DM 4/203.8386 (+2.37%)3.8602 (+2.29%)3.9233 (+1.87%)3.7713 (+2.18%)3.5647 (+1.86%)3.7916 (+2.11%)
MAAD 4/203.8755 (+0.96%)3.9090 (+1.26%)3.9617 (+0.98%)3.7912 (+0.53%)3.5905 (+0.72%)3.8256 (+0.90%)

The results show that both contribute, to different aspects. The extra steps improve every dimension, notably vocal clarity (Vocal +5.21%). At equal steps, DM/DM 4/20 keeps slightly clearer vocals, but MAAD improves Melody, Structure, Arrangement, Mixing, Musicality and all SongEval dimensions; its Structure and Arrangement gains (+0.1666, +0.0603) exceed those from extra steps (+0.0708, +0.0282).

Does the objective's gain follow long-range structure, as our analysis predicts? (UrUh W1/Q1, Mysh W1) Prop. 1 shows that DM supervision weakens cross-time structure more than first-order statistics, so replacing DM with trajectory supervision for audio should help most where the output depends most on long-range order. UrUh and Mysh asked whether MAAD's gain reflects this mechanism; we therefore test this prediction directly. We cut the first 5 s of each teacher output into five 1 s blocks, randomly reorder them (eight random orders), and record how much the teacher's score drops (SongEval Coherence for audio, AMT for video; audio crossfades are matched so that only the order differs). We then split the 350 prompts into thirds by this drop (Low/Middle/High, separately for audio and video) and, within each third, compare MAAD with DM/DM at equal steps (4 video / 20 audio updates).

MAAD − DM/DMLow (n = 117)Middle (n = 116)High (n = 117)High − Low
SongBench Melody+0.0087+0.0943+0.1429+0.1342
SongEval Coherence+0.0306+0.0386+0.0415+0.0109
Video motion smoothness (AMT)−0.0022−0.0025−0.0030−0.0008
Video Visual Reality+0.0000+0.0000−0.0342−0.0342

The results show that, from low to high sensitivity, MAAD's Melody gain rises from +0.0087 to +0.1429 (95% CI of the High − Low difference: [+0.013, +0.243]) and its Coherence gain from +0.0306 to +0.0415, while video differences stay close to zero. The audio trends support our long-range motivation, and the video metrics show that MAAD keeps video performance comparable to DM/DM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.