acceptodds
Under review as a conference paper at ICLR 2027

Modality-decoupled Distribution Matching Distillation for Few-step Audio-visual Generation

Abstract

Distribution matching distillation (DMD) is an effective route to few-step generation, but its standard formulation treats the audio and video outputs of a joint generator as if they shared the same noise geometry. We find that this coupling is harmful in audio-visual distillation: the two modalities prefer different renoising scales, while a joint fake-score objective can underfit one modality even when its aggregate loss appears well behaved. We introduce , a modality-decoupled DMD formulation with two changes. First, it uses separate audio and video noise schedules while retaining in-distribution joint model evaluations. Second, it trains the fake score with branch-specific objectives at the noise scale used by each modality. On a four-step LTX-2.3 audio-visual generator, these changes improve the normalized full VBench score from 0.642 to 0.674 and the CLAP score from 0.268 to 0.306. These results identify noise-scale and fake-marginal mismatch as central obstacles in few-step joint audio-visual distillation.

Primary area:
generative models
Keywords:
joint audio-visual generation, distribution matching distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Fgn8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

AV-DMD adapts DMD2 to four-step joint audio-visual generation. It reads video and audio score differences from separate noise schedules while keeping each joint score-network input internally synchronized, then trains one shared fake score with branch-specific losses on the corresponding states. On LTX-2.3, VBench Overall rises from 0.6417 to 0.6740 and JavisBench-mini CLAP from 0.372 to 0.416 against matched Joint DMD, with improved lip-sync scores but lower dynamic degree.

Critical strengths
  1. The two synchronized joint states permit modality-specific score estimation while retaining one shared fake score and the unchanged four-step inference path.
  2. Matched Joint DMD comparisons across VBench, JavisBench-mini, and AVBench identify gains in Semantic and CLAP and report regressions rather than only aggregate scores.
  3. The digital-human analysis repeats SyncNet comparisons on the same successfully tracked clips, strengthening the lip-sync result against a face-track coverage confound.
Critical weaknesses
  1. VBench Dynamic Degree falls from 77.8 to 62.5, Quality from 0.8419 to 0.8345, and AV-Align barely changes (0.2420 to 0.2431). The submitted materials provide no playable videos or user study to assess this perceptual trade-off.
  2. The synchronization constraint is not tested against mixed-noise evaluation, and the time/noise conditioning of the second score pass is unclear. This leaves the central in-distribution rationale insufficiently verified.
  3. The manuscript has no explicit limitations or failure-case discussion.
Questions for authors
  1. Can the authors provide matched-prompt playable examples, including failures, and a blinded user study of motion, audio quality, and synchronization?
Reviewer_Qj4rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AV-DMD for distilling joint audio-visual diffusion models into four-step generators. It constructs two joint noisy states at different noise levels, while keeping audio and video synchronized within each network evaluation. Video and audio score corrections are extracted from their respective states, and the shared fake-score network is supervised on the corresponding modality-specific inputs.

Critical strengths
  • A relevant problem and a practical construction. The method addresses modality-dependent denoising difficulty while respecting the synchronized-noise interface of the pretrained joint model. It requires no additional inference-time network or change to the student architecture.

  • Meaningful improvements under matched inference settings. VBench Overall increases from 0.6417 to 0.6740, Semantic from 0.4861 to 0.5491, and JavisBench-mini CLAP from 0.372 to 0.416.

Critical weaknesses
  • The novelty is limited and insufficiently positioned against prior work. Modality-specific noise scheduling predates this paper. Ishii et al. (https://arxiv.org/abs/2409.17550v1) introduced different audio/video timestep mappings. MOVA (https://arxiv.org/abs/2602.08794v1) explicitly used asymmetric sigma shifts. These works do not propose the same DMD construction, but they establish that assigning different schedules to audio and video is not itself new.

  • The ablations do not isolate the benefit of the proposed decoupling. In Table 4, DecShift alone lowers VBench Overall from 0.6417 to 0.6283, although CLAP improves from 0.268 to 0.304. The full method reaches 0.6740 Overall, but there is no standalone DFake control. Consequently, the results do not establish whether the improvement requires modality-specific schedules or could largely be obtained through additional fake-score training and score evaluations.

  • The necessity of the two-pass construction remains experimentally unverified.

Questions for authors

See weaknesses; the main concern is novelty.

Reviewer_mEtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AV-DMD, a few-step distillation method for joint audio-visual diffusion transformers. Standard multimodal DMD perturbs video and audio at the same noise level, although the two modalities have different denoising difficulty, while directly using mixed noise may be out of distribution for a jointly pretrained score model. AV-DMD instead constructs two internally synchronized joint states, reads the video target under the video schedule and the audio target under the audio schedule, and trains a shared fake-score network with corresponding branch losses. The inference architecture remains unchanged. Experiments report improved semantic composition, text-audio alignment, and lip synchronization, with mixed changes in visual quality and motion.

Critical strengths
  1. Synchronization-preserving modality decoupling AV-DMD decouples modality-specific schedules across score evaluations while keeping each joint input internally synchronized. This avoids the obvious mixed-noise distribution shift and preserves audio-visual context, providing a clean solution for distilling a frozen jointly pretrained score model.

  2. No additional inference overhead The method changes only the distillation-time score evaluations. It retains the original student architecture, parameter count, and sampling trajectory, so its deployment benefits do not require extra inference modules or steps.

Critical weaknesses
  1. Insufficient component isolation and unmatched training compute Table 6 lacks a DFake-only configuration, while AV-DMD also doubles the score-network forward passes. The current results therefore cannot separate state matching and branch supervision from additional computation or noise-state coverage. This should be addressed with DFake-only, compute-matched, and modality-normalized baselines, together with multiple seeds and held-out shift selection.

  2. Mixed evaluation protocols and no matched teacher comparison Table 1 combines several incompatible protocols, inviting invalid cross-row comparisons and leaving many entries missing. Moreover, LTX-2 is not evaluated on the same Mini-1K protocol as Joint DMD and AV-DMD, so teacher-to-student retention is unknown. The teacher should be evaluated under the matched protocol, and the table should be split or clearly grouped with protocol details in captions or footnotes.

Questions for authors
  1. What time/noise conditioning is used for each score evaluation? For ztAz_t^A, do the real and fake score networks receive the original shared tt, the shifted timestep or sigma corresponding to σA(t)\sigma_A(t), or an inverse-mapped local timestep? Please provide concise pseudocode and the actual tensors passed to both score networks. A mismatch between latent corruption and timestep conditioning would raise a soundness concern, whereas consistent conditioning would resolve this ambiguity.
  2. What are the respective contributions of DFake and the additional score computation? Please report a native/shared-schedule DFake-only configuration and a baseline matched to AV-DMD in score-network forward passes or training FLOPs. These controls are needed to distinguish the effect of state-matched branch supervision from additional computation and broader noise-state coverage.
  3. Why are standard modality-balancing methods insufficient? Does the audio fake-score error persist under per-modality normalization, explicit loss reweighting, GradNorm, or gradient-conflict methods such as PCGrad? Please compare these baselines under matched states and computation, reporting per-modality score errors and final generation metrics. This would clarify whether DFake addresses more than conventional multitask imbalance.
  4. Can Table 1 be reorganized by evaluation protocol and include a matched teacher comparison? Please evaluate LTX-2, Joint DMD, and AV-DMD using the same Mini-1K prompts, generation settings, seeds, and evaluators. Results from incompatible protocols should be split or clearly grouped. Protocol details should appear in the caption, with supplementary information presented as proper table footnotes rather than title-like text below the table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.