acceptodds
Under review as a conference paper at ICLR 2027

Lost in the Rollout: Rethinking On-Policy Distillation for Multimodal Video Diffusion

Abstract

On-policy distillation, exemplified by Self Forcing, has become the leading approach for turning video diffusion models into fast, interactive few-step generators. Applying it under joint text, image, and audio conditioning for avatar generation has recently attracted great interest, but rare insights have been exposed on the corresponding training recipe. This paper first identifies the lost in the rollout issue in this regime, where the video rollouts generated by the student often lose what they should carry at two levels. First, they lose the conditions: within a few hundred steps, training collapses into black frames, flickering, or blur, and the identity given by the image condition is gone. Unlike a text prompt, the image condition becomes the first frame of every rollout, and the teacher and the critic score each rollout under that same image, so a flaw in the image enters the very signal meant to correct the student. (1) Curated conditions and (2) a converged ODE initialization keep the conditions intact and stabilize training; together they cut FID from 27.10 to 11.67 and raise Sync-C from 3.13 to 4.15. Second, they lose the teacher's trajectory: after the ODE initialization, the student sees only the DMD score of its own rollouts, and at standard learning rates and guidance it learns too little lip-sync. We keep a small piece of the regression onto teacher samples that DMD2 removed: (3) an auxiliary target that matches the audio energy of each generated clip to that of a full teacher sample for the same condition. Because the target is computed on the generated audio, it applies to any model that generates sound. (4) Higher learning rates and teacher guidance make the student learn more from the teacher's score, raising Sync-C further to 4.50. With these remedies, a collapsing recipe becomes a stable four-step causal student of a Wan2.1-based multimodal model, which improves lip-sync, image quality, and aesthetics over the bidirectional model of the same size on HDTF, AVSpeech, and CelebV-HQ, with over 20× higher throughput, running at 24.82 FPS on a single GPU. On MiniMax-H3, a 33B model that generates video and audio jointly, the auxiliary target further sharpens lip-sync, raising Sync-C from 7.37 to 8.02. The lesson reaches beyond these models: an on-policy student excels when its rollouts start well and the teacher's signal keeps reaching them.

Primary area:
generative models
Keywords:
on-policy distillation, diffusion distillation, distribution matching distillation, Self Forcing, few-step video generation, autoregressive video diffusion, video diffusion models, multimodal conditioning, audio-conditioned video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YAVUOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies the "lost in the rollout" problem in on-policy distillation (exemplified by Self Forcing) for multimodal video diffusion, and proposes four targeted improvements: curating and optimizing image and text conditions to fix input defects in conditioning; training the ODE trajectory distillation to convergence to solidify the initialization foundation; introducing an auxiliary audio energy loss to restore the teacher’s trajectory signal; and increasing the learning rate and tuning classifier-free guidance strength to amplify the teacher’s learning signal. These remedies effectively stabilize training and improve lip-sync performance. Validated on OmniAvatar, MiniMax-H3 and multiple benchmarks, the 4-step model achieves improved image quality and lip-sync, with over 20× inference speedup.

Critical strengths
  1. Within a relatively short submission timeframe, the authors conduct experiments on the MiniMax-H3 base model and verify the effectiveness of the proposed method on state-of-the-art larger-scale base models, which is beneficial to the research community. The ablation study is thorough and detailed, demonstrating the effectiveness of each component of the method, which helps readers intuitively grasp the detailed effects of the proposed approach.
  2. The insights and findings regarding image conditions in on-policy DMD training are interesting. I believe the community needs to pay more attention to this issue, and this observation is instructive even in non-avatar scenarios. In addition, the converged ODE initialization is well-justified; while the community has previously noted the importance of ODE initialization, few have provided in-depth analysis of the training process.
  3. This paper is well organized and easy to understand.
Critical weaknesses
  1. The visual quality and lip-sync alignment of the "livetalk_interactive" demo are noticeably inferior to the generation results in the "h3_samples". Since the supplementary material does not include a README or directory structure guide to clarify the experimental setup, I can only infer that the proposed method suffers from noticeable performance degradation in real-world real-time interactive scenarios.
  2. Some baselines are missing. Given the fast-growing interest in this task and the large number of recent works, I consider it reasonable to include more baselines for a more comprehensive comparison of model performance, e.g., AvatarForcing (Cui et al., 2026), daVinci-MagiHuman (Chern et al., 2026), Hallo-Live (Li et al., 2026), and LongCat-Video-Avatar 1.5 (Cai et al., 2026).
Questions for authors
  1. Please explain the noticeable performance degradation of the LiveTalk demo mentioned in the weaknesses.
  2. Please add more baselines.
Reviewer_Tr3WOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for distillation of multimodal video diffusion models (for generation). The paper identifies errors which are introduced into the rollout of the model via poorly chosen/curated image inputs, and improves other components of the video distillation pipeline. In particular, the proposed improvements are:

  • better choosing/curating the input images and text prompts to the model. This is done using other external models
  • increasing the training duration of the causal student model during the "ODE initialisation" step to improve convergence
  • imposing the mean energy of the audio of the teacher
  • adjusting learning rates to improve results The visual results are compelling (especially in the supplementary material) and the quantitative results are also good.
Critical strengths
  • The paper considers a very timely and important problem, ie the drift and accumulated errors induced during rollouts of video generation models distilled into few-step, autoregressive models from many-step and (usually) bidirectional teachers.
  • The visual results, if I have understood them correctly (see question 4), seem to be of very good quality, especially when compared with wan livetalk, for example.
Critical weaknesses

1/ I found the paper to be written in quite an unclear, fuzzy and often confusing manner. I will detail this further in both the weaknesses and questions.

2/ The contributions are quite slim I feel. They boil down to:

  • improving the quality of the input images
  • further training of the initialisation step
  • imposing the mean energy of the audio (it was not clear to me why this was important, see questions)
  • changing the learning rate of the student This all seems just to be adjusting existing elements of the distillation pipeline.

3/ The abstract is very unclear, and I feel this is quite egregious: this is where you sell your paper! For example:

  • Line 14: "Applying it under joint text, image, and audio conditioning for avatar generation has recently attracted great interest, but rare insights have been exposed on the corresponding training recipe". What do you mean? There are not enough insights? Or on the contrary, there are recent important insights?
  • Line 16: "This paper first identifies the lost in the rollout issue in this regime, where the video rollouts generated by the student often lose what they should carry at two levels", there is no introduction of the notion of student/teacher, at least a sentence should explain this. "often lose what they should carry at two levels" is not very crisp writing.
  • Line 22: "(1) Curated conditions and (2) a converged ODE initialization keep the conditions intact and stabilize training". Again, very unclear. This way of writing is not super easy to read (starting a sentence with a numbered list), and "curated conditions" is an extremely unclear notion at this point. What conditions? What curation?
  • Line 26: "We keep a small piece of the regression onto teacher samples that DMD2 removed". This is very vague language, it is not clear to talk about "a small piece of the regression".
  • Line 27: "(3) an auxiliary target that matches the audio energy of each generated clip to that of a full teacher sample for the same condition.", again, starting with a numbered list, and the sentence is very difficult to understand.
  • Line 32: "With these remedies, a collapsing recipe becomes a stable four-step causal student of a Wan2.1-based multimodal model". Again, unclear. What is a "collapsing recipe" ? You have not talked about collapsing before.

4/ Line 262: "We therefore curate the image and text conditions cimg and ctext per dataset. For Hallo3, characterized by overall low image quality, we employ Qwen-Image (Wu et al., 2025) to generate semantically consistent yet high-quality reference frames cimg . For HDTF, which primarily suffers from facial blurriness, we apply super-resolution (Zhou et al., 2022) to obtain clear facial details. We also use Qwen2.5-VL-72B (Bai et al., 2025) to refine text prompts ctext , emphasizing dynamic motion and facial expressions. The configuration with curated conditions is the most stable". This is one of the main, key, points of your paper, and one of the main contributions, and it is explained very vaguely. You say that you use large, foundation, VLMs such as Qwen-Image to improve the inputs, and a super-resolution method, without ever specifying how you used these methods. What input did you give to the VLMs?

5/ The notation in equation 1 has some issues:

  • $t \sim {t_{iN/K} \dots $} does not specify which distribution it is. I assume a uniform distribution, so in this case you would need $t \sim U ( $ { $ t_{iN/K} $ } $ \dots )$

  • gϕg_\phi is not defined.

  • define block more clearly. xtbx_t^b is never rigorously defined (only xtjx_{t_j})

Questions for authors

1/ Line 18 "Unlike a text prompt, the image condition becomes the first frame of every rollout, and the teacher and the critic score each rollout under that same image, so a flaw in the image enters the very signal meant to correct the student.". This argument is repeated at several places in the paper. I do not understand it: you are saying that the image condition directly affects and creates the first image, but this can equally be said of text. For sure, the image condition has a more strong visual impact, but the text does too. Furthermore, as in other places, the language is fuzzy: "image condition becomes the first frame of every rollout", what does "become" mean here? One could equally say that a text "becomes" the first frame. What if my text prompt was "black cat"? Then the first frame would be a black cat. The argument is unclear for me, and does not seem convincing. Also, why would a "flaw enter"? This is not clear to me. If you are trying to create a video with this image as a template, why would copy/pasting the image into the first frame be a "flaw"? This is not clearly explained.

2/ Line 212: "The first clue came from the image condition: filtering the training conditions by brightness and image quality helped (Fig. 2, middle), which pointed to the image condition and to how it enters training". This experiment is not explained at all. What does "filtering by brightness mean"?

3/ Line 245: "Under multimodal conditioning, the conditions themselves enter the rollout: the image condition, unlike the text prompt, becomes part of it". Again, I do not agree, and again the language is fuzzy. What does "becomes part of it" actually mean?

4/ In the supplemental material results, you have a folder "H3". Are these your results ? It is not clear in the paper, since your method does not have a name as far as I can tell.

5/ Section 3.2 I did not understand why imposing the mean energy of the audio made sense here. Why not, but this surely could be done by some sort of normalisation no ? Isn't the audio normalised anyway ? It is like trying to turn the general volume up or down when playing the video: you can do it, but it will not affect the quality of the output (which is given by the video appearance/motion and the oscillations of the audio waveform). It is like multiplying an image by a constant: you will change the contrast, but the content of the image will not change. Furthermore, I did not understand the explanation of the paper as to why this was important: "Before training, the teacher generates one full sample for each training condition. A full regression onto this sample does not fit on-policy training: the rollout starts from its own noise, so matching the whole clip would pull the student toward one particular video. We therefore match one statistic of the whole clip instead of the clip itself." This is not clear to me.

Small questions and further clarity problems:

  • Line 45: "appealing visual fidelity for video generation" fidelity to what? Prompts or anything else have not been explained, so this is unclear. Fidelity to prompts ? Fidelity to the training database generation ?
  • Line 45: "The sampling involves denoising all video frames simultaneously using bidirectional attention via an iterative process". This is very difficult to read as the second sentence of the main paper: nothing has been introduced (attention, bidirectionality, sampling).
  • Line 75: "These works, however, treat distillation as one component of a real-time system; why the distillation itself breaks under the simultaneous integration of text, images, and audio, as in audio-driven human video generation, remains underexplored." This feels like a non-sequitur: why does the second part of the sentence have anything to do with the second? If I rephrase your sentence, it says "These works, however, treat distillation as one component of a real-time system, but the question of when and why distillation fails is underexplored". It is like saying: "the restaurant is one building of the town, but why it was painted blue is under-explored", the sentence does not follow logically. Again, the writing makes reading difficult.
  • Line 86: However, naively applying Self Forcing training produces extensive visual artifacts, e.g., flickering (row 1 of Fig. 5), and a related issue has been reported in the community (BrianChen1129, 2025)". What community? What related issue?
  • Line 113: "Curated multimodal conditions": again, at this point, it is very unclear what these conditions could be, you do not explain them precisely at all.
  • Line 116: "A target that matches the audio energy of each generated clip to that of a full teacher sample brings back a piece of the teacher’s trajectory and raises Sync-C by 0.65 on MiniMax-H3," As mentioned above "brings back a piece of the teacher's trajectory" is not very rigorous or scientific language.
  • Line 190: "For multimodal conditioning, sθs_\theta is estimated with classifier-free guidance (Ho & Salimans, 2022) using separate scales per condition." What is a scale?
  • Line 289: S is not defined (give precise definition of mean energy).
Reviewer_q4jcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why on-policy distillation (Self Forcing/DMD) becomes unstable under multimodal conditioning (text+image+audio) for video diffusion, identifying a “lost in the rollout” failure: (i) conditions are lost because the reference image becomes the first frame of every rollout and contaminates the training signal, and (ii) the teacher’s trajectory is lost because, post-ODE initialization, the student only sees score signals on its own rollouts. The authors propose a practical recipe: curate multimodal conditions, train ODE initialization to convergence, add a lightweight auxiliary audio-energy target from a full teacher sample, and use higher learning rates with tuned teacher CFG. They demonstrate a stable 4-step causal student that improves lip-sync and perceptual scores over a same-size bidirectional baseline and achieves 20×+ throughput, and show the auxiliary target further improves lip-sync on MiniMax-H3.

Critical strengths
  1. Identifies a concrete, under-discussed pathology specific to multimodal on-policy distillation: the image condition’s insertion into the rollout corrupts both critic training and the teacher signal; post-ODE, teacher trajectories stop reaching the student.
  2. Proposes simple, actionable mitigations (condition curation, fully converged ODE initialization, higher LR/CFG) that are easy to reproduce and transfer across backbones.
  3. Introduces a clip-level audio-energy auxiliary target computed on generated audio that is teacher-agnostic and applicable to joint audio–video models; shows measurable Sync-C gains on MiniMax-H3.
Critical weaknesses
  1. Many remedies are engineering choices already explored in similar form (e.g., condition curation, stronger ODE initialization, raising CFG/LR), limiting methodological novelty beyond the diagnosis and the audio-energy target.
  2. The auxiliary audio-energy target is very coarse; it constrains loudness dynamics rather than timing or phoneme-level alignment and could bias the student toward high-amplitude mouth motion without guaranteeing better sync.
Questions for authors
  1. The auxiliary objective matches only the mean energy of the generated audio latent to that of a teacher sample. This statistic does not directly constrain the temporal correspondence between speech and lip movements. Could the authors clarify how this objective improves audiovisual synchronization?

  2. While stronger CFG improves lip synchronization, how do the authors balance this gain against potential color oversaturation, particularly around the mouth? Some generated examples also appear to exhibit exaggerated mouth shapes and facial/body movements. Qualitative comparisons across CFG scales, together with a perceptual assessment of naturalness, would help clarify this trade-off. Could the authors also discuss possible ways to mitigate these unnatural effects?

  3. Although Figure 5 provides qualitative ablations, the benefit of improving reference-image quality could be demonstrated more directly. Could the authors present paired original and enhanced reference images alongside their generated results, keeping the audio, text prompt, and generation settings fixed? Facial close-ups and short video comparisons would help reveal improvements in detail, identity preservation, and temporal consistency beyond aggregate metric gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.