acceptodds
Under review as a conference paper at ICLR 2027

Aligning the Clocks: Alignment-Aware Causal Distillation for Streaming Audio-Driven Avatars

Abstract

Streaming audio-driven avatars require a video generator to produce each new chunk using only an image identity, past visual context, and a temporally bounded audio signal. Existing diffusion distillation recipes largely treat this problem as model compression. We argue that causal avatar distillation has an additional failure mode: the clocks of waveform time, audio features, video frames, VAE latents, and autoregressive blocks are not automatically aligned. This mismatch is amplified when a bidirectional teacher is distilled into a causal student, and can manifest as lip lag, weak mouth motion, chunk-boundary discontinuities, identity drift, and exposure drift. We introduce , an alignment-aware causal distillation framework combining causal projection of teacher trajectories, explicitly bounded audio visibility with monotonic audio–latent alignment, mouth-dynamics-aware score matching, and partition-invariant chunk consistency. The design preserves the original OmniAvatar audio-conditioning path rather than introducing an artificial audio cross-attention module. Our current implementation provides a reproducible baseline suite over OmniAvatar and CausVid. Preliminary results show that ODE initialization improves selected visual and synchronization dimensions but does not dominate direct causal DMD on every metric. We therefore separate completed evidence from the proposed controlled study, which tests whether temporal alignment, rather than ODE initialization alone, is the central bottleneck.

Primary area:
generative models
Keywords:
video generation, avatar generation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_z4yhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an approach for causal distillation of video generation model. The main components of the method include distilling a teacher model into a causal student using Distribution Matching Distillation and CausVid. The method attempts to resolve issues such as poor lip alignment with what is referred to as monotonic audio-latent alignment. Quantitative experiments are carried out in comparison with several other approaches, but favour these latter approaches mostly.

Critical strengths

The problem discussed in the paper is a key, and timely, issue in the field of causal video generation.

Critical weaknesses

1/ I found the paper very poorly written and unclear. Many of the sentences' meanings are not at all explicit, and it is left to the reader to fill in the holes. Techniques, concepts and acronyms are not introduced or explained, among other things:

  • Line 38: "Wan" Wan is not explained, no reference given ?
  • Line 39: "DMD ..." acronym not explained
  • Line 40: "Yin et al, 2024b;a" weird citation formatting
  • Line 42: "The preliminary results show why trajectory intialization alone is not a sufficient research claim". This sentence does not make sense. "Trajectory intialization alone" cannot be a "resarch claim".
  • Line 47: "The key observation is that OmniAvatar (Gan et al., 2025) does not inject audio through an audio K/V cross-attention stream. Its audio path uses a Wav2Vec encoder (Schneider et al., 2019), temporal AudioPack, linear projection, and additive injection into selected DiT blocks." Impossible to understand, because nothing is introduced. What is an "AudioPack"
  • Line 183: "The student is optimized with the CausVid DMD framework". Sentence is unclear, CausVid and DMD are two different papers.
  • Line 189: "Let It\mathcal{I}_t be the ..." This notation is not used anywhere else in the paper. Why introduce it ?
  • Line 260: "BAII CS-Dialogue": put the reference again.
  • Line 308: "Figure 1 reports the engineering benchmark that has already been recorded, but it is not directly interchangeable with the FPS column in Table". What does this mean ? Not clear. Not the same conditions? If so, what changes?

2/ There are very few visual comparisons with other methods. The only examples are with AvatarForcing and "Direct Causal DMD", and it is impossible to see anything because the images are so squashed. There are no results in any supplementary material (there is no supplementary material), and no link to any video results. Thus, it is almost impossible to correctly evaluate the results visually.

3/ The quantitative results are heavily in favour of other methods, with only speed being better for the proposed method. All quantitative quality metrics favour competing methods. This, in conjunction with the fact that no video results are shown (no supplementary material), calls into question the

4/ The visual presentation of the results is extremely poor. Figure 2 shows images which are compressed in the yy axis, and no effort has been made to put them into the original aspect ratio. In this same Figure, the box in green has text ("Keys/values visible ...") which is partially hidden by other text. In Figure 3, it is even worse, the images are completely flattened. I really feel you need to re-read this paper in its entirety, it gives the impression that it was done extremely quickly, with no re-reading.

Questions for authors

1/ Line 206: "The alignment can first be evaluated by a grid search and then learned with a monotonic regularizer". What is this monotonic regularizer ? τa(b)\tau_a(b) (equation 3) is not used anywhere else in the manuscript.

2/ Line 212: "Let RbR_b be a detected mouth region and ϕ\phi a local feature map". Detected with what ? How are these defined? There are no details!

2/ Line 199: What is an "audio center" ?

2/ Why is Section 2.4 in the related work? It should be at the beginning of the results, since it explains your metrics. The current organisation makes things difficult to read.

Reviewer_ReR7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Clockwork Avatar, an alignment-aware causal distillation framework for streaming audio-driven avatar generation. Building on OmniAvatar and CausVid-style DMD, it aims to address temporal mismatches between audio features, video latents, and causal generation blocks through bounded audio lookahead, temporal alignment, mouth-motion supervision, and chunk consistency. Preliminary experiments evaluate generation quality and audio-visual synchronization.

Critical strengths

The paper identifies a relevant practical issue in streaming avatar generation: temporal misalignment and potential future-information leakage in audio conditioning. The investigation of audio lookahead and temporal offsets is potentially useful.

Critical weaknesses
  1. Incomplete and insufficient experimental validation. The submission appears unfinished. Table 2 includes projected rather than measured results, explicitly stating that "They must be replaced by matched-protocol measurements before submission." Table 1 uses inconsistent evaluation sample sizes, with some methods evaluated on only 4 or 10 samples. Important evaluations, including FVD and long-horizon stability, are missing. These issues substantially undermine the experimental conclusions.

  2. Limited novelty and unclear implementation. The framework largely combines existing techniques, including causal DMD, bounded audio conditioning, and auxiliary consistency losses, without demonstrating a sufficiently novel methodological contribution. Several components are described as proposed or remain unimplemented, making it unclear what was actually evaluated. The absence of an appendix further limits reproducibility. The presentation is also below expectations, with poorly organized and difficult-to-read figures. Additionally, the mandatory ICLR 2027 AI use statement is missing.

Questions for authors
  1. Which proposed components are actually implemented and included in the reported results? Can the authors provide component-wise ablations to justify their contributions?

  2. Why does Table 2 contain unmeasured, projected results? Can the authors provide reliable matched-protocol comparisons and long-horizon evaluations?

  3. Can the authors clarify the training configurations and implementation details that are currently missing?

Reviewer_BJjqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an alignment-aware framework for causal distillation in audio-driven avatar generation, combining an OmniAvatar teacher with a CausVid-style student. The proposed method addresses temporal alignment across audio features, video representations, and generation chunks. This framework uses bounded audio visibility and explicit timing offsets to control the audio context available during streaming generation.

Critical strengths

The paper identifies how future audio can influence the features used for streaming generation. Specifically, slicing features after bidirectional audio encoding does not prevent future audio from affecting the retained features. Truncating the waveform before encoding limits access to future audio. The lookahead and offset experiments reveal trade-offs between synchronization confidence, timing accuracy, and image quality.

Critical weaknesses
  1. The abstract and conclusion describe the full framework as established, while Sec. 3.3 presents the causally projected teacher as a proposed component and Sec. 4.5 states that AR-teacher initialization and causal consistency are not implemented. The setup table in Sec. 4.4 lists a bidirectional teacher and trajectory initialization for the alignment-aware run, but does not specify which additional alignment components are active. The partition term in Eq. 7 is also absent from the total objective in Eq. 6. It therefore remains unclear which proposed components are evaluated in Table 1 and how much each contributes.
  2. Table 1 and Fig. 1 provide preliminary measurements, but the comparison with other systems in Table 2 remains incomplete because the performance values for Ours are estimates. Table 1 also mixes runs evaluated on very different numbers of clips, and the trajectory-only row is computed on only a handful, yet it is used to argue that initialization alone is insufficient (Sec. 4.1). Measurements under a shared protocol with equal sample counts are needed to establish comparative performance.
Questions for authors
  1. Please identify the components used in each Table 1 experiment, including the teacher's information boundary, alignment settings, and active loss terms. How is the partition-consistency term in Eq. 7 incorporated into training?
  2. Sec. 4.2 already commits to replacing the projected entries. Can the authors provide measured results for Ours and LiveTalk on the same clips used in Table 1 during the rebuttal, with the device and precision stated?

Comments on the paper

1 comment

"Values marked with † are proportional projections used only to complete the planned comparison and are not measured results. They must be replaced by matched-protocol measurements before submission."

#7 of https://academiahome.wordpress.com/wp-admin/post.php?post=420&action=edit

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.