acceptodds
Under review as a conference paper at ICLR 2027

DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Model

Abstract

Generating realistic, dyadic talking head video requires ultra-low latency. Existing chunk-based methods require full non-causal context windows, introducing significant delays. This high latency critically prevents the immediate, non-verbal feedback required for a realistic listener. To address this, we present DyStream, a flow matching-based autoregressive model that could generate video in real-time from both speaker and listener audio. Our method contains two key designs: (1) we adopt a stream-friendly autoregressive framework with flow-matching heads for probabilistic modeling, and (2) we minimize reliance on future context by using a causal encoder with a minimal (e.g., 60ms) lookahead module. We analyzed alternative causal strategies, including distillation and generative encoders. Our results show that a simple lookahead module is both simple and effective. Extensive experiments show that \ourmethod could generate video within 34ms per frame, guaranteeing the entire system latency remains under 100ms. Besides, it achieves state-of-the-art lip-sync quality, with offline and online LipSync Confidence scores of 8.13 and 7.67 on HDTF, respectively. The model, weights and codes are available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming generation, conversational avatars, listening and talking, real-time interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zwf3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DyStream, which generates dyadic talking head video from a reference image and dual-track audio. It combines an autoregressive motion model with a single-frame flow-matching head and introduces a causal Wav2Vec2 variant with controlled audio lookahead. On HDTF and RealTalk, the method reports strong lip-sync quality, 68.49 FPS with 14.6 ms model-side APD at five flow steps, and a response time of about 0.11 s under its stated measurement protocol.

Critical strengths
  1. The paper addresses the latency of existing chunk-based dyadic talking-head methods (e.g., INFP) with autoregressive frame-by-frame generation and bounded lookahead, and the reported latency represents a meaningful advance for deployable conversational AI systems.
  2. The causal audio-encoder design is practical and cleanly isolated in the experiments. The lookahead analysis clearly illustrates the quality/latency trade-off, and the paper reports throughput as well as several latency definitions. The frame-wise flow-matching head is a sensible way to preserve short-term one-to-many listener motion.
Critical weaknesses
  1. The paper relies on a bounded lookahead mechanism to enable low-latency streaming generation, but its implementation and operation are not described in sufficient detail. In particular, it is unclear how the lookahead constraint is enforced throughout the audio encoder and how it translates into the reported streaming latency.
  2. The dyadic listener evaluation keeps only 38 clips in which one participant listens. It relies on automatic motion statistics and does not report online listener quality under overlap, crosstalk, or leakage between tracks. Consequently, the main listener-generation claim is less well supported than the speaker lip-sync result.
Questions for authors

Clarification:

  1. How is the 60 ms lookahead constraint enforced throughout the audio encoder, and how does it relate to the reported streaming latency?
  2. How are speaker identities distributed across the training and test sets, and are they disjoint? Given the limited number and short duration of the listener clips, it would be helpful to discuss how representative this evaluation is and whether the reported metrics consistently support the conclusions about listener-motion quality and diversity.

Suggestion for future work: Evaluate listener quality and stability in free-running online streams with overlap, crosstalk, and simultaneous speaker/listener audio.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Zwf3 (1/2)

We thank the reviewer for the positive assessment and the precise questions.

Q1/W1. How the 60 ms lookahead is enforced, and how it maps to latency

Enforcement. The lookahead is used only in the first Transformer layer of the audio encoder; all later layers are strictly causal, so the future context stays at exactly 60 ms. In the first layer, audio frame t (20 ms per frame) attends up to frame t + n, i.e. 60 ms ahead for n = 3; in every later layer, each frame attends only to itself and earlier frames. If later layers also looked ahead, the leakage of future information would accumulate with depth.

The attention masks fully control the future context because we redesigned Wav2Vec2 into Rope-Wav2Vec2 (Sec. 3.3), removing its two non-causal parts: GroupNorm in the convolutional feature extractor, replaced by per-timestep LayerNorm, and the convolutional positional embedding (kernel 128, i.e. 1.28 s of implicit lookahead), replaced by RoPE. Because the architecture differs from the original, we pretrain the full-attention Rope-Wav2Vec2 from scratch with the Wav2Vec2 self-supervised objective and distill the lookahead variant from it.

Latency (RTX 4090, Sec. 4.7). The lookahead delays only the speaker track:

  • Per-frame generation takes at most 34 ms (audio features ≈4 ms, motion generation 14.6 ms APD from Table 4, rendering ≈10 ms), below the 40 ms frame interval.
  • Speaker track: 60 ms lookahead + per-frame time = 94 ms model-side latency.
  • With 100 ms audio packets, the response time is 0.11 s (Table 5).

During the rebuttal, we also sped up inference without retraining, using fused GPU kernels, fp16 and CUDA graphs. On the same GPU, one streaming step (audio encoding, motion generation and rendering) is now 3× faster than our original implementation, and the generated frames are visually unchanged. A step takes 11.4 ms in total: audio encoding 2.7 ms, motion generation 3.0 ms and the warping decoder (rendering) 5.7 ms. Sec. 4.7 will report the new latency. Even with 80 ms lookahead, the total latency is then about 91 ms (80 + 11.4 ms), below 100 ms.

AuthorsOpenReview ↗

Response to Reviewer Zwf3 (2/2)

Q2. Identity split; representativeness of the listener evaluation

Identities. Training and test identities are not disjoint: the official RealTalk split is clip-level (as for every method trained on RealTalk, including INFP and Avatar Forcing), and HDTF-100 was also sampled at the clip level. We therefore also evaluated the speaker mode on VFHQ, which is not in our training data: on 63 clips from different videos, Sync-C matches the real videos (6.17 vs. 6.26; difference not significant).

Representativeness. We replaced the 38 clips of Sec. 4.4 with 378 listening clips (target speaks in fewer than 10% of frames) from 84 videos of the official RealTalk test split, with 95% confidence intervals from a bootstrap over videos, and compared with Avatar Forcing (official code and weights; it also uses the partner's video). The mismatched rows drive the listener with another video's partner audio. The table uses the metrics of Table 2, on the 250 clips without camera cuts (at a cut, the GT head pose jumps):

FD-Exp ↓FD-Pose ↓MSE-Exp ↓MSE-Pose ↓SID-Exp ↑SID-Pose ↑Var-Exp ↑Var-Pose ↑
GT––––4.994.200.2660.954
Static head0.0530.1170.01440.4284.834.1300
DyStream0.0810.1480.01420.3094.524.100.1580.412
DyStream, mismatched0.0860.1570.01600.3174.474.070.1650.419
Avatar Forcing0.1310.3070.01500.2864.754.140.1620.304
Avatar Forcing, mismatched0.1590.3400.01750.2914.614.100.1810.295
DyStream − Avatar Forcing−0.051 [−0.087, −0.009]−0.16 [−0.23, −0.10]−0.0009 [−0.0026, 0.0009]+0.023 [0.001, 0.044]−0.23 [−0.38, −0.05]−0.04 [−0.18, 0.09]−0.004 [−0.034, 0.027]+0.11 [0.02, 0.20]
DyStream − mismatched−0.005 [−0.015, 0.004]−0.01 [−0.05, 0.03]−0.0019 [−0.0026, −0.0010]−0.008 [−0.026, 0.011]+0.05 [−0.06, 0.11]+0.03 [−0.01, 0.07]−0.007 [−0.027, 0.016]−0.01 [−0.07, 0.07]

Protocol of Table 2 (Appendix D): 52 MediaPipe blendshapes (Exp) and 6-D head pose, translation and rotation (Pose); SID with k = 40 / 20 clusters fitted on the GT frames. Each video is first cropped to the same face-centred box, since the methods frame the face differently; absolute values therefore differ from Table 2. Italic rows: paired differences with 95% confidence intervals (bootstrap over videos).

The results support three conclusions:

  • Realism is better than Avatar Forcing's: FD-Exp and FD-Pose are both significantly lower.
  • The listener depends on the partner's speech. A static head has the best FD-Exp and FD-Pose and a mismatched partner scores close to the matched one, so unpaired metrics cannot show this; in a paired test, however, the correct partner audio significantly lowers expression MSE (last row).
  • Reactions are timed to the partner. At 1,549 partner pause onsets, real listeners move more right after the partner pauses (+0.175 [0.053, 0.304], relative to pseudo-pairs). DyStream reproduces this response almost exactly (+0.176 [0.070, 0.291]; difference from GT +0.001); it vanishes with a mismatched partner, and Avatar Forcing shows none.

They do not show a diversity advantage: Avatar Forcing has a higher SID-Exp (4.75 vs. 4.52), while DyStream's pose variance is closer to GT (0.41 vs. 0.30; GT 0.95). We will word Sec. 4.4 accordingly.

W2. Online listener quality under overlap, crosstalk and leakage

With separated tracks, the listener shows no spurious lip motion, but leakage is a real failure mode: even −20 dB of partner speech in the agent's own channel doubles the listener's mouth opening. Setup: on 100 RealTalk listener clips, we leaked the partner's speech into the agent's own channel and measured the listener's mouth opening while the partner speaks.

Own-channel conditionLip apertureFrames with open mouth
GT listener0.0228.2%
Clean separated tracks0.0206.8%
Leak −20 dB0.04816.4%
Leak −10 / 0 dB0.048 / 0.04916.4% / 16.6%

In deployment, the agent's own channel is its TTS output and contains no partner speech. Training with leakage augmentation is the principled fix and future work. We will add this analysis and state that DyStream assumes separated or echo-cancelled tracks, which full-duplex speech systems provide.

I thank the authors for the detailed clarifications. The lookahead mechanism and latency decomposition are now clear, and the expanded listener evaluation with confidence intervals is helpful. I acknowledge the non-disjoint split is a dataset-level limitation and appreciate the authors scoping the diversity claim accordingly. My remaining concern is minor.

Reviewer_EHTpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies low-latency talking-head generation from two audio streams and a reference image. DyStream combines an autoregressive motion model with a flow-matching head. The results show a useful quality–latency trade-off.

Critical strengths
  • The paper tackles a practical bottleneck. It clearly distinguishes fast generation from low response latency, which matters for interactive avatars.
  • The paper is self-contained and well-written.
  • The lookahead ablation is informative. It shows how lip-sync quality changes as future audio is restricted, making the main design choice easy to assess.
Critical weaknesses
  • The methodological novelty is limited. Autoregressive generation, flow-based prediction heads, and short-lookahead audio processing are established ideas. The contribution is mainly their combination for streaming dyadic avatars, so the strength of the paper rests largely on the evaluation.
  • The listener evaluation uses only 38 clips and measures motion statistics rather than whether reactions are timely and appropriate.
  • The baseline coverage is limited. Some stronger methods [1,2] are missed. The results also do not establish an overall SOTA claim.
  • The two-stage design limits joint motion and visual modeling. Motion is generated in a separate latent space and then rendered, so the model does not directly capture how motion interacts with visual details. This limits its ability to handle occlusions, moving accessories, and other complex motion issues.
  • The video in the suppl html show unsatisfactory quality (especially tooth details).

[1] Ki, T., Jang, S., Jo, J., Yoon, J., & Hwang, S. J. (2026). Avatar forcing: Real-time interactive head avatar generation for natural conversation. arXiv preprint arXiv:2601.00664.

[2] Xie, Y., Gu, T., Li, Z., Zhang, C., Song, G., Zhao, X., ... & Luo, L. (2025). X-streamer: Unified human world modeling with audiovisual interaction. arXiv preprint arXiv:2509.21574.

Questions for authors

Please refer to the Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer EHTp (1/2)

We thank the reviewer for the constructive comments. We added substantial new experiments, including a 10× larger listener evaluation with timing and paired tests (W2) and comparisons with new baselines (W3).

W1. Novelty

Our contributions lie in what is needed to make these ideas work for streaming dyadic generation:

  1. An audio encoder whose future context is set exactly by the attention mask. Standard Wav2Vec2 hides future context in GroupNorm and in its convolutional positional embedding (1.28 s), so masking attention alone does not make it streaming. Rope-Wav2Vec2 removes both (Sec. 3.3), so the lookahead is exactly what the mask allows.
  2. Showing that a short lookahead, on the speaker track only, recovers streaming lip sync. Purely causal encoders, including pretrained causal codecs and distillation, are poor for lip sync, while a short lookahead recovers most of the gap (Sync-C 5.98 / 6.90 / 7.39 / 7.67 at 20 / 40 / 60 / 80 ms, Table 3). The partner track needs none, so listener reactions are not delayed.
  3. A real-time dyadic system: each frame is generated within 34 ms on one RTX 4090, with a model-side latency within 94 ms (Sec. 4.7) and a 0.11 s response time, vs. 1.21–3.27 s for SoulX-FlashHead (Table 5); simply shrinking the chunk of a chunk-based model instead collapses lip sync (INFP_I*: Sync-C 1.72 at 40 ms, Table 1).

W3. Missing baselines; SOTA claim

Avatar Forcing (official code and weights). DyStream is better as both listener and speaker and has lower latency:

  • Listener (RealTalk, W2; Avatar Forcing also receives the partner's video, which DyStream does not use): FD-Exp 0.081 vs. 0.131 and FD-Pose 0.15 vs. 0.31 (both significant), and only DyStream reacts to partner pauses.
  • Speaker (63 VFHQ clips, not in our training data): Sync-C 6.17 vs. 5.54 and FVD 99.6 vs. 129.2.
  • Latency. Avatar Forcing generates 10-frame (400 ms) blocks conditioned on the whole block's audio, so a block's first frame cannot appear before its 400 ms of audio has arrived; DyStream generates single frames with a 60 ms lookahead. During the rebuttal, we also sped up our inference 3× without retraining (fused GPU kernels, fp16 and CUDA graphs; generated frames are visually unchanged): a streaming step now takes 11.4 ms in total, so even with 80 ms lookahead the total latency is about 91 ms, below 100 ms.

X-Streamer. Without released code or weights, a quantitative comparison is not possible. The two methods address different settings: X-Streamer is a full agent that generates its own text and speech and renders them with chunk-wise video diffusion on two A100s, whereas DyStream is a lightweight visual front-end driven by given dual-track audio. We will discuss both works with a design/latency comparison table.

SOTA. We will remove the overall SOTA claim; the abstract will claim the best lip sync among the streaming/causal methods we evaluated.

W4. Two-stage design

Sec. 5 notes this limitation, and we will expand it. We chose the two-stage design for speed: autoregression in a 512-dimensional motion latent allows per-frame generation within 34 ms on a single consumer GPU, whereas pixel- or video-latent AR models need several high-end GPUs for real time (e.g., X-Streamer: 2×A100).

W5. Video quality, especially teeth

The teeth artifacts come from the warping-based renderer rather than the motion generator: when the reference image shows a closed mouth, the renderer has to synthesize the teeth without any visual reference. In future work, we will fine-tune the renderer on more data with clearly visible teeth and retrain DyStream with a stronger warping-based renderer.

AuthorsOpenReview ↗

Response to Reviewer EHTp (2/2)

W2. Listener evaluation: 38 clips, motion statistics only

We replaced it with a 10× larger evaluation that tests appropriateness and timeliness directly. We use 378 listening clips from 84 videos of the official RealTalk test split (target speaks in fewer than 10% of frames, both faces visible), with 95% confidence intervals from a bootstrap over videos. We add Avatar Forcing with its official code and weights (W3), a static head, and a mismatched partner (listener driven by another video's partner audio). With the metrics of our Table 2, on the 250 clips without camera cuts (at a cut, the GT head pose jumps):

FD-Exp ↓FD-Pose ↓MSE-Exp ↓MSE-Pose ↓SID-Exp ↑SID-Pose ↑Var-Exp ↑Var-Pose ↑
GT––––4.994.200.2660.954
Static head0.0530.1170.01440.4284.834.1300
DyStream0.0810.1480.01420.3094.524.100.1580.412
DyStream, mismatched0.0860.1570.01600.3174.474.070.1650.419
Avatar Forcing0.1310.3070.01500.2864.754.140.1620.304
Avatar Forcing, mismatched0.1590.3400.01750.2914.614.100.1810.295
DyStream − Avatar Forcing−0.051 [−0.087, −0.009]−0.16 [−0.23, −0.10]−0.0009 [−0.0026, 0.0009]+0.023 [0.001, 0.044]−0.23 [−0.38, −0.05]−0.04 [−0.18, 0.09]−0.004 [−0.034, 0.027]+0.11 [0.02, 0.20]
DyStream − mismatched−0.005 [−0.015, 0.004]−0.01 [−0.05, 0.03]−0.0019 [−0.0026, −0.0010]−0.008 [−0.026, 0.011]+0.05 [−0.06, 0.11]+0.03 [−0.01, 0.07]−0.007 [−0.027, 0.016]−0.01 [−0.07, 0.07]

Protocol of Table 2 (Appendix D): 52 MediaPipe blendshapes (Exp) and 6-D head pose, translation and rotation (Pose); SID with k = 40 / 20 clusters fitted on the GT frames. Each video is first cropped to the same face-centred box, since the methods frame the face differently; absolute values therefore differ from Table 2. Italic rows: paired differences with 95% confidence intervals (bootstrap over videos).

  • Realism. DyStream is significantly better than Avatar Forcing on FD-Exp and FD-Pose and closer to GT in pose variance; Avatar Forcing has a higher SID-Exp and a lower MSE-Pose. Unpaired statistics, however, cannot separate good listening from no listening: the static head has the best FD-Exp and FD-Pose.
  • Appropriateness. The listener's expression depends on the partner's speech: in a paired test (same clip, matched vs. mismatched partner), the matched partner gives significantly lower expression MSE (by 0.0019 [0.0010, 0.0026], lower in 68% of clips).
  • Timeliness. Like real listeners, DyStream reacts to partner pauses; Avatar Forcing does not. At 1,549 partner pause onsets (energy VAD, ≥0.2 s of silence), we measure how much more the listener's head and face move in the second after the onset than in the second before, relative to pseudo-pairs (the same listener aligned with other videos' pauses):
Motion increase after pauseDifference from GT (0 is best)
GT+0.175 [0.053, 0.304]–
DyStream+0.176 [0.070, 0.291]+0.001 [−0.151, 0.148]
DyStream, mismatched−0.011 [−0.089, 0.074]−0.186 [−0.353, −0.037]
Avatar Forcing+0.020 [−0.072, 0.113]−0.154 [−0.310, −0.005]

Motion speed is normalized within each clip by its median and median absolute deviation (MAD).

DyStream's response matches that of real listeners (difference +0.001). With a mismatched partner, the response disappears at the true partner's pauses (paired Δ = 0.187 [0.084, 0.300]) but appears at the pauses in the audio the model actually receives (+0.194 [0.098, 0.291]); the reaction is thus driven by the partner's speech.

Reviewer_EHTpOpenReview ↗

Concerns remain

Thanks for your rebuttal, but some of these parts cannot convince me. I still have some concerns:

  • My main concern is that the methodological novelty is limited. The framework is a combination of existing techniques and the claimed main target is efficiency. However, a publication paper must bring some technical insights for the community.
  • Video quality is a crucial point for video generation work. However, the authors blame the unsatisfactory quality to the backbone model instead of solving it in the original proposed method.
AuthorsOpenReview ↗

Response to Reviewer EHTp

Thank you for reading our rebuttal carefully. We are glad the expanded evaluation to address those concerns. Alongside addressing your two points below, we conducted a user study to evaluate perceptual visual quality, speaking/listening quality, and overall preference directly.

1. Technical insights

Beyond system speed, our paper contributes two sets of empirical findings for streaming avatars:

  1. Why naive streaming fails, and the ~60 ms coarticulation horizon (Sec. 3.3, 4.3, 4.6). Shrinking the generation window of a chunk-based model collapses lip sync (INFP_I*: Sync-C 1.72 at 40 ms, Table 1), while masking attention in standard Wav2Vec2 still leaks 1.28 s of future audio via GroupNorm and convolutional positional embeddings. Once hidden paths are removed (Rope-Wav2Vec2), every purely causal alternative, including pretrained codecs (Mimi, EnCodec, Wav2Vec), causal distillation (Sync-C 3.02), and VAE distillation (3.50), hits a clear ceiling (Table 3), whereas 40–60 ms lookahead recovers most of the gap to full attention (3.02 → 5.98 → 6.90 → 7.39 at 0 / 20 / 40 / 60 ms). Coarticulation thus cannot be learned away by purely causal encoders or distillation alone, yet needs only ~40–60 ms of lookahead rather than 400–960 ms chunks.
  2. Making frame-wise motion autoregression expressive and drift-free (Sec. 3.2, 4.5, Appendix B). Three findings stand out from our ablations: (i) a small per-frame flow-matching MLP avoids the mean collapse of deterministic L2 loss (which lowers listener variance and speaker Sync-C from 8.14 to 7.87) at only 0.5–0.7 ms/step; (ii) frame-aligned audio addition outperforms cross-attention in both Sync-C (8.14 vs. 7.66) and speed (+23.1%), showing local alignment beats global audio cross-attention; and (iii) sampling training anchors from the last 10 frames eliminates cumulative AR drift in long continuous streams (Table 6 shows 3-minute generation results).

2. Video quality and user study

That is a fair criticism. Our previous reply was poorly worded and sounded like shifting blame to the renderer, whereas final visual quality of the full pipeline is our responsibility.

To evaluate perceptual quality directly, we ran a preference study comparing DyStream and Avatar Forcing on VFHQ and RealTalk (21 raters, 10 speaking and 10 listening samples each, 7.96 s per clip, identical 512×512 face framing, with the partner's video shown for listening):

DyStream vs. Avatar ForcingSpeakingListening
Visual quality71.9% / 28.1%79.0% / 21.0%
Lip-audio match / Listening reaction60.5% / 39.5%88.1% / 11.9%
Overall preference63.8% / 36.2%79.5% / 20.5%

DyStream clearly outperforms Avatar Forcing, which likewise supports both speaker and listener generation, across every criterion, winning 71.9% vs. 28.1% (speaking) and 79.0% vs. 21.0% (listening) on Visual Quality, 88.1% vs. 11.9% on Listening Reaction, and 73.8% vs. 26.2% across all six criteria combined. This is consistent with our VFHQ FVD (99.6 vs. 129.2) and HDTF-100 VBench Quality (0.642 vs. 0.642 GT, 0.638 Hallo3, 0.640 SoulX-FlashHead-Pro).

Regarding teeth specifically, artifacts indeed appear mainly when the reference image shows no teeth at all. This is a known limitation shared by single-image warping methods (e.g., LivePortrait and LIA), which trade heavy pixel-level diffusion priors for lightweight real-time rendering (our full pipeline uses under 4 GB of GPU memory), whereas visible reference teeth are preserved well. We will include the user study, explicitly document this closed-mouth limitation in Sec. 5, and explore dedicated inner-mouth synthesis in future work.

Reviewer_p4XQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DyStream generates dyadic talking-head video from a single image and two audio tracks (the agent's own speech and the partner's speech) frame by frame. It uses a LivePortrait/LIA-style warping autoencoder for appearance and motion, and an autoregressive Transformer with a per-frame flow-matching head (MAR-style) for audio-to-motion generation. A modified Wav2Vec2 (LayerNorm instead of GroupNorm, RoPE instead of the convolutional position encoding) makes the future receptive field controllable by the attention mask, allowing 60 ms lookahead for the speaker track and none for the listener track. The paper reports offline Sync-C 8.13 and online 7.67 on HDTF, 34 ms per frame on an RTX 4090, and a listener evaluation on 38 RealTalk clips against a re-implemented INFP.

Critical strengths
  1. A real deployment problem, measured carefully. Chunk-based dyadic models must buffer about 1 s of audio before reacting. The paper separates per-frame generation time, model-side latency, and response time (Sec. 4.7), and it trains a latency-matched INFP variant (INFP_I) instead of only quoting the original. The comparison with SoulX-FlashHead (0.11 s vs. 1.21–3.27 s response) makes the practical gain concrete.
  2. The audio-encoder analysis transfers beyond this paper. Table 3 shows that pretrained causal codecs (Mimi, EnCodec) are weak for lip sync, that distillation alone helps little, and how Sync-C recovers with 20/40/60/80 ms of lookahead (5.98/6.90/7.39/7.67). This is useful for any streaming audio-driven generation work.
Critical weaknesses

Update after rebuttal. Both conditions stated in my review are met: the abstract now reports one consistent configuration (60 ms lookahead, 94 ms, Sync-C 7.39), and CFG weights selected on two held-out splits match the submitted weights closely. W1 is largely resolved by Sync-D / FID / FVD and a lip-amplitude analysis showing no exaggerated articulation. The listener evaluation was expanded to 378 clips with floors, a paired test and a timing analysis, which shows partner-dependent responses. W2 is resolved for cost and long-horizon stability. Remaining: no perceptual study, train/test identity overlap on HDTF and RealTalk, and the revisions are promised but not yet in the PDF. Rating raised from 2 to 3.

  1. The abstract combines numbers from different settings, and the quality evidence rests mainly on Sync-C. The online Sync-C of 7.67 is the 80 ms lookahead result (Tables 1 and 3), while the "under 100 ms" latency assumes 60 ms lookahead, where Sync-C is 7.39; 80 ms lookahead would give about 114 ms. "State-of-the-art lip sync" also needs a qualifier, since Sonic reaches 8.50. Moreover, the offline model's Sync-C (8.14) exceeds that of the ground-truth videos (7.61), so Sync-C alone cannot separate accurate from exaggerated articulation, and the other reported metrics (content similarity, image quality, dynamic degree) do not measure articulation. Appendix F and Fig. 4 also show that the CFG weights were chosen by maximizing Sync-C over a grid, where Sync-C varies from about 6.2 to 8.05; the paper does not say which split was used for this choice. FID/FVD, Sync-D, or a user study are needed, and the guidance weights should be selected on a held-out split.
  2. The flow-matching head and the AR design choices are weakly supported. The flow-matching head helps only a little and its cost is not reported: removing it lowers offline Sync-C from 8.136 to 7.867 (Table 1) and slightly lowers listener variance (Table 2), but the speed of the deterministic variant is not reported, and more steps cost speed for little gain (Table 4: 1/5/10 steps give 84.8/68.5/57.8 FPS and Sync-C 7.17/7.37/7.39). Known AR weaknesses are only partly handled: anchor conditioning limits pose drift (Table 6, 10 clips, still 7.6° mean rotation drift after 3 minutes), but exposure bias and long-horizon lip-sync or image quality are not evaluated.
Questions for authors
  1. (Decision-relevant) Please report Sync-C and end-to-end latency for one consistent configuration in the abstract. What is the total latency at 80 ms lookahead?
  2. (Clarification) The listener evaluation that motivates the paper uses 38 clips, one re-implemented baseline (INFP), and MediaPipe statistics; expression FD is better (0.074 vs. 0.141), but INFP is better on pose FD and MSE (Table 2: 3.158/1.286 vs. 3.192/1.636). Can you add a perceptual or timing-based evaluation of listener behaviour (e.g., a user study, or backchannel onset latency relative to partner speech)? As floors for Table 2, please report the same metrics for a static head and for a mismatched listener (motion generated from another clip's partner audio), with confidence intervals over the 38 clips; the differences to INFP may otherwise be within sampling error.
  3. (Clarification) What is the speed of the variant without the flow-matching head, and how do Sync-C and image quality change over a 3-minute stream?
  4. (Clarification) Is the test set HDTF-100 only, or HDTF plus 100 internal clips? Sec. 4.2 says the collected data is not used for evaluation, but Sec. 4.3 samples test clips from it.
  5. (Decision-relevant) On which data were the CFG weights in Fig. 4 selected? What is Sync-C on HDTF-100 with weights chosen on a separate validation split?
  6. (Suggestion, not required) How does the method differ in design and latency from Teller (CVPR 2025) and ARIG (ICCV 2025), which also generate motion autoregressively for real-time use, and from real-time speaking-and-listening avatars such as StreamAvatar (CVPR 2026) and DualTalk (CVPR 2025), which are not cited? Missing citations are not a reason for my rating.
  7. (Suggestion, not required) The head samples each frame from fresh noise. Streaming schemes that carry noise across frames (Rolling Diffusion, ICML 2024; Diffusion Forcing, NeurIPS 2024) may lower the per-frame sampling cost; a short discussion would help.

If the abstract numbers come from one consistent configuration (Q1) and Sync-C holds with guidance weights chosen on a held-out split (Q5), I would raise my rating to 3.

4 comments
AuthorsOpenReview ↗

Response to Reviewer p4XQ (1/3)

We thank the reviewer for the careful review and answer the two decision-relevant questions, Q1 and Q5, first.

Q1. Abstract numbers and latency

At 80 ms lookahead, the total latency is 114 ms (80 ms lookahead + 34 ms per-frame generation), with online Sync-C 7.67. As in Sec. 4.7, latency is measured on an RTX 4090 from the current audio timestamp to the delivery of its rendered frame.

LookaheadOnline Sync-CPer-frame generationTotal latency
60 ms (default)7.3934 ms94 ms
80 ms7.6734 ms114 ms

The abstract mixed the two rows. The revised abstract reports only the 60 ms default: "DyStream generates each frame within 34 ms, keeping the model-side latency within 94 ms, with an online Sync-C of 7.39 (8.13 offline)." We keep 60 ms because it is the largest lookahead with which the first frame can be produced as soon as the first 100 ms audio packet arrives (Sec. 3.3); Tables 4 and 5 also use this model. We will also replace "state-of-the-art lip-sync" with "the best lip sync among the streaming/causal methods we evaluated", since Sonic (offline) reaches 8.50.

During the rebuttal, we also sped up inference without retraining, using fused GPU kernels, fp16 and CUDA graphs. On the same GPU, one streaming step (audio encoding, motion generation and rendering) is now 3× faster than our original implementation, and the generated frames are visually unchanged. A step takes 11.4 ms in total: audio encoding 2.7 ms, motion generation 3.0 ms and the warping decoder (rendering) 5.7 ms. Sec. 4.7 will report the new latency. Even with 80 ms lookahead, the total latency is then about 91 ms (80 + 11.4 ms), below 100 ms.

Q5. CFG weights selected on a validation split

Data used for Fig. 4. Fig. 4 was computed on HDTF-100, so the submitted weights (w_s = 0.5, w_all = 1.0) were indeed chosen on the test clips. They are not the grid maximum: as stated in Appendix F, we picked a point on the ridge w_s + w_all ≈ 1.5, along which Sync-C changes little.

Sync-C with validation-selected weights. We used two validation splits, 30 HDTF clips not used for training and 32 VFHQ clips from different videos; both select w_s = 0, w_all = 1.5, which is also the best setting in Fig. 4. On each split, we re-ran the 3×3 block of the Fig. 4 grid that contains its best point and the submitted weights (w_l, w_r fixed):

w_s / w_allHDTF validationVFHQ validation
0 / 0.56.285.35
0 / 1.07.906.53
0 / 1.5 (validation-selected)8.126.77
0.5 / 0.57.786.50
0.5 / 1.0 (submitted)8.096.72
0.5 / 1.58.046.66
1 / 0.58.046.67
1 / 1.08.066.65
1 / 1.57.936.47

The submitted weights rank second on both validation splits, and the nine settings rank almost identically on the two sets. Selecting on the test clips therefore did not inflate Table 1. VFHQ is not in our training data, and its in-the-wild interview footage gives lower Sync-C overall (real videos: 6.64). We keep Table 1, add this experiment to Appendix F, and state there that Fig. 4 uses HDTF-100.

Q4. Test set

The test set is HDTF-100 only (100 HDTF clips excluded from training); the mention of internal clips in Sec. 4.3 is an error and will be removed.

Q6. Teller, ARIG, StreamAvatar and DualTalk

Teller generates only the speaker and cannot listen. StreamAvatar builds on a 5B video diffusion model and generates in chunks; its paper reports 1.2 s latency on two H800s, so both its hardware requirements and its latency are far higher than ours. Besides both audio tracks, ARIG needs the partner's facial motion, i.e. an extra camera on the user and face tracking, which raises the deployment cost; its reported motion-generation speed is 31 FPS, below DyStream's 68.5 FPS (Table 4); and its code and weights are not released, so we cannot compare with it directly. DualTalk generates 3D face parameters rather than video and runs a bidirectional LSTM over the whole sequence, so it is not causal and cannot stream.

Q7. Carrying noise across frames

We will discuss Rolling Diffusion and Diffusion Forcing. Since our head denoises one frame with 1–5 steps, the saving is bounded, but staggered noise levels could let 1 or 2 steps approach 5-step quality.

AuthorsOpenReview ↗

Response to Reviewer p4XQ (2/3)

W1. Sync-C alone cannot separate accurate from exaggerated articulation

We added Sync-D, FID, FVD and lip amplitude on the 63 VFHQ test clips (the clips not used for validation in Q5). In short: at our weights the articulation is not exaggerated, and FVD catches over-strong guidance that Sync-C misses.

Setting (w_s / w_all)Sync-CSync-D ↓FID ↓FVD ↓Lip amplitude / GT
Real videos6.267.55––1.00
0 / 0.54.968.9330.2106.20.95
0 / 1.05.997.9929.697.61.05
0 / 1.56.237.8330.5101.01.08
0.5 / 0.55.928.0430.8105.31.04
0.5 / 1.06.177.8530.799.61.07
0.5 / 1.56.157.9130.6102.81.08
1 / 0.56.097.8931.3109.11.07
1 / 1.06.167.8830.8105.41.07
1 / 1.56.068.0131.2106.71.08
Avatar Forcing (official weights)5.548.6230.5129.21.33

Lip amplitude: how much the mouth opens and closes over time (std of the inner-lip gap, MediaPipe landmarks 13/14, normalized by the inter-ocular distance), relative to the real videos (= 1). FID/FVD: on face crops, against the real videos of the same clips; FVD on 16-frame windows.

At the submitted weights, lip amplitude is 1.07× that of the real videos (95% confidence interval [0.98, 1.16], not significantly different), and Sync-C matches the real videos (6.17 vs. 6.26; paired difference −0.08 [−0.34, 0.15]); neither difference is significant, so the articulation is not exaggerated.

FVD does pick up what Sync-C misses. With w_s = 1, Sync-C still looks fine (6.06–6.16), but FVD is significantly worse (105–109); too weak overall guidance (w_all = 0.5) also raises FVD. The submitted and validation-selected weights are in the group with the lowest FVD (97.6–102.8, differences not significant). FID, which looks only at single frames, barely changes (29.6–31.3).

A wider mouth on its own does not raise Sync-C. Avatar Forcing's lip amplitude is 1.33× that of the real videos, yet its Sync-C is only 5.54.

We will add Sync-D, FID and FVD for all methods to Table 1 and report the lip-amplitude analysis.

W2/Q3. Flow-matching head; long-horizon behaviour

Benefit. The head is mainly designed for listener motion, and it also improves speaker lip sync (offline Sync-C 8.136 vs. 7.867 without it, Table 1). Many listener reactions fit the same audio, and a deterministic ℓ2 regressor is pulled toward their mean (Sec. 3.2). Without the head, listener motion becomes flatter (Table 2: Var-Exp 0.275 → 0.246, Var-Pose 0.596 → 0.556) while pose MSE drops (1.636 → 1.472), as expected when predictions move toward the mean. For the same reason INFP, whose pose variance is much lower (0.200 vs. 0.596), has the lower pose MSE in Table 2.

Cost. The head is cheap: each flow step adds only 0.5–0.7 ms (APD 11.8 / 14.6 / 17.3 ms at 1 / 5 / 10 steps, Table 4), as it is a small MLP. Most of the time is spent in the AR Transformer, which the deterministic variant shares.

3-minute streams. Sync-C and image quality do not change over 3 minutes, and the other metrics settle within the first 30 s. To separate the long rollout from changes in content, we repeated the 10-s audio of each of 72 VFHQ clips (not in our training data) 18 times, generated one continuous 180-s stream per clip, and compared every 10-s repetition with the real 10-s clip, which has the same audio:

Real clip0–30 s30–60 s60–90 s90–120 s120–150 s150–180 s
Sync-C ↑6.586.566.576.576.606.626.60
VBench Imaging Quality ↑0.7220.6880.6880.6880.6880.6880.688
FID ↓–27.128.128.228.128.228.1
FVD ↓–86.590.290.589.289.989.6
Head-pose deviation from first frame (°)3.544.224.464.484.464.484.46
Lip amplitude / GT1.001.051.041.041.041.031.04

Mean over the 72 clips. VBench Imaging Quality: MUSIQ (SPAQ) score / 100, on one frame per second. FID/FVD: against the real clips, on face crops as in W1. Head pose: mean absolute difference in pitch, yaw and roll to the first frame, i.e. the reference image (the real clip's first frame). Lip amplitude: as in W1.

Sync-C matches the real clips throughout (6.56–6.62 vs. 6.58), and image quality and lip amplitude do not change. FID, FVD and the pose deviation are slightly lower in the first 30 s, while the head is still close to the reference pose, and stay flat afterwards: between the second and the last 30-s window, no metric changes significantly (all 95% confidence intervals, from a bootstrap over clips, include zero). This matches Table 6, where pose drift saturates within the first minute. The randomized anchor conditioning used during training is thus enough: over 3 minutes, we see no accumulation of error.

AuthorsOpenReview ↗

Response to Reviewer p4XQ (3/3)

Q2. Listener evaluation: baselines, confidence intervals and response timing

We added both floors with confidence intervals, a paired test, a timing-based evaluation and a second baseline, on a 10× larger set. Instead of the 38 partner-only clips of Sec. 4.4, we use 378 clips from 84 videos of the official RealTalk test split (target speaks in fewer than 10% of frames, both faces visible), with 95% confidence intervals from a bootstrap over videos. Since INFP* is our re-implementation, the second baseline is Avatar Forcing (CVPR 2026) with its official code and weights; it receives the same audio plus the partner's video. The mismatched floor drives the listener with another video's partner audio. The table uses the metrics of Table 2, on the 250 clips without camera cuts (at a cut, the GT head pose jumps):

FD-Exp ↓FD-Pose ↓MSE-Exp ↓MSE-Pose ↓SID-Exp ↑SID-Pose ↑Var-Exp ↑Var-Pose ↑
GT––––4.994.200.2660.954
Static head0.0530.1170.01440.4284.834.1300
DyStream0.0810.1480.01420.3094.524.100.1580.412
DyStream, mismatched0.0860.1570.01600.3174.474.070.1650.419
Avatar Forcing0.1310.3070.01500.2864.754.140.1620.304
Avatar Forcing, mismatched0.1590.3400.01750.2914.614.100.1810.295
DyStream − Avatar Forcing−0.051 [−0.087, −0.009]−0.16 [−0.23, −0.10]−0.0009 [−0.0026, 0.0009]+0.023 [0.001, 0.044]−0.23 [−0.38, −0.05]−0.04 [−0.18, 0.09]−0.004 [−0.034, 0.027]+0.11 [0.02, 0.20]
DyStream − mismatched−0.005 [−0.015, 0.004]−0.01 [−0.05, 0.03]−0.0019 [−0.0026, −0.0010]−0.008 [−0.026, 0.011]+0.05 [−0.06, 0.11]+0.03 [−0.01, 0.07]−0.007 [−0.027, 0.016]−0.01 [−0.07, 0.07]

Protocol of Table 2 (Appendix D): 52 MediaPipe blendshapes (Exp) and 6-D head pose, translation and rotation (Pose); SID with k = 40 / 20 clusters fitted on the GT frames. Each video is first cropped to the same face-centred box, since the methods frame the face differently; absolute values therefore differ from Table 2. Italic rows: paired differences with 95% confidence intervals.

The floors turned out to be informative. The static head has an even lower FD than both methods, and its SID-Exp is not low either (the SID histogram pools frames across clips, whose reference frames differ); the mismatched floors are also close to the matched models. Such unpaired distribution metrics compare only overall statistics, so a head that stays at its reference pose and ignores the partner can score well. Whether a listener actually listens must therefore be judged by the paired test and the timing analysis below; a static head does not change with the partner and by construction shows no response on either. Even with 250 clips, the confidence intervals on FD-Pose remain wide, so INFP*'s ~1% advantage in pose FD in Table 2 (3.158 vs. 3.192) is likely within sampling error.

Against Avatar Forcing, DyStream is significantly better on FD-Exp and FD-Pose and closer to GT in pose variance; Avatar Forcing has a higher SID-Exp and a lower MSE-Pose (MSE favours low-variance motion, see W2/Q3).

Paired test (same clip, matched vs. mismatched partner). With the matched partner, expression MSE is significantly lower: 0.0142 vs. 0.0160, about 12% lower (paired difference 0.0019 [0.0010, 0.0026]), and lower in 68% of clips. DyStream's listener expression thus depends on the partner's speech.

Timing. Like real listeners, DyStream reacts to partner pauses; Avatar Forcing does not. At 1,549 partner pause onsets (energy VAD, ≥0.2 s of silence), we measure how much more the listener's head and face move in the second after the onset than in the second before, relative to pseudo-pairs (the same listener aligned with other videos' pauses):

Motion increase after pauseDifference from GT (0 is best)
GT+0.175 [0.053, 0.304]–
DyStream+0.176 [0.070, 0.291]+0.001 [−0.151, 0.148]
DyStream, mismatched−0.011 [−0.089, 0.074]−0.186 [−0.353, −0.037]
Avatar Forcing+0.020 [−0.072, 0.113]−0.154 [−0.310, −0.005]

Motion speed is normalized within each clip by its median and median absolute deviation (MAD).

DyStream's response matches that of real listeners (difference +0.001). With a mismatched partner, the response disappears at the true partner's pauses (paired Δ = 0.187 [0.084, 0.300]) but appears at the pauses in the audio the model actually receives (+0.194 [0.098, 0.291]); the reaction is thus driven by the partner's speech.

We will add these results to Sec. 4.4.

AuthorsOpenReview ↗

Follow-up Response to Reviewer p4XQ (Updated PDF and User Study)

Thank you for the update and for raising your rating. We have uploaded the revised PDF (additions in blue, deletions in red), which incorporates all promised revisions (Sec. 2–5, Appendices D and F–H) and adds a perceptual user study (Appendix I). (Note: the abstract and introduction report our 3× optimized streaming runtime from Q1—11.4 ms/frame and ~71 ms model-side latency, vs. 34 ms and 94 ms unoptimized—for the same 60 ms lookahead, Sync-C 7.39 model without retraining.)

1. Perceptual Study (Appendix I, Table 13)

We ran a blinded, randomized side-by-side preference study comparing DyStream and Avatar Forcing on VFHQ and RealTalk (21 raters, 10 speaking and 10 listening samples each, 7.96 s per clip, identical 512×512 face framing, with the partner's video shown for listening):

DyStream / Avatar ForcingSpeakingListening
Visual quality71.9% / 28.1%79.0% / 21.0%
Lip-audio match / Listening reaction60.5% / 39.5%88.1% / 11.9%
Overall preference63.8% / 36.2%79.5% / 20.5%

DyStream is preferred on every criterion (p < 0.003, two-sided binomial test), and on 73.8% vs. 26.2% across all 1,260 judgments over the six criteria combined.

2. Identity Overlap (Sec. 5, Appendices G–H)

We explicitly note the clip-level identity overlap of HDTF-100 and the official RealTalk split as a limitation in Sec. 5. Meanwhile: (i) VFHQ has zero identity overlap with our training data, where DyStream matches real videos in Sync-C (6.17 vs. 6.26) and lip amplitude (1.07×), outperforms Avatar Forcing in FVD (99.6 vs. 129.2), and is included in the user study above; and (ii) our motion generator operates solely on 512-D motion latents without appearance input, and both our RealTalk paired test and 1,549-pause timing analysis (Appendix H) hold the test clip and reference identity fixed while swapping only the partner's audio, ruling out identity memorization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.