- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DyStream, which generates dyadic talking head video from a reference image and dual-track audio. It combines an autoregressive motion model with a single-frame flow-matching head and introduces a causal Wav2Vec2 variant with controlled audio lookahead. On HDTF and RealTalk, the method reports strong lip-sync quality, 68.49 FPS with 14.6 ms model-side APD at five flow steps, and a response time of about 0.11 s under its stated measurement protocol.
Critical strengths
- The paper addresses the latency of existing chunk-based dyadic talking-head methods (e.g., INFP) with autoregressive frame-by-frame generation and bounded lookahead, and the reported latency represents a meaningful advance for deployable conversational AI systems.
- The causal audio-encoder design is practical and cleanly isolated in the experiments. The lookahead analysis clearly illustrates the quality/latency trade-off, and the paper reports throughput as well as several latency definitions. The frame-wise flow-matching head is a sensible way to preserve short-term one-to-many listener motion.
Critical weaknesses
- The paper relies on a bounded lookahead mechanism to enable low-latency streaming generation, but its implementation and operation are not described in sufficient detail. In particular, it is unclear how the lookahead constraint is enforced throughout the audio encoder and how it translates into the reported streaming latency.
- The dyadic listener evaluation keeps only 38 clips in which one participant listens. It relies on automatic motion statistics and does not report online listener quality under overlap, crosstalk, or leakage between tracks. Consequently, the main listener-generation claim is less well supported than the speaker lip-sync result.
Questions for authors
Clarification:
- How is the 60 ms lookahead constraint enforced throughout the audio encoder, and how does it relate to the reported streaming latency?
- How are speaker identities distributed across the training and test sets, and are they disjoint? Given the limited number and short duration of the listener clips, it would be helpful to discuss how representative this evaluation is and whether the reported metrics consistently support the conclusions about listener-motion quality and diversity.
Suggestion for future work: Evaluate listener quality and stability in free-running online streams with overlap, crosstalk, and simultaneous speaker/listener audio.
3 comments
Response to Reviewer Zwf3 (2/2)
Q2. Identity split; representativeness of the listener evaluation
Identities. Training and test identities are not disjoint: the official RealTalk split is clip-level (as for every method trained on RealTalk, including INFP and Avatar Forcing), and HDTF-100 was also sampled at the clip level. We therefore also evaluated the speaker mode on VFHQ, which is not in our training data: on 63 clips from different videos, Sync-C matches the real videos (6.17 vs. 6.26; difference not significant).
Representativeness. We replaced the 38 clips of Sec. 4.4 with 378 listening clips (target speaks in fewer than 10% of frames) from 84 videos of the official RealTalk test split, with 95% confidence intervals from a bootstrap over videos, and compared with Avatar Forcing (official code and weights; it also uses the partner's video). The mismatched rows drive the listener with another video's partner audio. The table uses the metrics of Table 2, on the 250 clips without camera cuts (at a cut, the GT head pose jumps):
| FD-Exp ↓ | FD-Pose ↓ | MSE-Exp ↓ | MSE-Pose ↓ | SID-Exp ↑ | SID-Pose ↑ | Var-Exp ↑ | Var-Pose ↑ | |
|---|---|---|---|---|---|---|---|---|
| GT | – | – | – | – | 4.99 | 4.20 | 0.266 | 0.954 |
| Static head | 0.053 | 0.117 | 0.0144 | 0.428 | 4.83 | 4.13 | 0 | 0 |
| DyStream | 0.081 | 0.148 | 0.0142 | 0.309 | 4.52 | 4.10 | 0.158 | 0.412 |
| DyStream, mismatched | 0.086 | 0.157 | 0.0160 | 0.317 | 4.47 | 4.07 | 0.165 | 0.419 |
| Avatar Forcing | 0.131 | 0.307 | 0.0150 | 0.286 | 4.75 | 4.14 | 0.162 | 0.304 |
| Avatar Forcing, mismatched | 0.159 | 0.340 | 0.0175 | 0.291 | 4.61 | 4.10 | 0.181 | 0.295 |
| DyStream − Avatar Forcing | −0.051 [−0.087, −0.009] | −0.16 [−0.23, −0.10] | −0.0009 [−0.0026, 0.0009] | +0.023 [0.001, 0.044] | −0.23 [−0.38, −0.05] | −0.04 [−0.18, 0.09] | −0.004 [−0.034, 0.027] | +0.11 [0.02, 0.20] |
| DyStream − mismatched | −0.005 [−0.015, 0.004] | −0.01 [−0.05, 0.03] | −0.0019 [−0.0026, −0.0010] | −0.008 [−0.026, 0.011] | +0.05 [−0.06, 0.11] | +0.03 [−0.01, 0.07] | −0.007 [−0.027, 0.016] | −0.01 [−0.07, 0.07] |
Protocol of Table 2 (Appendix D): 52 MediaPipe blendshapes (Exp) and 6-D head pose, translation and rotation (Pose); SID with k = 40 / 20 clusters fitted on the GT frames. Each video is first cropped to the same face-centred box, since the methods frame the face differently; absolute values therefore differ from Table 2. Italic rows: paired differences with 95% confidence intervals (bootstrap over videos).
The results support three conclusions:
- Realism is better than Avatar Forcing's: FD-Exp and FD-Pose are both significantly lower.
- The listener depends on the partner's speech. A static head has the best FD-Exp and FD-Pose and a mismatched partner scores close to the matched one, so unpaired metrics cannot show this; in a paired test, however, the correct partner audio significantly lowers expression MSE (last row).
- Reactions are timed to the partner. At 1,549 partner pause onsets, real listeners move more right after the partner pauses (+0.175 [0.053, 0.304], relative to pseudo-pairs). DyStream reproduces this response almost exactly (+0.176 [0.070, 0.291]; difference from GT +0.001); it vanishes with a mismatched partner, and Avatar Forcing shows none.
They do not show a diversity advantage: Avatar Forcing has a higher SID-Exp (4.75 vs. 4.52), while DyStream's pose variance is closer to GT (0.41 vs. 0.30; GT 0.95). We will word Sec. 4.4 accordingly.
W2. Online listener quality under overlap, crosstalk and leakage
With separated tracks, the listener shows no spurious lip motion, but leakage is a real failure mode: even −20 dB of partner speech in the agent's own channel doubles the listener's mouth opening. Setup: on 100 RealTalk listener clips, we leaked the partner's speech into the agent's own channel and measured the listener's mouth opening while the partner speaks.
| Own-channel condition | Lip aperture | Frames with open mouth |
|---|---|---|
| GT listener | 0.022 | 8.2% |
| Clean separated tracks | 0.020 | 6.8% |
| Leak −20 dB | 0.048 | 16.4% |
| Leak −10 / 0 dB | 0.048 / 0.049 | 16.4% / 16.6% |
In deployment, the agent's own channel is its TTS output and contains no partner speech. Training with leakage augmentation is the principled fix and future work. We will add this analysis and state that DyStream assumes separated or echo-cancelled tracks, which full-duplex speech systems provide.
I thank the authors for the detailed clarifications. The lookahead mechanism and latency decomposition are now clear, and the expanded listener evaluation with confidence intervals is helpful. I acknowledge the non-disjoint split is a dataset-level limitation and appreciate the authors scoping the diversity claim accordingly. My remaining concern is minor.
Response to Reviewer Zwf3 (1/2)
We thank the reviewer for the positive assessment and the precise questions.
Q1/W1. How the 60 ms lookahead is enforced, and how it maps to latency
Enforcement. The lookahead is used only in the first Transformer layer of the audio encoder; all later layers are strictly causal, so the future context stays at exactly 60 ms. In the first layer, audio frame t (20 ms per frame) attends up to frame t + n, i.e. 60 ms ahead for n = 3; in every later layer, each frame attends only to itself and earlier frames. If later layers also looked ahead, the leakage of future information would accumulate with depth.
The attention masks fully control the future context because we redesigned Wav2Vec2 into Rope-Wav2Vec2 (Sec. 3.3), removing its two non-causal parts: GroupNorm in the convolutional feature extractor, replaced by per-timestep LayerNorm, and the convolutional positional embedding (kernel 128, i.e. 1.28 s of implicit lookahead), replaced by RoPE. Because the architecture differs from the original, we pretrain the full-attention Rope-Wav2Vec2 from scratch with the Wav2Vec2 self-supervised objective and distill the lookahead variant from it.
Latency (RTX 4090, Sec. 4.7). The lookahead delays only the speaker track:
During the rebuttal, we also sped up inference without retraining, using fused GPU kernels, fp16 and CUDA graphs. On the same GPU, one streaming step (audio encoding, motion generation and rendering) is now 3× faster than our original implementation, and the generated frames are visually unchanged. A step takes 11.4 ms in total: audio encoding 2.7 ms, motion generation 3.0 ms and the warping decoder (rendering) 5.7 ms. Sec. 4.7 will report the new latency. Even with 80 ms lookahead, the total latency is then about 91 ms (80 + 11.4 ms), below 100 ms.