acceptodds
Withdrawn from ICLR 2027

FlowTalker: Causal Streaming Audio-Driven 3D Head Animation via Factorized Motion Modeling with Rectified Flow

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Causally generating expressive 3D head animation as audio streams in is essential for real-time digital human interaction. However, most existing methods rely heavily on future audio context, leading to noticeable performance degradation when only limited or no look-ahead is available in real-time streaming inference. Current streaming-oriented approaches achieve low latency, yet their animation quality remains suboptimal, particularly in terms of expressive facial motion and long-term temporal consistency. We present FlowTalker, a framework for audio-driven 3D head animation under causal streaming inference. Pose, expression, and mouth motion evolve at different temporal scales and respond to distinct audio cues, but prior methods typically model them in a shared motion space. FlowTalker preserves these distinct motion dynamics by encoding them into separate token streams using spatially factorized vector quantization. We further introduce a Draft-then-Refine paradigm, in which a kinematics-aware autoregressive model predicts factorized motion tokens, and a selective rectified-flow module subsequently refines the expression and mouth latents to enhance expressive facial dynamics. Extensive experiments show that FlowTalker achieves state-of-the-art lip synchronization and facial motion quality under causal streaming inference, while showing the smallest performance degradation when transitioning from non-causal to causal settings. FlowTalker further enables high-fidelity photorealistic animation of 4D head avatars reconstructed from a single image.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Head Animation, Audio Driven, Causal Streaming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rZeVOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a well-developed system with reasonably comprehensive experiments, and the proposed components are technically plausible. However, my main concern is the limited methodological novelty and the significance of the resulting contribution. FlowTalker largely builds on established techniques in talking-head animation, including quantized motion representations, autoregressive generation, motion-history conditioning, and continuous latent refinement. Its main additions—spatially factorized token streams, directed cross-stream attention, and selective rectified-flow refinement—form a sensible integration, but the paper does not yet establish a sufficiently substantial advance over closely related approaches.

The experiments support the effectiveness of the complete system, while leaving the distinctive benefits of these design choices insufficiently isolated. Stronger comparisons under matched causal training and latency constraints, together with capacity-matched representation and refinement controls, would help establish their significance. Despite the paper’s good overall execution, I currently lean toward a weak rejection since there is no borderline option.

Critical strengths

Relevant problem and coherent system design. Generating expressive head animation under streaming constraints is practically meaningful. The factorized representation, structured autoregressive drafting, and selective continuous refinement form a coherent pipeline, with reasonable motivations for the individual components.

Good experimental coverage. The paper evaluates short and long sequences, cross-dataset transfer, and photorealistic avatar driving. The appendix provides useful analyses of window length, update stride, token specialization, directed attention, training–inference mismatch, and loss components. These experiments support the effectiveness of the complete system.

Useful implementation and evaluation details. The reported steady-state inference time includes audio and emotion encoding as well as motion generation. The photorealistic evaluation fixes the avatar, camera, and renderer, helping isolate differences in motion quality. The blinded and randomized user study provides additional perceptual evidence.

Critical weaknesses

Limited methodological novelty and contribution. The main contribution appears to be the integration of established modeling choices: quantized motion representations, autoregressive generation, structured conditioning, and continuous latent refinement. Spatial factorization, directed cross-stream attention, and selective rectified-flow refinement are sensible additions, but their combination does not yet establish a sufficiently substantial advance over closely related talking-head approaches. The paper would benefit from a clearer account of the new modeling insight and its broader significance. Integration with an existing avatar reconstruction system primarily demonstrates applicability.

The distinctive benefits of the proposed components are insufficiently isolated. The ablations show improvements within the proposed pipeline, but leave important alternative explanations unresolved. It is unclear whether the unfactorized representation matches the factorized model’s latent capacity and token budget. Similarly, removing the flow refiner does not establish its advantage over a comparable deterministic residual refiner. These controls are particularly important because the contribution depends on the significance of these architectural choices.

The causal robustness comparison mixes architecture and training conditions. FlowTalker is trained under the causal sliding-window protocol, whereas the baseline causal-gap evaluation restricts checkpoints trained under their original protocols. This provides useful deployment evidence, but does not isolate architectural robustness to limited context. Training the strongest baselines under the same causal constraints would provide a more convincing assessment of the proposed streaming contribution.

Some broader claims exceed the evaluation provided. The principal quantitative metrics remove global head pose, leaving the claimed benefits for full-head motion insufficiently evaluated. Several improvements over strong baselines are relatively modest, without uncertainty estimates or significance analysis. The latency discussion would also benefit from a clearer distinction between the 3.2-second initialization, steady-state chunk availability, and actual frame display latency.

Questions for authors

How does rectified-flow refinement compare with a deterministic residual MLP of comparable capacity? What quality–runtime trade-off is obtained with different integration steps and noise levels?

Can the strongest baselines be trained and evaluated under the same causal window, update stride, and conditioning constraints as FlowTalker?

How is the initial 3.2-second motion window emitted and played? Are initialization frames included in the reported causal metrics, and what is the actual audio-to-display latency during startup and steady-state operation?

Equation (8) applies a linear projection to flattened window-level audio features. How does this fixed-dimensional projection accommodate the full-utterance evaluation with longer windows? Are the same parameters used for both settings?

Reviewer_tDCCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper incrementally generates full-head motion from audio to drive FLAME meshes / CAP4D photo-realistic avatars

More specifically, a learnt model for audio-based face animation; i.e. audio as input and animated FLAME model as output. The method can also be combined with an image input that is fed into CAP4D that allows the creation of an avatar that is animated by the audio signal.

  • sliding window of audio and past motion for low latency;-> autoregressive model predicts discrete pose, expression, and mouth
  • rectified-flow refines expression and mouth embeddings in continuous latent space
  • VQ-VAE decoder maps pose embedding and refined facial latents to FLAME parameters.
Critical strengths
  • interesting direction and important task.
  • focus on speed makes sense and faster latency makes sense
Critical weaknesses
  • not great paper and supplemental polish; this includes both the low-resolution video as well as the figure presentation.

  • the results are not super convincing; I would've hoped for more detailed results in the faces (I feel even the already low-dimensional FLAME space is not that much exploited)

  • generally, the animated faces look not very natural and I feel there is a significant step missing to make this work in practice.

  • overall novelty is not that high - yes it's a flow-matching model but the architecture itself is relatively straightforward.

  • Two of the contribution bullets are confusing: 1) describes task; 3) says they have results but no information; 2) seems the actual contribution; i.e., representation of framework with drafter + refiner.

  • method is relatively basic: 3.3 and 3.4 -- the rest is intro, preliminaries and results.

  • Fig 5./6. not convincing in terms of presentation (this needs more polish).

Questions for authors

I feel animating 3d meshes and training models is interesting but compared to recent trends (e.g., use a diffusion model as a baseline). I'd be interesting to understand about the design choices. E.g., why use CAP4D as an avatar over other approaches.

Reviewer_pbmWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I carefully read the entire paper, and I believe that the authors have made a serious effort to address several challenging problems in 3D talking-head generation, particularly under causal streaming constraints. The system is thoughtfully designed, and the experimental results suggest that it can be effective in practice. However, from the overview, the use of style and emotion embeddings does not appear particularly novel, as similar ideas have already been explored in works such as EmoTalk, TalkingStyle, EmoFace, and other related studies. Likewise, the Draft and Refine modules seem to primarily combine existing attention-based techniques with Rectified Flow. Although causal streaming generation may be the authors’ most important claimed contribution, it appears to rely largely on a combination of existing techniques rather than introducing a fundamentally new generation mechanism. Therefore, despite the authors’ serious effort and the practical value of the system, I remain concerned that the paper does not provide sufficiently substantial algorithmic or conceptual novelty to meet the innovation bar of ICLR.

Critical strengths

None

Critical weaknesses

The main contribution of this paper appears to lie in combining existing techniques and improving the empirical results, while its originality at the theoretical or algorithmic level remains relatively limited. Nevertheless, I would be very happy to hear the authors’ responses to these concerns and am open to further discussion. If the authors believe that the core contributions are sufficiently solid, I would welcome additional clarification and justification.

Questions for authors

as my comment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.