acceptodds
Withdrawn from ICLR 2027

FacePlex: Toward Natural Full-Duplex Conversational Avatars

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

88 views

Abstract

Natural human conversation is inherently a real-time interaction in which speech and facial behavior continuously evolve. Enabling such interaction requires a conversational avatar to jointly generate speech and facial motion in real time, prepare facial motion for upcoming speech before the corresponding audio is emitted, and produce non-verbal reactions that reflect the ongoing dialogue. However, existing conversational avatar systems cannot address such requirements: audio-driven methods rely on pre-given speech, while joint streaming generation alone does not ensure anticipatory articulation or semantically appropriate reactions. We propose **FacePlex**, a unified framework for full-duplex speech–facial motion generation and real-time avatar rendering. FacePlex jointly coordinates speech, facial motion, and Gaussian splatting rendering on a shared streaming timeline. To prepare facial motion for upcoming speech, FacePlex predicts a short speech continuation and uses its future hidden states through late facial denoising, while continuously updating them as new user audio arrives without observing future user input. For dialogue-grounded non-verbal behavior, we construct *SemReact*, a dataset aligning dialogue context, reaction semantics, and facial motion, and introduce a semantic behavior router that guides continuous motion during both speaking and listening. Extensive experiments show improved audio-visual synchronization and facial articulation in offline evaluation, natural dialogue-grounded non-verbal responses, and low-latency of 122 ms end-to-end avatar interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-duplex, Avatar, 3D Motion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FL9EOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper constructs an overall head motion modeling method for real-time interaction between two people. The task of this paper has been improved on existing related head motion tasks, with better applicability. As a reviewer, I had already seen this article from arXiv and logged into the paper homepage before reviewing it. Overall, this article does have some innovation, but from a holistic perspective, it does not deviate from the architecture of head motion, and the generated effects still have some issues with head motion itself, such as image jitter, large action offset, and insufficient parallel synchronization of dialogue actions. The specific opinions are as follows. I believe the author still needs to further polish the article.

Critical strengths

1.A new task with greater potential for application, which has made new improvements on existing head motion tasks and achieved preliminary dialogue effects. 2.The author conducted extensive experimental verification and demonstrated real-life scenarios. 3.By defining some basic action units, enrich the prominent features of head motion.

Critical weaknesses

There has been innovation in the task define, but there has been no significant improvement in terms of the results and effects. There are still many traditional problems with 3D head generation, especially in the display of dynamic effects. The biggest problem with the article is that the methodology section is somewhat disorganized, such as the messy presentation of Figure 2, which fails to highlight the core innovation of the methodology.

Questions for authors
  1. There has been innovation in the task define, but there has been no significant improvement in terms of the results and effects. There are still many traditional problems with 3D head generation, especially in the display of dynamic effects.

  2. The biggest problem with the article is that the methodology section is somewhat disorganized, such as the messy presentation of Figure 2, which fails to highlight the core innovation of the methodology

  3. The hypothesis of the paper was validated through several indicators in Figure 3. This section is a bit self explanatory and does not effectively convince readers. It feels like an analysis is being conducted for the sake of rationality

4.The innovation at the methodological level is not strong, and many of them are conceptual integrations.

5.The academic nature of the entire work is not strong enough, it is more like an engineering pipeline.

6.The dynamic head motion effects presented by the author are not comprehensive. Some display effects still have unreasonable situations, such as unreasonable opening and closing of the mouth, understanding of the head, and rapid action conversion.

Reviewer_xoRiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FacePlex, a unified framework for full-duplex joint speech–facial motion generation and real-time avatar rendering. It addresses three requirements: continuous full-duplex speech–motion streaming, anticipatory facial articulation and dialogue-grounded non-verbal behavior. Using a frozen PersonaPlex-7B speech backbone batched run-ahead prediction recomputes future hidden states as new user audio arrives while keeping buffered avatar audio tokens fixed. Asymmetric conditioning and denoising use the latest current and future hidden states for expression and jaw motion while progressively denoising head and eye motion across successive windows. The authors construct SemReact to align dialogue context, reaction semantics and facial motion and train a semantic behavior router that guides speaking and listening motion through style token routing. Streaming-optimized FlexAvatar rendering completes the visual pipeline.

Critical strengths

Clear problem formulation and organization of contributions: Section 1 explicitly formulates three requirements for natural conversational avatars: continuous full-duplex speech–motion streaming (R1), anticipatory facial articulation (R2) and dialogue-grounded non-verbal behavior (R3). The introduction connects each requirement to the corresponding components of FacePlex while Figure 1 provides an accessible overview. This structure makes the motivation, scope and intended contributions easy to follow.

Empirically motivated use of future hidden states: Section 3.1 supports the use of anticipatory context through temporal analysis. Figure 3(a) shows that prediction of jaw and expression motion is most accurate when using hidden states 160ms ahead of the target motion. Appendix D.7 further examines the hidden-state horizon: the four-state window achieves the lowest errors across all five reported metrics among the evaluated configurations in Table 15. These results provide empirical support for the selected conditioning window.

Promising motion-quality and perceptual results: Table 1 reports the lowest errors across all five motion-quality metrics for FacePlex under the stated streaming evaluation compared with baselines in their default offline settings. For example FacePlex achieves S-LVE of 7.162 and S-MHD of 1.674 compared with 7.691 and 1.819 for UniLS the strongest baseline on these metrics. The 31-participant user study also reports the highest mean ratings across all four criteria (Table 2). These results provide encouraging evidence of motion quality and perceived responsiveness.

Critical weaknesses

Inconsistent evaluation conditions and insufficient validation of causal streaming motion quality: Section 4.1 states “We evaluate FacePlex in streaming condition” (L369). However Appendix I.1 acknowledges that the main motion-quality comparison uses recorded conversations “with future user audio available” and that these results “do not directly establish motion quality under causal run-ahead conditioning.” (L1727) This distinction is central because the deployed system must use predicted future user audio. Consequently Table 1 does not directly establish the reported motion quality under the intended causal setting.

Indirect evidence for anticipatory facial articulation: Anticipatory facial articulation is a central requirement of FacePlex (R2) but the presented evidence primarily measures predictive information and motion reconstruction quality. Figure 3(a) shows that future hidden states better predict current jaw and expression motion while Figure 3(b) and Tables 15, 16 evaluate motion errors under different conditioning windows. These results support the usefulness of future context but do not directly establish whether generated articulatory movements precede the corresponding acoustic events with appropriate timing. Lower LVE alone cannot distinguish improved mouth-shape reconstruction from improved anticipation. The authors should provide event aligned analysis of generated articulatory timing such as lip rounding or closure onset relative to the corresponding sounds and compare it with ground truth and a past/current-only conditioning baseline.

Limited evaluation of adaptation to unexpected user input: Section 3.2 keeps buffered avatar audio tokens fixed and Appendix D.3 explicitly states that “Already committed avatar tokens cannot be revised in response to later user input” (L1230). These do not directly measure how quickly the system changes its response when the user interrupts, corrects a statement or diverges from the predicted continuation. This is important for the claimed full-duplex interaction since updating hidden states does not revise already committed speech. The authors should evaluate response adaptation under such events reporting the delay before inappropriate speech stops or a revised response begins, together with the effect on speech–motion consistency.

Limited validation of the adapted Moshi-Face baseline: Moshi-Face is a particularly relevant baseline because it also jointly generates speech and facial motion. However Appendix E describes substantial changes to its implementation: the original motion representation is replaced with 108-dimensional FLAME parameters, the explicit text- and audio-token embedding sums are omitted and only the first conversational-model adaptation stage is implemented while PersonaPlex remains frozen. Although these deviations are disclosed their effects on performance are not isolated.

Minor comment:

Unexplained differences between quantitative and perceptual rankings: DualTalk has higher S-LVE than UniLS in Table 1 (11.785 vs. 7.691), yet receives a higher Lip-Sync rating in Table 2 (3.161 vs. 2.594). Such differences are possible because geometric error and perceived synchronization measure different properties but the paper does not analyze this discrepancy.

Questions for authors
  1. Section 4.1 describes FacePlex as evaluated in a “streaming condition” whereas Appendix I.1 states that the main motion-quality comparison uses recorded conversations with future user audio available. What conditioning was used for Table 1? Can the authors compare motion quality using received future user audio versus predicted future user audio under an otherwise identical evaluation protocol?
  2. Does the reported 122.16 ms represent an observed delay from user speech onset to the first audible response or the processing time of a streaming step?
  3. When newly received user audio conflicts with the predicted continuation how quickly can the avatar stop or revise its already committed speech? Can the authors evaluate interruptions and user corrections across different run-ahead horizons reporting response adaptation delay and speech–motion consistency?
  4. Does FacePlex reproduce the timing of anticipatory articulation beyond reducing aggregate motion errors? Can the authors compare the onset of visible articulatory events relative to the corresponding acoustic events against ground truth and a model using only past and current hidden states?
Reviewer_dHezOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

FacePlex is a framework for full-duplex conversational avatars that jointly generates speech and facial motion while continuously processing user audio. Built on a pretrained PersonaPlex backbone, it uses batched run-ahead prediction to obtain short-term future speech representations, allowing lip motion to prepare for upcoming sounds without accessing future user input. An asymmetric denoising strategy progressively generates head and eye movements while refining expressions and jaw motion just before output. The system also integrates an optimized Gaussian splatting renderer for real-time visualization. Experiments report improved facial-motion metrics and user ratings over the evaluated baselines.

Critical strengths
  1. FacePlex integrates full-duplex speech generation, continuous facial motion, semantic reactions, and real-time rendering within a shared streaming timeline, addressing several practical requirements of interactive avatars together.
  2. The temporal analysis supports using near-future speech representations to guide current facial motion. Batched run-ahead prediction and asymmetric denoising translate this finding into an efficient design that accommodates different temporal dependencies across facial components.
  3. SemReact provides aligned dialogue, reaction, and motion supervision, while the semantic router connects conversational context to continuous non-verbal behavior during both speaking and listening.
Critical weaknesses
  1. Sec. 4 describes FacePlex’s evaluation as streaming, whereas the Appendix explicitly states that the main comparison uses recorded conversations with future user audio available. This inconsistency leaves the central claim insufficiently supported: the reported improvements need to be verified using predicted future inputs throughout actual online execution.
  2. The Moshi-Face implementation changes the motion representation, omits explicit text and audio token conditioning, and implements only the first training stage. These differences confound the comparison and make it difficult to attribute the performance gap to FacePlex’s proposed contributions.
  3. The reported 80.26% accuracy assumes ground-truth reaction presence, excludes idle, and measures agreement with synthetic reaction labels (Table 14). It therefore leaves reaction detection, false-positive behavior, and human-perceived semantic appropriateness insufficiently evaluated.
  4. Buffered avatar tokens cannot be revised when new user input arrives, even though their hidden states are updated. The token consistency analyses and prerecorded video ratings do not establish how this commitment affects interruption handling, corrections, or turn-taking. Live interaction experiments are necessary to assess this trade-off.
Questions for authors
  1. Regarding weakness 1, how should readers reconcile the streaming evaluation described in Section 4.1 with Appendix I.1’s statement that future user audio is available in the main comparison? Can you report motion-quality results under strictly causal run-ahead inference?
  2. Does the reported 122 ms correspond to directly measured user-speech response time? Could you report startup latency, steady-state processing time, and complete audiovisual output latency, including mesh construction and rendering?
  3. How do the omitted text/audio conditioning and incomplete training procedure affect the reproduced Moshi-Face baseline? Could you provide a faithful reproduction or controlled experiments quantifying these modifications?
  4. Could you report reaction detection, false-positive rates, and human judgments of semantic appropriateness on real conversations?
  5. Should weakness 4 be considered a limitation? How might this affect responsiveness to interruptions or sudden changes in user intent?

I would be willing to raise my rating if the authors satisfactorily clarify the above concerns and provide supporting evidence where needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.