acceptodds
Withdrawn from ICLR 2027

DuplexAvatar: Unified 3D Motion Understanding and Generation for Full-Duplex Spoken Dialogue

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

An interactive avatar must understand behavioral context and generate appropriate 3D motion during real-time spoken interaction. This requires behavioral understanding, multimodal response generation, and motion realization to operate within a shared conversational state. We introduce DuplexAvatar, a unified full-duplex speech–language–motion model that maintains speech, language, behavior plans, and holistic 3D motion within a shared causal state, supporting both behavior understanding and multimodal response generation. At the core of DuplexAvatar is the Executable Behavior Plan, a structured and executable representation of behavioral intent that specifies what to do, where and when it applies, and what must remain unchanged. The plan connects semantic decisions to controllable motion realization and provides an explicit interface for localized intervention. Structured operations are executed by a typed executor, while open-ended requests are realized through masked residual flow. Before commitment, geometry-based checks and motion-only readback can trigger a bounded local correction; only uncommitted future motion remains revisable, while committed motion is re-encoded into the conversational state. We further compile Spear, a multi-source, multi-view supervision suite that provides motion events, executable plans, geometry-grounded queries, and controlled counterfactual edits, together with an intervention-based benchmark for controllable behavior realization. Experiments under matched evaluation settings show consistent improvements across speech-conditioned motion generation, motion understanding, and behavioral control, achieving 73% human preference, 91.8% Event F1, and 78.95% constraint success. Together, DuplexAvatar and Spear provide an inspectable path from behavioral intent to realized 3D motion within full-duplex spoken interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-duplex spoken dialogue, 3D human motion, motion understanding and generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hbdxOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DuplexAvatar, a unified speech–language–motion framework for controllable full-duplex conversational avatars. Its central interface is an Executable Behavior Plan (EBP), which specifies behavioral intent, spatial and temporal support, and preservation constraints. A typed executor realizes precise geometric operations, while a masked residual-flow module handles open-ended semantic edits over a speech-driven motion base. Motion-only readback and geometric verification support bounded local correction before commitment. At runtime, the system separates committed motion from pending future output, allowing interventions to revise uncommitted behavior while re-encoding executed motion into the shared conversational state. To train and evaluate this framework, the authors compile SPEAR, a multi-source, geometry-grounded supervision suite and intervention-based benchmark, evaluating the system across motion generation, motion understanding, and behavioral control.

Critical strengths

1.Well-motivated control architecture. Separating exact geometric operations from learned semantic realization is a sensible way to combine precise control with expressive co-speech motion. Explicit preservation constraints and commitment boundaries address concrete requirements of interactive avatars. Integrating these mechanisms alongside executed-state feedback within a shared causal conversational backbone is a well-considered system design.
2. Good offline behavioral-control results. Table 1 reports 78.95% constraint success, compared with 59.13% for SimMotionEdit and 51.68% for MotionReFit. These improvements are accompanied by substantially lower preservation drift (8.64 mm) and target error (18.73 mm), indicating that the method effectively localizes requested edits while preserving unrelated motion outside the specified support.
3. Ablations and dataset organization support component utility. Removing EBP, the typed executor, masked residual flow, or readback correction reduces constraint success to 31.58%, 52.63%, 60.53%, and 68.42%, respectively. These results provide substantive evidence that the proposed planning, realization, and verification modules work complementarily. In addition, organizing heterogeneous datasets into unified plan views under a speaker- and recording-disjoint splitting protocol is a sound experimental design.

Critical weaknesses
  1. The central full-duplex claim lacks quantitative validation. The paragraph on page 9 ("Full-duplex interaction") introduces time to first audio (TTFA), time to first motion (TTFM), real-time factor (RTF), and barge-in recovery as the evaluation metrics for online interaction. However, neither the main paper nor the supplementary material reports any numerical results, tables, or curves for these measures.
  2. Generation evaluation and SPEAR documentation leave key details underspecified. For speech-conditioned motion generation, Table 1 reports only upper-body smoothness, beat consistency, and human preference. While useful, these metrics do not separately characterize distributional realism (e.g., FGD), generation diversity, or facial quality. Smoothness can favor damped, low-amplitude motion, while beat consistency measures rhythmic alignment rather than semantic appropriateness.
  3. The comparisons do not sufficiently isolate the incremental value of the conversational control interface. EMAGE and SemTalk are reasonable co-speech generation baselines, and StreamTalk provides a relevant streaming reference. However, these comparisons do not isolate the benefit of EBP, typed/residual execution, verification, and commitment over a compatible conversational architecture. The paper itself discusses closely related full-duplex and conversational behavior systems such as DuplexSLA, ViBES, Motion-Omni, and DyaPlex. While differences in input modalities and body coverage make an unadapted end-to-end comparison across all of them impractical, comparing on shared capabilities would clarify what the structured EBP and verification loop add beyond backbone scaling and multi-view SPEAR supervision.
Questions for authors
  1. Can you provide the missing quantitative results for the online full-duplex evaluation (TTFA, TTFM, RTF, and barge-in recovery), along with hardware specifications, precise timing definitions, and an evaluation of motion continuity at the commitment boundary?
  2. What are the test-set sizes for generation, understanding, and control? What participant counts, trial counts, and comparison protocol produced the 73% human-preference result?
Reviewer_BpXCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper present DuplexAvatar, which integrates speech dialogue, motion understanding, and SMPL-X motion generation within a shared model. Executable behavior plans guide deterministic or learned motion modifications, followed by verification and bounded correction before playback. SPEAR provides supervision for generation, understanding, and behavioral control.

Critical strengths

The explicit behavior interface is well motivated. Localized editing, revision of uncommitted outputs, and feedback from executed motion are sensible designs for interactive avatars. The reported offline results and component ablations are promising.

Critical weaknesses
  1. The central full-duplex capability lacks runtime evidence. Section 5.2 names TTFA, TTFM, RTF, and barge in recovery, but provides no measurements or hardware details. The 0.8 second interaction schedule does not establish that the complete pipeline can operate in real time, including motion realization, readback, verification, and correction. An end-to-end runtime breakdown and interruption experiments are essential to substantiate the main claim.

  2. Several evaluation details are missing. The paper describes the dataset size and speaker-disjoint splits, but does not report test-set sizes for each task or the number of raters and trials in the human study. It also leaves the event-matching rules and constraint-success thresholds unspecified. These details are needed to interpret and reproduce Table 1.

Questions for authors
  1. Please provide a runtime breakdown and end-to-end latency on specified hardware, including verification, correction, and interruption recovery.
  2. Please provide the test and human-study counts, event-matching rules, and constraint-success thresholds used for Table 1.
Reviewer_2k1eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents DuplexAvatar, a model that can understand and generate 3D motion during full-duplex conversations. The main idea is to use an Executable Behavior Plan to control and edit specific movements without changing the rest of the motion. The authors also introduce a dataset for training and evaluating motion understanding and generation. Experiments show better results than several baselines.

Critical strengths
  1. The idea of using Executable Behavior Plans to control motion is interesting. It makes it easier to edit specific movements without changing the rest, which is useful for interactive avatars.
  2. The paper brings motion understanding, generation, and editing into one framework. The SPEAR dataset is also a useful contribution, and the ablation results show that the behavior planning components do help.
Critical weaknesses
  1. The paper claims to support real-time full-duplex interaction, but I couldn't find any actual results for latency, real-time performance, or interruption recovery. There are also no demo videos in the provided supplementary material, which makes it hard to judge the actual motion quality and interaction performance. I think this is a major issue given the paper's focus.
  2. The paper reports strong numbers, but important details are missing, such as the test set size and human evaluation setup. Also, much of SPEAR is based on rule-generated annotations, so it's unclear how well the results would generalize to real-world interactions.
Questions for authors
  1. The paper mentions latency and interruption recovery, but I couldn't find the actual results. Could the authors share these numbers? Are there any demo videos available?
  2. Could the authors clarify how many test samples and human evaluators were used? Also, since much of SPEAR uses rule-generated annotations, how well does the model work on real-world interactions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.