acceptodds
Withdrawn from ICLR 2027

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

64 views

Abstract

An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding quality-ranked pairs ( hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding faster (, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a word error rate, the lowest among the omni-modal systems compared.

Primary area:
generative models
Keywords:
spoken dialogue model, co-speech motion generation, joint speech and motion generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_fFnzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Motion-Omni generates facial, hand, upper-body, and lower-body motion codes directly from speech-generator hidden states and speech tokens, avoiding a separate waveform-to-motion inference pass. Motion-Omni-Q7 is trained on LOM-pseudo-labeled samples and evaluated on SwDA-500 using matched-audio comparisons.

Critical strengths
  1. Directly conditioning motion generation on speech-generation representations removes a separate inference stage and substantially reduces generation time relative to the teacher cascade.

  2. The matched-audio comparisons are a strong experimental choice, controlling for differences in speech content and prosody.

  3. The model produces full-body motion while retaining competitive speech performance on the reported benchmarks.

Critical weaknesses
  1. The claim that speech-pathway co-adaptation is necessary for alignment lacks a matched quantitative comparison. Section 3.3 describes a frozen-pathway pilot, whereas Section 4.6 and Appendix G evaluate fusion mechanisms and context sources. Neither isolates the effect of co-adaptation. Stage-wise FGD results also conflate changes in data exposure and training budget.

  2. The interpretation of the real-time metric is problematic. Section 4.5 defines RTF as total response-generation time divided by user-input duration, rather than generated-response duration. This normalization does not establish output-generation throughput relative to playback. The offline timing comparisons also do not measure latency to the first output.

  3. The validity of the lip-synchronization metrics warrants further examination. LSE-C and LSE-D rely on a pretrained SyncNet, whose reliability on visibly synthetic avatar renderings is unclear. In the examples I examined, lip movements appeared imperfectly synchronized with the audio despite favorable reported scores. This discrepancy calls for validation of the metric in this rendering domain.

  4. The method relies heavily on teacher-generated motion supervision. Although this limitation is acknowledged, teacher fidelity alone does not establish motion quality beyond the teacher’s existing limitations. The current evaluation provides limited independent evidence that these limitations are mitigated.

Questions for authors
  1. Starting from the same checkpoint, with matched training data and optimization, how do motion alignment and speech quality compare when the speech pathway is frozen versus co-adapted?

  2. Can the authors report RTF relative to generated-response duration, alongside total generation time, hardware, and the exact timing scope? Please distinguish offline completion time from latency to the first output.

  3. Can the authors validate SyncNet-based scores on the rendered avatars, for example using controlled audio–video offsets and a focused human lip-synchronization assessment?

Reviewer_NpvFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Motion-Omni, a framework for jointly generating spoken responses and full-body co-speech motion. Instead of generating motion from synthesized audio in a separate stage, the model predicts facial and body motion directly from the speech generator's hidden states. The authors also introduce a teacher-based data pipeline with 1,402 hours of speech-motion pairs and a new evaluation benchmark. Experiments show competitive motion quality compared to cascaded baselines and a 5.4× speedup over the teacher-based cascade.

Critical strengths
  1. The paper tackles an interesting problem by generating speech and full-body motion together. The approach is practical and avoids the extra inference step used in traditional cascaded methods.
  2. The experiments are solid, with multiple baselines and matched-audio comparisons. The 5.4× speedup over the teacher-based cascade is impressive, while maintaining comparable motion quality.
Critical weaknesses
  1. The paper relies a lot on LOM. The training data is generated by LOM, and several evaluation metrics also use LOM as the reference. It's hard to tell how good the generated motion really is beyond just matching the teacher.
  2. I'm not fully convinced by the joint training experiments. This is one of the main contributions, but there isn't a clear ablation showing how much better it is than keeping the speech generator frozen.
  3. There's also a mistake in the references. The ViBES paper lists "Philip Chen" instead of "Changan Chen". The authors should probably double-check the other references as well.
Questions for authors
  1. Since the model is trained on LOM-generated motion, how well does it perform when evaluated against real human motion? Is there any evidence that it can produce natural and meaningful gestures beyond imitating LOM?
  2. Can the authors provide a more complete ablation on joint training? In particular, how much does performance change if the speech generator is kept frozen throughout training, with everything else remaining the same?
Reviewer_ANXoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Motion-Omni, an end-to-end framework for generating spoken dialogue responses together with synchronized facial and full-body motion. Specifically, the authors build four part-aware motion decoders that use the speech generator's hidden states and emitted speech tokens to produce face, hand, upper-body, and lower-body motion without a separate waveform-conditioned motion pass. A four-stage curriculum progressively trains the speech and motion pathways and then jointly fine-tunes them for dialogue. To obtain supervision, the authors construct 422,856 speech-motion pairs using a pretrained motion teacher and a quality-ranked training schedule. They also introduce SwDA-500 for evaluating open-ended spoken motion generation. Experiments compare speech quality, motion alignment, human preference, and response latency against cascaded systems.

Critical strengths
  1. The end-to-end design generates motion from the representations used for speech generation, avoiding a separate waveform-to-motion inference pass. Extending this interface to facial expression and full-body movement addresses a limitation of conventional spoken dialogue systems.

  2. The pseudo-labeling pipeline produces 422,856 speech-motion pairs in a consistent target voice, with a quality-ranked curriculum based on motion reconstruction and rhythmic alignment. This provides a practical training recipe for a setting with little directly paired supervision.

Critical weaknesses
  1. The methodological novelty is narrower than the framing suggests. The fusion design builds on Ex-Omni and the part-wise motion representation comes from LOM, while MIBURI already generates conversational motion from internal speech states. The paper does not clearly establish what new modeling capability follows from their integration.

  2. The claimed necessity of jointly updating the speech pathway is supported mainly by a brief frozen-pathway pilot. The reported controlled ablation changes the fusion mechanism and context source, but does not isolate the effect of updating the language model and speech generator during motion training. Meanwhile, much of the motion evaluation relies on the same LOM model used to generate training labels. Teacher-referenced distributional and facial metrics therefore measure imitation fidelity. Evidence for independently improved conversational motion remains limited.

Questions for authors
  1. How does Motion-Omni differ methodologically from Ex-Omni and MIBURI beyond extending the output to four pretrained motion-code streams? In particular, how do the independently decoded body parts achieve coordinated behavior beyond sharing the speech-conditioning states?

  2. The paper argues that joint optimization of the speech pathway is necessary for alignment. How do motion quality, speech intelligibility, and dialogue performance change when the language model and speech generator are frozen, updated separately, or jointly trained under matched conditions?

  3. Since LOM supplies the motion supervision and several evaluation references, what evidence establishes that Motion-Omni learns meaningful dialogue-conditioned movements beyond imitating its teacher? How should the semantic alignment and naturalness results be interpreted without captured motion ground truth?

  4. Table 4 reports a 5.4-fold speedup over the LOM cascade, but only a small latency difference relative to MO-audio + EMAGE. How much of the efficiency advantage comes from joint generation itself, and how should the reported real-time factor be interpreted when it is normalized by user-utterance duration rather than generated-response duration?

Reviewer_isapOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Motion-Omni, which couples a 3D full-body motion decoder directly to the speech generator of a Spoken Dialogue Model (SDM). Instead of waiting for full waveform synthesis, it uses the speech generator's hidden states and interpolated discrete speech units to predict VQ codebook indices for the face, hands, upper body, and lower body in parallel. The authors use a pretrained LOM model to generate approximately 1,400 hours of motion pseudo-labels for InstructS2S-200K and adopt a multi-stage training curriculum based on VQ reconstruction error and Beat Correlation (BC). On SwDA-500, Motion-Omni-Q7 achieves performance comparable to the LOM teacher cascade while providing a 5.4x speedup in the offline setting.

Critical strengths

The engineering integration is solid. Unifying speech and motion generation within a single forward pass eliminates redundant waveform decoding and audio feature re-extraction in the cascaded pipeline. Curriculum filtering based on VQ reconstruction error and BC is a practical technique that prevents training divergence on noisy pseudo-labels while maintaining a low word error rate (WER) on Seed-TTS-Eval.

Critical weaknesses

The methodological novelty is limited. The framework primarily combines existing components to distill LOM into an SDM. Furthermore, the experimental evaluation has several clear limitations. In Table 2, reference-based metrics (FGD, MSE, and LVD) use LOM outputs as "ground truth." Since Motion-Omni is trained on LOM pseudo-labels and shares its VQ-VAE codebook, its output distribution naturally aligns more closely with LOM than those of baselines such as EMAGE or GestureLSM. In addition, the training data were explicitly filtered using BC, which raises a fairness concern when BC is also used for test-time evaluation. The paper also repeatedly claims that end-to-end joint training of the speech pathway is essential and that freezing this pathway causes speech-motion misalignment. However, it provides no quantitative ablation comparing a frozen speech generator with joint training, relying instead on a single qualitative statement in Section 3.3. Finally, the speedup is overstated. The 5.4x speedup primarily stems from the slow LOM baseline, which takes 23.35 seconds. In Table 4, MO-audio + EMAGE takes 4.63 seconds, only about 0.3 seconds longer than the proposed method's 4.32 seconds, while the proposed method still operates offline and does not support streaming.

Questions for authors

To substantiate the claim in Section 3.3, please provide an ablation in Stages 3 and 4 in which the speech generator and LLM are frozen and only the motion decoder is trained, and report the corresponding metrics on SwDA-500. Otherwise, please refrain from making this claim. To remove the bias introduced by LOM pseudo-labels in Table 2, please evaluate Motion-Omni-Q7 on the BEAT2 test set against ground-truth human motion capture. This could use the Stage 3 TTSM text-input setting or teacher forcing with ground-truth GLM-4-Voice speech units u. Please report the corresponding metrics. In addition, in Table 9, Motion-Omni-Q7 trails Qwen2.5-Omni, which also uses a 7B backbone, on the VoiceBench reasoning subsets, with scores of 27.36 on MMSU and 26.59 on OBQA. Please report VoiceBench and Seed-TTS-Eval metrics for the Stage 2, Stage 3d, and Stage 4 checkpoints to clarify whether joint motion training degrades the backbone's reasoning capabilities. If it does, please explicitly acknowledge and explain this degradation. Finally, please break down the 4.32-second total latency in Table 4 into LLM text generation, speech generation, motion generation, CosyVoice 2 flow matching, and decoding. Please do not use inappropriate efficiency comparisons to obscure the actual gains or overstate the speedup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.