- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Motion-Omni generates facial, hand, upper-body, and lower-body motion codes directly from speech-generator hidden states and speech tokens, avoiding a separate waveform-to-motion inference pass. Motion-Omni-Q7 is trained on LOM-pseudo-labeled samples and evaluated on SwDA-500 using matched-audio comparisons.
Critical strengths
-
Directly conditioning motion generation on speech-generation representations removes a separate inference stage and substantially reduces generation time relative to the teacher cascade.
-
The matched-audio comparisons are a strong experimental choice, controlling for differences in speech content and prosody.
-
The model produces full-body motion while retaining competitive speech performance on the reported benchmarks.
Critical weaknesses
-
The claim that speech-pathway co-adaptation is necessary for alignment lacks a matched quantitative comparison. Section 3.3 describes a frozen-pathway pilot, whereas Section 4.6 and Appendix G evaluate fusion mechanisms and context sources. Neither isolates the effect of co-adaptation. Stage-wise FGD results also conflate changes in data exposure and training budget.
-
The interpretation of the real-time metric is problematic. Section 4.5 defines RTF as total response-generation time divided by user-input duration, rather than generated-response duration. This normalization does not establish output-generation throughput relative to playback. The offline timing comparisons also do not measure latency to the first output.
-
The validity of the lip-synchronization metrics warrants further examination. LSE-C and LSE-D rely on a pretrained SyncNet, whose reliability on visibly synthetic avatar renderings is unclear. In the examples I examined, lip movements appeared imperfectly synchronized with the audio despite favorable reported scores. This discrepancy calls for validation of the metric in this rendering domain.
-
The method relies heavily on teacher-generated motion supervision. Although this limitation is acknowledged, teacher fidelity alone does not establish motion quality beyond the teacher’s existing limitations. The current evaluation provides limited independent evidence that these limitations are mitigated.
Questions for authors
-
Starting from the same checkpoint, with matched training data and optimization, how do motion alignment and speech quality compare when the speech pathway is frozen versus co-adapted?
-
Can the authors report RTF relative to generated-response duration, alongside total generation time, hardware, and the exact timing scope? Please distinguish offline completion time from latency to the first output.
-
Can the authors validate SyncNet-based scores on the rendered avatars, for example using controlled audio–video offsets and a focused human lip-synchronization assessment?