- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents DirectBeat, a co-speech motion generation framework that separates behavioral planning from motion execution through explicit, editable language directives. A frozen multimodal language model plans gestures using speech, recent behavioral context, and retrieved examples, while a learned executor generates motion conditioned on these directives and time-aligned transcripts. The framework introduces Warp-VQVAE for audio-conditioned temporal warping of motion latents to improve rhythmic alignment, alongside BEAT2-Directive, a dataset of 101,660 aligned speech–directive–motion triplets. Validation includes single- and all-speaker BEAT2 benchmarks, component ablations, and human evaluation on unseen TED speech. The reported results show improved Fréchet Gesture Distance, competitive beat alignment, and the highest aggregate human preference among the compared methods. Qualitative demonstrations further illustrate persona editing through language prompts without retraining the executor.
Critical strengths
-
An explicit and controllable interface between behavioral planning and motion generation. DirectBeat uses structured language directives to connect a frozen multimodal language model with a learned motion executor, making intended gestures inspectable and editable. The demonstrations of persona editing without executor retraining, together with improvements across different director backends, support the utility of this interface. This is a meaningful contribution because it provides practical control over behavioral choices while retaining a specialized model for motion synthesis.
-
Empirical support across quantitative benchmarks and perceptual evaluation. The method achieves the lowest reported FGD among the compared methods in both BEAT2 protocols, with approximately 11% and 38% reductions relative to GestureLSM, while maintaining competitive beat alignment. Its highest aggregate human preference on unseen TED speech provides complementary evidence beyond in-domain metrics. Component ablations further support the contributions of the proposed design, giving reasonable confidence that the framework offers useful improvements in co-speech motion generation.
Critical weaknesses
Potential motion artifacts associated with rhythmic alignment. Some generated motions appear to accelerate abruptly, reducing their perceived smoothness and naturalness. This raises a concern that temporal warping may improve speech–motion synchronization at the expense of locally plausible motion dynamics, although the source of these artifacts remains unclear. Aggregate FGD and beat-alignment scores do not directly resolve this concern; a targeted comparison of motion smoothness with and without temporal warping would help establish the trade-off. This limits my confidence in the rhythmic execution component, but does not outweigh the framework’s overall contributions.
Questions for authors
Clarification request: Some generated motions appear to exhibit abrupt local accelerations. Can the authors clarify whether these artifacts originate from the temporal warping mechanism or other components? A controlled comparison with and without warping, using matched examples and motion velocity/acceleration statistics, would help determine whether improved rhythmic alignment comes at the expense of motion smoothness and naturalness.
Thank you for recognizing the explicit planning interface and the evidence across benchmarks and perceptual evaluation.
Q1. Abrupt acceleration and temporal warping
To assess the effect of temporal warping on motion smoothness, we measured joint acceleration and jerk for
Ours (w/ warping)andOurs w/o warping, alongside ground truth motions and three baseline methods. We also measured jerk at window boundaries.Although Warp-VQVAE provides a re-timing handle for applications, shows improvements over BC and FGD relative to Full-body RVQ (Table 4), it can slightly reduce motion smoothness.
Ours (w/ warping)exhibits 3.8% higher acceleration, 19.0% higher jerk, and 13.4% higher boundary jerk than thew/o warpingvariant. The overall smoothness remains comparable to ground truth and prior methods.The hyperactive style of some generated gestures may also reflect the training data. The BEAT2 authors describe gestures in acted speech as more expressive and varied than those in daily conversation. [Liu et al., EMAGE, Sec. 5]