acceptodds
Under review as a conference paper at ICLR 2027

DirectBeat: Behavioral Planning and Rhythmic Execution for Co-Speech Motion

Abstract

Speech-to-motion correspondence is highly ambiguous, making generation difficult to control and inspect. We present DirectBeat, which connects behavioral planning and motion execution through explicit directives. Using pretrained knowledge, speech context, recent memory, and retrieved examples, a frozen multimodal language model predicts two-second behavioral plans. A specialized executor realizes these plans as motion. Warp-VQVAE learns audio-conditioned latent space re-timing via temporal warping to synchronize motion with speech. BEAT2-Directive supplies 101,660 aligned speech–directive–motion triplets from BEAT2 to supervise this interface. DirectBeat achieves FGD scores of 3.631 and 2.76 on the single- and all-speaker benchmarks, reducing FGD by 11% and 38% compared with prior methods. On the unseen TED speech benchmark, it receives the highest aggregate human preference across realism, semantic consistency, synchrony, and diversity among the compared methods. We further demonstrate an application that directives support inspection and persona editing in language without retraining the executor, providing a reusable motion interface for LLM agents. Video results are provided in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Co-Speech Motion Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1CRZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents DirectBeat, a co-speech motion generation framework that separates behavioral planning from motion execution through explicit, editable language directives. A frozen multimodal language model plans gestures using speech, recent behavioral context, and retrieved examples, while a learned executor generates motion conditioned on these directives and time-aligned transcripts. The framework introduces Warp-VQVAE for audio-conditioned temporal warping of motion latents to improve rhythmic alignment, alongside BEAT2-Directive, a dataset of 101,660 aligned speech–directive–motion triplets. Validation includes single- and all-speaker BEAT2 benchmarks, component ablations, and human evaluation on unseen TED speech. The reported results show improved Fréchet Gesture Distance, competitive beat alignment, and the highest aggregate human preference among the compared methods. Qualitative demonstrations further illustrate persona editing through language prompts without retraining the executor.

Critical strengths
  1. An explicit and controllable interface between behavioral planning and motion generation. DirectBeat uses structured language directives to connect a frozen multimodal language model with a learned motion executor, making intended gestures inspectable and editable. The demonstrations of persona editing without executor retraining, together with improvements across different director backends, support the utility of this interface. This is a meaningful contribution because it provides practical control over behavioral choices while retaining a specialized model for motion synthesis.

  2. Empirical support across quantitative benchmarks and perceptual evaluation. The method achieves the lowest reported FGD among the compared methods in both BEAT2 protocols, with approximately 11% and 38% reductions relative to GestureLSM, while maintaining competitive beat alignment. Its highest aggregate human preference on unseen TED speech provides complementary evidence beyond in-domain metrics. Component ablations further support the contributions of the proposed design, giving reasonable confidence that the framework offers useful improvements in co-speech motion generation.

Critical weaknesses

Potential motion artifacts associated with rhythmic alignment. Some generated motions appear to accelerate abruptly, reducing their perceived smoothness and naturalness. This raises a concern that temporal warping may improve speech–motion synchronization at the expense of locally plausible motion dynamics, although the source of these artifacts remains unclear. Aggregate FGD and beat-alignment scores do not directly resolve this concern; a targeted comparison of motion smoothness with and without temporal warping would help establish the trade-off. This limits my confidence in the rhythmic execution component, but does not outweigh the framework’s overall contributions.

Questions for authors

Clarification request: Some generated motions appear to exhibit abrupt local accelerations. Can the authors clarify whether these artifacts originate from the temporal warping mechanism or other components? A controlled comparison with and without warping, using matched examples and motion velocity/acceleration statistics, would help determine whether improved rhythmic alignment comes at the expense of motion smoothness and naturalness.

1 comment

Thank you for recognizing the explicit planning interface and the evidence across benchmarks and perceptual evaluation.


Q1. Abrupt acceleration and temporal warping

To assess the effect of temporal warping on motion smoothness, we measured joint acceleration and jerk for Ours (w/ warping) and Ours w/o warping , alongside ground truth motions and three baseline methods. We also measured jerk at window boundaries.

MethodAcceleration (mm/s²)Jerk (mm/s³)Boundary jerk (mm/s³)
Ground truth3,87881,556—
EMAGE3,87984,045—
SemTalk4,21376,026—
GestureLSM4,838158,895—
Ours (w/ warping)4,325112,571125,289
Ours (w/o warping)4,16794,568110,525

Although Warp-VQVAE provides a re-timing handle for applications, shows improvements over BC and FGD relative to Full-body RVQ (Table 4), it can slightly reduce motion smoothness. Ours (w/ warping) exhibits 3.8% higher acceleration, 19.0% higher jerk, and 13.4% higher boundary jerk than the w/o warping variant. The overall smoothness remains comparable to ground truth and prior methods.

The hyperactive style of some generated gestures may also reflect the training data. The BEAT2 authors describe gestures in acted speech as more expressive and varied than those in daily conversation. [Liu et al., EMAGE, Sec. 5]

Reviewer_BAiqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DirectBeat, a system that takes speech audio, a timestamped transcript, and optional persona preferences as input and generates full-body and hand motion.

A frozen multimodal language model produces two-second behavioral directives. A masked motion-token generator executes these directives, while an audio-conditioned Warp-VQVAE adjusts motion timing.

The paper also introduces BEAT2-Directive, adding 101,660 structured motion descriptions to existing BEAT2 recordings.

Critical strengths
  1. BEAT2-Directive adds structured descriptions to existing motion data. The planned release could be a useful resource for future work on language-controlled gestures.
  2. The evaluation is comprehensive, including component ablations, different language-model directors, comparisons using the same predicted plans, and the inference speed reports.
  3. The user studies on BEAT2 and unseen TED speech provide useful evidence of perceptual improvements over the compared baselines.
Critical weaknesses
  1. The current system is designed for offline speech animation, with most of the runtime spent on the Director. Reducing planning latency would be a useful direction toward interactive applications.
  2. Some generated motions in the supplementary examples appear overly rhythmic to me. It would be helpful to discuss what might cause this behavior and whether the strength of rhythmic motion can be adjusted.
Questions for authors

Could the authors discuss the overly rhythmic motion? The latency comment is a suggestion for future work.

1 comment

Response to Reviewer BAiq

Thank you for recognizing the BEAT2-Directive dataset as a useful resource for future work and for appreciating our comprehensive evaluation.


Q1. Overly rhythmic motion

The perception of overly rhythmic motion could potentially stem from two factors: the hyperactive training data style and abrupt motion changes from temporal warping.

Hyperative style. The hyperactive style of some generated gestures may reflect the training data. The BEAT2 authors describe gestures in acted speech as more expressive and varied than those in daily conversation.

"While acted gestures in speech sections are more diverse and expressive, spontaneous gestures in conversational sections tend to be more natural yet less varied." -- [Liu et al., EMAGE, Sec. 5]

our directive interface provides a direct mechanism to steer gesture perference through the Director's system prompt. As shown in our teaser example (Figure 1 and persona_process.mp4), persona prompts such as Right-hand preference alter the predicted directives to encourage right-hand gestures while suppressing left-arm strength to keep it held at the side. System prompts can potentially instruct the Director to favor calmer, more restrained, or held postures to reduce gesture strength without retraining the executor. However, because the executor is trained on the benchmark dataset, the steered behavior remains bounded by the training motion distribution.

Abrupt motion changes. We investigate the abrupt acceleration that potentially results from the warping mechanism. we measured joint acceleration and jerk for Ours (w/ warping) and Ours w/o warping, alongside ground truth motions and three baseline methods. We also measured jerk at window boundaries.

MethodAcceleration (mm/s²)Jerk (mm/s³)Boundary jerk (mm/s³)
Ground truth3,87881,556—
EMAGE3,87984,045—
SemTalk4,21376,026—
GestureLSM4,838158,895—
Ours (w/ warping)4,325112,571125,289
Ours (w/o warping)4,16794,568110,525

Although Warp-VQVAE provides a re-timing handle for applications, shows improvements over BC and FGD relative to Full-body RVQ (Table 4), it can slightly reduce motion smoothness. Ours (w/ warping) exhibits 3.8% higher acceleration, 19.0% higher jerk, and 13.4% higher boundary jerk than the w/o warping variant. The overall smoothness remains comparable to ground truth and prior methods.


Q2. Planning latency

We appreciate this suggestion. Appendix K.1 (Table 11) separates planning latency from execution: the Executor takes approximately 0.03 s per two-second motion window, whereas the Director takes 4.1–11.4 s, depending on the backend and deployment. These results confirm that planning is the primary latency bottleneck and motivate faster local backends or policy distillation for interactive applications.

Reviewer_VNHHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DirectBeat, a human motion generation method conditioned on speech. DirectBeat features a planning phase, where a frozen LLM (Gemini 3.8 Flash) serves as the "director", predicting natural language directives (phase/class/posture/arm movements) from recent directive memory, corpus priors, and retrieved motion examples. Then, an executor generates motion tokens from the directive and time-aligned transcripts. Finally, a Warp-VQVAE with audio-conditioned warp decodes the tokens to human motion. The paper additionally presents the BEAT2-Directive dataset, which contains 101,660 triplets of (speech, directive, motion) that are labelled by a vision-language model and filtered by humans. BEAT2 significantly improves on the FGD metric compared to previous methods. Additionally, the persona can be edited via the system prompt without retraining the executor.

Critical strengths
  1. The LLM-based planning method is insightful and practical. The method decomposes speech-to-motion generation into an intermediate form (directives) that are editable, and changing persona does not require retraining the executor. The method is additionally not model-dependent: replacing Gemini 3.8 Flash with Qwen2.5/Qwen3-Omni still yields an improvement compared to a baseline without the directive.

  2. The BEAT2-Directive dataset is a useful contribution to the community if released.

Critical weaknesses
  1. The claim that LLM-based planning improves generation quality is not fully supported. The executor without directives achieves FGD 3.920 (Table 2), which still outperforms the next-best baseline in Table 1 (GestureLSM). Confidence intervals are also not reported, so a 0.05-0.1 FGD difference cannot be statistically distinguished from noise.

  2. TED Gesture generalization is only measured by a user preference study, without objective evaluations. There is also no evaluation of persona editing.

Questions for authors
  1. Can confidence intervals be calculated for results in Tables 1/2/3/4 using multiple director samples + multiple executor seeds?
  2. The executor without directives achieves FGD 3.920. Can you provide a baseline that trains the executor to directly use speech features (without the LLM) to isolate the contribution of LLM planning?
  3. Can you provide p-values for the TED user preference study results, and explain why SemTalk was excluded from the preference study?
  4. Does the executor generate the whole motion at once, or generate once for each time window? If the latter is the case, when generating for time window tt, can the executor see previous time windows' motion tokens? How are the boundaries between the time windows smoothed to prevent motion jerking? Can evaluations on the continuity of motion and velocity between time windows be provided?
  5. Can you provide quantitative evaluations for persona editing? For example, the frequency/average height of raising the right hand for different personas.
4 comments

Thank you for recognizing the value of explicit, editable planning and BEAT2-Directive. We address the contribution of planning, statistical reliability, generalization, and inference details below.


W1&Q2. Planning contribution and speech-only baseline

We agree that our carefully designed executor and Warp-VQVAE are strong components of DirectBeat. The contribution of planning is measured by comparisons within this architecture, rather than by attributing the entire improvement over prior methods to the LLM. With the executor fixed, predicted directives reduce FGD from 3.920 to 3.631 7.4%↓ (Table 2). We also include several controlled variants that probe what the directive plans contribute (Appendix M.2, Table 12):

Plans supplied to the same executorFGD ↓
Shuffled predicted plans5.891
No directives3.920
Predicted plans3.631
Target-derived plans (oracle)3.215

These comparisons support that appropriate directive plan improve co-speech motion quality.

Speech-only baseline. The No directives condition retains timed transcript conditioning and audio-conditioned decoding (Sections 3.3–3.4). It therefore uses speech information without LLM plans, although raw audio features are not injected into the token generator itself (Appendix E.2).

We additionaly trained a executor that conditioned directly on audio features and timed transcripts but no directives, using the same motion tokenizer and minimal adpatation in architecture. It obtains FGD 4.021 and BC 0.780.


Q1. Confidence intervals

To assess statistical stability, we evaluated 10 runs across Director samples and Executor random seeds on the single-speaker BEAT2 test set, computing 95% confidence intervals (mean ± 95% CI):

ConfigurationFGD ↓BC ↑L1 div.
GestureLSM4.0880.71413.24
Ours (w/o Directives)3.931 ± [0.025]0.778 ± [0.002]10.70 ± [0.07]
Ours (w/o Warping / Full-body RVQ)3.720 ± [0.069]0.775 ± [0.002]10.56 ± [0.10]
Ours (Qwen2.5-Omni-7B)3.747 ± [0.051]0.782 ± [0.002]10.29 ± [0.13]
Ours (Qwen3-Omni-30B)3.698 ± [0.038]0.780 ± [0.001]10.09 ± [0.13]
Ours (Full, Gemini 3.8 Flash)3.639 ± [0.022]0.781 ± [0.001]10.48 ± [0.12]

Across seeds and director samples, variance is small relative to the 0.292 FGD reduction from adding predicted directives (3.931 → 3.639) and the improvements across ablations, confirming that the gains from LLM planning and Warp-VQVAE are consistent.


Q3. TED significance and SemTalk

Statistical significance. We reported p-value below.

CriterionDirectBeat top-1EMAGE top-1GestureLSM top-1vs. EMAGE pvs. GestureLSM p
Realism36.6%31.4%32.0%0.0250.043
Semantics39.0%27.9%33.1%0.0000.015
Synchrony37.5%26.9%35.6%0.0000.253
Diversity35.2%32.0%32.8%0.1160.189

DirectBeat was preferred over both baselines for realism and semantic consistency at the unadjusted 0.05 level. For synchrony, DirectBeat was preferred over EMAGE (p<0.001), while the comparison with GestureLSM was inconclusive (p=0.253). Neither diversity comparison reached significance. The smaller gaps against GestureLSM on synchrony and diversity may reflect differences that were harder for participants to perceive.

SemTalk comparison. As described in Appendix L, our unseen TED evaluation follows a zero-shot multi-speaker protocol: all compared methods had to be retrained on the full all-speaker BEAT2 dataset with speaker-identity conditioning removed to generalize to unseen speakers without TED fine-tuning. Accordingly, we selected GestureLSM and EMAGE as representative full-body and holistic baselines from our all-speaker benchmark (Table 1b). SemTalk was originally released around the single-speaker (Speaker 2) protocol (Table 1a).

Q4. Windowed generation and boundary continuity

As noted in Appendix G, the Executor generates motion tokens in successive two-second windows. Following common practice on BEAT2 [2], when generating window tt, the Executor receives the last motion token of four frames from window t−1t-1 as a prefix to maintain local cross-window continuity, while the Director's two-window memory maintains high-level behavioral transitions.

To quantitatively evaluate motion continuity across window transitions, we measured joint acceleration, overall jerk, and boundary jerk (specifically at the two-second window boundaries) on the single-speaker BEAT2 test set:

MethodAcceleration (mm/s²)Overall Jerk (mm/s³)Boundary Jerk (mm/s³)
Ground truth3,87881,556—
EMAGE3,87984,045—
SemTalk4,21376,026—
GestureLSM4,838158,895—
Ours4,325112,571125,289

Boundary jerk (125,289 mm/s³) is moderately higher than overall jerk (112,571 mm/s³), showing that window transitions introduce minor local velocity changes. Nevertheless, even at window boundaries, our jerk remains lower than the overall jerk of the strongest FGD baseline, GestureLSM (158,895 mm/s³).


Q5&W2. Objective TED Gesture and persona control

For zero-shot evaluation on TED Gesture, there is no established benchmark, compatible 3D ground truth. More broadly across gesture and motion generation, human perceptual evaluation via user studies remains the practical protocol for comparing generative models [1].

We present persona editing as a zero-shot downstream application enabled by our explicit directive interface rather than as a supervised stylization benchmark. Because prior co-speech motion methods do not support training-free, language-driven persona customization via system prompts, there is no directly comparable baseline or established quantitative benchmark for this setting.

  • [1] Kucherenko*, Taras, et al. "Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022." ACM Transactions on Graphics 43.3 (2024): 1-28.
  • [2] Liu, Haiyang, et al. "Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Thank you for the authors' detailed response. They resolved most of my concerns. I would like to keep my rating.

Thank you for the follow-up. We’re glad our response addressed most of your concerns. We appreciate your decision to keep the positive rating. Your constructive feedback has helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.