acceptodds
Under review as a conference paper at ICLR 2027

StrokeForm: Semantic Co-Speech Gesture Generation with Kinematically Grounded Motion Tokens

Abstract

Semantic co-speech gestures express linguistic meaning through body and hand movements. Existing speech-conditioned methods generate natural, rhythmically synchronized motion but remain limited in generating contextually appropriate semantic gestures. Learning such gestures from speech–motion training data is challenging due to their sparse, long-tailed distribution and diverse realizations of the same meaning. Large language models can support context-aware gesture planning, but their plans require an explicit motion interface for reliable execution. We present StrokeForm, a framework that connects language-based gesture planning to motion generation through kinematically grounded motion tokens. An adapted language model plans semantic gestures by specifying their arm configurations, palm orientations, and handshapes, together with contextually relevant linguistic triggers. A grounded tokenizer maps these kinematic attributes to addressable stroke tokens. Anchored near their linguistic triggers, these tokens guide a speech-conditioned masked infiller to generate complete gesture events, while a separate phase condition guides temporal realization. Extensive experiments on BEAT2 demonstrate that StrokeForm achieves higher human-rated semantic appropriateness than the evaluated state-of-the-art baselines and enables controllable realization of specified gesture forms.

Primary area:
generative models
Keywords:
Co-speech gesture generation, Semantic gesture planning, Kinematically grounded motion tokens, Controllable motion generation, Language-guided motion synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_X3eMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents StrokeForm, a framework that connects language-based gesture planning to motion generation through kinematically grounded motion tokens. The authors demonstrate that StrokeForm achieves the highest mean human-rated semantic appropriateness among the evaluated methods on BEAT2 and supports controllable realization of gesture forms with motion-quality trade-offs.

Critical strengths
  1. The paper introduces an explicit interface between language-guided gesture planning and motion generation, enabling controllable gesture synthesis while preserving generative flexibility.
  2. The method achieves the best human-rated semantic appropriateness and speech–gesture alignment among the evaluated generation methods.
  3. The paper provides quantitative ablation studies demonstrating the effectiveness of its key architectural components, particularly anchor-centric remasking.
Critical weaknesses
  1. The paper does not provide an inference time comparison with existing methods, leaving the computational overhead introduced by language model-based gesture planning and the practicality of real-time gesture generation unclear.
  2. The paper lacks comparisons with more capable language models, such as Gemini or GPT, making it unclear how the choice of LLM backbone affects gesture planning performance.
  3. The gesture representation mainly focuses on static kinematic attributes around stroke peaks, without explicitly modeling dynamic properties such as movement trajectories, directions, amplitudes, and speeds, which may limit its ability to represent complex semantic gestures.
Questions for authors
  1. Could the authors report the inference latency of StrokeForm, including the language-model-based planning stage, and compare it with existing methods to evaluate its computational efficiency?
  2. Could the authors evaluate stronger LLMs, such as Gemini 2.5 Pro or GPT 5.6, under consistent input and evaluation settings to investigate the impact of different language models on gesture planning performance?
  3. Could the authors discuss how the proposed framework could incorporate dynamic gesture attributes, such as movement trajectories, directions, amplitudes, and speeds, to represent complex semantic gestures better?
2 comments
AuthorsOpenReview ↗

Part I: Inference Efficiency and Asynchronous Deployment

Thank you for requesting a latency comparison including language-based planning. We report motion-generation benchmarks, concurrent module timings, and a derived asynchronous timeline separately.

1. Motion-generation comparison

The following benchmarks use an RTX 3090, batch size 1, and FP32 motion networks. We report median wall-clock time over 10 runs after 3 warm-up runs. RTF is generation time divided by output motion duration. Native windowing accounts for the slightly different durations.

Method / modeMotion duration (s)Generation time (s) ↓RTF ↓
EchoMask10.1330.3830.0378
SemTalk10.1330.4940.0488
StrokeForm-VQ / audio-only10.0000.5860.0586
RAG-Gesture†10.0002.8850.2885

Timings exclude model loading, audio/text feature extraction, ASR/alignment, disk I/O, and rendering. Baseline timings are repeated measurements of one fixed BEAT2 clip, not test-set averages. †RAG-Gesture uses a fixed exemplar and 50-step sampling, including exemplar encoding, inversion, guidance, and decoding, but excluding LLM planning and database retrieval.

StrokeForm-VQ therefore generates motion faster than playback under this protocol. This table compares motion-generation cost, not complete-system latency, and does not cover the new RVQ variant.

2. Concurrent module timings, including the planner

We retain the paper’s 4-second model window. For a 10-second utterance containing two gesture events, the following module timings were measured on an RTX 3090 with the Qwen3-8B planner and motion branch running concurrently. Models are loaded and warmed up; speech, text, and required timestamps are available.

ModuleMeasured time
Planner prefill, once per request120 ms
Event decoding, approximately 64 tokens per event580 ms
Validation and compilation, per event12 ms
Initial audio-only window generation and decoding180 ms
Subsequent conditioned window generation and decoding160 ms
Transition handling, buffering, and submission, per update24 ms

The approximately 128 output tokens require 1.16 s of decoding, corresponding to about 110 tokens/s in this example. The 180/160 ms window-generation times correspond to RTFs of 0.045/0.040. Prefill and submission costs are reported separately.

3. Asynchronous system timeline

Initial audio-only generation runs alongside planning. Each complete event can be validated and compiled while the planner continues decoding subsequent events. Compiled constraints can then condition eligible motion intervals that remain editable.

Using the measured concurrent module times, the following milestones are derived under a schedule with no additional queueing:

MilestoneTime after input readiness
Initial segment ready for playback submission204 ms
First event validated and compiled712 ms
First conditioned segment ready for submission896 ms
Both events validated and compiled1,292 ms
Second conditioned segment ready for submission1,476 ms

For example, first-event readiness is 120 + 580 + 12 = 712 ms, followed by 160 + 24 ms to prepare its conditioned segment. First-event compilation overlaps second-event decoding, so both events are compiled at 120 + 2 × 580 + 12 = 1,292 ms.

These are module-based timing estimates, not directly measured end-to-end latency percentiles. Submission readiness also differs from actual display time or the scheduled appearance of a semantic gesture. On-time execution requires that the affected motion interval remain editable and that its update complete before the submission deadline. A 4-second model window does not by itself establish the playback-commit granularity.

4. Practical scope and deployment demonstration

The asynchronous schedule allows initial motion generation to proceed without waiting for the complete plan. Our measurements start after the utterance inputs are available; they do not include speech acquisition or establish the latency of processing continuously arriving speech. Such streaming performance also depends on input lookahead and playback-commit policies.

We encourage the reviewer to watch the Apple Vision Pro digital-human interaction demonstration in the supplementary video, VideoDemo_Compressed.mp4. It shows our system deployed in an interactive application and illustrates its practical use.

We will include the timing protocols and the distinction between measured module costs and derived system milestones in the revised manuscript.

AuthorsOpenReview ↗

Part II: Planner Choice and Dynamic Gesture Representation

Thank you for suggesting comparisons with stronger language models. We added Gemini 2.5 Pro and GPT-5.6 to examine whether greater general-purpose model capability can substitute for task-specific adaptation.

1. Controlled comparison of planners

We evaluate intent and form prediction with fixed reference triggers, using the same linguistic context, timing information, and interface schema. Retrieval-based configurations receive the same examples selected from the LoRA training-event pool; no test-event reference forms or motions are supplied as inputs.

PlannerIntent agreement ↑Arm agreement ↑Joint form agreement ↑Predicted-event compilability ↑
Qwen3-8B / Schema41.2%18.6%6%78%
Qwen3-8B / Schema + retrieval55%31%12%89%
Gemini 2.5 Pro / Schema + retrieval65%37%16%93%
GPT-5.6 / Schema + retrieval68%40%19%94%
Qwen3-8B + LoRA71.5%43.8%22%96%

Arm agreement follows the original planner-ablation metric. Joint form agreement requires all evaluated form fields to match the reference. Compilability measures acceptance by the interface’s compilation checks. Because triggers are supplied, this experiment evaluates conditional intent/form planning rather than automatic event selection.

Retrieval improves the base Qwen3-8B model, and stronger general-purpose models provide further gains. Nevertheless, task-adapted Qwen3-8B achieves the highest point estimates across all four metrics. Compared with GPT-5.6, it improves intent agreement by 3.5 percentage points, arm agreement by 3.8 points, joint form agreement by 3 points, and compilability by 2 points.

2. Task-specific adaptation and deployment considerations

Our planner learns associations between linguistic context, communicative intent, and motion-derived form specifications. These include dataset-specific preferences for handshape, palm orientation, and arm configuration within the supported vocabulary. Schema constraints enforce output legality, while LoRA adaptation targets which form to select for a given context.

The results support the value of this adaptation even when stronger models receive retrieved training examples. They do not establish general superiority of the Qwen backbone: the LoRA model receives task-specific training, whereas the other configurations use prompting and in-context examples. Reference agreement also does not capture every valid alternative gesture realization.

Deployment cost and responsiveness are additional considerations. Our 8B planner can run locally alongside the motion branch on a single RTX 3090, without requiring a hosted proprietary-model call for each plan. In the concurrent profile reported in Part I, planner prefill takes 120 ms and decoding an approximately 64-token event takes 580 ms. Planning overlaps with motion generation, so initial motion generation need not wait for the complete plan.

This provides a practical deployment option combining strong task-specific performance with modest hardware requirements and control over serving costs. We therefore select the planner based on both interface-specific effectiveness and deployability, rather than model scale alone.

3. Dynamic gesture attributes

StrokeForm generates complete motion sequences, but its explicit interface currently specifies kinematic form near a shared stroke peak. The onset–peak–offset schedule and phase conditioning convey temporal structure; they do not explicitly prescribe a full trajectory or speed profile. Likewise, body-relative position and reach describe a configuration, not the direction or amplitude of an entire movement.

A concrete extension would represent each event using multiple time-stamped form anchors, optionally augmented with trajectory waypoints and velocity constraints. Anchor-centric infilling could provide a basis for generating the intervening motion. This would require extending the event representation and compiler, adding dynamic supervision, and evaluating trajectory adherence and transition naturalness.

We will clarify that trajectory-dependent, multi-stroke, and asynchronously coordinated gestures remain outside the current demonstrated control scope. The proposed extension is a future direction, not a capability established by the present experiments.

We will include the additional planner comparisons, evaluation details, and scope clarifications in the revised manuscript.

Reviewer_dTvMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an LLM-based motion planning method for the co-speech motion generation task. It proposes a method that encodes LLM motion planning into an anchor token, and generates co-speech motion conditioned on the anchor token. The anchor token is a fixed address for a structured gesture form (arm configuration, palm orientation, handshape), and a speech-conditioned masked infiller generates the remaining motion around it. The method can run in three modes: pure audio-driven generation without any plan, LLM-planned generation, and user-specified instruction. Experiments on BEAT2 compare with EMAGE, SemTalk, EchoMask, and RAG-Gesture on FGD/BC/DIV and a human evaluation. With the LLM planner, the semantic appropriateness score is the highest among the generation methods (0.40 vs 0.23-0.25), but FGD and BC get worse. The controllability is evaluated by slot agreement and joint success against the requested specs (44.2% joint success; 68% of the annotated test events can be compiled). An edit experiment shows that changing the request under the same speech raises joint success from 1.2% to 33.8%.

Critical strengths
  1. The stroke-peak event formulates semantic co-speech gestures as stroke-peak-centered events, which link measurable form specifications to linguistic triggers while leaving the surrounding motion generative. This captures where the meaning is, without over-constraining the rest of the motion.

  2. The kinematically grounded motion tokens are an interesting idea. Compiling supported specifications into fixed, partwise stroke-token addresses gives an explicit and verifiable interface between LLM planning and motion generation: the planner's output can be checked and anchored, instead of being a free-text hint. Together with the stroke-anchored masked infiller and the phase-conditioned decoder, this design connects language-guided planning to motion generation in a principled way.

Critical weaknesses
  1. In the experiment part, in the overall comparison, the audio-only base model performs worse than the sota method SemTalk on FGD and BC, while the LLM-planning method is further worse than the audio-only base model. Author seem not explain the reason why.

  2. For what is essentially a command-following task, the human evaluation compares the semantic-driven method only with audio-driven methods that never receive a command. The instruction-conditioned methods cited by the paper (Semantic Gesticulator, Semantic Motion Anchors) are never compared on command alignment, so the comparison shows the value of having a command channel, not the value of the proposed command channel.

  3. The instruction-following evaluation has no human aspect. The instructions are reference specs instead of real user commands, the scoring uses the same geometric readouts as the training supervision.

Questions for authors

Please provide clarification on the listed weaknesses, I will consider raise to weak acceptance based on the clarifications.

2 comments
AuthorsOpenReview ↗

Part I: Motion Quality and Planner Conditioning

Thank you for highlighting the motion-quality gap and the effect of planner conditioning. We distinguish the capacity of the motion representation from the role of our control interface.

Motion representation and the RVQ extension.

We originally chose single-level VQ to validate the addressable interface with a simple motion representation. This version achieves lower FGD than VQ-based EMAGE in both settings and is competitive with RVQ-based EchoMask. To address the comparison with SemTalk, we added an RVQ extension with matching quantizer settings while retaining the form-control interface. This tests whether higher representational capacity can improve quality without sacrificing control.

Objective results. S = single speaker; M = all 25 speakers. ↓ means lower is better; → means closer to GT. VQ/RVQ denote StrokeForm variants.

Method / modeS-FGD ↓S-BC →S-DIV →M-FGD ↓M-BC →M-DIV →
GT—0.70311.97—0.4777.29
EMAGE0.5510.77213.060.6920.2846.06
EchoMask0.4620.77413.370.5660.4959.30
SemTalk0.4280.77712.910.3560.8418.41
RAG-Gesture0.8080.73411.920.4870.5149.94
VQ / audio-only0.4970.73613.180.4210.7849.23
VQ / LLM-plan0.6040.79812.680.5880.8288.26
RVQ / audio-only0.3780.72712.480.3480.5087.74
RVQ / LLM-plan0.4250.72212.660.4160.4987.96
VQ / reference†0.4080.73212.760.3240.4947.63
RVQ / reference†0.3560.71812.240.3020.4867.48

†Reference-plan uses motion-derived reference specifications and schedules as an oracle diagnostic. The original “with planner” setting uses predicted Qwen3-8B plans (LLM-plan). RAG-Gesture's objective metrics are quoted from its paper.

Quality and control are compatible.

RVQ reduces audio-only FGD by 23.9%/17.3%, reaching 0.378/0.348 versus SemTalk's 0.428/0.356. Reference-conditioned joint execution remains comparable (44.2% vs. 45.4%). Our tokenizer provides a structured, compact control interface through stable partwise form addresses, while free-motion codes model surrounding motion. The extension shows that the interface accommodates higher-capacity quantization while preserving form execution.

BC is evaluated relative to GT.

Under our protocol, higher BC is not uniformly better. For 25 speakers, GT BC is 0.477: the original audio-only value of 0.784 is closer than SemTalk's 0.841. RVQ reduces the absolute deviation to 0.031 for audio-only and 0.021 for LLM-plan.

LLM planning and distributional fidelity.

Plans predicted from speech need not reproduce the gesture form recorded in GT, even when they express the same meaning. We therefore evaluate semantic matching separately from distributional fidelity:

Method / modeSemantic ↑Speech ↑Realism ↑
GT0.410.451000
EMAGE−0.050.05934
EchoMask−0.04−0.10935
SemTalk0.230.07942
RAG-Gesture0.04−0.09947
VQ / audio-only0.250.16969.5*
VQ / LLM-plan0.400.25917.4*
RVQ / audio-only0.260.17995.4*
RVQ / LLM-plan0.420.26988.3*

Semantic/Speech are matching scores on [−1,1]. Realism uses Elo with GT anchored at 1000. *Updated realism estimates from the supplementary evaluation; baseline human scores are retained from the original evaluation. These values do not establish cross-study Elo rankings.

With RVQ, LLM planning improves semantic/speech matching from 0.26/0.17 to 0.42/0.26, while 25-speaker FGD rises from 0.348 to 0.416. Reference-plan FGD is 0.302, illustrating sensitivity to plan selection. Thus, distributional fidelity and semantic matching capture different aspects of the output. Valid alternative forms can contribute to the FGD difference, but planning and execution errors remain possible.

These results support improved motion quality while retaining the interface. Part II addresses whether that interface offers benefits over existing controllable methods.

AuthorsOpenReview ↗

Part II: Interface Effectiveness and Human Evaluation

We agree that audio-only comparisons establish the value of adding control, but cannot establish the value of our particular interface. We therefore added comparisons with existing controllable methods, human evaluation, and interactive qualitative examples.

Comparison with controllable methods.

We compare with Semantic Gesticulator [1] and RAG-Gesture [2] under the same speech, event timing, and requested forms. Each baseline uses its native reference-motion interface, rather than being required to accept StrokeForm tokens.

MethodInstruction following (1–5) ↑Edit success ↑Naturalness (1–5) ↑
Semantic Gesticulator3.445%2.6
RAG-Gesture2.723%3.4
StrokeForm-RVQ4.688%4.2

StrokeForm-RVQ achieves higher mean instruction-following and naturalness scores, together with a higher edit-success rate. These human judgments complement the geometric readouts shared with training. Human edit success is a separate measure from geometric joint success.

Interactive qualitative examples.

Please download and unzip the updated supplementary material, then open Qualitative_Comparison_of_Form-Controlled_Gestures.html in a browser. The page contains the comparison figure and synchronized videos with complete transcripts, word highlighting, slow playback, and a Key pose button. All media are embedded; no server or installation is required.

The two cases illustrate how the same semantic intention can be expressed through different physical forms:

  • Self-reference: “…by myself…” At approximately 3.53 s, our index-finger and open-hand variants both express self-reference near the upper chest. The baseline outputs contain related gestural cues, but their handshape and/or chest-relative placement differ from the displayed target forms. The hand close-ups make these differences easier to inspect.
  • Leftward indication: “…the left of the frame…” At approximately 4.37 s, our variants use either the right index finger across the body or the left open hand extended laterally. Thus, the alternatives change active-hand choice and arm configuration as well as handshape. “Left” denotes the speaker’s own left. Semantic Gesticulator does not produce a clear leftward indication at the selected time; RAG-Gesture’s handshape and pointing configuration are less explicit in the displayed output.

Each baseline column shows one selected reference-guided output per case; the two Ours columns illustrate alternative form specifications. These are examples of multi-attribute form variation, not handshape-only selective edits. The figure illustrates individual outcomes, while the human evaluation above provides the aggregate comparison.

Retrieval-based control and its limitations.

Semantic Gesticulator retrieves semantic gestures from a motion library and aligns their tokens with speech-driven generation. RAG-Gesture retrieves exemplar motions and incorporates them through DDIM inversion and retrieval guidance.

In both cases, a selected motion instance conveys the requested form. Precise control therefore depends on finding a suitable reference and preserving its relevant configuration during generation. If the pool lacks the desired combination of handshape, palm orientation, and arm placement, retrieval alone cannot supply an exact reference; the method must rely on adapting a nearby exemplar or constructing an additional reference. Both methods remain generative and can adapt references.

StrokeForm instead exposes supported arm, wrist, and hand forms through stable partwise addresses. A supported combination can be specified directly without first locating a complete exemplar. The examples make the distinction concrete: semantic relevance to “myself” or “left” does not by itself establish adherence to the requested physical form.

Scope of the claim.

The results support more reliable form execution under shared control requirements, beyond the benefit of simply adding a command channel. Our vocabulary remains finite: 68% of the original reference requests pass compilation, and direct addressing does not guarantee successful execution of every supported combination.

Together, Part I addresses motion quality and representation capacity, while these comparisons and qualitative examples demonstrate the practical value of explicit form control. We will consolidate the additional results and evaluation protocols in the revised manuscript.

[1] Zhang et al. Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis. SIGGRAPH 2024.

[2] Mughal et al. Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis. CVPR 2025.

Reviewer_qFkCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on creating a semantically grounded tokenizer to improve the semantic appropriateness of co-speech gestures. The authors do this by creating tokens that are trained to capture certain strokes that are kinematically grounded using a schema (this is done offline) and then free motion tokens are introduced to take care of the rest of the motion. Overall, the approach achieves comparatively better semantic appropriateness with a trade-off with motion quality. The main contribution is the proposal of a semantically grounded tokenizer which is not currently used in co-speech gesture generation works but is definitely an important direction. The design of the stroke-form tokens is done kinematically and is also split based on the stroke which is also interesting and is grounded in gesture research.

Critical strengths

The main strength is the fact that this is an important start to looking at more semantically grounded tokenizer, or basically bringing in the structure of gestures that we already know of in gesture research for modelling. This first good attempt (at least from my knowledge of existing works) is the main strength of this paper. Overall, it’s a good proposal rather than a robust solution for now.

Critical weaknesses

It’s an interesting proposal but having said that the trade-off in motion quality is why I said it’s not a robust solution yet. It needs to achieve similar performance to existing works on motion quality to make it a usable solution. While this is mentioned as a limitation, it nevertheless is a very important issue to be tackled as good motion quality is the base and semantic control is an important addition to it. Below is one of the more important gesture feature which should be looked at potentially that could improve quality degradation.

We know that gestures vary in duration quite a lot, for instance, if we want to say “this and that” with Palm Up Hand Open from one side to another is a longer duration than saying “The one time” with a pointing up gesture with one finger for “one”. Now it’s a bit unclear how this is encoded currently since I see that this is evaluated only with joint success at the peak and 140 ms error seems to be a bit too high. This also makes me wonder how the retraction and preparation phases look especially for gestures with varying lengths as they are important for the gestures to look natural but currently it seems to be determined by the free motion tokens and my thought is that the interaction between anchored stroke tokens and surrounding free motion generation could be a source of quality degradation. In addition, there are gestures in BEAT2 which have hold phases as well (defined in gesture research), it’s also unclear how those are represented.

Questions for authors

It would evaluate the generations to track down where the motion quality degradations come from for instance maybe seeing how the preparation and retractions look and maybe computing motion quality metrics such as FGD over these surrounding phases specifically would provide more complete evidence of whether the free motion token generation produces naturalistic gesture transitions.

Since when to gesture is determined by GPT 5.6 it would be good to show qualitative examples from RAGesture as an LLM is used there as well to know where to gesture

4 comments
AuthorsOpenReview ↗

Part I: Motion Quality and Gesture Transitions

Thank you for emphasizing that motion quality is essential and for suggesting an evaluation around stroke peaks. We added a local-context diagnostic and a transition-specific human evaluation.

Tokenizer choice and the RVQ extension.

We originally used single-level VQ as a simple implementation for establishing and evaluating our form-addressable interface. This version achieves lower FGD than VQ-based EMAGE in both settings and is competitive with RVQ-based EchoMask. To address the quality concern, we extended motion quantization to RVQ, with quantizer settings aligned with SemTalk, retaining the same control interface and anchor-centric generation strategy.

Audio-only FGD improves from 0.497/0.421 to 0.378/0.348 in the single-/25-speaker settings, compared with SemTalk’s 0.428/0.356. Reference-conditioned joint execution remains comparable (44.2% vs. 45.4%). These results support our view that representation capacity contributes to the original quality gap, which should not be interpreted as an inherent limitation of explicit form control.

Pre- and post-peak context evaluation.

We use 24 clips containing two reference events each (48 events). For each reference peak ta, we extract [ta−4 s, ta) and [ta, ta+4 s): 120 frames per window at 30 fps. GT and all conditions share the same event list and windows. Selection is independent of generation quality and is not restricted to successfully executed events. FGD is computed between feature collections separately for each context, not averaged over event-wise FGD values.

VQ and RVQ reference-plan conditions share the same motion-derived specifications and schedules, isolating motion realization from planner prediction. These are oracle diagnostic conditions. Audio-only outputs use the same reference-aligned evaluation windows. The windows measure surrounding-motion distributions, not exclusively linguistic preparation/retraction phases.

ConditionPre-peak context FGD ↓Post-peak context FGD ↓
VQ / audio-only0.5640.582
VQ / reference-plan0.5240.568
RVQ / audio-only0.3770.392
RVQ / reference-plan0.3520.383

Within the original VQ setting, reference conditioning reduces pre-/post-peak FGD from 0.564/0.582 to 0.524/0.568. Explicit anchoring therefore does not increase the measured local distributional discrepancy in this diagnostic. RVQ further improves both conditions, with reference-conditioned FGD remaining lower than its audio-only counterpart. These findings support compatibility between form anchoring and surrounding-motion quality already in the VQ framework, with additional improvements from RVQ.

Supplementary GT-referenced human comparisons also show more favorable entry and exit preferences for RVQ than for VQ, consistent with the objective results. The detailed human-evaluation protocol, scoring procedure, and complete results will be included in the appendix of the revised manuscript.

How anchor-centric remasking supports transitions.

The transition mechanism is already present in the original VQ model. Stroke anchors remain visible and fixed while surrounding free-motion tokens are generated from speech, visible context, and phase information. Our retention rule combines proximity to the stroke peak with prediction confidence, prioritizing context completion near anchors. Lower-scoring predictions are remasked, and the phase-conditioned decoder converts the completed sequence into motion.

The original ablation supports this strategy under identical decoding and update budgets:

Remasking strategyJoint success (%) ↑Peak error (ms) ↓FGD ↓
Anchor-centric44.21400.571
Anchor-agnostic29.71760.689

These FGD values follow the original reference-event ablation protocol and are not directly comparable to the local-context FGD above.

Semantic expressiveness and practical quality.

Our original semantic-appropriateness evaluation involved 80 participants. StrokeForm achieves the highest mean semantic matching score among the evaluated generation methods: 0.40, compared with 0.23 for the strongest existing baseline. Because this task assesses correspondence between motion and utterance meaning, it supports the communicative value of explicit semantic gestures beyond rhythmic synchronization.

We agree that this expressive benefit must be accompanied by adequate motion quality. The original ablation supports our anchor-centric generation mechanism; the new within-tokenizer comparisons examine the effect of anchoring, and the VQ/RVQ comparisons assess the benefit of increased representation capacity. Together, these results strengthen the practical evidence for combining semantic expressiveness, explicit control, and motion quality within the supported single-peak gesture scope.

AuthorsOpenReview ↗

Part II: Temporal Modeling, Hold Phases, and Qualitative Examples

Thank you for asking how gesture duration and hold phases are represented. We clarify the distinction between the temporal information supported by the model and the controls explicitly exposed by the current interface.

Gesture duration and phase conditioning.

Each gesture event is represented by its onset, stroke peak, and offset, (t0, ta, t1). The pre- and post-peak durations can therefore differ across events; the representation does not impose one fixed duration on all gestures.

The phase condition combines progress normalized by the corresponding pre-/post-peak duration with an absolute offset from the stroke peak. This preserves both relative event progress and absolute time scale. These conditions guide the infiller and decoder in realizing the surrounding motion.

For automatically predicted plans, the current scheduler uses intent-conditioned training medians for the pre-/post-peak durations and the peak location relative to the linguistic trigger. This provides a simple scheduling rule, but does not fully capture context-dependent duration variation within the same intent. We will clarify this distinction: the motion representation accommodates variable event durations, while the current automatic scheduler remains comparatively coarse.

Hold phases.

The current interface does not contain a separate hold-duration parameter. Generated motion may contain a period of low velocity near the stroke, but this does not constitute explicit control of a hold phase. Extending an event’s duration also does not necessarily maintain the same form throughout that interval.

A concrete extension would add hold onset and offset to the event schedule and apply form-maintenance constraints over the resulting interval. This would require corresponding annotations and evaluation. We will explicitly state this limitation rather than imply that the present single-peak formulation fully models preparation, stroke, hold, and retraction as independently controllable phases.

Interpretation of the 140 ms timing error.

The reported 140 ms is the median absolute difference between the detected and requested stroke-peak times. Joint success is evaluated directly at the requested time, without searching within a 140 ms tolerance window.

The original phase-conditioning ablation increases median peak error from 140 ms to 518 ms when phase conditioning is removed, supporting its role in temporal alignment. Nevertheless, 140 ms represents a remaining alignment error, not exact peak execution. The local-context evaluation in Part I assesses surrounding-motion quality and should not be interpreted as eliminating this timing limitation.

Planner clarification and RAG-Gesture examples.

GPT-5.6 is used offline to produce candidate semantic annotations for planner supervision. At inference, plans are predicted by LoRA-adapted Qwen3-8B, and the scheduler converts the predicted trigger information into event timings. GPT-5.6 therefore does not determine gesture timing during inference.

We have added qualitative comparisons with RAG-Gesture in the updated supplementary material. After downloading and unzipping it, please open Qualitative_Comparison_of_Form-Controlled_Gestures.html in a browser. The page includes synchronized motion videos, complete transcripts, keyword highlighting, slow playback, and target-pose inspection.

The examples cover self-reference near “myself” and leftward indication near “left,” allowing inspection of the gesture form and its surrounding motion in the speech context. RAG-Gesture uses preselected exemplars in these examples, so they compare motion realization under specified semantic targets rather than automatic event selection by different LLMs.

We will incorporate these temporal-modeling clarifications, the explicit hold limitation, and the additional evaluation details into the revised manuscript.

Dear Reviewer qFkC,

We would appreciate your feedback on the additional motion-quality and transition experiments in our response, including VQ/RVQ comparisons and matched pre-/post-peak context evaluations. Could you please let us know whether these address your concerns, or which issues remain unresolved?

We would particularly appreciate your viewing the supplementary application demo, VideoDemo_Compressed.mp4. It demonstrates our method in a real-time interactive avatar application deployed on Apple Vision Pro, providing a direct view of its semantic gesture expressiveness, motion naturalness, and transitions in practice.

Thank you for your time and consideration.

I thank the authors for the detailed response and the additional experiments. My main concern was the trade-off in motion quality, and the RVQ extension largely resolves it. I also appreciate the clarifications on how gesture duration is represented and the explicit statement that hold phases are currently not modelled.

The human evaluation of semantic appropriateness was however carried out with the original VQ model, so it is not yet clear whether the semantic advantage carries over to the RVQ version. The 4 s windows in the local-context analysis are a lot longer than typical preparation and retraction phases. In addition, the reference-plan condition is an oracle derived from the ground-truth motion, so its lower FGD may partly reflect the additional information and does not show that the transitions around the anchors are natural. A comparison of transition and non-transition windows, or of StrokeForm and the baselines within the same windows, would be more informative.

Overall, the paper now offers a semantically grounded tokenizer that does not come at the cost of motion quality, which I consider an important step for co-speech gesture generation. I will therefore raise my score to weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.