- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents StrokeForm, a framework that connects language-based gesture planning to motion generation through kinematically grounded motion tokens. The authors demonstrate that StrokeForm achieves the highest mean human-rated semantic appropriateness among the evaluated methods on BEAT2 and supports controllable realization of gesture forms with motion-quality trade-offs.
Critical strengths
- The paper introduces an explicit interface between language-guided gesture planning and motion generation, enabling controllable gesture synthesis while preserving generative flexibility.
- The method achieves the best human-rated semantic appropriateness and speech–gesture alignment among the evaluated generation methods.
- The paper provides quantitative ablation studies demonstrating the effectiveness of its key architectural components, particularly anchor-centric remasking.
Critical weaknesses
- The paper does not provide an inference time comparison with existing methods, leaving the computational overhead introduced by language model-based gesture planning and the practicality of real-time gesture generation unclear.
- The paper lacks comparisons with more capable language models, such as Gemini or GPT, making it unclear how the choice of LLM backbone affects gesture planning performance.
- The gesture representation mainly focuses on static kinematic attributes around stroke peaks, without explicitly modeling dynamic properties such as movement trajectories, directions, amplitudes, and speeds, which may limit its ability to represent complex semantic gestures.
Questions for authors
- Could the authors report the inference latency of StrokeForm, including the language-model-based planning stage, and compare it with existing methods to evaluate its computational efficiency?
- Could the authors evaluate stronger LLMs, such as Gemini 2.5 Pro or GPT 5.6, under consistent input and evaluation settings to investigate the impact of different language models on gesture planning performance?
- Could the authors discuss how the proposed framework could incorporate dynamic gesture attributes, such as movement trajectories, directions, amplitudes, and speeds, to represent complex semantic gestures better?
2 comments
Part II: Planner Choice and Dynamic Gesture Representation
Thank you for suggesting comparisons with stronger language models. We added Gemini 2.5 Pro and GPT-5.6 to examine whether greater general-purpose model capability can substitute for task-specific adaptation.
1. Controlled comparison of planners
We evaluate intent and form prediction with fixed reference triggers, using the same linguistic context, timing information, and interface schema. Retrieval-based configurations receive the same examples selected from the LoRA training-event pool; no test-event reference forms or motions are supplied as inputs.
| Planner | Intent agreement ↑ | Arm agreement ↑ | Joint form agreement ↑ | Predicted-event compilability ↑ |
|---|---|---|---|---|
| Qwen3-8B / Schema | 41.2% | 18.6% | 6% | 78% |
| Qwen3-8B / Schema + retrieval | 55% | 31% | 12% | 89% |
| Gemini 2.5 Pro / Schema + retrieval | 65% | 37% | 16% | 93% |
| GPT-5.6 / Schema + retrieval | 68% | 40% | 19% | 94% |
| Qwen3-8B + LoRA | 71.5% | 43.8% | 22% | 96% |
Arm agreement follows the original planner-ablation metric. Joint form agreement requires all evaluated form fields to match the reference. Compilability measures acceptance by the interface’s compilation checks. Because triggers are supplied, this experiment evaluates conditional intent/form planning rather than automatic event selection.
Retrieval improves the base Qwen3-8B model, and stronger general-purpose models provide further gains. Nevertheless, task-adapted Qwen3-8B achieves the highest point estimates across all four metrics. Compared with GPT-5.6, it improves intent agreement by 3.5 percentage points, arm agreement by 3.8 points, joint form agreement by 3 points, and compilability by 2 points.
2. Task-specific adaptation and deployment considerations
Our planner learns associations between linguistic context, communicative intent, and motion-derived form specifications. These include dataset-specific preferences for handshape, palm orientation, and arm configuration within the supported vocabulary. Schema constraints enforce output legality, while LoRA adaptation targets which form to select for a given context.
The results support the value of this adaptation even when stronger models receive retrieved training examples. They do not establish general superiority of the Qwen backbone: the LoRA model receives task-specific training, whereas the other configurations use prompting and in-context examples. Reference agreement also does not capture every valid alternative gesture realization.
Deployment cost and responsiveness are additional considerations. Our 8B planner can run locally alongside the motion branch on a single RTX 3090, without requiring a hosted proprietary-model call for each plan. In the concurrent profile reported in Part I, planner prefill takes 120 ms and decoding an approximately 64-token event takes 580 ms. Planning overlaps with motion generation, so initial motion generation need not wait for the complete plan.
This provides a practical deployment option combining strong task-specific performance with modest hardware requirements and control over serving costs. We therefore select the planner based on both interface-specific effectiveness and deployability, rather than model scale alone.
3. Dynamic gesture attributes
StrokeForm generates complete motion sequences, but its explicit interface currently specifies kinematic form near a shared stroke peak. The onset–peak–offset schedule and phase conditioning convey temporal structure; they do not explicitly prescribe a full trajectory or speed profile. Likewise, body-relative position and reach describe a configuration, not the direction or amplitude of an entire movement.
A concrete extension would represent each event using multiple time-stamped form anchors, optionally augmented with trajectory waypoints and velocity constraints. Anchor-centric infilling could provide a basis for generating the intervening motion. This would require extending the event representation and compiler, adding dynamic supervision, and evaluating trajectory adherence and transition naturalness.
We will clarify that trajectory-dependent, multi-stroke, and asynchronously coordinated gestures remain outside the current demonstrated control scope. The proposed extension is a future direction, not a capability established by the present experiments.
We will include the additional planner comparisons, evaluation details, and scope clarifications in the revised manuscript.
Part I: Inference Efficiency and Asynchronous Deployment
Thank you for requesting a latency comparison including language-based planning. We report motion-generation benchmarks, concurrent module timings, and a derived asynchronous timeline separately.
1. Motion-generation comparison
The following benchmarks use an RTX 3090, batch size 1, and FP32 motion networks. We report median wall-clock time over 10 runs after 3 warm-up runs. RTF is generation time divided by output motion duration. Native windowing accounts for the slightly different durations.
Timings exclude model loading, audio/text feature extraction, ASR/alignment, disk I/O, and rendering. Baseline timings are repeated measurements of one fixed BEAT2 clip, not test-set averages. †RAG-Gesture uses a fixed exemplar and 50-step sampling, including exemplar encoding, inversion, guidance, and decoding, but excluding LLM planning and database retrieval.
StrokeForm-VQ therefore generates motion faster than playback under this protocol. This table compares motion-generation cost, not complete-system latency, and does not cover the new RVQ variant.
2. Concurrent module timings, including the planner
We retain the paper’s 4-second model window. For a 10-second utterance containing two gesture events, the following module timings were measured on an RTX 3090 with the Qwen3-8B planner and motion branch running concurrently. Models are loaded and warmed up; speech, text, and required timestamps are available.
The approximately 128 output tokens require 1.16 s of decoding, corresponding to about 110 tokens/s in this example. The 180/160 ms window-generation times correspond to RTFs of 0.045/0.040. Prefill and submission costs are reported separately.
3. Asynchronous system timeline
Initial audio-only generation runs alongside planning. Each complete event can be validated and compiled while the planner continues decoding subsequent events. Compiled constraints can then condition eligible motion intervals that remain editable.
Using the measured concurrent module times, the following milestones are derived under a schedule with no additional queueing:
For example, first-event readiness is 120 + 580 + 12 = 712 ms, followed by 160 + 24 ms to prepare its conditioned segment. First-event compilation overlaps second-event decoding, so both events are compiled at 120 + 2 × 580 + 12 = 1,292 ms.
These are module-based timing estimates, not directly measured end-to-end latency percentiles. Submission readiness also differs from actual display time or the scheduled appearance of a semantic gesture. On-time execution requires that the affected motion interval remain editable and that its update complete before the submission deadline. A 4-second model window does not by itself establish the playback-commit granularity.
4. Practical scope and deployment demonstration
The asynchronous schedule allows initial motion generation to proceed without waiting for the complete plan. Our measurements start after the utterance inputs are available; they do not include speech acquisition or establish the latency of processing continuously arriving speech. Such streaming performance also depends on input lookahead and playback-commit policies.
We encourage the reviewer to watch the Apple Vision Pro digital-human interaction demonstration in the supplementary video, VideoDemo_Compressed.mp4. It shows our system deployed in an interactive application and illustrates its practical use.
We will include the timing protocols and the distinction between measured module costs and derived system milestones in the revised manuscript.