acceptodds
Under review as a conference paper at ICLR 2027

SpeakingRobot: Semantic-Event-Driven Robot-Space Co-Speech Generation with Lower-Body Coordination

Abstract

Humanoid robots are increasingly built for speaking roles, such as guides, re- ceptionists, and tutors, where the body should convey what the speech means: semantic gestures must land on the right words, and the physical robot must be able to execute them. Existing approaches typically inject a gesture either as a soft condition, which lets it drift away from its word, or as a hard constraint, which copies a retrieved clip together with dynamics from its source context that the robot cannot reliably execute. We formulate this tension as a preservation– adaptation problem and address it with editable semantic motion priors: a re- trieved gesture is treated as structured but uncertain, fixing which action occurs and at which word while leaving its dynamics and the whole-body support free to adapt. We present SpeakingRobot, which schedules sparse word-level semantic events with a language model and retrieves a robot-space prior for each event. Its heterogeneous whole-body diffusion initializes only the scheduled upper-body re- gions from lightly corrupted priors and everything else, including the torso, pelvis, and legs, from Gaussian noise; one denoiser refines them jointly, so the support can anticipate each gesture. Training exposes the denoiser to the same heteroge- neous states and, through cross-instance priors, rewards adapting a prior rather than copying it. SpeakingRobot improves generation quality over the compared co-speech generators on ZeroEGGS and BEAT2 even without priors, and raises tracking success on BEAT2 under a shared whole-body tracker; with priors, it keeps gestures on their words nearly as precisely as hard constraints while track- ing more reliably, and participants in a blind study on a physical humanoid rate the full system higher on every criterion.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Robot

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_eSL1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose SpeakingRobot, a diffusion model for humanoid co-speech motion generation. The framework uses a language model to schedule gestures at specific words and retrieves corresponding motion clips as priors. The model initializes the scheduled upper-body regions with noisy priors and jointly generates gestures and whole-body support. Experiments on ZeroEGGS and BEAT2 show improved motion quality over the evaluated baselines. A user study using real-robot videos also reports higher ratings for naturalness, rhythm, and semantic appropriateness.

Critical strengths
  1. The framework integrates dynamic gesture generation with coordinated whole-body execution on a humanoid robot.
  2. Experiments on two benchmarks and a real-robot user study show improvements in motion quality, tracking success, and semantic alignment over the evaluated baselines.
Critical weaknesses
  1. The paper does not explain how category-specific timing offsets are determined or how retrieved clips are cropped or resampled. Robustness to timestamp errors and motion-duration mismatches is also not evaluated.
  2. The current ablations do not isolate the contribution of the LLM-based semantic scheduler. A comparison with a simple rule-based scheduler, using the same gesture vocabulary and motion generator, would clarify its contribution to overall performance.
  3. Although the gesture bank is constructed from BABEL, the selection of clips within each category is not explained. The paper should also clarify whether the quantitative benchmarks use the same retrieved-prior protocol as the real-robot experiments.
  4. RAG-Gesture is discussed as a closely related approach but is absent from the experimental comparisons. Including it in a comparable robot-space setting would better establish the benefit of the proposed prior initialization and training strategy.
Questions for authors
  1. How does the prior corruption level (τ=10\tau=10) relate to the sampled diffusion timestep during training?
  2. How are the torso and leg reference trajectories and contact labels constructed for training? Are these references checked for physical feasibility through simulation or execution with the tracking controller?
6 comments
AuthorsOpenReview ↗

1. Category Offsets, Peak-Preserving Alignment, and Timing Robustness

We thank the reviewer for requesting clarification of peak annotations, timing offsets, crop/resampling, and temporal robustness. We provide implementation details and additional experiments (Tables R9, R16–R17).

Peak annotation and diversity-oriented offsets. Six independent annotators identify each gesture's expressive-stroke peak. A clip is retained only when their maximum pairwise disagreement is ≤5 frames; 682/794 clips (85.9%) satisfy this criterion. The source peak is the floor of the six annotations' mean (Table R9). We schedule the target peak as

fj=round⁡(Ffpstijmid)+Δkj,jf_j=\operatorname{round}(F_{\mathrm{fps}}t_{i_j}^{\mathrm{mid}})+\Delta_{k_j,j}

where Δkj,j\Delta_{k_j,j} is a category-conditioned, event-level timing offset. This offset is specifically introduced to improve motion diversity (DIV) by allowing limited variation in expressive-peak placement, rather than always aligning gestures rigidly to word midpoints. Most events use zero offset; categories with more than three exemplars allow bounded variations within ±3 frames (±0.1 s at 30 fps). This is a scheduling heuristic, not a learned timing estimator.

Diversity–alignment trade-off. Table R16 shows that the proposed offsets improve DIV from 8.95 to 9.03, demonstrating their intended contribution to motion diversity. Meanwhile, PTE-Scheduled remains nearly unchanged (0.07→0.08 s), indicating that generated gestures still closely follow their scheduled peaks. PTE-Word increases from 0.07 to 0.10 s because it measures deviation from the original word midpoint and therefore includes the intentionally introduced offsets. Thus, the offset mechanism is designed to enhance diversity, not to minimize word-centered timing error, while maintaining accurate alignment with the actual scheduled targets.

Peak-preserving alignment. After uniformly sampling a validated exemplar, we account for crop/resampling when updating its peak position. Given source peak pjp_j, crop start cjc_j, and resampling factor rjr_j, the transformed peak is pj′=round⁡[rj(pj−cj)]p'_j=\operatorname{round}[r_j(p_j-c_j)]. The clip is shifted by fj−pj′f_j-p'_j frames to preserve peak alignment. Gestures spanning multiple 72-frame windows continue using preceding generated frames as history; overlapping priors are resolved by event salience.

Timing robustness. We independently perturb timestamps by ±0.1/0.2 s and prior durations by 0.75×/1.25×, keeping the checkpoint, exemplar, offset realization, and tracker fixed (Table R17). PTE-Scheduled measures error against the offset-adjusted scheduled peak, whereas PTE-Word uses the original unperturbed word midpoint. Under ±25% duration changes, SR remains 96.0%–96.2% (baseline: 96.8%), with PTE-Word changing only 0.10→0.11 s. Under ±0.2 s timestamp shifts, SC remains 0.65 and SR 95.8%–95.9%, although PTE-Word increases to 0.22–0.23 s. These results show that semantic and execution performance remains relatively stable under bounded perturbations, while the method follows supplied timestamps rather than correcting ASR errors.

We will clarify the diversity-oriented offset design, peak-preserving alignment, and timing metrics in the revised manuscript. We sincerely appreciate the reviewer's constructive suggestions.

Table R9. Expressive peak annotation protocol.

ItemProtocol / Result
Candidates after quality filtering1,684 clips
Visual inspection794 retained
Peak annotation6 independent annotators; 15 pairs/clip
Agreement criterionMaximum difference ≤5 frames
Accepted / rejected682 (85.9%) / 112
Final peakFloor of mean annotated frame
Final vocabulary133 categories; 682 clips

Table R16. Effect of diversity-oriented timing offsets.

SettingPTE-Scheduled (s) ↓PTE-Word (s) ↓DIV ↑
Without Offset0.070.078.95
With Offset (Ours)0.080.109.03

Table R17. Robustness to timestamp and duration perturbations.

PerturbationSettingSC ↑PTE-Scheduled ↓PTE-Word ↓FGD ↓SR (%) ↑
Timestamp−0.2 s0.650.100.239.7295.8
Timestamp−0.1 s0.670.090.159.5596.2
None0 s0.690.080.109.3696.8
Timestamp+0.1 s0.680.090.149.5196.3
Timestamp+0.2 s0.650.100.229.7895.9
Duration0.75×0.670.090.119.5696.0
Duration1.00×0.690.080.109.3696.8
Duration1.25×0.680.090.119.4996.2
AuthorsOpenReview ↗

2. BABEL retrieval consistency between quantitative and physical tests

Vocabulary construction and instance selection. Table R8 documents the staged BABEL ([Punnakkal et al., CVPR 2021]) vocabulary construction, beginning with 260 action categories and 5,472 semantically selected candidates, followed by ≤2 s segmentation (4,028), 29-DoF retargeting, joint/safety filtering (3,385), quantitative retargeting screening (1,684), human visual inspection (794), and six-annotator peak validation (682 final clips across 133 categories; Table R9). During inference, Qwen3 ([Yang et al., 2025]) first schedules the semantic event and its gesture category, after which an exemplar is uniformly sampled from the validated clips within that category, independent of any test ground-truth motion.

Identical offline pipeline. SpeakingRobot is an offline co-speech generator, not an online dialogue system. Audio and transcript are prepared in advance; Qwen3 schedules events, the same BABEL vocabulary is retrieved, peaks are aligned and semantic upper-body regions are initialized from forward-noised priors. This exact scheduling/retrieval/generation workflow is used in quantitative benchmarks and in the real-robot demonstration. The benchmark evaluates the generated references numerically; the robot demonstration sends the offline-generated full-body reference directly to the whole-body tracker. No additional gesture search, online scheduling or robot-specific motion regeneration occurs during tracking. For identical inputs and sampled priors, the reference generation is identical, and only the downstream metric or physical execution step differs. The w/o-priors control retains schedules and conditioning while replacing structured initialization in semantic regions with Gaussian noise; BEAT2 ([Liu et al., CVPR 2024]) test GT is not used as an inference prior.

Table R8. Semantic gesture vocabulary construction and filtering.

StageSelection / Filtering RuleCategoriesClips
BABEL taxonomy15,472 unique raw language labels grouped into 260 action categories260—
Semantic selectionSelect interpretable upper-body semantic gestures1335,472
Temporal segmentationBABEL action boundaries; duration ≤2 s1334,028
Robot retargetingConvert to 29-DoF robot motion1334,028
Safety filteringRemove unsafe poses and joint-limit violations1333,385
Retargeting accuracyMPJAE < 0.3 and MPKPE < 0.31331,684
Visual inspectionExclude unnatural or semantically distorted motions133794
Peak consistencySix annotators; maximum disagreement ≤5 frames133682
AuthorsOpenReview ↗

3. Controlled Comparison of Qwen3 and Rule-Based Scheduler

We thank the reviewer for suggesting a controlled comparison to isolate the contribution of LLM-based semantic scheduling. We compare Qwen3 with a rule-based scheduler while keeping the 133-category gesture vocabulary, retrieval strategy, peak-aligned initialization, diffusion checkpoint, event validator, and GMT tracker identical (Table R18).

Scheduler comparison. The rule-based scheduler performs exact/longest-phrase matching against predefined gesture triggers, followed by deterministic conflict resolution. In contrast, Qwen3 considers the full word-timed transcript to jointly determine gesture categories, trigger-word positions, and event salience. This enables context-dependent interpretation of ambiguous expressions (e.g., “turn right” versus “you are right”) and paraphrases beyond fixed keyword matching. Importantly, Qwen3 only schedules semantic events; motion generation remains entirely handled by the same diffusion model.

Quantitative results. As shown in Table R18, Qwen3 improves SC from 0.58 to 0.69, reduces PTE from 0.17 to 0.08 s (52.9%), and lowers FGD from 12.04 to 9.36 (22.3%), while maintaining high execution success (96.8% vs. 96.1%). SC is evaluated against the original transcript rather than the scheduler's predicted labels. PTE measures alignment with the scheduled target peak, not the correctness of trigger-word selection. Thus, these results establish the downstream benefit of contextual scheduling, rather than directly measuring event-level classification accuracy.

We will incorporate this controlled ablation and clarify the scheduler's role in the revised manuscript. We sincerely appreciate the reviewer's constructive suggestion, which helps strengthen the attribution of our system's improvements.

Table R18. Controlled ablation of semantic event scheduling.

SchedulerSC ↑PTE (s) ↓FGD ↓SR (%) ↑
Rule-based Scheduler0.580.1712.0496.1
Qwen3 Scheduler (Ours)0.690.089.3696.8
Reviewer_WuZHOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper describes a system/method for generating co-speech gestures for a humanoid robot. the system uses a gesture vocabulary and places tempate gestures on a timeline based on text semantic matching. Then the full body motion is geneated via denoising, where the upper body starts from the template key frames, and the lower body starts from noise, in order to give it freed om to generate plausible supporting lower-body motion (to reduce risk of robot falling over etc). The system is compared against other gesture generation systems, retrained to generate in robot joint angle space, and objective metrics include FGD, Beat concistency and diversity, as well as sucess rate that measures how often the robot can carry out the sequence without falling over. A small user study is also reported.

Critical strengths

The core contribution and strength of this paper is the implementation of semantic co-speech gestures on a humanoid robot. The paper cites a few recent papers adressing the same problem, where the proposed approach is differentiated in terms of the retrieval based approach of using gesture templates in combination with lower-body denoising in order to retain holistically balanced body pose. The evaluation against co-speech gesutre baseline models, re-trained on robot-space angle reresentations is valid. The inclusion of a subjective user study (albeit limited) is a strength.

Critical weaknesses

the gesture vocabulary is under-specified - this is a central part of the method yet it is given very little space in the paper. it is based on BABEL which contains general actions, (not specifically communicative gestures), and it is unclear how the 133 categories werer selected. Also not clear how peak annotation was done.

The templates are corrupted with noise level τ, how was the level decided? there is no ablation to investigate how this affects the other metrics SC/PTE/PD, could be done with a sweep across noise levels.

The user study asks three questions at the same time for each clip (naturalness, rythm and semantic appropriateness), which is known to confound motion quality and appropriateness (see e.g. GENEA 2020). The established alternative is to do a mismatch parwise study of rythmic & semantic appropriateness (same generated motion, different speech pairing) and evaluate motion quality separately (see Yoon et al, ICMI 2022).

Also, the user study does not evaluate the papers main claim, that an editable prior is better than a hard constraint.

Questions for authors

How were the BABEL categories selected and the upper-body clips segmented? Who annotated the expressive peaks, and was inter-annotator agreement measured?

Which motion and text encoders are used for SC, and what data were they trained on?

How do SC/PTE/PD/SR vary with τ?

Injecting out-of-domain BABEL clips lowers FGD against BEAT2 ground truth (10.18 → 9.36). How can this be explained?

6 comments
AuthorsOpenReview ↗

1. BABEL category selection, clip filtering and peak annotations

We thank the reviewer for highlighting the central importance of reconstructing the gesture vocabulary and expressive-peak annotation protocol. BABEL ([Punnakkal et al., CVPR 2021]) contains general labeled actions and their temporal boundaries, rather than paired speech–gesture recordings or manually supplied expressive stroke peaks. Our construction therefore includes semantic selection, robot-space quality control, and a separate human annotation stage (Tables R8 and R9).

Semantic selection and segmentation. From BABEL's taxonomy of 260 action categories (its 15,472 distinct original textual labels are not a count of gesture clips), we select 133 categories appropriate for semantic upper-body gestures, normalize descriptions and trigger phrases, and obtain 5,472 candidate clips. Using BABEL action intervals and a maximum 2 s clip length leaves 4,028 gesture segments.

Robot-space validation. We retarget all 4,028 candidates into the common 29-DoF humanoid representation. Filtering joint-limit violations, unsafe poses and kinematic anomalies yields 3,385; retargeting accuracy thresholds MPJAE < 0.3 and MPKPE < 0.3 yield 1,684. Human visual inspection removes distorted, unnatural or semantically corrupted gesture clips, leaving 794.

Independent peak annotation. Six independent annotators identify the most expressive stroke frame in each of the 794 visually accepted clips. All 15 annotator pairs are compared: the clip is retained only when the greatest pairwise difference is at most five frames (equivalently, max annotation minus min annotation ≤ 5). Of the 794 clips, 682 pass (85.9%); the other 112 are rejected. For each accepted clip we use the floor of the mean of six peak-frame annotations to obtain one integer source peak. The final vocabulary therefore comprises 133 categories and 682 validated robot-space clips with descriptions, trigger phrases, and human-consensus temporal anchors. During generation, source peaks are placed at the scheduled target frame specified by Sec. 3.2. We will add the complete filtering statistics and annotation protocol to the revision.

Table R8. Semantic gesture vocabulary construction and filtering.

StageSelection / Filtering RuleCategoriesClips
BABEL taxonomy15,472 unique raw language labels grouped into 260 action categories260—
Semantic selectionSelect interpretable upper-body semantic gestures1335,472
Temporal segmentationBABEL action boundaries; duration ≤2 s1334,028
Robot retargetingConvert to 29-DoF robot motion1334,028
Safety filteringRemove unsafe poses and joint-limit violations1333,385
Retargeting accuracyMPJAE < 0.3 and MPKPE < 0.31331,684
Visual inspectionExclude unnatural or semantically distorted motions133794
Peak consistencySix annotators; maximum disagreement ≤5 frames133682

Table R9. Expressive peak annotation and quality control.

ItemProtocolResult
Retargeted candidatesAfter quantitative quality filtering1,684 clips
Visual inspectionVisual quality inspection of retargeted candidate gestures794 retained
AnnotatorsIndependent human annotators6
Peak definitionMost semantically representative stroke momentOne per clip
Peak annotationAnnotators independently select peak framesManual
Pairwise checksAll annotator pairs15 pairs/clip
Agreement criterionMaximum pairwise difference ≤5 frames≤5 frames
Rejected by consistency checkDisagreement exceeds threshold112 clips
Agreement pass rate682 / 79485.9%
Final source peakFloor of mean of six annotated peak framesInteger frame
Final vocabularyValidated gesture exemplars133 categories / 682 clips
AuthorsOpenReview ↗

2. Sensitivity and Selection of Prior Corruption Level τ

We thank the reviewer for asking why τ=10 was selected and how semantic preservation, prior deviation, and robot execution are affected by this parameter. We conduct a fixed-checkpoint inference sweep in 29-DoF robot space with τ ∈ {2, 5, 10, 15, 20, 30}, without retraining the motion generator (Table R10).

Preservation–adaptation trade-off. As τ increases from 2 to 30, prior deviation (PD) increases from 0.043 to 0.358, while semantic consistency (SC) decreases from 0.73 to 0.51 and peak timing error (PTE) increases from 0.05 to 0.20 s. Lower corruption better preserves the retrieved gesture's semantic structure and temporal alignment, whereas higher corruption permits greater adaptation but progressively weakens these properties.

Selection of τ=10. We choose τ=10 for its favorable balance between motion tracking accuracy and semantic preservation, rather than optimizing any single metric. Although τ=2 achieves the highest SC (0.73) and lowest PTE (0.05 s), τ=10 yields the lowest MPJAE (0.097) and MPKPE (0.106), reducing tracking errors by 24.2% and 21.5%, respectively, compared with τ=2. It also achieves the highest execution success (96.8%), while retaining strong SC (0.69) and PTE (0.08 s). Across τ=2–15, SR remains between 96.1% and 96.8%, indicating robustness to moderate variations in corruption strength. These results support τ=10 as a practical trade-off for physically executable motion, rather than an optimum across all metrics. We will include Table R10 and this analysis in the revision.

Table R10. Sensitivity to prior corruption level τ.

τSC ↑PTE (s) ↓PDSR (%) ↑MPJAE ↓MPKPE ↓
20.730.050.04396.60.1280.135
50.710.060.08296.20.1080.115
100.690.080.13796.80.0970.106
150.670.090.18696.10.1010.109
200.620.120.24194.80.1130.118
300.510.200.35894.70.1390.147
AuthorsOpenReview ↗

3. Disentangled Perceptual Study and Editable-vs-Hard Comparison

We sincerely thank the reviewer for the valuable suggestion regarding potential halo effects when motion naturalness, rhythm, and semantic appropriateness are evaluated jointly. Following the evaluation principles of the GENEA Challenge ([Yoon et al., ICMI 2022]; [Kucherenko et al., ICMI 2023]) and the recent community-driven benchmark ([Nagy et al., CVPR 2026 Findings]), we conduct an additional disentangled perceptual study with 48 participants, separating motion naturalness from speech–gesture and semantic correspondence (Tables R12–R13).

Disentangled evaluation protocol. We design four separate tasks: (1) Naturalness: five-point ratings of silent motion; (2) Speech Match: forced-choice identification of correctly versus incorrectly paired speech for the same motion; (3) Semantic Match: forced-choice identification of matching versus mismatched descriptions for the same silent motion; and (4) Rhythm: five-point ratings of speech–motion synchrony in an independent task. Methods are anonymized, rendering conditions are standardized, and presentation order is randomized. This design reduces potential halo effects by separating visual naturalness from speech and semantic judgments. The protocol follows GENEA-inspired evaluation principles rather than reproducing its official evaluation.

Perceptual results. As shown in Table R12, SpeakingRobot achieves the highest silent naturalness (4.08), speech matching accuracy (74.6%), semantic matching accuracy (77.2%), and rhythm rating (4.26) among the evaluated methods. These independent assessments provide complementary evidence of improved motion quality and speech–gesture correspondence.

Editable versus hard constraints. We further conduct a silent-motion paired-preference study (Table R13) to directly assess perceived naturalness without speech or semantic cues. SpeakingRobot is preferred over Hard Anchor in 77% of comparisons and over RAG-style Guidance in 62%, whereas RAG-style Guidance is preferred over Hard Anchor in 56%. We report these preferences descriptively without claiming statistical significance.

Together, these results provide perceptual evidence consistent with the benefits of editable semantic-prior adaptation over rigid motion anchoring, complementing our quantitative ablations. We will incorporate the newly conducted perceptual experiments, their results, and the above references into the revised manuscript. We sincerely appreciate the reviewer's constructive suggestions, which have helped us strengthen the evaluation protocol and the presentation of our work.

Table R12. GENEA-inspired disentangled perceptual evaluation (48 participants).

MethodSilent Naturalness (1–5) ↑Speech Match (%) ↑Semantic Match (%) ↑Rhythm (1–5) ↑
Hard Anchor3.5165.269.33.58
RAG-style Guidance3.8367.565.83.92
SpeakingRobot (Ours)4.0874.677.24.26

Table R13. Silent-motion paired preference for naturalness.

ComparisonFirst Method Preferred (%)
SpeakingRobot vs. Hard Anchor77%
SpeakingRobot vs. RAG-style Guidance62%
RAG-style Guidance vs. Hard Anchor56%
Reviewer_EA9AOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper presents a system for generating semantic co-speech gestures on a humanoid robot. It schedules semantic events at the word level and retrieves motion priors for those events. Lightly noised priors initialize selected upper-body regions of a diffusion model, which jointly generates the remaining motion. Training uses a matching heterogeneous-noise setup and cross-instance priors to encourage the model to adapt the retrieved motions. The evaluation includes ZeroEGGS and BEAT2, simulation with a fixed GMT tracker, and a user study on a physical humanoid.

Critical strengths

I found the execution results in Table 3 the strongest part of the paper. Keeping GMT fixed makes the comparison informative: the improvement in execution success provides evidence that the generated reference motions are easier for this tracker to follow. This has practical value for humanoid gesture generation, although I am less sure which parts of the proposed method account for the gains.

Critical weaknesses
  1. My main concern is what the method adds beyond existing work on motion-prior injection. LivelySpeaker already adds noise to semantically generated motion and refines it with diffusion. RAG-Gesture places retrieved motion at the corresponding words and limits injection to the upper body and hands. Here, the differences include forward noising instead of DDIM inversion, heterogeneous-noise training, and cross-instance priors. These changes may help, but the experiments do not show how much they contribute. The hard-anchor baseline in Table 4 does not settle this question, since RAG-Gesture allows its guidance strength to be adjusted. A direct comparison is still needed.

  2. I had trouble interpreting the anticipatory-support ablation. Removing this component lowers execution success from 96.8% to 82.2%, but what happens to the leg and root references in this setting is unclear. Without knowing what GMT receives, I cannot attribute the 14.6-point gap specifically to the support head. The comparison also leaves open whether ordinary joint whole-body generation, or motion inpainting that allows the upper body to change, would provide similar benefits. The event-weighted loss needs a separate ablation as well.

  3. The supplementary videos left me less convinced about motion naturalness and expressiveness. Seeing the generated motions in the original SMPL-X representation would help. I would also like to see comparisons with GestureLSM (ICCV 2025), MECo (SIGGRAPH 2025), and StreamTalk (ECCV 2026) under a common BEAT2 evaluation protocol. All three provide public code and pretrained weights, so these comparisons seem feasible.

GestureLSM (ICCV 2025) Paper: https://arxiv.org/abs/2501.18898 Code: https://github.com/andypinxinliu/GestureLSM

MECo (SIGGRAPH 2025) Paper: https://arxiv.org/abs/2507.20220 Code: https://github.com/RobinWitch/MECo

StreamTalk (ECCV 2026) Paper: https://arxiv.org/html/2608.01643 Code: https://github.com/Xiangyue-Zhang/StreamTalk

The first two concerns are the main reasons for my Weak Reject recommendation.

Questions for authors
  1. In the variant without anticipatory support, what leg and root references are passed to GMT?

  2. How does the proposed prior initialization compare with RAG-Gesture’s injection mechanism when both methods use the same retrieved priors and event timing?

  3. Can you retain the backbone, future window, and whole-body outputs, and remove only the explicit connection from the upper-body prediction to the support head? This would help isolate the value of that connection. Please also report a separate ablation of the event-weighted loss.

These clarifications and comparisons would help me reassess the paper.

As an additional experiment, testing another strong tracker, such as SONIC, would help show whether the execution gains carry over to a different controller. The paper explicitly limits its evaluation to a fixed tracker, so the lack of this experiment is not a main concern for my recommendation.

9 comments
AuthorsOpenReview ↗

1. Direct Comparison with RAG-Gesture under Aligned Evaluation Protocols

We thank the reviewer for emphasizing the importance of directly comparing SpeakingRobot with RAG-Gesture (Mughal et al., CVPR 2025), particularly its adjustable retrieval-guidance mechanism. We apologize for not including RAG-Gesture as a direct experimental baseline in our original submission. In the revised manuscript, we will establish RAG-Gesture as a primary comparison baseline and strengthen its discussion and citation as closely related work. To address this omission, we provide direct comparisons in both SMPL-X and 29-DoF robot space, following RAG-Gesture's evaluation protocol and explicitly aligning the retrieved semantic priors and insertion positions in our controlled robot-space evaluation.

SMPL-X comparison under aligned evaluation protocols. We additionally train SpeakingRobot in SMPL-X space and evaluate it on BEAT2 All Speakers using RAG-Gesture's benchmark metrics, including FID, Beat Alignment (BA), L1 Diversity (L1Div), and MPJPE. We adopt its discourse-based retrieval procedure for semantic gesture selection and temporal alignment. Table R1(a) compares SpeakingRobot with the original published RAG-Gesture results under the corresponding benchmark setting.

SpeakingRobot achieves 0.419 FID and 31.6 mm MPJPE, compared with 0.447 and 35.0 mm for RAG-Gesture with Latent Inversion and Retrieval Guidance (LI+RG). Under the additional semantic and timing evaluations, SpeakingRobot achieves 0.73 SC and 0.06 s PTE, compared with 0.59 and 0.14 s for RAG-Gesture. BA and L1Div are interpreted relative to their ground-truth references rather than simply maximized.

Matched-prior robot-space comparison. To directly isolate the generation mechanism, we further retrain RAG-Gesture in the same 29-DoF X2 robot representation, retaining its original LI+RG inference mechanism. Crucially, both methods receive identical retrieved gesture instances, target-word timestamps, and temporal insertion positions and intervals. Their complete generated whole-body references are evaluated using the same fixed GMT tracker and evaluation criteria.

As shown in Table R1(b), SpeakingRobot improves SC from 0.62 to 0.69, reduces PTE from 0.13 s to 0.08 s and FGD from 10.82 to 9.36, and increases execution success from 94.9% to 96.8%. By controlling the retrieved priors, insertion positions, robot representation, and tracking controller, this comparison directly evaluates how the two approaches integrate semantic motion priors into physically executable whole-body generation.

Mechanistic distinction and controlled ablations. RAG-Gesture uses DDIM inversion and adjustable inference-time retrieval guidance to steer generation toward retrieved motions. SpeakingRobot instead explicitly trains its denoiser on heterogeneous-noise states, where structured semantic upper-body regions coexist with highly noisy surrounding whole-body motion. Furthermore, cross-instance priors provide different gesture instances from the same semantic category during training, discouraging direct copying of the target and encouraging context-dependent adaptation of gesture dynamics and transitions.

The B0-B4 ablations in Table R2 further isolate the contributions of prior initialization, heterogeneous-noise training, and cross-instance adaptation. Together, the SMPL-X comparison and matched-prior robot-space evaluation support our central contribution: learning to adapt retrieved semantic gestures to the current speech context while jointly generating physically executable whole-body support, beyond inference-time retrieval guidance alone.

Anonymous SMPL-X comparisons using aligned speech and semantic-prior conditions will also be provided: https://anonymous.4open.science/r/review-videos-5192/.

Table R1(a). Comparison with RAG-Gesture on BEAT2 All Speakers (SMPL-X).

MethodFID↓BA→L1Div→MPJPE (mm)↓SC↑PTE (s)↓
RAG-Gesture (LI Only)0.5890.4869.2954.60.550.23
RAG-Gesture (LI + RG)0.4470.4719.0435.00.590.14
SpeakingRobot (SMPL-X)0.4190.4938.9131.60.730.06

Table R1(b). Matched-Prior Comparison in X2 Robot Space (29-DoF).

MethodSCPTE (s)↓FGD↓SR (%)
RAG-Gesture-Robot (LI + RG, Retrained)0.620.1310.8294.9
SpeakingRobot (Ours)0.690.089.3696.8
AuthorsOpenReview ↗

2. Initialization, Heterogeneous Training, and Cross-Instance Adaptation

We thank the reviewer for requesting a clearer attribution of improvements beyond conventional motion-prior injection. We conduct staged ablations on BEAT2 in the same 29-DoF robot space, using consistent data splits, speech/event conditioning, and a fixed GMT ([Chen et al., IROS 2026]) tracker. Table R2 isolates the contributions of prior initialization, heterogeneous-noise training, and cross-instance adaptation.

B0→B1: Prior initialization. Using the same standard diffusion checkpoint, replacing Gaussian initialization with a forward-noised semantic prior improves SC 0.44→0.51, PTE 0.28→0.19 s, FGD 11.03→10.72, and SR 88.3%→90.6%. This confirms that retrieved priors provide useful semantic and temporal structure even without specialized training, although improvements remain limited by the mismatch between standard diffusion training and heterogeneous inference initialization.

B1→B2: Heterogeneous-noise training. We introduce heterogeneous-noise training with self-instance priors while retaining the same inference initialization. SC improves 0.51→0.61, PTE decreases 0.19→0.14 s, FGD decreases 10.72→9.83, and SR increases 90.6%→93.0%. This demonstrates the benefit of training the denoiser to jointly process partially corrupted semantic regions and noisy surrounding motion.

B2→B3: Cross-instance adaptation. Both variants use heterogeneous-noise training and identical initialization; B2 uses self-instance training priors, whereas B3 uses different instances from the same semantic category. Cross-instance training improves SC 0.61→0.69, PTE 0.14→0.08 s, FGD 9.83→9.36, and SR 93.0%→96.8%. Meanwhile, Prior Deviation (PD) increases 0.058→0.137. PD is a diagnostic measure rather than a metric to maximize or minimize; together with the improved generation metrics, this supports contextual adaptation instead of simply copying retrieved trajectories.

B4→B3: Prior initialization in the full model. Holding the heterogeneous-noise and cross-instance-trained checkpoint fixed, restoring forward-noised initialization improves SC 0.45→0.69, PTE 0.24→0.08 s, FGD 10.18→9.36, and SR 90.4%→96.8%. Importantly, B0 and B4 both use Gaussian initialization but differ in their training regimes; they are not interchangeable baselines.

Overall, these controlled comparisons establish complementary roles: initialization supplies semantic and timing structure, heterogeneous training enables refinement of structured priors within noisy whole-body motion, and cross-instance training encourages context-dependent adaptation. Together with the RAG-Gesture comparison (Table R1), they support our contribution of learning semantic-prior adaptation during training rather than relying solely on inference-time prior injection.

Table R2. Ablation of prior initialization, heterogeneous-noise training, and cross-instance adaptation.

VariantTraining ConfigurationInitializationSC ↑PTE (s) ↓PDFGD ↓SR (%) ↑
B0Standard DiffusionGaussian0.440.280.37711.0388.3
B1Standard DiffusionForward-Noised Prior0.510.190.22110.7290.6
B2Heterogeneous Noise, Self-InstanceForward-Noised Prior0.610.140.0589.8393.0
B3 (Ours)Heterogeneous Noise, Cross-InstanceForward-Noised Prior0.690.080.1379.3696.8
B4Same Trained Model as B3Gaussian0.450.240.36210.1890.4
AuthorsOpenReview ↗

3. Clarification and Isolation of Anticipatory Support

We thank the reviewer for requesting clarification of the anticipatory-support ablation, particularly how the leg and root references are generated and provided to GMT. We clarify the original ablation and introduce an additional controlled comparison to isolate the contribution of explicit upper-body-guided support prediction.

Clarification of the original ablation. In the original w/o anticipatory support variant, we remove the separate gesture-guided support head and merge the upper-body and support motion channels into a single whole-body prediction head. The model directly generates complete whole-body motion, including upper-body joints, torso, pelvis, root trajectories, legs, and contact states, through standard joint prediction from the shared temporal representation. Thus, the ablation does not remove lower-body generation; instead, it replaces our explicit upper-to-support formulation with conventional joint whole-body generation.

In this setting, the torso, root, and leg references are dynamically generated together with the upper-body motion and passed to the same fixed GMT tracker. The difference is that support motion is predicted implicitly as part of the unified whole-body output, rather than explicitly conditioned on the evolving upper-body gesture. This original variant achieves 82.2% execution success.

Isolating the explicit upper-to-support connection. To distinguish the effect of explicit gesture guidance from the architectural difference between single-head and two-head prediction, we further introduce an independent two-head variant. We retain the same bidirectional temporal Transformer, future window, upper-body and support heads, full-body representation, training data, and GMT controller, removing only the explicit input of the predicted upper-body motion to the support head.

Specifically, the independent two-head variant predicts

\hat{Y}_0^s = h_s(H),$$ whereas SpeakingRobot uses $$\hat{Y}_0^u = h_u(H), \qquad \hat{Y}_0^s = h_s(H, \hat{Y}_0^u).$$ Both variants generate complete dynamic whole-body references. Their key difference is whether the support head explicitly receives the predicted gesture, rather than relying solely on shared Transformer features. **Quantitative results.** As shown in Table R3, standard joint whole-body generation achieves **82.2% SR**, while independent two-head generation reaches **85.9% SR**. Introducing explicit upper-to-support guidance further improves SR to **96.8%**, with MPJAE decreasing from **0.151 to 0.097** and MPKPE from **0.144 to 0.106** relative to the independent-head variant. Importantly, the **10.9-percentage-point improvement from 85.9% to 96.8%** isolates the contribution of explicit upper-to-support conditioning, rather than merely introducing separate prediction heads. This distinction is important because expressive gestures may require coordinated torso adjustments, root displacement, and foot-contact transitions before the gesture reaches its expressive peak. Explicitly conditioning support prediction on the evolving upper-body gesture facilitates such coordinated whole-body responses. Together, these comparisons demonstrate that **the benefit of anticipatory support extends beyond conventional joint whole-body generation and independent body-part prediction**. The improvement is associated with explicitly modeling the dependency between the predicted semantic gesture and its corresponding whole-body support. Synchronized comparisons of generated gestures, root motion, foot contacts, and GMT execution will be provided: https://anonymous.4open.science/r/review-videos-v02-823E/. **Table R3. Ablation of Explicit Gesture-Guided Support Generation (29-DoF).** | Method | Prediction Architecture | Whole-Body Output | SR (%)↑ | MPJAE↓ | MPKPE↓ | |---|---|---|---:|---:|---:| | Joint Whole-Body Generation (w/o Anticipatory Support) | Single whole-body head | Yes | 82.2 | 0.142 | 0.149 | | Independent Two-Head Generation | h_u(H), h_s(H) | Yes | 85.9 | 0.151 | 0.144 | | **SpeakingRobot (Ours)** | **h_u(H), h_s(H, predicted upper-body)** | **Yes** | **96.8** | **0.097** | **0.106** |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.