- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose SpeakingRobot, a diffusion model for humanoid co-speech motion generation. The framework uses a language model to schedule gestures at specific words and retrieves corresponding motion clips as priors. The model initializes the scheduled upper-body regions with noisy priors and jointly generates gestures and whole-body support. Experiments on ZeroEGGS and BEAT2 show improved motion quality over the evaluated baselines. A user study using real-robot videos also reports higher ratings for naturalness, rhythm, and semantic appropriateness.
Critical strengths
- The framework integrates dynamic gesture generation with coordinated whole-body execution on a humanoid robot.
- Experiments on two benchmarks and a real-robot user study show improvements in motion quality, tracking success, and semantic alignment over the evaluated baselines.
Critical weaknesses
- The paper does not explain how category-specific timing offsets are determined or how retrieved clips are cropped or resampled. Robustness to timestamp errors and motion-duration mismatches is also not evaluated.
- The current ablations do not isolate the contribution of the LLM-based semantic scheduler. A comparison with a simple rule-based scheduler, using the same gesture vocabulary and motion generator, would clarify its contribution to overall performance.
- Although the gesture bank is constructed from BABEL, the selection of clips within each category is not explained. The paper should also clarify whether the quantitative benchmarks use the same retrieved-prior protocol as the real-robot experiments.
- RAG-Gesture is discussed as a closely related approach but is absent from the experimental comparisons. Including it in a comparable robot-space setting would better establish the benefit of the proposed prior initialization and training strategy.
Questions for authors
- How does the prior corruption level () relate to the sampled diffusion timestep during training?
- How are the torso and leg reference trajectories and contact labels constructed for training? Are these references checked for physical feasibility through simulation or execution with the tracking controller?
6 comments
2. BABEL retrieval consistency between quantitative and physical tests
Vocabulary construction and instance selection. Table R8 documents the staged BABEL ([Punnakkal et al., CVPR 2021]) vocabulary construction, beginning with 260 action categories and 5,472 semantically selected candidates, followed by ≤2 s segmentation (4,028), 29-DoF retargeting, joint/safety filtering (3,385), quantitative retargeting screening (1,684), human visual inspection (794), and six-annotator peak validation (682 final clips across 133 categories; Table R9). During inference, Qwen3 ([Yang et al., 2025]) first schedules the semantic event and its gesture category, after which an exemplar is uniformly sampled from the validated clips within that category, independent of any test ground-truth motion.
Identical offline pipeline. SpeakingRobot is an offline co-speech generator, not an online dialogue system. Audio and transcript are prepared in advance; Qwen3 schedules events, the same BABEL vocabulary is retrieved, peaks are aligned and semantic upper-body regions are initialized from forward-noised priors. This exact scheduling/retrieval/generation workflow is used in quantitative benchmarks and in the real-robot demonstration. The benchmark evaluates the generated references numerically; the robot demonstration sends the offline-generated full-body reference directly to the whole-body tracker. No additional gesture search, online scheduling or robot-specific motion regeneration occurs during tracking. For identical inputs and sampled priors, the reference generation is identical, and only the downstream metric or physical execution step differs. The w/o-priors control retains schedules and conditioning while replacing structured initialization in semantic regions with Gaussian noise; BEAT2 ([Liu et al., CVPR 2024]) test GT is not used as an inference prior.
Table R8. Semantic gesture vocabulary construction and filtering.
| Stage | Selection / Filtering Rule | Categories | Clips |
|---|---|---|---|
| BABEL taxonomy | 15,472 unique raw language labels grouped into 260 action categories | 260 | — |
| Semantic selection | Select interpretable upper-body semantic gestures | 133 | 5,472 |
| Temporal segmentation | BABEL action boundaries; duration ≤2 s | 133 | 4,028 |
| Robot retargeting | Convert to 29-DoF robot motion | 133 | 4,028 |
| Safety filtering | Remove unsafe poses and joint-limit violations | 133 | 3,385 |
| Retargeting accuracy | MPJAE < 0.3 and MPKPE < 0.3 | 133 | 1,684 |
| Visual inspection | Exclude unnatural or semantically distorted motions | 133 | 794 |
| Peak consistency | Six annotators; maximum disagreement ≤5 frames | 133 | 682 |
3. Controlled Comparison of Qwen3 and Rule-Based Scheduler
We thank the reviewer for suggesting a controlled comparison to isolate the contribution of LLM-based semantic scheduling. We compare Qwen3 with a rule-based scheduler while keeping the 133-category gesture vocabulary, retrieval strategy, peak-aligned initialization, diffusion checkpoint, event validator, and GMT tracker identical (Table R18).
Scheduler comparison. The rule-based scheduler performs exact/longest-phrase matching against predefined gesture triggers, followed by deterministic conflict resolution. In contrast, Qwen3 considers the full word-timed transcript to jointly determine gesture categories, trigger-word positions, and event salience. This enables context-dependent interpretation of ambiguous expressions (e.g., “turn right” versus “you are right”) and paraphrases beyond fixed keyword matching. Importantly, Qwen3 only schedules semantic events; motion generation remains entirely handled by the same diffusion model.
Quantitative results. As shown in Table R18, Qwen3 improves SC from 0.58 to 0.69, reduces PTE from 0.17 to 0.08 s (52.9%), and lowers FGD from 12.04 to 9.36 (22.3%), while maintaining high execution success (96.8% vs. 96.1%). SC is evaluated against the original transcript rather than the scheduler's predicted labels. PTE measures alignment with the scheduled target peak, not the correctness of trigger-word selection. Thus, these results establish the downstream benefit of contextual scheduling, rather than directly measuring event-level classification accuracy.
We will incorporate this controlled ablation and clarify the scheduler's role in the revised manuscript. We sincerely appreciate the reviewer's constructive suggestion, which helps strengthen the attribution of our system's improvements.
Table R18. Controlled ablation of semantic event scheduling.
| Scheduler | SC ↑ | PTE (s) ↓ | FGD ↓ | SR (%) ↑ |
|---|---|---|---|---|
| Rule-based Scheduler | 0.58 | 0.17 | 12.04 | 96.1 |
| Qwen3 Scheduler (Ours) | 0.69 | 0.08 | 9.36 | 96.8 |
1. Category Offsets, Peak-Preserving Alignment, and Timing Robustness
We thank the reviewer for requesting clarification of peak annotations, timing offsets, crop/resampling, and temporal robustness. We provide implementation details and additional experiments (Tables R9, R16–R17).
Peak annotation and diversity-oriented offsets. Six independent annotators identify each gesture's expressive-stroke peak. A clip is retained only when their maximum pairwise disagreement is ≤5 frames; 682/794 clips (85.9%) satisfy this criterion. The source peak is the floor of the six annotations' mean (Table R9). We schedule the target peak as
fj=round(Ffpstijmid)+Δkj,j
where Δkj,j is a category-conditioned, event-level timing offset. This offset is specifically introduced to improve motion diversity (DIV) by allowing limited variation in expressive-peak placement, rather than always aligning gestures rigidly to word midpoints. Most events use zero offset; categories with more than three exemplars allow bounded variations within ±3 frames (±0.1 s at 30 fps). This is a scheduling heuristic, not a learned timing estimator.
Diversity–alignment trade-off. Table R16 shows that the proposed offsets improve DIV from 8.95 to 9.03, demonstrating their intended contribution to motion diversity. Meanwhile, PTE-Scheduled remains nearly unchanged (0.07→0.08 s), indicating that generated gestures still closely follow their scheduled peaks. PTE-Word increases from 0.07 to 0.10 s because it measures deviation from the original word midpoint and therefore includes the intentionally introduced offsets. Thus, the offset mechanism is designed to enhance diversity, not to minimize word-centered timing error, while maintaining accurate alignment with the actual scheduled targets.
Peak-preserving alignment. After uniformly sampling a validated exemplar, we account for crop/resampling when updating its peak position. Given source peak pj, crop start cj, and resampling factor rj, the transformed peak is pj′=round[rj(pj−cj)]. The clip is shifted by fj−pj′ frames to preserve peak alignment. Gestures spanning multiple 72-frame windows continue using preceding generated frames as history; overlapping priors are resolved by event salience.
Timing robustness. We independently perturb timestamps by ±0.1/0.2 s and prior durations by 0.75×/1.25×, keeping the checkpoint, exemplar, offset realization, and tracker fixed (Table R17). PTE-Scheduled measures error against the offset-adjusted scheduled peak, whereas PTE-Word uses the original unperturbed word midpoint. Under ±25% duration changes, SR remains 96.0%–96.2% (baseline: 96.8%), with PTE-Word changing only 0.10→0.11 s. Under ±0.2 s timestamp shifts, SC remains 0.65 and SR 95.8%–95.9%, although PTE-Word increases to 0.22–0.23 s. These results show that semantic and execution performance remains relatively stable under bounded perturbations, while the method follows supplied timestamps rather than correcting ASR errors.
We will clarify the diversity-oriented offset design, peak-preserving alignment, and timing metrics in the revised manuscript. We sincerely appreciate the reviewer's constructive suggestions.
Table R9. Expressive peak annotation protocol.
Table R16. Effect of diversity-oriented timing offsets.
Table R17. Robustness to timestamp and duration perturbations.