SpeakingRobot: Semantic-Event-Driven Robot-Space Co-Speech Generation with Lower-Body Coordination

Humanoid robots are increasingly built for speaking roles, such as guides, re- ceptionists, and tutors, where the body should convey what the speech means: semantic gestures must land on the right words, and the physical robot must be able to execute them. Existing approaches typically inject a gesture either as a soft condition, which lets it drift away from its word, or as a hard constraint, which copies a retrieved clip together with dynamics from its source context that the robot cannot reliably execute. We formulate this tension as a preservation– adaptation problem and address it with editable semantic motion priors: a re- trieved gesture is treated as structured but uncertain, fixing which action occurs and at which word while leaving its dynamics and the whole-body support free to adapt. We present SpeakingRobot, which schedules sparse word-level semantic events with a language model and retrieves a robot-space prior for each event. Its heterogeneous whole-body diffusion initializes only the scheduled upper-body re- gions from lightly corrupted priors and everything else, including the torso, pelvis, and legs, from Gaussian noise; one denoiser refines them jointly, so the support can anticipate each gesture. Training exposes the denoiser to the same heteroge- neous states and, through cross-instance priors, rewards adapting a prior rather than copying it. SpeakingRobot improves generation quality over the compared co-speech generators on ZeroEGGS and BEAT2 even without priors, and raises tracking success on BEAT2 under a shared whole-body tracker; with priors, it keeps gestures on their words nearly as precisely as hard constraints while track- ing more reliably, and participants in a blind study on a physical humanoid rate the full system higher on every criterion.