acceptodds
Withdrawn from ICLR 2027

Embodied Interaction Policy: A Unified Model for Real-Time Human-Robot Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

43 views

Abstract

Real-time human-robot social interaction requires continuous coordination between speech and physical behavior. Existing systems often separate interaction agents from motion generation, complicating coordination as conversational states change. We propose Embodied Interaction Policy (EIP), a unified streaming framework that couples an Audio-Interaction expert with a motion expert through layer-wise joint attention. This allows motion generation to directly access the evolving interaction context. A chunk-wise causal mask prevents access to future context, while a vocal token encodes speaking state and motion intensity derived from the response emotion. We construct Embodied Interaction Dataset (EI-Data) from co-speech and dyadic interaction data and train our model in two stages. EIP supports proactive responses to salient acoustic events, state adaptation between listening and speaking, and interruption-aware turn-taking. Quantitative evaluations demonstrate improved motion quality over representative gesture-generation baselines. Compared with RoboGesture, EIP achieves higher interaction success rates and reduces warm-start latency from speech input to the first generated motion chunk from 3.78 s to 1.35 s. Physical deployment and user studies further demonstrate improved motion naturalness and interaction comfort.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human--Robot Interaction, Unified, Real-Time, Streaming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QnMSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EIP, a system that takes streaming user audio, dialogue history, and recent robot motion as input and generates spoken responses and upper-body gestures in real time.

The system has two main components:

  • A frozen audio interaction model processes the user’s audio and dialogue history to generate a spoken response.
  • A flow-matching motion model reads the interaction model’s hidden states, recent robot state, and speaking/intensity conditions to generate upper-body motion in 400 ms chunks.

The paper also constructs EI-Data from existing human interaction and gesture datasets and demonstrates the system on an Astribot S1 robot.

Critical strengths
  • The problem is practical. A conversational robot needs to generate motion continuously and respond to changes between listening, speaking, and interruption.
  • The paper demonstrates the complete system on a physical robot. This is useful evidence that the proposed integration can operate beyond offline motion generation.
Critical weaknesses
  1. The motion conditioning could be better justified. The model does not explicitly use acoustic features of the robot’s generated speech. For lively, speech-aligned gestures, the speaking state and coarse intensity conditions seem insufficient. The gestures in the supplementary videos also do not look well aligned with the speech rhythm.

  2. The paper reports better FGD and comparable BC. It would be helpful to add subjective comparisons with baselines to show whether these results also reflect how natural the gestures look and how well they align with speech.

Questions for authors
  1. Could you clarify whether any acoustic or prosodic features of the robot’s generated speech reach the motion model?
  2. Could you provide subjective comparisons with external baselines on gesture naturalness and speech alignment?
Reviewer_EhWoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Embodied Interaction Policy, a framework that combines an audio interaction model with a motion expert for real-time human-robot interaction. It generates speech-aligned motions and supports interruptions, listening behaviors, and emotion-aware gestures. The authors also introduce EI-Data for training and evaluation. Experiments show improved motion quality over existing baselines, and real-world tests show higher interaction success rates and lower latency than RoboGesture.

Critical strengths
  1. The paper addresses an interesting problem by bringing real-time speech and robot motion together. Being able to handle interruptions and adjust gestures during conversations is useful for real-world human-robot interaction.
  2. I like that the authors tested the system on a real robot, not just in simulation. The results show better motion quality and lower latency, and the user study also suggests that the interactions feel more natural.
Critical weaknesses
  1. The baseline comparisons don't seem entirely fair. Some baselines use retargeted motion, while EIP is trained directly on robot motion. Even though a few baselines are retrained, it's still unclear how much of the improvement comes from the method itself.
  2. I think the paper needs more ablations. Layer-wise joint attention and chunk-wise causal masking are presented as key parts of the method, but their individual contributions aren't really tested. It's hard to tell how much these design choices actually help.
  3. The robot's speech in the supplementary demo videos is quite muffled and hard to understand.
Questions for authors
  1. Can the authors clarify how the baseline comparisons are kept fair, especially when some methods use retargeted motion while EIP is trained directly on robot motion?
  2. How much do the layer-wise joint attention and chunk-wise causal mask contribute to the final performance? It would be helpful to see ablations that isolate these two components.
Reviewer_iPtMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EIP, a streaming system that generates speech and upper-body robot motion on a shared 400 ms clock. A frozen full-duplex Audio Interaction Model (AIM) is coupled to a flow-matching motion expert through layer-wise joint attention. A chunk-wise causal mask keeps generation causal, and a vocal token conditions motion on speaking state and an emotion-derived intensity level. The motion expert is trained in two stages on EI-Data, built from co-speech and dyadic datasets. On an Astribot S1, EIP reports better gesture metrics than prior baselines, lower latency than RoboGesture, and real-time demos of proactive responses, barge-in handling, and expressive gestures.

Critical strengths
  1. A working end-to-end full-duplex system on a physical humanoid, with MPC collision avoidance and Ruckig smoothing downstream, an end-to-end RTF below 1, and real demos of proactive responses and barge-in.
  2. The streaming design is careful. The chunk-wise mask makes parallel training match chunk-by-chunk inference (argued in Appendix D), and the separate decoding and committed KV caches keep unplayed response text out of the motion context.
  3. The retargeting pipeline (SMPL-X → SOMA → robot IK) is well documented, and two baselines are retrained directly in the robot embodiment rather than only retargeted.
Critical weaknesses

I lean toward rejection mainly because the experiments do not isolate the paper's central claim: that layer-wise coupling to the interaction model's hidden states is what improves the robot's behavior.

  1. The interaction results measure the full-duplex backbone, not the coupling. RoboGesture is a turn-based ASR–LLM–TTS cascade, so failing on acoustic events (0%) and interruptions (15%) is expected by design. In EIP, the decision to respond or yield comes from the frozen AIM and the outer loop, which supplies the interruption flag ιt\iota_t. Motion tokens never feed back into the interaction context. The 100% vs. 18.3% gap in Table 3 therefore cannot be credited to the coupling. The comparison I would want is a cascade built on the same frozen AIM and streaming scheduler.

  2. Layer-wise coupling is not compared with simpler interfaces. The only evidence for Contribution 1 is Ours (w/o h_t). That shows conversational context helps, not that layer-wise joint attention beats final-layer conditioning, final-layer cross-attention, or late fusion. The largest FGD drop in the S1 ablation (126.84 → 69.33, Vanilla → +Stage I) comes from adding the speaking-state token and phase-specific losses together, so their individual effects are unknown. The chunk-wise mask, FiLM, and the two-stage schedule are not ablated either; the mode collapse under joint training (Sec. 2.3) is asserted but not shown.

  3. Table 2 is not an apples-to-apples comparison.

    • Inputs and data differ. All baselines see only target-speaker audio. The starred ones are retrained on the co-speech data, still without dyadic listening supervision, and the rest use original weights with post-hoc retargeting. EIP gets user audio, teacher-forced target text, speaking-state and intensity tokens, and dyadic supervision.
    • Rollout protocol. Eq. (13) uses the ground-truth previous chunk as qtq_t in training. If EIP also gets ground-truth states at evaluation while baselines roll out on their own, the numbers are not comparable.
    • What BC measures. Ours (w/o h_t) has no speech hidden states, yet its BC (0.677) is above MIBURI (0.666) and Photoreal (0.665). This makes me wonder whether BC here reflects fine-grained rhythmic alignment or mostly speaking/listening boundaries.
    • Test split. The 20 test episodes were selected partly for motion–emotion consistency and are separated only at the episode level, and synthetic episodes reuse a looped 9 s listening clip. Please confirm there is no session overlap between train and test.
  4. Two headline properties lack quantitative support.

    • Smooth transitions are supported only by Likert ratings and figures. There are no continuity metrics at state changes (jerk, velocity jumps, transition delay). Any observed smoothness is also confounded by Ruckig filtering and the 0.4 s linear blends in the training data.
    • Emotion awareness is a binary switch: six tags map to mt∈{low,high}m_t \in \{\text{low}, \text{high}\}, which scales motion through FiLM. What is validated is amplitude modulation, not emotion-specific gestures. Training labels come from Qwen3.6-27B while inference tags come from AIM, with no agreement reported, and emotion success in Table 3 is judged subjectively.
Questions for authors
  1. Can you add a cascade baseline built on the same frozen AIM, data, and state conditioning, and report motion-side metrics (gesture quality, stop delay after interruption, transition smoothness)?
  2. Under identical training, how does layer-wise coupling compare with final-layer conditioning or late fusion?
  3. In Table 2, is qtq_t ground truth or rolled out from the policy's own predictions? How do the metrics change in closed loop?
  4. In training, response tokens are teacher-forced by speech timestamps. At inference, text enters the committed cache only after its chunk is played (Sec. 2.4, Alg. 1). Does the motion expert see current-chunk text in training but not at inference?
  5. Appendix B.2 describes a response-onset decision loss. What does it train if AIM is fully frozen?
  6. Can you report continuity metrics across speaking/listening transitions, both from raw policy output and after Ruckig?

Answers to Q1–Q3 are the most likely to change my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.